You're reading from Azure Data Engineer Associate Certification Guide A hands-on reference guide to developing your data engineering skills and preparing for the DP-203 exam

Product type Paperback

Published in Feb 2022

Publisher Packt

ISBN-13 9781801816069

Length 574 pages

Edition 1st Edition

Tools

Azure

Concepts

Big Data

Author (1):

Newton Alex

View More author details

Table of Contents (23) Chapters

Preface

1. Part 1: Azure Basics

2. Chapter 1: Introducing Azure Basics FREE CHAPTER

3. Part 2: Data Storage

4. Chapter 2: Designing a Data Storage Structure

5. Chapter 3: Designing a Partition Strategy

6. Chapter 4: Designing the Serving Layer

7. Chapter 5: Implementing Physical Data Storage Structures

8. Chapter 6: Implementing Logical Data Structures

9. Chapter 7: Implementing the Serving Layer

10. Part 3: Design and Develop Data Processing (25-30%)

11. Chapter 8: Ingesting and Transforming Data

12. Chapter 9: Designing and Developing a Batch Processing Solution

13. Chapter 10: Designing and Developing a Stream Processing Solution

14. Chapter 11: Managing Batches and Pipelines

15. Part 4: Design and Implement Data Security (10-15%)

16. Chapter 12: Designing Security for Data Policies and Standards

17. Part 5: Monitor and Optimize Data Storage and Data Processing (10-15%)

18. Chapter 13: Monitoring Data Storage and Data Processing

19. Chapter 14: Optimizing and Troubleshooting Data Storage and Data Processing

20. Part 6: Practice Exercises

21. Chapter 15: Sample Questions with Solutions

22. Other Books You May Enjoy

Rewriting user-defined functions (UDFs)

User-defined functions are custom functions that can be defined in databases and in certain analytical and streaming engines such as Spark and Azure Stream Analytics. An example of a UDF could be a custom function to check whether a given value is a valid email address.

The DP-203 syllabus just mentions the topic as Rewriting user-defined functions. In a literal sense, this is just dropping a UDF and recreating a new one, as we do for SQL or Spark tables. However, I believe the syllabus committee might have referred to rewriting normal repetitive scripts as UDFs to make them efficient from a development perspective. Note that UDFs can also decrease runtime performance if not designed correctly. Let's look at the ways to create UDFs in SQL, Spark, and Streaming.