Chapter 11: Managing Batches and Pipelines
Welcome to Chapter 11! This is one of the smaller and easier chapters and will be a breeze to read through. In this chapter, we will be focusing on four broad categories: triggering Batch jobs, handling failures in Batch jobs, managing pipelines, and configuring version control for our pipelines. Once you have completed this chapter, you should be able to comfortably set up and manage Batch pipelines using Azure Batch, Azure Data Factory (ADF), or Synapse pipelines.
In this chapter, we will cover the following topics:
- Triggering Batches
- Handling failed Batch loads
- Validating Batch loads
- Managing data pipelines in Data Factory/Synapse pipelines
- Scheduling data pipelines in Data Factory/Synapse pipelines
- Managing Spark jobs in a pipeline
- Implementing version control for pipeline artifacts
