In today’s data-driven world, organizations face the challenge of handling vast amounts of data from diverse sources. Data orchestration is the key to managing this complex data landscape efficiently. It involves the coordination, automation, and monitoring of data workflows, ensuring the smooth execution of tasks and the timely delivery of valuable insights.

Orchestration, in the context of software development and data engineering, refers to the process of automating and managing the execution of interconnected tasks or processes to achieve a specific goal. These tasks might involve data processing, workflow scheduling, service provisioning, and more. The purpose of orchestration is to streamline the flow of operations, optimize resource utilization, and ensure that tasks are executed in a well-coordinated manner.

Traditional, manual orchestration is cumbersome and prone to errors, especially as the complexity of workflows increases. However...

Apache Airflow is known within the data engineering community as the go-to open source platform for “developing, scheduling, and monitoring batch-oriented workflows.” (Apache.org Airflow documentation: https://airflow.apache.org/docs/apache-airflow/stable/index.html)

Apache Airflow has emerged as the go-to open source platform for data orchestration and remains the leader as a result of its active development community. It offers a robust and flexible solution to the challenges of managing complex data workflows. Airflow enables data engineers, data scientists, artificial intelligence (AI)/ML engineering, and MLOps and DevOps professionals to design, schedule, and monitor data pipelines with ease.

The power of Apache Airflow lies in its ability to represent data workflows as directed acyclic graphs (DAGs). This intuitive approach allows users to visualize and understand relationships between tasks, making it easier to create and maintain...

Apache Airflow is a dynamic, extensible, and flexible framework that allows for the building of workflows as code. Airflow allows for the definition of these automated workflows as code. This allows for better code versioning, development through CI/CD, easy testing, and extensible components and operators from a thriving community of committers.

Airflow is known for its approach to scheduling tasks and workflows. It can take advantage of CRON scheduling or its built-in scheduling functions. In addition, features such as backfilling, allowing for the rerun of pipelines, allow for going back and updating pipelines if the logic changes. This means it has powerful operational components that need to be accounted for as part of the design.

Following these guidelines will help you lay a foundation for scaling your Airflow deployments and increase the effectiveness of your workflows from both authorship and operational viewpoints.

Why Airflow may not be...

It isn’t sufficient to just know Airflow; there are other skills that are required and will impact your team’s effectiveness at providing value with Airflow. As you’re reading this book, think about your teams’ skills in the following areas to understand if you need additional support or capabilities to make the most out of your platform:

• Python: DAGs and plugins are written in Python. As such, in order for Airflow to be effectively adopted, you should be able to write and understand Python code.
• Application testing: Your company should have relatively mature testing processes and practices in place to ensure that the plugins you write and the DAGs you author can be reasonably expected to function as intended. The team responsible for the health of your Airflow instance(s) and plugins should be able to help teams prepare for upgrades, and the teams responsible for writing workflows should be able to run...

In this chapter, we spent time learning the basics of what data orchestration is and what problems companies and engineers are facing today. In addition, we introduced Apache Airflow, the leading data orchestration and workflow management tool. We also covered what you can expect over the course of this book. It is important to remember that Apache Airflow requires multiple baseline tools and knowledge areas to be most successful. Although these areas are needed for best use, each is a learnable topic and can be picked up at a quick pace.

At the core of Airflow use is Python code. To be the best data engineer using Airflow, you need to understand the core concepts of Python code and how it will orchestrate your stack of data tools. Taking time to review these core concepts and understand the use cases that are being tackled by Airflow will lead to scalable systems of code and optimization opportunities.

In the next chapter, we will introduce the basics of DAGs and tasks...