Components of Airflow
Apache Airflow is a distributed system with multiple components. While distributed systems are inherently complex, the components themselves are relatively simple. It is important to understand each component’s specific role and the role they play in the operation of an Airflow application. Understanding their setup, operation, and maintenance will help you scale confidently and become experts at operating your Airflow environments in production. In this chapter, you will learn about each component’s responsibility and capabilities within the overall application, how to select certain configurations for certain components, and how to determine which capabilities you will need in order to achieve your business goals.
It is important to focus on understanding basic components such as these as there is often optimization and latent capacity opportunities to be found as the total number of tasks and jobs that Airflow orchestrates grows.
In this...
