Airflow Ops: Development and Deployment
In this chapter, we explore the application of modern ops practices within Apache Airflow, focusing on streamlining development and deployment workflows for your consuming teams.
We’ll begin with a brief exploration and discussion of Airflow deployment methodologies, including Kubernetes, distributed virtual machines, and service providers.
With those foundations established, we’ll dive into common patterns of the common DAG deployment methodologies: bundling, push, and pull. We’ll then address repository structures, secrets management, and localized development.
No matter what path you decide on, it will generally require that you figure out methods to do a few common tasks:
- Customizing your Airflow deployment with additional operators, Python packages, and system packages
- Versioning your DAGs
- Delivering your DAGs to your Airflow deployment
- Managing Airflow configurations
Finally, we&...
