Core concepts of Airflow
Apache Airflow is a dynamic, extensible, and flexible framework that allows for the building of workflows as code. Airflow allows for the definition of these automated workflows as code. This allows for better code versioning, development through CI/CD, easy testing, and extensible components and operators from a thriving community of committers.
Airflow is known for its approach to scheduling tasks and workflows. It can take advantage of CRON scheduling or its built-in scheduling functions. In addition, features such as backfilling, allowing for the rerun of pipelines, allow for going back and updating pipelines if the logic changes. This means it has powerful operational components that need to be accounted for as part of the design.
Following these guidelines will help you lay a foundation for scaling your Airflow deployments and increase the effectiveness of your workflows from both authorship and operational viewpoints.
