Overall architecture
Apache Airflow at its core is a set of components working together, allowing you to build and run workflows or Directed Acyclic Graphs (DAGs). These workflows run on top of several microservices that coordinate workers to execute tasks on a specified schedule.
Apache Airflow’s architecture comprises several key components that work together to orchestrate data pipelines efficiently. The primary components include:
- Metadata Database: Stores metadata related to DAG runs, task instance status, and other key metadata. It allows your Airflow instance to keep track of task states, DAG versions, and offers persistence.
- Scheduler: Responsible for triggering the task instances, based on assigned time or an external trigger. It will check the DAGs constantly to see if they can be triggered.
- Triggerer: responsible for storing and executing asynchronous functions that are instantiated from Trigger classes.
- Executor: Determines how tasks are going...
