Scheduler
The previous sections covered how tasks are executed and the best way to enable different use cases of tasks instances to be executed. To determine when these tasks should be scheduled for execution, we need to take a closer look at the Scheduler and its multiple responsibilities:
- DAG Parsing: The scheduler continuously parses DAG files in the DAG Directory to look for new tasks to schedule. It determines the execution order based on dependencies set within the DAGs.
- Heartbeat Mechanism: The scheduler operates in a loop, often referred to as the “heartbeat”, where it continually checks for tasks to run, schedules them, and then sleeps for a short duration before checking again.
- Dynamic Task Scheduling: Unlike traditional cron setups where jobs are fixed, the Airflow scheduler dynamically determines which tasks should run based on their dependencies and state. This allows for more complex workflows with conditional execution paths.
Some...
