Monitoring core Airflow components
All of the components we will discuss here are critical to ensuring a functioning Airflow deployment. Generally, all of them should be monitored with a bare minimum check of Is it on? and if a component is not, an alert should surface to your team for investigation. The easiest way to check this is to query the REST API on the web server at `/health/`
; this will return a JSON object that can be parsed to determine whether components are healthy and, if not, when they were last seen.
Scheduler
This component needs to be running and working effectively in order for tasks to be scheduled for execution.
When the scheduler service is started, it also starts a `/health`
endpoint that can be checked by an external process with an active monitoring approach.
The returned signal does not always indicate that the scheduler is working properly, as its state is simply indicative that the service is up and running. There are many scenarios where the...