Designing advanced monitoring
After spending some time learning and practicing logging concepts, we can advance a little more in the subject of monitoring. We can monitor results from all our logging collection work and generate insightful monitoring dashboards and alerts, with the right monitoring message stored.
In this recipe, we will cover the Airflow metrics integrated with StatsD, a platform that collects system statistics, and their purpose to help us achieve a mature pipeline.
Getting ready
This exercise will focus on bringing clarity to the Airflow monitoring metrics and how to build a robust architecture to structure it.
As a requirement for this recipe, it is vital to keep in mind the following basic Airflow architecture:
Figure 10.26 – An Airflow high-level architecture diagram
Airflow components, from a high-level perspective, are composed of the following:
- A web server, where we can access the Airflow UI.
- A...
