Summary
In this chapter, we learned about the ties between data quality and observability. We saw that data quality monitoring by itself is often not sufficient to ensure good data and trust in the pipeline.
We introduced the concept of data observability, which will help the monitoring of the data application by applying these three principles:
- Observability is put into context: Data issues must be put into context in order to avoid interfering with other lineages and applications
- Observability needs synchronicity: The sooner you detect the issue, the better you avoid other applications modifying the data source, as long as the appropriate mechanisms to do so are implemented in your pipeline
- Observability allows continuous validation: Use rules to validate data and ensure data quality at runtime
We learned how to measure the success of our projects and investments in data observability, identifying the fundamental objectives and metrics to monitor in order...
