Monitoring Delta Live Tables pipelines
Monitoring and observability are essential aspects of data engineering as they allow you to track, understand, and troubleshoot the state of your data pipelines. Delta Live Tables provides built-in features for monitoring and observability in the form of an event log that contains all the information about a pipeline, such as audit logs, data quality checks, pipeline progress, and data lineage.
In this recipe, you will learn how to use the event logs captured by Delta Live Tables to monitor Delta Live Tables pipelines in Databricks.
How to do it…
- Create a Delta Live Tables pipeline: Provide the following information:
- Name: This is the name of the Delta Live Tables pipeline. We will name it
Farmers MarketDLT Pipeline. - Product edition: Delta Live Tables comes in three flavors. We will choose the Advanced edition.
- Pipeline mode: We will choose Triggered:
- Choose the source code paths: We will choose the source code for this recipe...
- Name: This is the name of the Delta Live Tables pipeline. We will name it
