Summary

Implementing a data lake is a paradigm change within an organization. Delta Lake provides a solution for this when we are dealing with streams of data from different sources, when the schema of the data might change over time, and when we need to have a system that is reliable against data mishandling and easy to audit.

Delta Lake fills the gap between the functionality of a data warehouse and the benefits of a data lake while also overcoming most of its challenges.

Schema validation ensures that our ETL pipelines maintain reliability against changes in the tables. It informs us of this by raising an exception if any mismatches arise and the data becomes contaminated. If the change was intentional, we can use schema evolution.

Time travel allows us to access historic versions of data, thanks to its ordered transaction log. This keeps track of every operation that's performed in Delta tables. This is useful when we need to define pipelines that need to query different...