Integrating a Synapse workspace with Microsoft Purview and tracking data lineage
One of the key features of Microsoft Purview is tracking data lineage. Microsoft Purview offers end-to-end tracking of data lineage, using which you can track the life cycle of a particular data asset – for example, if data is loaded from a CSV file into a dedicated SQL pool table and then further processed and stored in a Parquet file. Purview will be able to represent the various phases (CSV file -> SQL table -> Parquet file) of data processing graphically and automatically. Tracking data lineage in data engineering projects is critical as you need to have a bird’s-eye view of the various processing phases of a data asset and its properties.
As you know, a Synapse workspace holds all the data engineering pipelines, which do all the data movement and processing. By integrating a Synapse workspace with Microsoft Purview, we can track data lineage with minimal effort.
In this...
