Extracting data from a Big Data Cluster
In this recipe, you are going to complete the development of an SSIS solution used to extract data from a database hosted on a Big Data Cluster instance in Azure and load it into a database hosted on the local SQL Server instance.
The purpose of this package is to reduce the size of the data that actually needs to be transferred from the external source into the line-of-business database, by restricting the remote source set to only contain data that does not yet exist in the local database.
In general, determining the delta hardly represents a task worthy of the capabilities available in Big Data Clusters or a typical case for using Spark; however, it does represent a very significant element in efficient data warehousing.
Getting ready
This recipe assumes that you have completed the preceding recipe in this chapter, entitled Loading data into a Big Data Cluster, and that the expected data is available in the dbo.NewPeople external...
