Exploring data pipelines
In Chapter 1, The Story of Data Engineering and Analytics, we talked about the journey of data. We equated data engineering to a vehicle that makes the journey of data possible through sharp turns and roadblocks to ultimately reach its destination as securely and timely as possible. If data engineering is a vehicle, then a data pipeline is the engine that makes the journey possible. The engine is simply a collection of components, each performing a specialized operation. Ultimately, all the parts and components working together can maneuver the vehicle in the desired direction.
In simple terms, a data pipeline is an engine that can move data through various stages of collection, curation, and aggregation to reach its analytics destination. As with the various parts and components of an engine, the data pipeline uses a series of actions to complete its work. Each action performs a specialized task (once or repeatedly) to contribute toward the end goal.
...
