Summary

In this chapter, we learned how to analyze a data engineering requirement from scratch, draw a definite conclusion, and extract facts that will help us in our architectural decision-making process. Next, we learned how to profile source data and how such an analysis helps us build better data engineering solutions. Going further, we used facts, requirements, and our analysis to build a robust and effective architecture for a batch-based data engineering problem with a low or medium volume of data. Finally, we mapped the design to build an effective ETL batch-based data ingestion pipeline using Spring Batch and test it. Along the way, you learned how to analyze a data engineering problem from scratch and how to build similar pipelines effectively for when you are presented with a similar problem next time around.

Now that we have successfully architected and developed a batch-based solution for medium- and low-volume data engineering problems, in the next chapter, we will...