Summary

So, here we are at the end of our project and data platform architecture journey. We covered topics including Spark, Delta Lake, Lambda architecture, Kafka, and MLOps. We also looked at how to build pipelines, package those pipelines, and deploy them centrally. Also, we delved into governance and platform design. This chapter and Chapter 11 built the initial beginnings of production-ready data pipelines. These pipelines can be used as the foundation of your data platform.

It’s my hope that you can take this project as a starting point for your next data platform project and build off it. I have presented many ideas and kernels of best practices. It’s up to you to continue the journey in your own projects. It doesn’t matter whether you choose the same tools or replace them with other tools that you prefer, the fundamental ideas should stay the same.