Summary

In this chapter, we started by discussing various popular batch processing patterns. We covered five commonly used patterns to solve batch processing problems. We also looked at examples of those patterns and real-world scenarios where such patterns are used. Then, we looked at five popular patterns available to architect stream processing pipelines and how they are used to solve real-world problems in data engineering. Next, we learned about the Lambda and Kappa architectures and how they are useful for both batch and stream processing. Finally, we learned what serverless architecture is and looked at two popular serverless architectures that are used to solve many data engineering problems in the cloud.

At this point, we know how to implement batch and streaming solutions, as well as have a fair idea of different data engineering patterns that are commonly used across the industry. Now, it is time to put some amount of security and data governance into our solutions. In...