Summary
In this chapter, we looked at how to take the ML solutions we have been building in the past few chapters and thought about how to scale them up to larger data volumes or higher numbers of requests for predictions. To do this, we mainly focused on Apache Spark as this is the most popular general-purpose engine for distributed computing. During our discussion of Apache Spark, we revisited some coding patterns and syntax we used previously in this book. By doing so, we developed a more thorough understanding of how and why to do certain things when developing in PySpark. We discussed the concept of UDFs in detail and how these can be used to create massively scalable ML workflows.
After this, we explored how to work with Spark on the cloud, specifically through the Elastic Map Reduce (EMR) service provided by AWS. Then, we looked at some of the other ways we can scale our solutions; that is, through serverless architectures and horizontal scaling with containers. In the former...
