Working with Large Data

There are several ways you can utilize Go concurrency primitives to process large amounts of data efficiently. Unlike threads, goroutines can be created without much overhead. Having thousands of goroutines in a program is common. With that in mind, we will look at some common patterns of dealing with large amounts of data concurrently.

This chapter includes the following recipes:

• Worker pools
• Connection pools
• Pipelines
• Working with large result sets