The pipeline pattern

In the previous sections, we concerned ourselves with organizing concurrency inside the functions themselves. However, we have chained them together pretty much as we would normally, by calling them in sequential order in the main function. In this section, we are going to look at the pipeline pattern, which will allow us to leverage goroutines and channels to chain function calls together. First, let’s discuss what a pipeline is exactly. In 1964, Doug McIlroy wrote the following:

We should have some ways of coupling programs like garden hose – screw in another segment when it becomes necessary to massage data in another way.

This quote neatly expresses the Unix philosophy of composing programs. Many of us are familiar with the concept of Unix pipes, denoted by the | symbol. By using pipes, we can chain Unix programs together. The output of one program becomes the input of the next. For example, we can use cat to read a file, and we can use...