Chapter 8: Understanding How Runners Execute Pipelines

So far in this book, we have focused on Apache Beam from the user's perspective. We have seen how to code pipelines in the Java Software Development Kit (SDK), how to use Domain-Specific Languages (DSLs) such as SQL, and how to use portability with the Python SDK. In this chapter, we will focus on how the runner executes the pipeline. This will help us if we want to develop a runner for a new technology, debug our code, or improve performance issues.

We will not try to implement our own runner in this chapter. Instead, we will focus on the theoretical concepts that underpin runners. We will explore the building blocks of a typical runner, and this will help us understand how a runner executes our user code.

After describing how runners implement the Beam model, we will conclude this chapter with an in-depth description of window semantics and using metrics for observability. Improving observability is key when attempting...