Core stream processing patterns
In the previous section, we learned about a few commonly used batch processing patterns. In this section, we will discuss various stream processing patterns. Let’s get started.
The outbox pattern
With modern data engineering, monolithic applications have been replaced by a series of microservices application working in tandem. Also, it is worth noting that microservices usually don’t share their databases with other microservices. The database session commits and interservice communications should be atomic and in real time to avoid inconsistencies and bugs. Here, the outbox pattern comes in handy. The following diagram shows the generic architecture of the outbox pattern:
Figure 7.7 – The outbox pattern
As we can see, a microservice (here, Service 1) writes a transaction to not only the required table where online reads and writes happen (denoted in the diagram as Online Table) but also to an Outbox...
