Understanding how Kafka uses logs

Kafka maintains logs that are written to by producers and read by consumers. The following sections will explain topics, consumers, and producers.

Topics

Apache Kafka uses logs to store data – records. Logs in Kafka are called topics. A topic is like a table in a database. In the previous chapter, you tested your Kafka cluster by creating a topic named dataengineering. The topic is saved to disk as a log file. Topics can be a single log, but usually they are scaled horizontally into partitions. Each partition is a log file that can be stored on another server. In a topic with partitions, the message order guarantee no longer applies to the topic, but only each partition. The following diagram shows a topic split into three partitions:

Figure 13.2 – A Kafka topic with three partitions

The preceding topic – Transactions – has three partitions labeled P1, P2, and P3. Within each partition, the...