Kafka architecture

This section aims to explain ins and outs of Apache Kafka. We will try to dive deep into its architecture and then, later on try expanding each part of it's architecture's components in a bit more detail.

So, let's stream forward.

Core architecture principles of Kafka

The main motivation behind Kafka when developed by LinkedIn’s engineering team was

To create a unified messaging platform to cater to real-time data from various applications in a big organization.

- LinkedIn

 

There are core architecture principles based on which Kafka was conceived and designed. The bulleted points sum up these principles:

• Maximize performance (compression and B-tree usage is an example)
• Wherever possible, core kernel capabilities to offload work to drive optimization and performance (zero-copy and direct use of Linux filesystem cache is an example)
• Distributed architecture
• Fault tolerance
• Durability of messages
• Wherever possible, eliminate redundant work
• Offload responsibility of tasks to consuming...