Publish-subscribe messaging system

In this section, we will take a look at a different messaging model called the publish/subscribe (Pub/Sub) messaging model.

In this type of model, a subscriber registers its interest in a particular topic or event and is subsequently notified about the event asynchronously. Subscribers have the ability to express their interest in an event, or a pattern of events, and are subsequently notified of any event generated by a publisher that matches their registered interest. These events are generated by publishers. It is different from the PTP messaging model in a way that a topic can have multiple receivers and every receiver receives a copy of each message. In other words, a message is broadcast to all receivers without them having to poll the topic. In the PTP model, the receiver polls the queue for new messages.

A Pub/Sub messaging model is used when you need to broadcast an event or message to many message consumers. Unlike the PTP messaging model, all message consumers (called subscribers) listening on the topic will receive the message.

One of the important aspects of Pub/Sub messaging models is that the topic abstraction is easy to understand and enforces platform interoperability. Moreover, messages can be retained in the topic until they are delivered to the active subscribers.

There is an option to have durable subscriptions in the Pub/Sub model that allows the subscriber to disconnect, reconnect, and collect the messages that were delivered when it was not active. The Kafka messaging system incorporates some of these important design principles.

The following figure describes a basic model of publish/subscribe messaging. Such event services are generally called queues. This kind of interaction need a service that provides storage of the event, a notification service, a way of managing subscriptions and ensuring the efficient guaranteed delivery of the event to destination. Generally, we call this service a queue. Queues act as a neutral mediator between the event producer and event consumer. The producer can produce all the data to queue that they want to and all the consumers will subscribe to the queue that they are interested in. The consumer does not care about the source and the producer does not care about consumers. Consumers can unsubscribe to a queue whenever they want to:

A graphical representation of a publish/subscribe message model

You can deduce the following important points about the Pub/Sub messaging system from the preceding figure:

• Messages are shared through a channel called a topic. A topic is a centralized place where producers can publish, and subscribers can consume, messages.
• Each message is delivered to one or more message consumers, called subscribers.
• The publisher generally does not know and is not aware of which subscribers are receiving the topic messages.
• Messages are pushed to consumers, which means that messages are delivered to consumers without their having to request them. Messages are exchanged through a virtual channel called a topic. Messages delivered to a topic are automatically pushed to all qualified consumers.
• There is no coupling of the producers to the consumers. Subscribers and publishers can be added dynamically at runtime, which allows the system to grow or shrink in complexity over time.
• Every client that subscribes to a topic receives its own copy of messages published to that topic. A single message produced by one publisher may be copied and distributed to hundreds, or even thousands, of subscribers.

You should use the Pub/Sub model when you want to broadcast a message or event to multiple message consumers. The important point here is that multiple consumers may consume the message.

By design, the Pub/Sub model will push copies of the message out to multiple subscribers. Some common examples are notifying exceptions or errors and change the notification of a particular data item in the database.

Any situation where you need to notify multiple consumers of an event is a good use of the Pub/Sub model. For example, you want to send out a notification to a topic whenever an exception occurs in your application or a system component. You may not know how that information will be used or what types of component will use it. Will the exception be e-mailed to various parties of interest? Will a notification be sent to a beeper or pager? This is the beauty of the Pub/Sub model. The publisher does not care or need to worry about how the information will be used. It simply publishes it to a topic.