Modern applications are becoming increasingly complicated. Communication across many services and processing large amounts of data are just a few of the issues developers encounter. As they help facilitate communication between different application services, message brokers have become a crucial and irreplaceable component of distributed systems. They can process a wide range of data in only a brief period of time.

A message broker is, in simple terms, software that enables message exchange between different services, applications, or systems. This message structure is independent of other services, therefore the services in question can become loosely coupled, and scalable.

Message brokers reliably handle storing, routing, and delivering messages to intended destinations. This structure allows senders to send messages without knowing the receiver’s state or availability. Services can be implemented in different programming languages and evolve independently, as long as the agreed-upon message format remains consistent.

Every message broker consists of a few key components:

• Producer - an endpoint that sends any type of data to the broker
• Consumer - an endpoint that receives and processes messages
• Queue - stores incoming messages until they are consumed, enabling decoupling between producers and consumers

There are multiple messaging models that brokers can be based on. Two main ones are point-to-point messaging and publish/subscribe model.

This type of messaging implies that each message is consumed by only one consumer - a one-to-one relationship. The point-to-point model is sometimes referred to simply as a queue model. Every message that gets sent will be consumed only once (or at-least-once). A queue is used to store the messages.

Here we have two types of message exchange:

• Fire-and-forget - the producer won’t wait for any further response from the message queue, it has no interaction with the consumer
• Request/reply model - the producer sends the message to the queue and waits for a response to continue with further action

In the Publish/Subscribe model (sometimes referred to as Fan-out model), the producer will be called publisher while the consumer plays the role of subscriber. Messages will be published on a topic. Multiple consumers can be subscribed to topics. From here, all messages that get sent to a topic will be further distributed to all subscribers, so each of them receives a copy of the message, independently of each other. Publishers don’t have any knowledge about when or where the message could be consumed.

Some of the benefits of using message brokers have been mentioned earlier in the article - improved scalability, enabling easier asynchronous communication, and improved reliability.

When it comes to facilitating asynchronous communication, separating the producer from the consumer makes the services less dependent on one another and thus effectively facilitating loose coupling. Information can be transmitted without thinking about the other service's availability or time frame. This can bring improved system performance and increased user experience.

Message brokers guarantee that all messages will be delivered and consumed, or they will at least attempt to be consumed. In case of failure on the consumer side, brokers provide their own error handling in the form of redelivering the message (immediately or after some specified time).

Decoupling the services ensures that applications can become more scalable. Application components are designed to communicate through these message brokers and to be independent of other points in the system. New services can be added to the architecture without tightly coupling them to existing components.

However, introducing a new structure does introduce some trade-offs as well. One of them being the increased complexity of the system. Message brokers, like any new component, introduce new architectural complexity. They require careful design and configuration to be set up correctly.

Another disadvantage is the so-called operational overhead. Tasks such as configuration, maintenance, and similar might become the responsibility of the development team. There could also be a potential learning curve if the team is new to the technology since every message broker has its own specific way of functioning.

Another concept that message brokers introduce as a trade-off is eventual consistency. Since the messages can be processed at different times, we technically don’t have the knowledge about when the data will be consumed. This leads to temporary inconsistencies in the system. If the system is not carefully designed and implemented, the inconsistencies might stay in the system for a prolonged time period, leaving it in an inconsistent state.

As we can see, some disadvantages are tightly related to the challenges of microservice architecture in general - complexity, consistency issues, harder debugging, and so on.

Many messaging systems are present today, but the ones that we can consider the most popular are Apache Kafka, RabbitMQ, and Amazon SQS.

Although Kafka is often grouped with traditional message brokers, it is more accurately described as an event streaming platform. It was created with the intention to handle large amounts of data in a short amount of time. Kafka can handle millions of messages per second with minimal impact on performance. It is based on the publish/subscribe messaging model. Kafka’s producer sends messages, which are referred to as records, to topics. Topics exist within a cluster. Consumers can, like in a classic pub/sub model, subscribe to one or multiple topics and consume records. While it is based on the pub/sub model, it can handle point-to-point messaging as well as using partitions. By default, it persists messages on the disk. It is usually used by services with stream processing, log aggregation, or similar, due to its high throughput. Kafka is used by some of the bigger companies, like Netflix, Spotify, Pinterest, Uber, etc.

RabbitMQ is considered the most popular and used message broker. Unlike Kafka, it has a simple configuration and is pretty lightweight. It was designed for moderate throughput but it still performs very well.

RabbitMQ supports both point-to-point and pub/sub models. The message is first published to exchange, which is responsible for routing the message to the appropriate queue. Exchange can route the message to one queue or to more of them who share a certain pattern - so-called topics. Also, an exchange can send messages to every queue it knows about and this process is called fanout.

SQS stands for Simple Queue Service. It is a fully managed message broker provided by Amazon. Its main advantage is that the responsibility for many technical details gets transferred to AWS, therefore the complexity of managing the service is removed. What is specific to SQS is that it recognizes a few types of queues.

Firstly, we have standard queues which guarantee us at-least-once delivery. This means that the message will be delivered at least once, but it is possible that another copy of the message exists - in case of an error handling process for example. Standard queues also introduce best-effort ordering. Simply, the messages might get delivered in an order different from the one they were received in.

Another type of queue is a FIFO (first-in-first-out) queue. These queues guarantee exactly once delivery - messages will be delivered only once. In practice, applications should still implement idempotent consumers to safely handle failure scenarios. Messages in this type of queue will be available until the consumer consumes or deletes them. FIFO Delivery is enabled, which means that the order of receiving and delivering the messages is always the same.

Messages that were not consumed properly can be put in a dead letter queue. DLQ will be of the same type as the basic queue (either standard or FIFO). It can be very useful for debugging and tracking irregularities that can occur because it gives us an insight into why the consumer failed to process a message.

Another fully managed messaging service from AWS is the Simple Notification Service (SNS). It follows the publish/subscribe messaging model, with publishers sending messages to topics and subscribers receiving messages from those topics.

When you publish a message to a topic, Amazon SNS replicates it and distributes it to applications that have subscribed to the topic.

Topics can be standard or FIFO (first-in-first-out). Standard topics are designed to support maximum throughput, a nearly unlimited number of messages per second. FIFO topics, on the other hand, can handle high but limited throughput. They are used to improve messaging between applications, when the order of actions and events is crucial, or duplicates cannot be permitted, for example, bank transaction logging or flight tracking.

Just like with SQS, messages that were not processed will end up in a Dead Letter Queue (DLQ), where they can be further analyzed.

Complex systems will always introduce more challenges in communication between components. A valuable and functional solution to some of these problems are message brokers. They offer us reliability, easier error handling, and other out-of-the-box solutions. Of course, they can introduce some trade-offs as well, but even with them, they have become a crucial part of every microservice-based system. See you on our blog on the next topic.