How to Handle Duplicate Messages and Message Ordering in Kafka

Dealing with the challenges faced when using Apache Kafka

Image by author

Event-driven architecture is one of the ways to implement interprocess communication between some services in a microservice architecture. A message-based application typically uses a message broker which acts as an intermediate layer between the services.

Apache Kafka is one of the tools which allows implementing a broker-based messaging approach. This tool resolves a lot of issues on its own from the box, but as a developer, we do come across challenges that should be taken into account. In the next few sections, we’ll consider some of them and have a look at how to resolve them.

1. Message Ordering

Some applications are pretty sensitive to the order of messages being processed by consumers. Let’s imagine that the user creates some Order, modifies it, and finally cancels. This means, that the Producer sends to Message Broker the following messages with commands: ORDER_CREATE, ORDER_MODIFY, ORDER_CANCEL.

In Kafka, we can create a special topic Order which will get all messages (commands) that will process any operation related to the orders. If our application has only one Consumer and one topic with one partition, you will never come across an ordering issue — since Kafka sends all messages within a topic Partition to Consumer in the same order which it gets from the Producer:

Figure 1 — Message ordering with one consumer and one partition in Kafka Topic

I suppose that you agree that having only one Consumer that processes all messages related to Order topic is not a scalable approach for an application that may have a lot of customers.

Hopefully, Kafka has a solution here and it offers to split any topic into partitions. Each read or write operation for a partition happens in parallel. It means that instead of having one Consumer we may have as many Consumers as there are partitions in a Topic. Kafka assigns each partition to a single Consumer.

Here, of course, we should pay our attention, since theoretically, all commands related to one Order…