Member-only story
An Introduction to Apache Kafka
Kafka’s basic components and how to write a basic producer and consumer
Kafka was developed by LinkedIn in 2010, and it has been a top-level Apache project since 2012. It is a highly scalable, durable, robust, and fault-tolerant publish-subscribe event streaming platform.
I spent some time working with Kafka as a software developer on a previous project. I want to share some things I’ve found useful to know when I was working with Kafka producers and consumers for the first time. Namely, this article is an introduction to Kafka — the basic components of Kafka, how to write a producer and consumer, and also what language support there is for it. Without further ado, let’s dive into the backbone of Kafka first and discuss some Kafka basics.
Let’s explore what some of the common use cases of Kafka are:
- Real-time processing of application activity tracking, like searches.
- Stream processing
- Log aggregation, where Kafka consolidates logs from multiple services (producers) and standardises the format for consumers.
- An interesting use case that has emerged is the microservices architecture. Kafka can be a suitable choice for event sourcing microservices where a lot of events are generated and we want to keep track of the sequence of events (i.e. what has happened).
There are multiple case studies on the use of Kafka, such as from The New York Times and Netflix.
Basic Components
Let’s talk a little about the basic components that Kafka uses for its publish-subscribe messaging system. A producer is an entity/application that publishes data to a Kafka cluster, which is made up of brokers. A broker is responsible for receiving and storing the data when a producer publishes. A consumer then consumes data from a broker at a specified offset, i.e. position.
That is, it’s a multi-producer, multi-consumer structure, and it looks something like this:
