How to Run Highly Available Kafka on Kubernetes

With Helm charts, stateful sets, and dynamic provisioning

Photo by Ayaz Lalani on Unsplash

Apache Kafka is one of the most popular event-based distributed streaming platforms. LinkedIn first developed it, and technology leaders such as Uber, Netflix, Slack, Coursera, Spotify, and others currently use it.

Though very powerful, Kafka is equally complex and requires a highly available robust platform to run on. Most of the time, engineers struggle to feed and water the Kafka servers, and standing one up and maintaining it is not a piece of cake.

With microservices in vogue and most companies adopting distributed computing, standing up Kafka as the core messaging backbone has its advantages. Kubernetes is a popular choice to run container-based microservices, and using Kafka as an eventing platform is another.

If you are running your microservices in Kubernetes, it will make sense to run your Kafka cluster within Kubernetes to take advantage of its in-built resilience and high-availability. The Kubernetes pods can also easily interact with the Kafka pods within the cluster using the inbuilt Kubernetes service discovery.

Let’s take a peek at how to build a distributed Kafka cluster on Kubernetes through a hands-on exercise. We will use Helm charts and stateful…