Kafka cluster without Zookeeper
Kafka 3.3 introduced KRaft for creating clusters without needing to create Zookeeper
Introduction
In the world of modern data processing and real-time event streaming, Apache Kafka has emerged as a powerful and versatile tool. Kafka is widely used for building scalable, fault-tolerant, and high-throughput data pipelines, making it a go-to choice for organizations dealing with vast amounts of data.
If you’re looking to set up a Kafka cluster for your data streaming needs, you’re in the right place. In this article, we’ll guide you through two popular methods for deploying a Kafka cluster. We’ll explore how to use Docker Compose and Helm Chart to set up and manage your Kafka cluster, each offering its own advantages and use cases.
Quick Start
In the following section, I’ll describe two popular types of deploying Kafka clusters using Docker Compose and Helm Chart.
1- Deploy with Docker Compose
Here’s a sample Kafka cluster configuration that you can run using Docker Compose:
version: '3'
services:
kafka-1:
image: 'bitnami/kafka:3.3.1'
container_name: kafka-1
environment:
- KAFKA_ENABLE_KRAFT=yes
- KAFKA_CFG_PROCESS_ROLES=broker,controller
- KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
- KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9094
- KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
- KAFKA_CFG_BROKER_ID=1
- KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@kafka-1:9094,2@kafka-2:9094,3@kafka-3:9094
- KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka-1:9092
- ALLOW_PLAINTEXT_LISTENER=yes
- KAFKA_KRAFT_CLUSTER_ID=r4zt_wrqTRuT7W2NJsB_GA
ports:
- 9192:9092
kafka-2:
image: 'bitnami/kafka:3.3.1'
container_name: kafka-2
environment:
- KAFKA_ENABLE_KRAFT=yes
- KAFKA_CFG_PROCESS_ROLES=broker,controller
- KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
- KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9094
- KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
- KAFKA_CFG_BROKER_ID=2
- KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@kafka-1:9094,2@kafka-2:9094,3@kafka-3:9094
- KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka-2:9092
- ALLOW_PLAINTEXT_LISTENER=yes
- KAFKA_KRAFT_CLUSTER_ID=r4zt_wrqTRuT7W2NJsB_GA
ports:
- 9292:9092
kafka-3:
image: 'bitnami/kafka:3.3.1'
container_name: kafka-3
environment:
- KAFKA_ENABLE_KRAFT=yes
- KAFKA_CFG_PROCESS_ROLES=broker,controller
- KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
- KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9094
- KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
- KAFKA_CFG_BROKER_ID=3
- KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@kafka-1:9094,2@kafka-2:9094,3@kafka-3:9094
- KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka-3:9092
- ALLOW_PLAINTEXT_LISTENER=yes
- KAFKA_KRAFT_CLUSTER_ID=r4zt_wrqTRuT7W2NJsB_GA
ports:
- 9392:9092
kafka-ui:
container_name: kafka-ui
image: 'provectuslabs/kafka-ui:latest'
ports:
- "8080:8080"
environment:
- KAFKA_CLUSTERS_0_BOOTSTRAP_SERVERS=kafka-1:9092
- KAFKA_CLUSTERS_0_NAME=r4zt_wrqTRuT7W2NJsB_GA
Save the above content into a file named docker-compose.yaml
and run the cluster using this command:
$ docker-compose up -d -f docker-compose.yaml
2- Deploy using Helm Chart
To deploy Kafka without Zookeeper inside a Kubernetes cluster, you can use the following commands:
repositories:
- name: kafka-repo
url: registry-1.docker.io/bitnamicharts
oci: true
releases:
- name: kafka
namespace: test
createNamespace: false
chart: kafka-repo/kafka
version: 26.8.3
values:
- ./values.yaml
Save the above content into a file named helmfile.yaml
and run the following command:
$ helmfile apply -f helmfile.yaml
$
$ kubectl -n test get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kafka ClusterIP 10.43.54.217 <none> 9092/TCP 3d4h
kafka-controller-headless ClusterIP None <none> 9094/TCP,9092/TCP,9093/TCP 3d4h
You can use the following information within your application to connect to the Kafka cluster above:
spring.kafka.bootstrap-servers=kafka:9092
spring.kafka.properties.security.protocol=SASL_PLAINTEXT
spring.kafka.properties.sasl.mechanism=PLAIN
spring.kafka.properties.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="user1" password="xxxxxxx";
you can find the password of user1
using the following command:
$ kubectl -n test edit secrets kafka-user-passwords
and here’s the password:
apiVersion: v1
data:
client-passwords: Nk5xWTRLNGJKWA== <==== password of user1
controller-password: TjRmeTRFRmlCcQ==
inter-broker-password: OTJzZWx0SUYwSQ==
system-user-password: Nk5xWTRLNGJKWA==
kind: Secret
decode using base64
$ echo "Nk5xWTRLNGJKWA==" | base64 -d
6NqY4K4bJX
Why does Kafka depend on Zookeeper?
Kafka uses Zookeeper as a Service Discovery tool, meaning each Kafka node that wants to find other nodes needs to discover their addresses and status through service discovery. Kafka also uses Zookeeper for other purposes such as Leader Election, Broker Discovery, Configuration Management, and Health Monitoring.
What is KRaft?
KRaft
is a new algorithm developed by Kafka based on the Raft Consensus algorithm, which is suitable for achieving consensus between trusted parties. Kafka no longer needs Zookeeper because it can achieve its goals using the Raft consensus algorithm
GitHub
You can find the code used in this documentation in the following GitHub repository:
Support My Blog ☕️
If you enjoy my technical blog posts and find them valuable, please consider buying me a coffee at here. Your support goes a long way in helping me produce more quality articles and content. If you have any feedback or suggestions for improving my code, please leave a comment on this post or send me a message on my LinkedIn.