Kafka cluster without Zookeeper

Kafka 3.3 introduced KRaft for creating clusters without needing to create Zookeeper

Photo by Shubham's Web3 on Unsplash

Introduction

In the world of modern data processing and real-time event streaming, Apache Kafka has emerged as a powerful and versatile tool. Kafka is widely used for building scalable, fault-tolerant, and high-throughput data pipelines, making it a go-to choice for organizations dealing with vast amounts of data.

If you’re looking to set up a Kafka cluster for your data streaming needs, you’re in the right place. In this article, we’ll guide you through two popular methods for deploying a Kafka cluster. We’ll explore how to use Docker Compose and Helm Chart to set up and manage your Kafka cluster, each offering its own advantages and use cases.

Quick Start

In the following section, I’ll describe two popular types of deploying Kafka clusters using Docker Compose and Helm Chart.

1- Deploy with Docker Compose

Here’s a sample Kafka cluster configuration that you can run using Docker Compose:

version: '3'
services:
  
  kafka-1:
    image: 'bitnami/kafka:3.3.1'
    container_name: kafka-1
    environment:
      - KAFKA_ENABLE_KRAFT=yes
      - KAFKA_CFG_PROCESS_ROLES=broker,controller
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9094
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      - KAFKA_CFG_BROKER_ID=1
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@kafka-1:9094,2@kafka-2:9094,3@kafka-3:9094
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka-1:9092
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_KRAFT_CLUSTER_ID=r4zt_wrqTRuT7W2NJsB_GA
    ports:
      - 9192:9092

  kafka-2:
    image: 'bitnami/kafka:3.3.1'
    container_name: kafka-2
    environment:
      - KAFKA_ENABLE_KRAFT=yes
      - KAFKA_CFG_PROCESS_ROLES=broker,controller
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9094
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      - KAFKA_CFG_BROKER_ID=2
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@kafka-1:9094,2@kafka-2:9094,3@kafka-3:9094
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka-2:9092
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_KRAFT_CLUSTER_ID=r4zt_wrqTRuT7W2NJsB_GA
    ports:
      - 9292:9092
      
  kafka-3:
    image: 'bitnami/kafka:3.3.1'
    container_name: kafka-3
    environment:
      - KAFKA_ENABLE_KRAFT=yes
      - KAFKA_CFG_PROCESS_ROLES=broker,controller
      - KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9094
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      - KAFKA_CFG_BROKER_ID=3
      - KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=1@kafka-1:9094,2@kafka-2:9094,3@kafka-3:9094
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka-3:9092
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_KRAFT_CLUSTER_ID=r4zt_wrqTRuT7W2NJsB_GA
    ports:
      - 9392:9092

  kafka-ui:
    container_name: kafka-ui
    image: 'provectuslabs/kafka-ui:latest'
    ports:
      - "8080:8080"
    environment:
      - KAFKA_CLUSTERS_0_BOOTSTRAP_SERVERS=kafka-1:9092
      - KAFKA_CLUSTERS_0_NAME=r4zt_wrqTRuT7W2NJsB_GA

Save the above content into a file named docker-compose.yaml and run the cluster using this command:

$ docker-compose up -d -f docker-compose.yaml

2- Deploy using Helm Chart

To deploy Kafka without Zookeeper inside a Kubernetes cluster, you can use the following commands:

repositories:
  - name: kafka-repo
    url: registry-1.docker.io/bitnamicharts
    oci: true

releases:
  - name: kafka
    namespace: test
    createNamespace: false
    chart: kafka-repo/kafka
    version: 26.8.3
    values:
      - ./values.yaml

Save the above content into a file named helmfile.yaml and run the following command:

$ helmfile apply -f helmfile.yaml
$ 
$ kubectl -n test get service
NAME                        TYPE        CLUSTER-IP        EXTERNAL-IP   PORT(S)                      AGE
kafka                       ClusterIP   10.43.54.217      <none>        9092/TCP                     3d4h
kafka-controller-headless   ClusterIP   None              <none>        9094/TCP,9092/TCP,9093/TCP   3d4h

You can use the following information within your application to connect to the Kafka cluster above:

spring.kafka.bootstrap-servers=kafka:9092
spring.kafka.properties.security.protocol=SASL_PLAINTEXT
spring.kafka.properties.sasl.mechanism=PLAIN
spring.kafka.properties.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="user1" password="xxxxxxx";

you can find the password of user1 using the following command:

$ kubectl -n test edit secrets kafka-user-passwords

and here’s the password:

apiVersion: v1
data:
  client-passwords: Nk5xWTRLNGJKWA==              <==== password of user1
  controller-password: TjRmeTRFRmlCcQ==
  inter-broker-password: OTJzZWx0SUYwSQ==
  system-user-password: Nk5xWTRLNGJKWA==
kind: Secret

decode using base64

$ echo "Nk5xWTRLNGJKWA==" | base64 -d
6NqY4K4bJX

Why does Kafka depend on Zookeeper?

Kafka uses Zookeeper as a Service Discovery tool, meaning each Kafka node that wants to find other nodes needs to discover their addresses and status through service discovery. Kafka also uses Zookeeper for other purposes such as Leader Election, Broker Discovery, Configuration Management, and Health Monitoring.

What is KRaft?

KRaft is a new algorithm developed by Kafka based on the Raft Consensus algorithm, which is suitable for achieving consensus between trusted parties. Kafka no longer needs Zookeeper because it can achieve its goals using the Raft consensus algorithm

GitHub

You can find the code used in this documentation in the following GitHub repository:

devops-diagrams/kafka-without-zookeeper at main · mlkmhd/devops-diagrams

Support My Blog ☕️

If you enjoy my technical blog posts and find them valuable, please consider buying me a coffee at here. Your support goes a long way in helping me produce more quality articles and content. If you have any feedback or suggestions for improving my code, please leave a comment on this post or send me a message on my LinkedIn.