Kafka Cluster Using Apache Kafka Docker Image
Created on: Jan 29, 2025
Quickly start kafka server using docker compose file.
services: controller-1: image: apache/kafka:latest container_name: controller-1 environment: KAFKA_NODE_ID: 1 KAFKA_PROCESS_ROLES: controller KAFKA_LISTENERS: CONTROLLER://:9093 KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER KAFKA_CONTROLLER_QUORUM_VOTERS: 1@controller-1:9093,2@controller-2:9093,3@controller-3:9093 KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0 controller-2: image: apache/kafka:latest container_name: controller-2 environment: KAFKA_NODE_ID: 2 KAFKA_PROCESS_ROLES: controller KAFKA_LISTENERS: CONTROLLER://:9093 KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER KAFKA_CONTROLLER_QUORUM_VOTERS: 1@controller-1:9093,2@controller-2:9093,3@controller-3:9093 KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0 controller-3: image: apache/kafka:latest container_name: controller-3 environment: KAFKA_NODE_ID: 3 KAFKA_PROCESS_ROLES: controller KAFKA_LISTENERS: CONTROLLER://:9093 KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER KAFKA_CONTROLLER_QUORUM_VOTERS: 1@controller-1:9093,2@controller-2:9093,3@controller-3:9093 KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0 broker-1: image: apache/kafka:latest container_name: broker-1 ports: - 29092:9092 environment: KAFKA_NODE_ID: 4 KAFKA_PROCESS_ROLES: broker KAFKA_LISTENERS: "PLAINTEXT://:19092,PLAINTEXT_HOST://:9092" KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://broker-1:19092,PLAINTEXT_HOST://localhost:29092" KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT KAFKA_CONTROLLER_QUORUM_VOTERS: 1@controller-1:9093,2@controller-2:9093,3@controller-3:9093 KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0 depends_on: - controller-1 - controller-2 - controller-3 broker-2: image: apache/kafka:latest container_name: broker-2 ports: - 39092:9092 environment: KAFKA_NODE_ID: 5 KAFKA_PROCESS_ROLES: broker KAFKA_LISTENERS: "PLAINTEXT://:19092,PLAINTEXT_HOST://:9092" KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://broker-2:19092,PLAINTEXT_HOST://localhost:39092" KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT KAFKA_CONTROLLER_QUORUM_VOTERS: 1@controller-1:9093,2@controller-2:9093,3@controller-3:9093 KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0 depends_on: - controller-1 - controller-2 - controller-3 broker-3: image: apache/kafka:latest container_name: broker-3 ports: - 49092:9092 environment: KAFKA_NODE_ID: 6 KAFKA_PROCESS_ROLES: broker KAFKA_LISTENERS: "PLAINTEXT://:19092,PLAINTEXT_HOST://:9092" KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://broker-3:19092,PLAINTEXT_HOST://localhost:49092" KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT KAFKA_CONTROLLER_QUORUM_VOTERS: 1@controller-1:9093,2@controller-2:9093,3@controller-3:9093 KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0 depends_on: - controller-1 - controller-2 - controller-3
Here we are creating three kafka broker and three controller. 💡 In a KRaft-based Kafka setup, controllers replace Zookeeper and form a quorum.
Run all the controller and broker.
docker compose up -d
[+] Running 6/6 ✔ controller-1 Pulled 2.9s ✔ controller-3 Pulled 2.9s ✔ broker-1 Pulled 2.9s ✔ broker-2 Pulled 2.9s ✔ broker-3 Pulled 2.9s ✔ controller-2 Pulled 2.9s [+] Running 7/7 ✔ Network apache-kafka-docker_default Created 0.1s ✔ Container controller-2 Started 0.7s ✔ Container controller-3 Started 0.7s ✔ Container controller-1 Started 0.7s ✔ Container broker-1 Started 0.9s ✔ Container broker-2 Started 0.8s ✔ Container broker-3 Started 0.9s
Let's create a topic and test consumer and producer using terminal of docker container.
docker exec --workdir /opt/kafka/bin/ -it broker-1 /bin/bash # Create a topic ./kafka-topics.sh --bootstrap-server broker-1:19092,broker-2:19092,broker-3:19092 --create --topic test-topic # Created topic test-topic. # command to produce the event ./kafka-console-producer.sh --bootstrap-server broker-1:19092,broker-2:19092,broker-3:19092 --topic test-topic
Open a another terminal and enter into another broker say broker 2.
docker exec --workdir /opt/kafka/bin/ -it broker-2 /bin/bash ./kafka-console-consumer.sh --bootstrap-server broker-1:19092,broker-2:19092,broker-3:19092 --topic test-topic --from-beginning
After this type any message in terminal 2, you will get same message in terminal 1.
Other helpful shell command for kafka.
# check all the topics ./kafka-topics.sh --list --bootstrap-server broker-2:19092 ./kafka-topics.sh --delete --topic orderEvent --bootstrap-server broker-1:19092,broker-2:19092,broker-3:19092