Running Kafka Locally with Docker and KRaft Mode

Laptop showing Kafka and Docker logos with KRaft mode enabled

Why KRaft Mode Matters

Kafka Raft Metadata (KRaft) is a new mode for Apache Kafka that eliminates the need for Zookeeper. It was introduced in KIP-500 to remove the kafka dependency on Zookeeper. Apache Kafka traditionally required Zookeeper for cluster coordination - until KRaft (Kafka Raft Metadata) mode arrived. This update eliminates Zookeeper dependencies while improving:

Image from Kafka Raft Metadata (KRaft) mode

Simplified architecture: Fewer moving parts
Faster recovery: Metadata changes happen in milliseconds
Easier scaling: More predictable cluster behavior

I recently wanted to setup Kafka for my local development, and after quite a few tries, ngl, I was finally able to do it. Let’s show you how it went.

Prerequisites

Docker (Latest version?)
4GB+ free memory (Kafka containers can be hungry)

Complete Docker Compose Setup

services:
  kafka:
    image: apache/kafka:latest
    container_name: broker
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://:9094
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:9092,EXTERNAL://localhost:9094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@broker:9093
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
    ports:
      - "9094:9094"
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "./opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092 > /dev/null 2>&1",
        ]
      interval: 10s
      timeout: 10s
      retries: 5

  kafka-ui:
    image: provectuslabs/kafka-ui:latest
    container_name: kafka-ui
    ports:
      - "8088:8080"
    environment:
      KAFKA_CLUSTERS_0_NAME: local
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: broker:9092
      KAFKA_CLUSTERS_0_READONLY: "false"
    depends_on:
      kafka:
        condition: service_healthy

Kafka Configuration:

KAFKA_NODE_ID: Unique identifier for the broker
KAFKA_PROCESS_ROLES: Broker and controller roles
KAFKA_LISTENERS: Listeners for different protocols
KAFKA_ADVERTISED_LISTENERS: Addresses advertised to clients
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: Security protocol map
KAFKA_INTER_BROKER_LISTENER_NAME: Inter-broker listener name
KAFKA_CONTROLLER_QUORUM_VOTERS: Controller quorum voters
KAFKA_CONTROLLER_LISTENER_NAMES: Controller listener names
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: Offsets topic replication factor
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: Transaction state log replication factor
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: Transaction state log minimum ISR

Cluster Management Commands

Start all services:

docker-compose up -d

Verify running containers:

docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"

Expected output:

NAMES     STATUS              PORTS
broker            Up 4 minutes (healthy)    9092/tcp, 0.0.0.0:9094->9094/tcp
kafka-ui          Up 4 minutes              0.0.0.0:8088->8080/tcp

Monitoring with Kafka UI

After starting containers, access http://localhost:8088 to:

View real-time broker status
Inspect topic configurations
Monitor consumer lag
Explore message payloads
Manage ACLs (access control lists)

Connection URLs to use in your applications

You will need to connect to the broker using the following URL:

If running outside of docker:

localhost:9094

If running inside of docker:

broker:9092

Topic Management: CLI vs UI

Command-line creation:

docker exec broker /opt/kafka/bin/kafka-topics.sh \
  --topic orders \
  --create \
  --bootstrap-server localhost:9092 \
  --partitions 3 \
  --replication-factor 1

UI alternative:

Navigate to Topics → Create Topic
Set partitions/replication
Click “Submit”

Message Production Patterns

Basic text messages:

docker exec -it broker /opt/kafka/bin/kafka-console-producer.sh \
  --topic orders \
  --bootstrap-server localhost:9092

Structured JSON:

# Using jq for JSON formatting
echo '{"id":1001, "amount":49.99}' | jq -c | docker exec -i broker /opt/kafka/bin/kafka-console-producer.sh \
  --topic orders \
  --bootstrap-server localhost:9092