Run Kafka Locally with Docker Compose and KRaft Mode

Why KRaft Mode Matters

Kafka Raft Metadata (KRaft) is a new mode for Apache Kafka that eliminates the need for Zookeeper. It was introduced in KIP-500 to remove the kafka dependency on Zookeeper. Apache Kafka traditionally required Zookeeper for cluster coordination - until KRaft (Kafka Raft Metadata) mode arrived. This update eliminates Zookeeper dependencies while improving:

Image from Kafka Raft Metadata (KRaft) mode

• Simplified architecture: Fewer moving parts
• Faster recovery: Metadata changes happen in milliseconds
• Easier scaling: More predictable cluster behavior

I recently wanted to setup Kafka for my local development, and after quite a few tries, ngl, I was finally able to do it. Let's show you how it went.

This guide focuses on the local broker loop. If you are also tightening deployment fundamentals, see the Next.js self-hosting guide and the Go Docker image optimization guide for the same container-first production mindset.

Prerequisites

• Docker (Latest version?)
• 4GB+ free memory (Kafka containers can be hungry)

Complete Docker Compose Setup

services:
  kafka:
    image: apache/kafka:latest
    container_name: broker
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://:9094
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:9092,EXTERNAL://localhost:9094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@broker:9093
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
    ports:
      - "9094:9094"
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "./opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092 > /dev/null 2>&1",
        ]
      interval: 10s
      timeout: 10s
      retries: 5

  kafka-ui:
    image: provectuslabs/kafka-ui:latest
    container_name: kafka-ui
    ports:
      - "8088:8080"
    environment:
      KAFKA_CLUSTERS_0_NAME: local
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: broker:9092
      KAFKA_CLUSTERS_0_READONLY: "false"
    depends_on:
      kafka:
        condition: service_healthy

Kafka Configuration:

• KAFKA_NODE_ID: Unique identifier for the broker
• KAFKA_PROCESS_ROLES: Broker and controller roles
• KAFKA_LISTENERS: Listeners for different protocols
• KAFKA_ADVERTISED_LISTENERS: Addresses advertised to clients
• KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: Security protocol map
• KAFKA_INTER_BROKER_LISTENER_NAME: Inter-broker listener name
• KAFKA_CONTROLLER_QUORUM_VOTERS: Controller quorum voters
• KAFKA_CONTROLLER_LISTENER_NAMES: Controller listener names
• KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: Offsets topic replication factor
• KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: Transaction state log replication factor
• KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: Transaction state log minimum ISR

Cluster Management Commands

Start all services:

docker-compose up -d

Verify running containers:

docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"

Expected output:

NAMES     STATUS              PORTS
broker            Up 4 minutes (healthy)    9092/tcp, 0.0.0.0:9094->9094/tcp
kafka-ui          Up 4 minutes              0.0.0.0:8088->8080/tcp

Monitoring with Kafka UI

After starting containers, access http://localhost:8088 to:

• View real-time broker status
• Inspect topic configurations
• Monitor consumer lag
• Explore message payloads
• Manage ACLs (access control lists)

Connection URLs to use in your applications

You will need to connect to the broker using the following URL:

• If running outside of docker:

localhost:9094

• If running inside of docker:

broker:9092

Topic Management: CLI vs UI

Command-line creation:

docker exec broker /opt/kafka/bin/kafka-topics.sh \
  --topic orders \
  --create \
  --bootstrap-server localhost:9092 \
  --partitions 3 \
  --replication-factor 1

UI alternative:

1. Navigate to Topics → Create Topic
2. Set partitions/replication
3. Click "Submit"

Message Production Patterns

Basic text messages:

docker exec -it broker /opt/kafka/bin/kafka-console-producer.sh \
  --topic orders \
  --bootstrap-server localhost:9092

Structured JSON:

# Using jq for JSON formatting
echo '{"id":1001, "amount":49.99}' | jq -c | docker exec -i broker /opt/kafka/bin/kafka-console-producer.sh \
  --topic orders \
  --bootstrap-server localhost:9092

You can also see the messages on the UI:

Troubleshooting Checklist

Brokers not connecting?

• Check port conflicts with netstat -tuln | grep 9092

UI not showing topics?

• Ensure bootstrap servers match in UI config
• Check container logs: docker logs kafka-ui

When to Choose This Setup

Good for:

• Local development
• CI/CD pipelines
• Prototyping new features

Not for:

• Production deployments
• High-availability requirements
• Sensitive data handling

Learn More

• Kafka Raft Metadata (KRaft) mode
• More details on KRaft