Kafka Offset Management Best Practices for Production Systems

Kafka Offset Management Best Practices for Production Systems

The consumer group you stopped worrying about just ate your data.

I watched it happen to a Fintech app in July 2026. Their consumer was committing offsets every 30 seconds, something a quick SO answer recommended years ago. Then a slow downstream API call caused a rebalance mid-poll. The broker replayed 15,000 messages, the consumer reprocessed a payment webhook twice, and their compliance team asked uncomfortable questions about duplicate transaction notifications.

The problem wasn't their code. It was their approach to offset management — treating it as an afterthought instead of the core contract between your consumer and your data.

In this guide, I'll walk you through what actually works for offset handling in production, based on what we've built and broken at SIVARO since 2018. You'll learn the precise semantics of auto.offset.reset, why enable.auto.commit=false is the only sane default for anything resembling critical infrastructure, and how rebalancing silently corrupts your offset state if you don't plan for it.

Let's start with the fundamentals, then get into the messy parts.

What Is an Offset, Really?

An offset is just a 64-bit integer — the position of a consumer within a partition. But anyone who's debugged a production incident knows it's a contract. It's the official record of what your application has processed, whether it actually has or not.

The Kafka broker stores committed offsets in the internal __consumer_offsets topic. Your consumer fetches messages sequentially, processes them, and periodically writes back "I've handled up to position 47." The broker doesn't care what you did with the message. It trusts that when you commit offset 47, you've genuinely finished it.

That trust is where production systems fail.

The Two Commit Strategies: Autocommit and Manual

Kafka gives you enable.auto.commit=true as a default. It's convenient. It's also dangerous — the auto-committer fires every auto.commit.interval.ms (default: 5000ms), regardless of whether your processing actually completed.

Here's the problem: your consumer polls, receives a batch, and starts processing. If the auto-committer fires while you're still working through that batch, you commit offsets for messages you haven't finished. Then your JVM crashes. The next consumer in the group picks up where the offset says — not where you actually were.

Autocommit is acceptable only for idempotent, low-stakes processing like metrics collection or analytics events where occasional duplicates are fine.

For everything else, use manual commits:

java
Properties props = new Properties();
props.put("enable.auto.commit", "false");
props.put("max.poll.interval.ms", "300000");
// 5 minutes max processing time per poll
props.put("max.poll.records", "500");

KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Arrays.asList("orders"));

try {
    while (true) {
        ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(1000));
        for (ConsumerRecord<String, String> record : records) {
            processRecord(record); // your actual work
        }
        consumer.commitSync();
    }
} finally {
    consumer.close();
}

This commits only after the entire batch is processed. If you crash between processRecord() and commitSync(), you get duplicates — but you never lose a message.

Kafka vs RabbitMQ: Which One Do You Choose?

Let's address the comparison that keeps coming up in every architecture review meeting. Kafka vs RabbitMQ which one to choose isn't about "better" — it's about semantic fit.

RabbitMQ is built around routing and queue semantics. Its consumer model is "one message, delivered once, ack when done." This makes it excellent for task distribution (RPC, work queues) where each message needs individual handling.

Kafka is not a queue in that sense. It's a log. Offsets are your acknowledgment mechanism, but they're batch-oriented by design. When you process 500 records and commit one offset, you're saying "I've handled all of these" — not "I've handled this specific one."

A real example: Telco company in 2025 kept their SMS delivery on RabbitMQ because each message needed individual acknowledgment from a stateful gateway. Meanwhile, their clickstream processing moved to Kafka because the replay capability and retention semantics made it possible to re-run analytics jobs without contacting the producers.

Pick RabbitMQ if your workload requires per-message ack with routing. Pick Kafka if you need replay, retention, or stream processing.

Rebalancing: Your Offsets' Silent Saboteur

When a consumer joins or leaves a group, the group coordinator triggers a rebalance. During rebalance, all consumers lose their partition assignments and wait for new ones. This is where offset management gets tricky.

The classic problem: your consumer is mid-processing, and a rebalance fires. You check isRunning() in your ConsumerRebalanceListener, but you haven't committed. The new partition owner starts from the old offset, and you get reprocessing in the middle of a transaction that thought it was done.

The Kafka Rebalancing Explained guide from Confluent covers this well: the coordination protocol has evolved significantly. The "eager" rebalance protocol — where all consumers stop consuming and rejoin the group entirely — was famously disruptive. Since Kafka 2.4, the "cooperative" protocol allows incremental rebalancing, where only the partitions being moved between consumers are revoked, not the entire set.

But here's what I've learned in production: even with cooperative rebalancing, your offsets can be committed at the wrong time.

When you use commitSync() after a long batch, you're making a trade-off between throughput and correctness. If your processing takes 30 seconds per batch of 500, a rebalance at second 29 wastes almost the entire batch because the new consumer will replay from the last commit.

The pragmatic solution: commit more granularly? Or accept duplicates?

You can't have both. You need to pick your poisoning — reprocessing duplicates or risking lost data with autocommit.

The Rebalance Listener: Your Safety Net

The one Kafka API most teams ignore is the ConsumerRebalanceListener. It lets you control what happens around rebalances. This is where you can commit remaining offsets before you lose partition ownership.

java
consumer.subscribe(Arrays.asList("topic"), new ConsumerRebalanceListener() {
    @Override
    public void onPartitionsRevoked(Collection<TopicPartition> partitions) {
        // Commit your offset BEFORE the partition is taken away
        consumer.commitSync();
        // Flush any state you were holding
        flushBufferedRecords();
    }

    @Override
    public void onPartitionsAssigned(Collection<TopicPartition> partitions) {
        // Load any state you need for the newly assigned partitions
        loadStateForPartitions(partitions);
    }
});

This listener is your best defense against wasted processing during rebalances. When you know the partition is about to be revoked, you do a final commit — even if you're mid-batch — because it's safer to have a few duplicates than it is to lose your position entirely.

The case study from Very Good Security shows this pattern in action. They were experiencing duplicate payment processing due to rebalances between processing and commit. Adding this listener with a final commitSync() eliminated the reprocessing window.

Eager Rebalance vs Cooperative Rebalance: Which to Use?

If you're on Kafka 2.4 or later, you have options. The Redpanda guide on rebalancing breaks down the two protocols:

• Eager rebalance: All consumers in the group stop consuming, give up all partitions, then get reassigned. Simple, safe, but with significant downtime for large groups.
• Cooperative rebalance: Only the partitions being reassigned get revoked. The rest keep consuming. Less downtime, but more complex to handle correctly.

For most production systems, I recommend cooperative rebalancing. It needs careful handling of the revocation callback — you must stop processing revoked partitions immediately — but the reduced time-to-consumption outweighs the complexity.

properties
partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor

This is especially vital for stateful consumers. If you're using Kafka Streams or a Kafka-backed microservice holding local state, cooperative rebalancing means you don't lose your entire state — just the parts being moved.

How to Set Kafka Retention Policy by Time

Retention policy shapes your offset management in ways people don't expect. If you set log.retention.hours too low, consumers that fall behind get their offsets deleted out from under them.

There's a fundamental tension here: you can't replay what you don't retain.

To set retention policy by time, you have a choice between topic-level and broker-level config:

bash
# Topic-level (applies only to this topic)
kafka-configs --bootstrap-server localhost:9092   --alter --entity-type topics --entity-name orders   --add-config retention.ms=604800

# Broker-level (applies as default to all topics)
kafka-configs --bootstrap-server localhost:9092   --alter --entity-type brokers --entity-name 0   --add-config log.retention.hours=72

I'm a fan of the compact cleanup policy for topics where you need to preserve state. Instead of deleting old records, Kafka keeps the latest value for each key. This is what makes the __consumer_offsets topic itself work — it never grows unboundedly.

The rule of thumb we use at SIVARO: set retention at least 3x your maximum consumer lag time. If your slowest consumer can be down for 2 days, you need at least 6 days of retention. Also, watch offsets.retention.minutes — if a consumer group becomes empty, its offsets get deleted after this period. On our brokers, that's set to 10080 (7 days), which gives enough time to scale down a group without losing the ability to resume exactly where they left off.

Idempotent Processing: The Ultimate Offset Secret

Here's the contrarian take: stop trying to guarantee exactly-once delivery. You'll burn engineering time and still fail. Instead, design for at-least-once and make your processing idempotent.

For a payment system in 2024, we had duplicate payment webhooks causing chaos. We added a simple idempotency table in PostgreSQL — the message key as a primary key. When the consumer reprocessed a message, the insert conflict no-op'd. The offset management problem went from "must guarantee exactly once" to "must commit promptly."

This is the philosophical shift: offset management best practices aren't about preventing duplicates. They're about minimizing the window for duplicates and making them harmless.

Monitoring: What Actually Matters for Offsets

You can't debug what you can't see. You're monitoring consumer lag, right? And you're tracking rebalance frequency?

The recommendations from Red Hat's developer platform highlight this: you should track lag per consumer group per partition, not just a single aggregate number. The mean hides the one partition that's 20,000 messages behind.

We track these metrics at SIVARO:

• Lag per partition — shows exactly which partition is struggling
• Rebalance rate per group — a group that rebalances more than once per 10 minutes needs intervention
• Commit latency — time between processing completion and offset commit
• Time since last rebalance — because a quiet group is a healthy group

bash
kafka-consumer-groups --bootstrap-server localhost:9092   --describe --group orders-service

Output shows each partition's current offset, log end offset, and lag. If lag keeps climbing, you're either behind on processing or about to get hit by the retention policy.

Common Pitfalls and Their Fixes

Pitfall 1: Committing too frequently. Every commitSync() is a round-trip to the broker. If you're committing every 10 records, you're adding 10x the network overhead. I've seen consumers slow down by 40%.

Fix: Use commitAsync() for regular commits and commitSync() only in the rebalance listener or on shutdown. Yes, commitAsync() can fail silently, but combining it with commitSync() on close handles the reliability gap.

Pitfall 2: Ignoring max.poll.interval.ms. If your processing takes longer than this (default: 300000ms), the consumer is removed from the group. This triggers a rebalance — and before you know it, the poisoning cycle from the beginning of this article happens.

Fix: Set max.poll.records low enough that your processing time stays under max.poll.interval.ms. And consider the rebalance protocol evolution mentioned in the SlideShare summary — the modern protocol gives you more time but doesn't save you from yourself.

Pitfall 3: Stateful consumers and rebalances. If your consumer keeps local state (a windowed count, for example), a rebalance means the new owner doesn't have that state. You're building from the last committed offset, but the state is gone.

Fix: Either persist state externally (Redis, database) or use Kafka Streams which handles state stores and their migration during rebalances.

Kafka's Own Internal Offset Management Isn't Perfect

Here's something that surprises people: Kafka's internal __consumer_offsets topic has its own retention policy. If you run into storage pressure, this topic can be compacted or deleted incorrectly.

We had a bank client in 2025 whose brokers ran out of disk. An engineer, in a panic, reduced offsets.retention.minutes to 60 minutes to free space. It worked immediately. And then all consumer groups with no members for more than an hour lost their offsets. When the clients reconnected, they couldn't resume — they were treated as new groups starting from the beginning.

The whole incident was viewable in the logs, and the fix was straightforward (restore the retention to 7 days, and accept the reprocessing), but the lesson stuck: offset management isn't just a client-side concern. It's a broker-side configuration that you must control carefully.

A Table of Offset Configuration Options

Configuration	Default	Recommended for Production	Rationale
enable.auto.commit	true	false	Manual commits give you control over when offsets are written
auto.commit.interval.ms	5000	N/A (when auto is false)	Only relevant if you keep autocommit on
auto.offset.reset	latest	earliest for event-driven systems	latest can silently skip messages for new consumers
max.poll.records	500	100-500	Smaller batches reduce processing time and rebalance window
max.poll.interval.ms	300000	Set based on your processing P99	The Kafka broker will remove you if you exceed this
session.timeout.ms	45000	10000-20000	Lower values detect failures faster but risk false rebalances
heartbeat.interval.ms	3000	Must be 1/3 of session.timeout	Keeps the coordinator informed of liveness

FAQ: Answering the Questions I Get Asked Most

Q: Should I use commitSync() or commitAsync()?

commitAsync() is faster but can fail silently. commitSync() blocks until the commit is acknowledged. Use commitAsync() for regular commits and commitSync() in the rebalance listener and at shutdown. The async failure is acceptable because you'll catch it in the next commit or the sync one.

Q: Do I have to set enable.auto.commit=false?

No. But if you don't, you're accepting that offsets may be committed before processing is complete. For idempotent workloads with rigorous monitoring, this might be fine. For anything financial, in production, at scale — set it to false.

Q: What's the auto.offset.reset behavior when a group has no committed offsets?

It depends on the value you set. latest starts consuming from the end of the log — you'll skip any messages that arrived before you started. earliest consumes from the beginning — you'll process everything retained. For event sourcing, I prefer earliest. And if you want to control this per consumergroup, you can manually commit offsets using kafka-consumer-groups.

Q: I'm using Kafka Streams. Do I need to worry about offset management?

Kafka Streams handles this automatically. The KafkaStreams application manages its own commits and state store restoration. You should still monitor lag and rebalances, but you don't write commit logic.

Q: How does cooperative rebalancing affect offset management?

With cooperative rebalancing, only the partitions being moved get revoked. Your callback should commit those specific partitions and stop processing them promptly. This minimizes the gap between processing and commitment for the other partitions.

Q: What's the best way to handle consumer shutdown cleanly?

Call consumer.close(). It triggers the rebalance listener, commits offsets if you disabled enable.auto.commit, and leaves the group. This is the cleanest way to avoid triggering forced rebalances.

Q: How often should I monitor consumer lag?

If you have a unified logging and alerting system, set a threshold for each group. "Consumer lag > 100,000" for any partition triggers a page. I prefer checking every minute for critical systems, every 5 minutes for standard systems. The real answer is: check whenever your broker dashboard updates, and ensure you're capturing the correct lag value, not just the total group lag.

Q: If I want to upgrade Kafka brokers, will my offsets survive?

Yes, as long as you move the __consumer_offsets topic along with the cluster. The Kafka rebalancing documentation describes how the coordination protocol depends on __consumer_offsets. If you're moving to a new cluster, you can export and import offsets using scripts, but the safest approach is to use MirrorMaker for migration and let consumers restart from the mirror's offsets (check the tool's exact behavior).

Q: How do I choose between kafka vs rabbitmq which one to choose for my stack?

If you need replay, retention, and pub/sub with multiple independent consumer groups, Kafka is the better choice. If you need per-message ack, complex routing, and native queue semantics, RabbitMQ is better. The offset management complexity is Kafka's unique burden. RabbitMQ's per-message ack is simpler but doesn't provide the same replay capabilities.

Kafka Offset Management: A Field-Tested Approach

Let's walk through what I consider the canonical production setup for a Kafka consumer. We use this at SIVARO for client systems that need to process financial transactions without losing data or processing duplicates beyond tolerance.

python
# Using confluent-kafka-python
conf = {
    'bootstrap.servers': 'broker1:9092,broker2:9092,broker3:9092',
    'group.id': 'orders-service',
    'enable.auto.commit': False,
    'auto.offset.reset': 'earliest',
    'max.poll.records': 200,
    'session.timeout.ms': 20000,
    'heartbeat.interval.ms': 6000,
    'max.poll.interval.ms': 400000,  # generous for a batch of 200
}

consumer = Consumer(conf)
consumer.subscribe(['orders'])

try:
    while True:
        msg = consumer.poll(timeout=1.0)
        if msg is None:
            continue
        if msg.error():
            print(f"Consumer error: {msg.error()}")
            continue

        store = process_message(msg)  # idempotent write to a database

        if store.success:
            consumer.commit(msg)  # manual async commit per record
        else:
            # Handle failure — maybe retry, maybe log and skip
            log_failure(msg)
finally:
    consumer.close()  # sync commit on shutdown

This pattern commits per record after success, uses earliest for new groups, and closes cleanly.

The Secret to Sane Offset State: Incremental Cooperative Rebalancing

One more thing. The Red Hat article brings up a point I want to echo: the ConsumerGroupMetadata rebalancing API change in RabbitMQ's use_partitions_aftersync in Spring for Kafka. But for vanilla Kafka, I want to emphasize the CooperativeStickyAssignor.

The cooperative protocol wasn't just an improvement — it fundamentally changed the rebalance trade-offs. With eager rebalancing, every rebalance revokes everything, triggering a full commit-reset cycle. With cooperative rebalancing, the time between "partition is revoked" and "new consumer starts processing" is drastically reduced. This matters for offset management because the revocation callback becomes the critical commit opportunity, not just an afterthought.

If you're on Kafka 2.4+, use the CooperativeStickyAssignor. It makes offsets more predictable and rebalances smaller.

What to Do When Offsets Go Wrong

You've got lag climbing past 100k per partition. Or worse, consumers are consuming from the beginning of the topic again because their offsets were lost.

Your first step should always be: check the __consumer_offsets topic. You can use the kafka-console-consumer script to read it like a regular topic, or use the ConsumerGroupCommand tool:

bash
kafka-consumer-groups --bootstrap-server localhost:9092   --group orders-service   --describe --members --verbose

This shows you each member's current assignment and their committed offsets. If the offsets look wrong — they're all at 0 when the topic has millions of messages — then check the retention policy on the topic.

If offsets are still available for consumption from the last committed position, you might be able to restart the consumer safely. If they're gone (you've hit the retention wall), you're facing the same choices as our bank client: reset to earliest or latest, and accept the consequences.

Wrapping Up: Your Next Steps

Kafka offset management isn't a one-time setup. It's an ongoing operational practice that you refine as your system evolves, your topics expand, and your consumer groups change.

Here's what to do today:

1. Set enable.auto.commit=false on all your critical consumers. Now.
2. Add a ConsumerRebalanceListener to your subscription.
3. Monitor lag per partition per group, and set alarms.
4. Set your retention policy thoughtfully, and revisit after any scaling changes.
5. Make your processing idempotent — it's the only safety net that truly works.

This is what I've built my company on: data infrastructure that doesn't break when you need it most. Offset management is the unglamorous core of that. Get it right, and your consumers will replay data when you need them to, avoid reprocessing when you don't, and keep your system honest.

---

Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.