Kafka Consumer Groups: The Hard-Won Lessons from 8 Years in Production
We were burning through rebalances like crazy back in 2021 at a fintech client. Every time we deployed a new consumer, the whole group would grind to a halt. Production alerts screaming. Lag climbing into the millions. The usual story.
But here's what took me too long to realize: the consumer group wasn't the problem. My assumptions about how it worked were the problem.
Let me save you those sleepless nights.
Apache Kafka consumer groups are the backbone of distributed event processing. They let you scale consumers horizontally, split partitions across workers, and maintain ordering guarantees within partitions. Elegant design. Brutal failure modes when you get it wrong.
This guide is everything I've learned building real systems on Kafka — the stuff that actually matters in production, not the textbook explanations.
What We Mean When We Say "Consumer Group"
A consumer group is a set of consumers that share a group ID and collectively consume from a set of topics. Kafka assigns each partition to exactly one consumer within the group. That's the core contract.
The broker coordinates this. When consumers join or leave — or when partitions are added — the group triggers a rebalance. The state of the world changes. Consumers get reassigned.
The rebalance protocol is where most of your pain will come from. Kafka's rebalancing logic determines how smoothly your consumption continues when membership changes. And the default behavior can absolutely wreck your throughput.
I've seen teams treat rebalances like unavoidable infrastructure noise. They're not. They're a design signal telling you something about your consumers is off.
The Rebalance Problem Nobody Warns You About
Let's talk about what actually happens during a rebalance.
Under the classic protocol, every consumer in the group must stop consuming. They all revoke their partitions. The group coordinator picks a group leader. That leader computes a new assignment. Then everyone receives their new partitions and resumes.
This is a full stop. Every consumer. At the same time.
Your processing pipeline doesn't gracefully pause — it just halts mid-message. If you're processing transactions or writing to downstream systems, you need to handle that disruption gracefully. Most teams don't.
According to Redpanda's analysis of rebalancing behavior, the frequency of rebalances is your single biggest predictor of consumer group instability. More rebalances equal more lag spikes, more duplication, more out-of-order processing.
And the trigger isn't always obvious. A consumer that takes too long to process a batch? Dead. A consumer that hits a GC pause? Also dead. A consumer that hiccups on a network call? You guessed it.
Set max.poll.interval.ms High Enough
The classic mistake is setting max.poll.interval.ms too low.
This is the amount of time the broker will wait between successful polls from a consumer before declaring it dead. Default is five minutes. But that's often too short.
I worked with a healthcare client in 2024 whose consumers were processing complex medical records with external API enrichment. Each record took 20-30 seconds to process. They were polling with a batch size of 500. Quick math — that's 2.5 to 4 hours of potential processing per poll return.
They kept getting killed off mid-processing. Clients disconnecting. Rebalance storms every few minutes.
The fix was simple:
java
Properties props = new Properties();
props.put("max.poll.interval.ms", "200000"); // 200 seconds
props.put("max.poll.records", "50"); // 50 records max per poll
We reduced the batch size and gave consumers more time to return from their processing loop. The rebalances stopped. The group stabilized.
But here's the thing — the heavy processing was still the bottleneck. We were just hiding the problem with tuning. The real solution was decomposing that work into a more appropriate pipeline, but the tuning kept us moving while we rearchitected.
Use Cooperative Rebalancing or Suffer
Here's where I take my strongest position: the cooperative sticky assignor. Use it. Don't argue with me.
The eager rebalance protocol was Kafka's original approach. It's simple. It's also brutal. Every rebalance triggers a full group stop.
The incremental cooperative rebalance protocol changed the game. It only revokes the partitions that need to move, not all of them. Your consumers keep processing the partitions they're keeping while the reassignment happens.
The difference in practice is massive. Full stop versus partial disruption. Millions of events processed versus thousands.
java
props.put("partition.assignment.strategy",
"org.apache.kafka.clients.consumer.CooperativeStickyAssignor");
That's it. Set it. Done.
But here's the caveat I always give people: cooperative rebalancing doesn't eliminate the need for proper processing timeouts. It just changes the rebalance dynamics. You still need to nail your max.poll.interval.ms and your batch sizes.
The Poll Loop Is a Contract, Not a Suggestion
Most consumer code looks like this:
java
while (true) {
ConsumerRecords<String, String> records = consumer.poll(100);
for (ConsumerRecord<String, String> record : records) {
process(record);
}
consumer.commitSync();
}
It's wrong. Well, not wrong — it's incomplete. The poll loop has contractual obligations that you're implicitly agreeing to by using the consumer.
- You must poll at least as often as
max.poll.interval.msminus the processing time - You must not spend longer than
max.poll.interval.msper poll return - You must complete the entire processing of the returned records before calling poll again
Break any of these and the consumer gets kicked from the group.
The recommended pattern from the Confluent team was discussed in our architecture reviews all the time: separate your polling from your processing. Use a dedicated polling thread and a processing threadpool.
java
ExecutorService executor = Executors.newFixedThreadPool(10);
while (offsetIsValid) {
ConsumerRecords<String, String> records = consumer.poll(100);
for (ConsumerRecord<String, String> record : records) {
executor.submit(() -> process(record));
}
consumer.commitAsync();
}
This decouples your slow processing from the poll loop. You can commit offsets without blocking on processing, and you don't risk rebalance kicks from slow code.
But careful — this introduces a new problem: offset commits might happen before processing completes. If your consumer crashes, you lose those records. That's the trade-off you're making for throughput. OneUptime's guide on handling rebalancing covers this exact pattern and its implications in detail.
You need to decide: at-least-once with possible duplicates, or exactly-once with more complexity. For most systems, at-least-once wins. Don't let perfect be the enemy of good.
Understanding Rebalance Triggers — All of Them
Most people know the obvious triggers. Consumer joins. Consumer dies. New partitions added. But there's a fifth one that rarely gets discussion: session timeout.
Red Hat's deep dive on avoiding rebalances highlights how session timeouts from GC pauses, network blips, and even CPU starvation on your brokers can trigger rebalances.
This happened to us at SIVARO. We had a consumer group processing clickstream data from e-commerce sites. The cluster was humming along fine until we pushed a new feature that increased heap usage by 30%. GC pauses went from 200ms to 4 seconds.
The session timeout was set to 10 seconds. The default heartbeat intervals meant the broker assumed the consumer was dead after 12 seconds of silence. Four seconds of GC time, queued up behind other work, and the consumer got disconnected.
We fixed it by setting session.timeout.ms to 20 seconds and heartbeat.interval.ms to 5 seconds. But the real fix was fixing the memory leak that caused the increased heap usage.
The lesson: rebalances are often a symptom, not the disease. Tune the timeouts to give yourself breathing room, but find the root cause.
Static Membership Is Your Friend for Deployments
Here's a technique that most teams don't know about until they've been burned a few times: static membership.
Kafka 2.3 introduced group.instance.id. It makes consumer membership sticky. The broker won't consider a consumer "gone" just because it missed a heartbeat — as long as it rejoins within session.timeout.ms, it gets its old partitions back.
This is brilliant for rolling deployments.
When you deploy a new version of your service, the old instance shuts down gracefully. The new instance starts with the same group.instance.id. There's no rebalance. Your consumers just pick up where they left off.
We used this at an e-commerce company in Atlanta processing ratings and reviews. Their deployments were causing rebalance storms that cascaded into 15-20 minutes of lag accumulation. Switching to static membership cut that to near zero.
java
props.put("group.instance.id", "consumer-" + instanceId);
Just make sure the instance ID is unique across your group. You're telling Kafka that this FQN is the owner of a specific consumer. If two consumers claim the same ID simultaneously, the group will reject one.
This doesn't fix all rebalances — new instances joining still cause them — but it eliminates the most common cause of disruption. Confluent's rebalancing documentation emphasizes this as the single highest-impact mitigation for deployment-related rebalances.
Monitor the Right Metrics or You're Flying Blind
You cannot fix what you cannot see. Anyone who's managed a Kafka cluster in production knows this.
The critical consumer group metrics you need:
Consumer Lag — the difference between the latest offset and the committed offset. This is your number one signal. Lag that stays flat means you're keeping up. Lag that grows means you're falling behind.
Rebalance Count and Duration — how often rebalances happen and how long they last. If you see rebalance counts per hour climbing, you've got a problem.
Max Poll Interval — how close you are to hitting max.poll.interval.ms. If you're routinely at 80%+ of the limit, you're living on the edge.
Session Timeout Rate — how often consumers miss their session timeout. This indicates instability.
A practical approach: track these through Kafka's JMX metrics or OpenTelemetry. Most teams I've worked with use Starburst or Confluent's metrics, but you can get by with vanilla JMX and a Prometheus setup.
Very Good Security's case study tells the story of how they diagnosed rebalancing issues using consumer lag metrics — the rebalancing problem became apparent only after they started tracking lag per partition over time. The pattern — steep lag increases followed by flat periods — pointed to rebalances as the culprit.
The Poll Thread Mistake That Cost Us 3 Days
One of the most insidious bugs I've seen: blocking the poll thread with logging.
At SIVARO, we had a consumer that processed transaction data. It was working fine until we added a new logging framework that was blocking on disk I/O. The poll loop was stalled in a synchronous log write.
By the time we caught it, the group had rebalanced 14 times in one production day. Lag went from near-zero to 200K messages. Downstream systems were timing out.
The fix: put logging on a separate thread. Use async appenders. Never block the poll thread with anything I/O-bound.
There's also the hidden trap of the GC pause problem. Most JVM-based consumers face this at some point. A 5-second GC pause while your session timeout is 10 seconds will get you disconnected. And on Kubernetes, where pods don't shut down politely when a node goes bad — immediate disconnection.
The lesson is that your consumer's runtime environment is often the root cause. Monitor CPU, heap, and GC behavior before blaming Kafka.
Configuring Consumers for Production — The Hard Numbers
Most teams I've interviewed at startups and enterprises alike under-configure their consumers. The defaults are conservative, designed for safety not performance. Here's my recommended production configuration:
enable.auto.commit=false
session.timeout.ms=30000
heartbeat.interval.ms=10000
max.poll.interval.ms=300000
max.poll.records=500
partition.assignment.strategy=CooperativeStickyAssignor
auto.offset.reset=earliest
Let me unpack the logic:
enable.auto.commit=false— You own the offset commit logic. No surprises.session.timeout.ms=30000— 30 seconds gives you room for GC pauses and transient network issues without waiting forever to detect a dead consumer.heartbeat.interval.ms=10000— One third of the session timeout. Leaves you breathing room.max.poll.interval.ms=300000— Five minutes. Enough for most processing.max.poll.records=500— Balanced batch size.CooperativeStickyAssignor— As discussed. Use it.earliest— Pull from the beginning on a fresh group, never lose data.
Here's a committed example:
java
Properties props = new Properties();
props.put("bootstrap.servers", "broker1:9092,broker2:9092,broker3:9092");
props.put("group.id", "core-processing-group");
props.put("enable.auto.commit", "false");
props.put("session.timeout.ms", "30000");
props.put("heartbeat.interval.ms", "10000");
props.put("max.poll.interval.ms", "300000");
props.put("max.poll.records", "500");
props.put("partition.assignment.strategy",
"org.apache.kafka.clients.consumer.CooperativeStickyAssignor");
props.put("auto.offset.reset", "earliest");
Not meant to be a one-size-fits-all perfect config — but a baseline. Strictly production-oriented.
Offsets and Commit Modes
Choosing how and when to commit offsets is one of the trickiest parts of building reliable Kafka consumers. Most people default to auto-commit. It's convenient. It's also dangerous.
Auto-commit means Kafka commits your offsets at a fixed interval — auto.commit.interval.ms, traditionally 5000ms. But it commits the offset of the last poll response, regardless of whether you've processed those records.
So imagine this. You poll 500 records. The consumer is fine. 5 seconds later, auto-commit marks your position after those 500 records. But you're only 100 records deep in processing when you crash. When you restart, you resume at position 500, meaning records 101-499 are processed twice, or skipped entirely if you're not idempotent.
Manual commit gives you control:
java
while (running) {
ConsumerRecords<String, String> records = consumer.poll(1000);
for (ConsumerRecord<String, String> record : records) {
process(record);
}
// Commit the current position
consumer.commitSync();
}
This commits the offset of the last record processed — at the end of the batch. If you crash mid-batch, those records get reprocessed. At-least-once.
But what if you need to track offsets per record? Kafka's ConsumerRebalanceListener is the answer:
java
consumer.subscribe(Arrays.asList("my-topic"), new ConsumerRebalanceListener() {
@Override
public void onPartitionsRevoked(Collection<TopicPartition> partitions) {
// Commit any uncommitted offsets before losing partitions
consumer.commitSync();
}
@Override
public void onPartitionsAssigned(Collection<TopicPartition> partitions) {
// Prepare for new partitions
}
});
The listener lets you handle intermediate states — commit what you've processed before giving up your partitions. In cooperative rebalancing, this is your key to avoiding duplicate processing.
Idempotent Processing — The Final Safety Net
All of the tuning and configuration can't eliminate duplicate processing. Not entirely. No matter how careful you are with offsets and rebalances, your consumer will occasionally process the same record twice. It's inherent to distributed systems.
The only absolute protection is idempotent processing. Design your downstream systems to handle duplicate records gracefully.
For a database, use upserts or natural keys. For a cache, write operations are naturally idempotent. For an HTTP endpoint, you need to send an idempotency key.
One pattern I've used with success at SIVARO:
java
// Process transaction with dedupe key
String dedupeKey = record.key() + ":" + record.offset();
// cache.putIfAbsent(dedupeKey, record.value());
// or store in DB with unique constraint
This is extra engineering effort, but it's what separates production-ready systems from demos.
The Human Element
Kafka consumer group management isn't just a technical problem. It's an operational discipline.
You need to think about rollouts. How do you deploy changes without disrupting the group? You need to think about failure. What happens when a consumer dies? What happens when the broker goes down? What happens when you need to add partitions?
The answers to these questions shape your consumer group's behavior in production. The tech is just one part of the equation.
I've seen teams with perfect configurations still fail because they didn't have a playbook for handling production incidents. Rebalance storms might not be avoidable — but they should be predictable and manageable.
The Hardest Lesson
If I had to distill everything I've learned about apache kafka consumer group best practices into one sentence, it would be this: treat consumer groups like the infrastructure they are, not an afterthought.
Your consumer group is the entry point for your stream processing pipeline. Get it wrong and everything downstream suffers. Get it right and you've got a system that handles failures gracefully, processes millions of events per second, and scales without drama.
At SIVARO, we're building data infrastructure that handles 200K+ events per second across multiple production environments. Consumer group tuning is still the highest-leverage lever for performance and reliability.
Test your configurations. Measure your metrics. Simulate failures. Do this before you're in a crisis.
And when you're in the crisis — because you will be — remember: the rebalance isn't the enemy. It's your system telling you something is wrong. Listen.
FAQ
What happens when a consumer group rebalances?
The group's members negotiate partition ownership. With eager rebalancing, all consumers stop consuming. With cooperative rebalancing, only affected partitions pause. Rebalances are triggered by membership changes — joins, leaves, or new partitions.
How do I prevent consumer groups from rebalancing too often?
Set session.timeout.ms high enough, use static membership for deployments, implement cooperative rebalancing, and monitor your poll behavior. Most rebalances come from consumers timing out — fix those causes.
What's the difference between static and dynamic membership?
Dynamic membership means each consumer is uniquely identified by a random UUID, and losing a connection means an instant rebalance. Static membership sets a fixed group.instance.id that allows the same consumer to reconnect without triggering a rebalance.
Does increasing max.poll.records help performance?
It can, but it also means you're processing larger batches, which increases the likelihood of hitting max.poll.interval.ms. There's always a trade-off. Test at multiple batch sizes to find the sweet spot.
How should I handle offset commits?
Choose what your system can tolerate. At-least-once with manual commits gives you data preservation. Exactly-once requires transactional boundaries and idempotent sinks. There's no universal answer; understand your business requirements.
Is it possible to eliminate rebalances entirely?
No. Rebalances will happen — partitions get added, consumers get killed by infrastructure failures. Your goal is to minimize their impact through configuration, monitoring, and architectural design. Cooperative rebalancing and session timeout tuning are the best levers.
What should I check first when lag spikes?
Monitor the four metrics I listed earlier: consumer lag per partition, rebalance counts, max poll interval hits, and session timeout rates. If lag spikes align with rebalances, your group membership is unstable.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.