HomeFind your caseOur Kafka consumers keep falling behind
Case · Enterprise
A common situation · not a client storyOur Kafka consumers keep falling behind.
Teams streaming events through Kafka whose consumers lag at peak, rebalance at the worst moment, or lose and duplicate messages.
What you're probably seeing
If this sounds familiar, you're in the right place.
- Consumer lag climbs at peak hours and takes hours to clear.
- Rebalances pause processing at the worst moments.
- Messages are processed twice, or not at all, after a restart.
- Adding consumers doesn't help.
- Downstream numbers don't match the source, and nobody knows why.
The usual ways out
What teams usually try. Each works somewhere. None works everywhere.
| The way | Works when | The catch |
|---|---|---|
| Add more consumers | There are more partitions than consumers | Does nothing past one consumer per partition |
| Add partitions | Throughput per partition is the limit | Changes key ordering, and can't easily be undone |
| Tune consumer settings | Batching, polling or commits are wrong | Easy to make one problem into another |
| Make processing faster or asynchronous | The consumer's own work is the bottleneck | Needs careful offset handling |
| Move to a managed service or another broker | Running Kafka itself is the pain | A migration, and the consumer problems come along |
| Rethink keys and topics | A few hot keys overload a few partitions | A design change across producers and consumers |
| Route work away from lagging partitions | Lag sits on a few partitions and producers can't change soon | Treats the symptom, and double-processes when several consumer groups read the topic |
What other companies did. Their published results, not ours.
We read each company's own write-up and asked one question: would our diagnostic's tests have led to the same call? We judge the approach, not the result.
| Company | What they did | What they published | Source |
|---|---|---|---|
| AgodaTravel platform moving hundreds of TB a day through Kafka on its private Kubernetes cloud. One supplier can send 1.5M price updates in a minute. | Traced lag to round-robin assignment meeting mixed hardware speeds and messages of uneven cost. Built a lag-aware producer that sends fewer messages to lagging partitions. A lag-aware consumer that triggers rebalances is still an experiment. | Resources for the system were cut by 50%, and the SLA held without manual mitigation. Before, 5 of 40 partitions showed significant lag. After, there was no significant high lag.They noted: Lag-aware producing does not suit topics read by several consumer groups, because lag is per group. Triggering a rebalance was historically costly, because eager rebalances stop all processing in a consumer group. Newer incremental rebalancing reduces that cost. There should be more partitions than workers. | How We Solve Load Balancing Challenges in Apache KafkaMay 2024 |
| CloudflareKafka carries time-sensitive workflows such as emails and alerts. Consumers run as microservices on Kubernetes, and some topics have hundreds of partitions. | Replaced a connection-only liveness check with one that fails when the committed offset stops moving, so Kubernetes restarts consumers that are connected but not processing. | Not stated (they expect fewer incidents with unhealthy consumers)They noted: After rollout, the check caused cascading restarts during rebalances. Replicas kept stale offsets for partitions they no longer owned. A known client-library issue could also deadlock consumers after a rebalance. | Intelligent, automatic restarts for unhealthy Kafka consumersJan 2023 |
| DropboxIngestion for Dropbox Dash, an AI assistant. The backlog needed about two weeks to clear, with a customer onboarding due the next morning. | An early key choice had routed every message to one partition. Producers could not change quickly, so they split messages into sub-messages and processed them in parallel on the consumer side, keeping order. No new partitions were added. | Roughly 100x ingestion throughput gain, and the customer was onboarded on time.They noted: The talk names the early key choice as the cause. Re-keying at the producer would have affected other consumers of the stream. | Beyond Message Key Parallelism: Increasing Dropbox Dash AI Ingestion Throughput by 100xUndated |
| CrowdStrikeKafka consumer library in an ecosystem of more than 200 microservices. | Added a 'redistribution' mode that can be switched on and off. Consumers of a lagging partition re-produce its messages to other partitions instead of processing them, and the mode is set through Redis. | Not stated.They noted: Adding consumers barely helped when the lag sat on one partition. Redistribution double-processes messages when a topic has several consumer groups. It treats symptoms, not causes, and must be turned off once lag clears. | Addressing Uneven Partition Lag in KafkaUndated |
Our read
Reasonable call. One thing showed up late. In most cases the lag sat on a few partitions, and in two write-ups the key spread or the restart behaviour came to light late.
Agoda measured lag per partition (5 of 40 lagging) and traced it to uneven hardware and message cost before choosing a fix. That is the call our per-partition test points to. Dropbox found under a deadline that an early key choice sent every message to one partition, which a check of how keys spread would show. Cloudflare's new offset-based health check caused cascading restarts during rebalances, found only after rollout, which is our test of commits and restarts. CrowdStrike's note that adding consumers barely helps matches the catch for that option in the table above.
What we would test first: Measure lag per partition and per consumer at peak.
What the diagnostic would test
The tests that rule the wrong ways out.
On your own systems and data, read-only first. The number of tests sets the diagnostic's price, quoted before it starts.
- 1
Lag per partition and per consumer, at peak.
- 2
How evenly keys spread across partitions.
- 3
How offsets are committed, and what happens on restart.
- 4
Where the time goes inside one message's processing.
- 5
What 'exactly once' means for your downstream systems.
This week, without us
Three things you can do yourself.
- 1
Chart consumer lag per partition for a full day, not the total.
- 2
Check whether a few keys carry most of the messages.
- 3
Find where your consumer commits offsets: before or after the work is done.
Sound like you? Bring it to a 30-minute call. We'll tell you which tests your case needs, and what they cost.
This page describes what teams in this situation commonly face. It is not a client story.