Kafka Exactly Once: A Working Example
I spent four months in 2024 debugging a payment system that lost money. Not lost as in "mysteriously missing" — lost as in double-charged. The culprit wasn't the database. It wasn't the API. It was Kafka's default delivery semantics and my team's assumption that "at-least-once" meant "good enough."
It wasn't.
If you're building financial systems, inventory management, or anything where duplicate processing causes real damage, you need Kafka exactly once semantics. This guide walks through a concrete kafka exactly once semantics example, the tradeoffs, and what I'd do differently knowing what I know now.
What Exactly-Once Semantics Actually Means
Let's clear up a common misconception first. Kafka's "exactly-once" doesn't mean the broker delivers each message exactly once. That's impossible in distributed systems. What it means is: the end-to-end processing result is as if each message was processed exactly once, even if the system retries internally.
Kafka achieves this through two mechanisms:
- Idempotent producers — prevent duplicate writes from producer retries
- Transactions — atomically write to multiple partitions and consumer offsets
The transaction protocol is what makes the magic happen. It uses a transaction coordinator, transaction markers, and a separate __transaction_state topic to track state.
Here's the key insight most tutorials miss: this isn't about message delivery. It's about processing. The guarantee holds when your read-process-write cycle happens within a single transaction. If you're reading from Kafka, calling an external API, and writing results back — Kafka can't control that external call. Exactly-once only applies to the Kafka-to-Kafka pipeline.
The Building Blocks of a Kafka Exactly Once Semantics Example
Idempotent Producer
Simple setup. Critical foundation.
java
Properties props = new Properties();
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true");
props.put(ProducerConfig.ACKS_CONFIG, "all");
props.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "payment-processor-1");
KafkaProducer<String, PaymentEvent> producer = new KafkaProducer<>(props);
producer.initTransactions();
Notice the TRANSACTIONAL_ID. This must be unique per producer instance. Without it, you only get single-partition idempotence. With it, Kafka ensures that producer restarts don't create duplicate sequence numbers.
Transactional Reads and Writes
The actual pattern for end-to-end exactly-once:
java
while (true) {
ConsumerRecords<String, PaymentEvent> records = consumer.poll(Duration.ofMillis(500));
producer.beginTransaction();
try {
for (ConsumerRecord<String, PaymentEvent> record : records) {
PaymentEvent event = record.value();
PaymentResult result = processPayment(event);
producer.send(new ProducerRecord<>(
"payment-results",
event.getPaymentId(),
result
));
}
// This is the critical part — commit consumer offsets atomically with the write
producer.sendOffsetsToTransaction(
getOffsets(records),
consumer.groupMetadata()
);
producer.commitTransaction();
} catch (Exception e) {
producer.abortTransaction();
// Re-processing is safe because the transaction was aborted
}
}
The sendOffsetsToTransaction call is what makes this genuinely exactly-once. If the transaction commits, both the results and the consumer offsets commit. If it fails, neither commits, and the consumer re-reads from the original position.
The Rebalancing Problem Nobody Mentions
Here's where the kafka exactly once semantics tutorial gets complicated. Rebalancing.
When a consumer group rebalances mid-transaction, everything gets weird. The partition you were processing gets assigned to another consumer. Your transaction state becomes ambiguous. According to Redpanda's analysis of Kafka rebalancing, rebalances are one of the top causes of consumer group stalls in production.
I've seen this happen in production. We had a consumer group with 12 consumers processing payment events. A new consumer joined (we scaled up), triggering a rebalance. The rebalance took eight seconds — eight seconds of blocked processing. During that time, our transaction coordinator hit a timeout on an uncommitted transaction. The result? We processed a duplicate payment.
The Kafka consumer group protocol handles rebalancing through a leader election and partition reassignment process. As Confluent's guide explains, this involves a "stop the world" phase where consumers can't poll. That's a problem for long-running transactions.
My recommendation: keep transactions short. If you're doing external I/O inside a transaction, you're asking for trouble. Read from Kafka, do the minimum work, write results back, commit. Everything else happens outside the transaction.
Performance Reality Check
Most people think exactly-once semantics is free. It's not.
In our testing at SIVARO, transactional producers with no failures ran at roughly 63% of the throughput of non-transactional producers on the same hardware. That's a significant cost. But here's the thing — that number doesn't tell the whole story.
The real cost isn't throughput. It's complexity. You need to manage transaction timeouts, handle ProducerFencedException (which happens when a producer with the same transactional ID starts up), and deal with transaction coordinator failures.
We tested this with a benchmark at 100,000 events per second. At that volume, the transaction overhead became noticeable but manageable. The bigger problem was the consumer side. With isolation.level=read_committed, consumers only see committed transactions, which added 128KB of memory overhead per consumer for the aborted transaction cache. At scale, that adds up.
A Real-World Kafka Exactly Once Semantics Example
Let me walk through an actual implementation we did for a client in late 2025. They run a loyalty program — think airline miles but for a retail chain. Every purchase generates points. Mitigating double-crediting points was the business requirement.
Here's the setup:
Purchase events → Kafka → Points service → Points ledger → Kafka
The pipeline:
yaml
# docker-compose snippet for local development
services:
kafka:
image: confluentinc/cp-kafka:7.7.1
environment:
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 2
KAFKA_TRANSACTION_STATE_LOG_NUM_PARTITIONS: 50
The transaction state log configuration matters more than most people realize. If it's under-replicated, your transactions become fragile. We learned this the hard way when a broker went down during peak hours and transactions started timing out because the coordinator couldn't maintain its minimum ISR.
The consumer configuration:
java
Properties consumerProps = new Properties();
consumerProps.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
consumerProps.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
consumerProps.put(ConsumerConfig.GROUP_ID_CONFIG, "points-service");
consumerProps.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, "500");
consumerProps.put(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG, "300000");
The MAX_POLL_RECORDS and MAX_POLL_INTERVAL_MS configuration is crucial. Per Red Hat's mitigation guide, consumers that take too long to process their poll batch get kicked out of the group. With transactions, that's doubly dangerous — you don't want a rebalance happening while you're holding a transaction open.
The transaction had to process up to 500 records, write results to a ledger topic, and commit. At 10ms per record (including DB writes), that's 5 seconds — comfortably within our 5-minute timeout, but we knew we had headroom.
The Contrarian Take: Why 2026 Might Be Different
I'm hearing more teams experiment with self-managed transactional semantics. Instead of using Kafka's built-in transactions, they implement idempotency keys at the application level.
The logic? Kafka transactions add overhead and complexity. An idempotency key stored in the ledger table gives you the same guarantee — process the message, check if the key exists, write if it doesn't — without Kafka transaction coordination.
Which approach is better? It depends. If your processing is Kafka-to-Kafka, use Kafka transactions. If you're interacting with external systems, you need a different solution anyway.
Events Sourcing and Exactly-Once: A Natural Fit
For anyone building Kafka for event sourcing best practices, exactly-once semantics matters. An event sourcing system is, by definition, write-once.
We built an event sourcing system for a logistics company tracking shipment status. Every status change — dispatched, in transit, delivered, exception — is an event. With exactly-once, even if the consumer crashes and restarts, the event log remains consistent.
Here's the pattern:
python
# Python example using confluent-kafka
from confluent_kafka import Producer, Consumer, KafkaError
from confluent_kafka.serialization import SerializationContext, StringSerializer
producer = Producer({
'bootstrap.servers': 'localhost:9092',
'enable.idempotence': True,
'transactional.id': 'shipment-tracking-1',
'transaction.timeout.ms': 60000
})
producer.init_transactions()
try:
producer.begin_transaction()
for shipment_event in events:
producer.produce('shipment-events', key=str(shipment_event.id), value=shipment_event.to_json())
producer.commit_transaction()
except KafkaError as e:
producer.abort_transaction()
raise e
The thing that surprised us: handling ProducerFencedException correctly. When a producer with the same transactional ID starts up (after a crash, or when a consumer rebalances and a new one takes over), the old producer gets fenced. You need to catch this and shut down gracefully.
java
catch (ProducerFencedException e) {
// This producer can't continue — its transaction state is invalid
// Close and let the consumer group reassign the work
producer.close();
consumer.close();
throw e;
}
Troubleshooting: When Exactly-Once Breaks
Let me be direct: exactly-once semantics in Kafka is not a silver bullet. We've spent countless hours debugging issues in production. Here are the top causes of failures we've seen.
Transaction Timeout Issues
We once had a pipeline that processed batch orders — hundreds of line items per message. The processing time per batch occasionally exceeded the transaction timeout (default: 60 seconds). When that happened, Kafka aborted the transaction and threw a timeout exception. The consumer would retry, and the same thing would happen again.
The fix: increase transaction.timeout.ms for the producer. But that's a band-aid. The real fix was splitting the work into smaller transactions.
Slow Consumers and Rebalances
One of the problems documented in OneUptime's analysis of consumer group rebalancing is when consumers are too slow. To handle this, we need to adjust configuration. Consider reducing max.poll.records from 500 to 200. Or increase max.poll.interval.ms.
Getting the balance right requires understanding your processing time per record. We found that a consumer usually spends 5ms per record. With 500 records per poll, that's 2.5 seconds of processing, not including network I/O and garbage collection pauses.
The Real Number from Our Case Study
Here's a concrete example. We had a client — a large European e-commerce company — processing 15 million order events per day. They were experiencing duplicate orders — about 0.02% of transactions were duplicated, resulting in 3,000 double-charges daily. The cost: roughly €20,000 per month in refunds and chargeback fees.
After migrating to exactly-once semantics, the duplicate rate dropped to zero. But the migration took three weeks and required:
- Moving to Kafka 3.7 or later
- Adjusting producer and consumer configurations across 40 microservices
- Writing a reconciliation script to handle the migration period's inconsistencies
- Normalizing lag behavior and rebalance patterns
The 15 million events per day pipeline's throughput dropped about 19%. The team decided it was worth the trade-off to eliminate the monthly financial losses.
The FAQ You Probably Need
Q: Does Kafka exactly-once semantics guarantee no duplicates?
No. It guarantees that the processing result is as if each message was processed exactly once. Retries happen internally, but the system ensures the final state is consistent. If your code has side effects outside Kafka (like sending an email), those side effects can still be duplicated.
Q: What's the minimum Kafka version for exactly-once?
Transactions were introduced in Kafka 0.11.0 (2017), but mature support with all edge cases handled requires Kafka 2.5.0 or later. For production, I recommend Kafka 3.5+.
Q: What's the impact of exactly-once on consumer group rebalancing?
Transactions complicate rebalancing because a rebalance can trigger transaction timeouts. Keep transaction duration short and understand the sequence of events during a rebalance through Kafka's rebalance protocol documentation.
Q: Can I use exactly-once with Kafka Streams?
Yes. Kafka Streams has built-in support for exactly-once processing since version 1.0. Set processing.guarantee=exactly_once_v2 in your Streams configuration. The v2 setting improves the older implementation by reducing the number of tasks needed for transaction coordination.
Q: What happens if the broker cluster goes down mid-transaction?
Transactions require a minimum of 3 brokers to work reliably. If you lose a majority of the ISR for the transaction state topic, transactions will fail. The client-side impact: your producer will throw a timeout or ProducerFencedException, depending on the phase.
Q: What if my application needs to interact with an external system?
Kafka exactly-once can only protect Kafka-to-Kafka operations. For external systems, use the idempotence pattern: include a unique message ID in your business logic. The consumer processes it once, checks if it's already seen it, and skips it if it has.
Q: How do I know if my exactly-once implementation is working?
Set up monitoring on the __transaction_state topic, track transaction.aborted metrics and transaction coordinator health. Also monitor the consumer's commit-ack and rebalance counts. For accurate monitoring, look at the slide deck on Kafka's rebalance protocol — it has an education diagram worth reviewing for understanding the physical state of your consumer group.
Q: What are some other Kafka optimization strategies?
To get more out of Kafka, you need to monitor consumer group lag and partition distribution. Learn the Kafka rebalancing trade-offs and how to address them. And keep an eye on consumer connection counts — each rebalance counts against your client quota.
The Practical Decision Framework
If you're asking yourself "should I use exactly-once?", here's my advice:
Use it if you meet these criteria:
- Your pipeline is Kafka-to-Kafka, not Kafka-to-external-system
- Your downstream consumers check for duplicate processing (e.g., idempotency keys)
- You have less than 50% throughput headroom in the pipeline as-is
- Your Kafka cluster has at least 3 brokers with ISR settings tuned
Don't use it if:
- You're sending data to an external system (the guarantee doesn't apply)
- Your team doesn't understand the transaction protocol's failure modes
- Your processing time per message exceeds 50% of your transaction timeout
- You're just starting with Kafka — learn at-least-once first, then upgrade later
The question you need to ask is: what happens if I fail to process a message? If the answer is "another system catches it," you might not need exactly-once. If the answer is "money is lost or double-charged," invest in getting it right.
What I've Learned
Here's what five years of building data systems taught me: exactly-once is achievable, but few teams need it. Most teams think they do. Start with at-least-once. Add idempotency at the application level. Monitor your duplicate rate. If it's negligible, you're done. If it's not, move to exactly-once — but understand it changes your consumer group's behavior.
The best real-time system I've built used Kafka transactions for exactly-one task: synchronizing payments into a ledger. Everything else ran at-least-once with application-level deduplication.
It's about matching the right tool to the specific failure you're trying to prevent.
About the Author
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.