Kafka Exactly Once Semantics: The Real-World Guide

Kafka Exactly Once Semantics: The Real-World Guide

I’ll never forget the night of April 12, 2024. A fintech client called me at 2 AM. Their payment pipeline had just credited $2.3 million twice to the same merchant. The root cause? “At-least-once” delivery combined with a broker failover. The duplicate slipped through.

That’s when I stopped treating exactly-once semantics as an academic feature and started treating it as a critical constraint. It saved that client — and since then, we’ve deployed it across a dozen production systems at SIVARO.

Here’s the honest guide I wish I had when I started.

Exactly-once semantics (EOS) in Kafka means each message is delivered exactly one time — no duplicates, no gaps. It’s not a single setting; it’s a protocol combining idempotent producers, transactional writes, and consumer isolation. If you’ve ever googled “kafka exactly once semantics” and felt overwhelmed by conflicting advice, this article is for you.

What you’ll learn: when exactly-once actually helps, how to configure it correctly (without shooting yourself in the foot), the three killer pitfalls I see teams hit, and why most people misunderstand the trade-off between correctness and throughput.

Why Exactly-Once Matters (But Not Always)

Most people think they need exactly-once. They’re wrong.

Let me show you two pipelines from SIVARO’s own stack:

Pipeline A: Financial settlement. For a Singapore-based payments processor, every transaction must be applied exactly once. A double credit means losing money. An omission means regulatory fines. We run this pipeline with full EOS — idempotent producers, transactions, read_committed consumers. Throughput? ~8,000 messages per second. Acceptable for the use case.

Pipeline B: Clickstream analytics. For a retail client in 2025, we process 150,000 events per second. A duplicate click on a “Buy Now” button? Who cares — we deduplicate after the fact in the data lake. Running exactly-once would have crushed throughput and added no value. We use at-least-once with a lightweight dedup layer.

The difference is stark. Kafka’s exactly-once semantics solve a real problem: message duplication caused by retry logic, broker crashes, and consumer rebalances. But they come with a cost — latency, throughput, and operational complexity. Kafka vs Pulsar comparisons often highlight that Pulsar’s exactly-once model is simpler for single-topic scenarios, but Kafka’s cross-topic transactional support is more powerful when you need atomic writes to multiple partitions.

The Building Blocks: Idempotent Producers and Transactions

Exactly-once in Kafka rests on two pillars: idempotent producers and the transactional API. You need both.

Idempotent Producers

Set enable.idempotence=true. That’s the single most important configuration.

What happens under the hood: each producer gets a unique producer.id and stamps each message with a monotonically increasing sequence number. The broker deduplicates by (producer.id, sequence.number). So if a producer retries after a timeout, the broker silently drops the duplicate.

Without idempotence, retries can cause duplicates. With it, you get per-producer sequential exactly-once. Simple.

java
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka1:9092");
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
// These are forced when idempotence is true:
// props.put(ProducerConfig.ACKS_CONFIG, "all");
// props.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE);
// props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 5);

KafkaProducer<String, String> producer = new KafkaProducer<>(props);
producer.send(new ProducerRecord<>("orders", "key1", "value1"));
producer.flush();

Note: idempotence works only within a single producer session. If the producer crashes and restarts with a new producer.id, the sequence resets. That’s where transactions come in.

The Transactional API

Transactions let you group multiple produce requests and even consumer offset commits into an atomic unit. All messages in a transaction are committed together — or none are.

The flow:

1. Initialize the transactional producer.
2. Begin a transaction.
3. Send messages.
4. Commit or abort.

java
producer.initTransactions();

try {
    producer.beginTransaction();
    producer.send(new ProducerRecord<>("account", "credit", "100"));
    producer.send(new ProducerRecord<>("audit", "event", "credit_applied"));
    producer.commitTransaction();
} catch (ProducerFencedException e) {
    producer.abortTransaction();
}

This example atomically writes to two topics: account and audit. If the write to audit fails, neither message is committed. That’s cross-topic atomicity — something Pulsar doesn’t natively support (see Pulsar vs Kafka - Comparison and Myths Explored for more nuance).

But here’s what most tutorials skip: transactions alone do not give you exactly-once to a sink. You also need to commit consumer offsets within the same transaction.

The Consumer Side: Isolation Level and the Read-Process-Write Loop

This is where I’ve seen teams fail the most.

Excerpt from a debugging session in 2025: A team had enabled transactions on the producer side but still saw duplicates downstream. Why? Because their consumer was committing offsets independently, outside the transaction. When the consumer crashed and restarted, it re-read messages that had already been processed but not yet committed to the sink.

The solution: use the Kafka transactions API on the consumer side too.

The EOS Pattern

1. Consumer polls messages.
2. For each batch, producer begins a transaction.
3. Producer sends output records to sink topics.
4. Producer commits the offsets of the consumed messages inside the same transaction using sendOffsetsToTransaction().
5. Producer commits the transaction.

This ensures that if the whole process fails, both the output and the offset advancement are rolled back. The consumer, when restarted, will start from the last committed offset.

java
consumer.subscribe(Collections.singletonList("input-topic"));
producer.initTransactions();

while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
    producer.beginTransaction();

    for (ConsumerRecord<String, String> record : records) {
        // Process and produce
        producer.send(new ProducerRecord<>("output", record.key(), process(record.value())));
    }

    // Commit offsets atomically
    producer.sendOffsetsToTransaction(getOffsets(records), consumer.groupMetadata());
    producer.commitTransaction();
}

Set isolation.level=read_committed on the consumer. Without that, the consumer will see uncommitted messages (including ones from aborted transactions). I’ve seen devs accidentally set it to read_uncommitted and then wonder why duplicates appear.

The Ugly Truth: Exactly-Once in Practice

In theory, theory matches practice. In practice, it doesn’t.

Performance overhead. In SIVARO’s load tests (June 2025, 3-broker cluster, 6 partitions), turning on transactions caused a 25% throughput drop for a simple produce-consume pipeline. The overhead comes from the two-phase commit protocol on the broker side. If you're processing 100K messages/second, you’ll feel it.

Zombie producers and consumer rebalancing. Kafka uses a fencing mechanism: when a new producer instance starts with the same transactional.id, it fences the old one. But consumer group rebalancing can trigger race conditions. A long-running consumer might still hold the old producer instance and try to produce within a transaction after being fenced. The result? ProducerFencedException and a crash.

The kafka consumer group rebalancing fix for EOS: use static group membership. Set group.instance.id to a unique identifier per consumer instance. This prevents unnecessary rebalances when a consumer restarts. Also, configure partition.assignment.strategy to CooperativeStickyAssignor to minimize the number of partitions that change hands during a rebalance.

From What's the Difference Between Kafka and RabbitMQ?, RabbitMQ enthusiasts often claim their message broker handles exactly-once more naturally via publisher confirms and consumer acknowledgements — but that’s only for single message, not atomic multi-sink scenarios.

Cross-system exactly-once is a fantasy. Kafka’s exactly-once semantics only apply within a single Kafka cluster. If you’re writing to a database or an S3 bucket, you need either idempotent writes at the sink or a distributed transaction (like two-phase commit). Many teams I know use a “best-effort exactly-once” approach: make the sink idempotent (e.g., use upserts with a dedup key) and accept the occasional skew.

Kafka Connect vs Flink: Which Handles Exactly-Once Better?

We get this question constantly. “Should I use Kafka Connect or Flink for ETL?”

My take: Kafka Connect with exactly-once sink connectors is brittle. Most connectors don’t support it fully. For example, the JDBC Sink connector in version 3.x had a long-standing bug where it would commit offsets but sometimes fail to flush the writes to the database, causing duplicates. We’ve seen it.

Flink’s checkpointing mechanism is a better fit for end-to-end exactly-once. Flink checkpoints state and exactly-once sinks by implementing two-phase commit (Kafka’s own TwoPhaseCommitSinkFunction). It’s more mature, more configurable, and handles rebalances gracefully.

I wrote about the kafka connect vs flink trade-offs in a SIVARO internal memo last year. The short version: if your pipeline is simple source-to-sink with no transformations, Connect is fine with at-least-once. If you need stateful processing or EOS, use Flink.

Configuration Checklist: The Right Settings

You don’t need to remember everything. Keep this list handy.

Producer:

• enable.idempotence=true (forces acks=all, retries=Integer.MAX_VALUE, max.in.flight=5)
• transactional.id must be unique per producer instance (e.g., txn-order-processor-1)
• max.in.flight.requests.per.connection=5 is safe with idempotence (Kafka 2.0+)

Consumer:

• isolation.level=read_committed
• enable.auto.commit=false
• group.instance.id=<unique-id> (for static membership)

Broker:

• transaction.state.log.replication.factor >= 3 (production)
• transaction.max.timeout.ms — keep default 900000 (15 minutes) or lower if you want stricter timeout

Properties file example:

properties
bootstrap.servers=kafka1:9092,kafka2:9092,kafka3:9092
enable.idempotence=true
transactional.id=payments-txn-1
acks=all
retries=2147483647
max.in.flight.requests.per.connection=5

When NOT to Use Exactly-Once (And What to Use Instead)

I’ll be blunt: most pipelines don’t need EOS. You’re over-engineering if you add transactions to every topic.

Don’t use it for:

• Log aggregation: duplicates don’t matter, throughput does.
• IoT sensor streams: one missing reading is fine; retries handle it.
• High-volume analytics: at-least-once + dedup at query time costs less.
• Stateless transformations: a simple map/filter can’t create chaos.

What to use instead:

• At-least-once with idempotent sinks (e.g., Redis SET commands are idempotent).
• At-most-once for monitoring (loss is OK).
• Effectively-once via application-level deduplication (e.g., using a dedup key in Cassandra).

Kafka vs Pulsar vs RabbitMQ vs NATS: What's Actually Best for Your Use Case has a good comparison table. For low-latency, non-critical data, NATS with at-most-once is often the right choice. Kafka’s EOS is overkill there.

Alternatives and Comparisons (Brief)

Pulsar’s exactly-once model uses producer epochs and broker-side deduplication per producer. It’s simpler — no transactions needed for single-topic writes. But it lacks atomic multi-topic commits (as of 2026, though there’s work on Pulsar transactions). Kafka vs Pulsar: Streaming Platform Comparison goes deep into the architectural differences.

RabbitMQ? Its confirmation mechanism is at-most-once on publish, and exactly-once is handled via idempotent consumers. Not a replacement for Kafka when you need streaming replay or cross-service atomicity.

Frequently Asked Questions

Q: Does exactly-once mean no duplicates ever?
A: Within a single Kafka cluster, yes — as long as you use idempotent producers and commit offsets in transactions. But cross-cluster mirroring or sink writes can still duplicate.

Q: What happens if a transaction times out?
A: The broker aborts it. The consumer will skip those messages (if read_committed). The producer must retry from the last committed offset — which requires restarting the entire transaction.

Q: Can I use exactly-once with Kafka Streams?
A: Yes. Kafka Streams enables exactly-once via the processing.guarantee config set to exactly_once_v2 (Kafka 3.0+). It handles the transactional semantics internally.

Q: How much performance loss should I expect?
A: In our tests, 15–30% throughput drop for write-heavy workloads. Read-heavy pipelines less affected. Always benchmark with your data shape.

Q: Is exactly-once the same as idempotent?
A: No. Idempotent producers prevent duplicates within a single producer session. Transactions add atomicity across sessions and topics. You need both for true exactly-once.

Q: How do I fix Kafka consumer group rebalancing when using transactions?
A: Use static group membership (group.instance.id), cooperative rebalancing, and ensure your transactional.id is unique per consumer instance. Also, handle ProducerFencedException gracefully by restarting the consumer.

Q: Should I use Kafka Connect or Flink for exactly-once?
A: Flink, for anything beyond simple S3 sinks. Connect’s exactly-once is half-baked for most connectors as of 2026.

Q: Does Kafka exactly-once work with external databases?
A: Not natively. You need a two-phase commit coordinator or write idempotent droplets (e.g., upsert with dedup key).

Conclusion

Exactly-once semantics in Kafka is a powerful tool — but it’s a tool, not a goal. Every time I see a team enable transactions without understanding the trade-offs, I brace for a late-night call.

At SIVARO, we use EOS sparingly. About 20% of our pipelines. The rest run on at-least-once with careful sink idempotency. That’s pragmatism, not laziness.

If you need cross-topic atomicity, use the transactional API. If you need per-producer dedup, enable idempotence. If you need both, implement the read-process-write loop correctly — with offset commits inside the transaction.

And remember: test your rebalance behavior. Simulate broker failures. Measure the throughput cost. Your architecture will thank you.

---

Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.