Kafka Producer Callback Example: Real-World Async Patterns
I’m writing this on July 30, 2026. Two years ago, I watched a production pipeline at SIVARO silently drop 12% of events for six hours. We had a Kafka producer pushing to a topic, no callbacks, no logging. The cluster was healthy. The broker acknowledged the batch. But a transient network blip meant half the records never made it to disk. We only noticed when downstream ML models started hallucinating on incomplete data.
That’s when I stopped treating producer callbacks as optional decoration. If you’re running Kafka in 2026 — whether you’re comparing it to RabbitMQ, Pulsar, or NATS — you need to understand how and when to use callbacks. Not just the API, but the trade-offs: latency, ordering, error handling.
This guide is the one I wish I’d read in 2021. You’ll learn how to implement a kafka producer callback example that actually works in production. We’ll cover retries, metrics, consumer group rebalancing impacts, and why the async paradigm beats sync in every real-world system I’ve built.
Why Callbacks Matter More Than You Think
Most developers treat producer.send() like a fire‑and‑forget API. It’s not. Kafka’s producer is asynchronous by design — the send method blocks only long enough to enqueue the record in an internal buffer. The actual network I/O happens on background threads. Without a callback, you lose all visibility into whether the message was acknowledged.
Here’s the hard truth: Kafka’s durability guarantees (acks=all, min.insync.replicas) only apply if the broker sends back a response. If your client disconnects mid‑write, the batch could be lost. You won’t know unless you check.
I’ve seen teams switch from RabbitMQ to Kafka because of throughput Kafka vs RabbitMQ 2026 comparisons — RabbitMQ’s confirms are synchronous by default, Kafka’s are async. They migrate, then spend weeks debugging silent data loss because no one added a callback.
The Anatomy of a Kafka Producer Callback
Let’s start with the basics. In the Java client (still the most widely used in 2026), a callback is an object implementing org.apache.kafka.clients.producer.Callback with a single method:
java
public interface Callback {
void onCompletion(RecordMetadata metadata, Exception exception);
}
The metadata is non‑null on success — it contains the partition, offset, and timestamp. The exception is non‑null on failure.
Here’s a kafka producer callback example you can drop into any project:
java
import org.apache.kafka.clients.producer.*;
public class CallbackExample {
public static void main(String[] args) {
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("acks", "all"); // strongest durability
props.put("retries", 3); // default retries are finite now (was infinite before 3.0)
props.put("max.in.flight.requests.per.connection", 5);
Producer<String, String> producer = new KafkaProducer<>(props);
for (int i = 0; i < 100; i++) {
ProducerRecord<String, String> record = new ProducerRecord<>("my-topic", "key-" + i, "value-" + i);
producer.send(record, (metadata, exception) -> {
if (exception == null) {
System.out.printf("Sent to partition %d offset %d%n", metadata.partition(), metadata.offset());
} else {
System.err.printf("Failed to send record: %s%n", exception.getMessage());
}
});
}
producer.close();
}
}
That’s the skeleton. But in production, you won’t just print to stdout. You’ll route failures to a dead‑letter topic, increment a counter, maybe trigger an alert.
Example: Handling Delivery Semantics with Callbacks
This is where the kafka producer callback example gets real. Delivery semantics — at‑most‑once, at‑least‑once, exactly‑once — affect what you do in the callback.
For at‑least‑once, you want to retry on transient errors but not duplicate. Kafka’s retries config handles the retry logic internally, but if you set max.in.flight.requests.per.connection=1 and enable.idempotence=true, you get strong ordering guarantees and no duplicates. The callback then becomes a monitoring hook: log success or escalate failure.
java
props.put("enable.idempotence", true); // prevents duplicates on retry
props.put("max.in.flight.requests.per.connection", 1);
In the callback, you can track how many records failed permanently:
java
producer.send(record, (metadata, exception) -> {
if (exception != null) {
// Not a transient error — retries exhausted or non‑retriable
metrics.failedRecords.increment();
deadLetterProducer.send(new ProducerRecord<>("dead-letter-topic", record.key(), record.value()));
}
});
I’ve seen teams treat callbacks as a logging afterthought. Don’t. In 2024, a fintech client of mine lost $40K because their callback only printed to a log file that nobody read. The failure was RecordTooLargeException — a 5 MB payload hitting the default 1 MB limit. A dead‑letter queue would have caught it instantly.
Retry Logic: The Callback That Saves Your Pipeline
Kafka’s built‑in retries handle broker‑side timeouts and leader elections. But what about client‑side issues — like a down DNS or a misconfigured serializer?
At SIVARO, we wrap the producer in a custom retry layer that uses the callback as the trigger:
java
public void sendWithRetry(ProducerRecord<String, String> record, int maxRetries) {
sendWithRetryInternal(record, maxRetries, 0);
}
private void sendWithRetryInternal(ProducerRecord<String, String> record, int maxRetries, int attempt) {
producer.send(record, (metadata, exception) -> {
if (exception == null) {
return; // success
}
if (attempt < maxRetries) {
// Exponential backoff: 100ms, 200ms, 400ms, ...
long backoff = (long) Math.pow(2, attempt) * 100;
scheduler.schedule(() -> sendWithRetryInternal(record, maxRetries, attempt + 1), backoff, TimeUnit.MILLISECONDS);
} else {
log.error("Failed after {} retries for record {}", maxRetries, record);
deadLetterProducer.send(record);
}
});
}
Notice: we use the callback’s exception to decide whether to retry. Kafka’s internal retries may already have been exhausted — we’re adding a second layer. Why? Because leader elections can take longer than retry.backoff.ms. With this pattern, we recovered from a 7‑second ZooKeeper hiccup last year. Internal retries gave up after 3 seconds. Our custom retry succeeded on the 4th attempt.
Caveat: this breaks ordering. If you need strict per‑partition ordering, you can’t reorder retries. In that case, use max.in.flight.requests=1 and accept the throughput hit. Or use idempotent writes with a transactional producer.
Async vs Sync: When to Use Callbacks (and When Not To)
The synchronous alternative to callbacks is producer.send(record).get(). This blocks the calling thread until the broker acknowledges. It simplifies error handling — you get an exception immediately. But it kills throughput.
| Approach | Throughput (records/sec) | Latency p99 | Error visibility |
|---|---|---|---|
Sync .get() |
1,200 | 12ms | Immediate |
| Async with callback | 28,000 | 3ms | Delayed, but complete |
| Fire‑and‑forget | 32,000 | 2ms | None |
Numbers from a 3‑node Kafka cluster I benchmarked in March 2026. Fire‑and‑forget looks fast but has no error path. Callbacks give you 98% of the throughput with full observability.
When should you use sync? Almost never. The only case I accept is in a test‑harness or a trivial script that processes a few hundred records. Anything in production demands async with callbacks.
One exception: exactly‑once semantics with transactions. producer.initTransactions() and producer.beginTransaction() are inherently synchronous because they require fencing. Even then, callbacks for individual records are still useful for metrics.
What About At‑Most‑Once, At‑Least‑Once, Exactly‑Once?
Callbacks aren’t just for error logging. They’re your hook into delivery semantics.
At‑most‑once: set acks=0. The producer never waits for a response. No callback fires on success (the broker doesn’t reply). You might still get a callback on a client‑side serialization error. This is for metrics where losing a few values is acceptable. I’ve used it for page‑view counts. Speed over reliability.
At‑least‑once: acks=all with idempotence. The callback fires for every acknowledged record. On failure, you can retry in your own layer (as above) or send to a dead‑letter topic. This is 95% of production use cases.
Exactly‑once: requires transactions. The producer sends within a transaction, and the transaction is committed atomically. The callback fires per record inside the transaction, but the records aren’t visible until commit. If a callback reports an error, you can abort the entire transaction. This is complex and often unnecessary — most teams over‑engineer exactly‑once. In 2026, with idempotent consumers and deduplication at the application layer, exactly‑once through transactions is rarely worth the overhead.
Advanced: Callback with Custom Metrics and Logging
Here’s a kafka producer callback example from our production stack at SIVARO — we use Micrometer for metrics and centralised logging:
java
public class MetricsCollectingCallback implements Callback {
private final ProducerRecord<String, String> record;
private final MeterRegistry registry;
private final Timer timer;
public MetricsCollectingCallback(ProducerRecord<String, String> record, MeterRegistry registry) {
this.record = record;
this.registry = registry;
this.timer = Timer.start(registry);
}
@Override
public void onCompletion(RecordMetadata metadata, Exception exception) {
timer.stop();
if (exception == null) {
registry.counter("producer.success", "topic", record.topic()).increment();
registry.counter("producer.bytes", "topic", record.topic()).increment(record.serializedValueSize());
} else {
registry.counter("producer.failure",
"topic", record.topic(),
"exception", exception.getClass().getSimpleName()).increment();
// Structured logging for alerting
log.error("Kafka send failed", Map.of(
"topic", record.topic(),
"partition", metadata != null ? metadata.partition() : null,
"exception", exception.getClass().getName(),
"message", exception.getMessage()
));
}
}
}
Use a Timer.Sample from Micrometer — it tracks both success and failure latency. I’ve caught slow broker responses (p99 spiking from 2ms to 200ms) this way. The callback captures context (record key, topic) so you can trace failures.
Kafka Consumer Group Rebalancing Fix — How Callbacks on the Producer Side Help (or Don’t)
Let’s address the elephant in the room: kafka consumer group rebalancing fix. Rebalancing is a consumer‑side phenomenon. When a consumer joins or leaves a group, partitions are reassigned. During rebalancing, all consumers in the group stop processing (revoke then assign). This can cause backpressure upstream if producers keep sending.
How does a producer callback help? Indirectly.
If your consumer processes messages and produces results to a downstream topic (a common Kafka‑to‑Kafka pipeline), the producer callback tells you the downstream write succeeded. During a rebalance, the consumer may pause processing for several seconds. The producer buffered records from processing — but those records aren’t flushed until the consumer resumes. A callback that fires after the rebalance ends can confirm delivery.
At SIVARO, we added a thin wrapper: after the consumer finishes processing each batch, it checks if it’s still assigned to the partition. If not (due to rebalance), we don’t process further records until the assignment stabilizes. The producer callback then confirms the final state.
More directly, if you’re using Kafka Streams or KSQL, the producer callback is buried inside the framework. But for custom consumers with manual offset management, the callback is your only view into whether the output topic was written.
The real fix for rebalancing issues is beyond callbacks — it’s about cooperative rebalancing (KIP-429, default since Kafka 3.1) and using static group membership. But never underestimate the insight a callback gives you. In 2025, a client saw 30‑second consumer rebalances every hour. The producer callbacks revealed that the downstream topic was being written with acks=1, causing occasional loss. Switched to acks=all after seeing the callback metrics.
Kafka vs RabbitMQ 2026: Callback Patterns Differ
RabbitMQ uses publisher confirms — a synchronous pattern where the channel waits for a basic.ack. That’s analogous to producer.send().get(). Kafka’s callbacks are fundamentally different: they’re non‑blocking and decoupled from the sending thread.
In 2026, the debate between Kafka and RabbitMQ persists. AWS’s comparison highlights throughput vs. routing flexibility. For callbacks, the pragmatic difference is that RabbitMQ confirms are simpler to reason about (one ACK per message) but don’t scale as well under high concurrency. Kafka’s callbacks batch acknowledgements — a single callback may represent many records in one request.
I’ve built systems that bridge both: RabbitMQ for request‑reply, Kafka for event streaming. The callback pattern in Kafka is more performant, but you need better monitoring. RabbitMQ gives you a single point of truth (the confirmation). Kafka gives you a stream of metadata. Both work — just don’t mix up the paradigms.
Pulsar vs Kafka: Producer Callbacks in a Serverless World
Pulsar’s producer API also supports callbacks, but there’s a twist: Pulsar uses a segmented architecture with separate serving and storage layers. That means the acknowledgement might come from a broker that’s not the eventual leader. Kai Wähner’s comparison points out that Pulsar’s message‑ack flow is similar but with slightly different semantics around batching.
In Kafka, the callback’s RecordMetadata includes the partition and offset. In Pulsar, the MessageId is a composite that includes a ledger ID and entry ID. The callback pattern is the same conceptual hook, but the metadata objects differ.
For 2026, Pulsar’s advantage is native multi‑tenancy — you don’t need separate clusters for different teams. But Kafka’s stream processing ecosystem (KSQL, connectors) is still more mature. Choose based on your operations team’s expertise, not the callback API.
OneUptime’s comparison notes that both platforms handle millions of messages per second. The callback library matters less than the monitoring infrastructure around it.
Common Mistakes and Hard Lessons Learned
I’ve seen the same three mistakes across a dozen teams:
-
Blocking in the callback. Never sleep, never do synchronous I/O in
onCompletion. The callback runs on the producer’s I/O thread. A 100ms delay there blocks other sends. Use an executor for any heavy work. -
Assuming the callback is called exactly once. With
enable.idempotence=false, a retry might result in both a success and a failure callback (if the first attempt timed out but later succeeded). Handle duplicates in the callback — idempotent metrics counters, not additive. -
Forgetting to flush on shutdown. The producer’s
close()flushes pending sends. But if you callSystem.exit()or a container kills you with SIGKILL, in‑flight callbacks never fire. Graceful shutdown matters.
In 2023, we had a Kubernetes pod crash before producer.close() ran. Five thousand records vanished because the callbacks never executed. The fix: use a pre‑stop hook that flushes the producer and waits for pending callbacks.
FAQ
Q: Do I need a callback if I use synchronous send().get()?
Technically no, but you’re sacrificing throughput. Use callbacks for any system processing more than a few hundred messages per second.
Q: Can I run out of memory if callbacks fire slower than sends?
Yes. The producer has a buffer (controlled by buffer.memory). If callbacks are blocked (e.g., because your callback is slow or the retry queue is full), the buffer fills up and send() blocks. Monitor buffer.memory and max.block.ms.
Q: What happens if the callback itself throws an exception?
Kafka’s producer catches it and logs a warning. Your exception handling code must not throw. Wrap everything in try‑catch.
Q: Is the callback guaranteed to run on the same thread as the send?
No. The callback runs on the producer’s I/O thread (or a separate thread pool in newer versions). Never rely on thread‑local state.
Q: How does the callback affect ordering guarantees?
The callback order matches the order records were acknowledged by the broker. With max.in.flight.requests=1 and idempotence, callbacks arrive in send order. With higher concurrency, they may arrive out of order.
Q: Should I use the same producer instance for callbacks that write to a dead‑letter topic?
You can, but the dead‑letter producer should be a separate instance to avoid contention. A blocked callback on the main producer could prevent it from writing the dead‑letter too.
Q: How do I test callbacks in unit tests?
Use a mock Producer that captures the callback and invokes it manually. The Kafka test utilities have MockProducer — pass it a Callback and call completeNext().
Conclusion
The kafka producer callback example I’ve shared here isn’t academic. It’s the exact pattern we run at SIVARO across 12 production clusters processing over 200K events per second. Callbacks are your safety net, your observability layer, and your retry engine.
Skip them, and you’re flying blind. One transient failure, one misconfigured topic, one rebalance — and your data goes missing. I’ve been there. You don’t want that call at 2 AM.
Start with the simple callback. Add metrics. Add a dead‑letter queue. Test your retry logic. Then move on to the hard problems — like exactly‑once semantics or multi‑region replication.
The code is the easy part. The discipline to make callbacks a first‑class citizen? That’s what separates production systems from prototypes.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.