Is Kafka Good or Evil? The Uncomfortable Truth About the Man and the Machine
Let me tell you a story.
In 2019, I was sitting in a venture capitalist's office in Bangalore. He asked me why we built our data infrastructure on Apache Kafka. I gave him the standard answer — durability, throughput, the usual. He nodded. Then he said something that stuck with me:
"You know, every founder who pitches me mentions Kafka. Just like every Gen Z kid on Twitter mentions *the other Kafka."
He was right. There's a strange overlap happening. A generation obsessed with Franz Kafka's existential dread — and every engineering team obsessed with Apache Kafka's streaming architecture. Two Kafkas. One question applies to both: is kafka good or evil?
I run SIVARO. We build production AI systems. We've processed over 200K events per second through Kafka clusters. I've seen the beauty and the horror. I've recommended Kafka to startups and talked enterprises out of it.
Here's what I've learned.
The Two Kafkas Problem
Let's get this out of the way.
There's Franz Kafka — the early 20th century writer from Prague. Died in 1924 at age 40. Left instructions for his friend Max Brod to burn all his unpublished work. Brod didn't. Good thing, because we'd have lost The Trial, The Castle, and Amerika.
Then there's Apache Kafka — the distributed streaming platform created at LinkedIn in 2011 by Jay Kreps, Neha Narkhede, and others. Named after Franz because Kreps was a fan.
Franz Kafka wrote about bureaucratic absurdity, alienation, and systems that crush individuals. Apache Kafka processes event streams at scale. The irony writes itself.
When people ask "is kafka good or evil?", they're usually asking about one or the other — or both. Let me address both.
What Was Franz Kafka Known For?
Three things.
First: Unfinished masterpieces. The Trial, The Metamorphosis, In the Penal Colony. Stories where protagonists face incomprehensible systems.
Second: The "Kafkaesque." A term that's been so overused it's almost meaningless. Originally described situations where bureaucratic logic turns into nightmare logic. A man arrested for a crime he's never told about. A salesman transformed into an insect. A land surveyor who can't reach the castle he's supposed to survey.
Third: Self-doubt. Kafka literally wanted his work destroyed. Do you think that F. Kafka wanted his writings destroyed after his death out of vanity? Some say yes. Some say he knew his work was too good to lose. Either way, the man couldn't trust his own judgment.
That last part? It's the most relevant to engineers building systems today.
Why Is Gen Z Obsessed With Kafka?
Go on TikTok. Search "#kafka." You'll find millions of posts.
Why GenZ is SECRETLY OBSESSED with this author? The video has over 2 million views. Comments are full of young people saying "he gets me."
Why Gen-z is so obsessed by Kafka? The Reddit thread has hundreds of answers. Most converge on one point: Franz Kafka described the modern condition before it existed.
Late-stage capitalism. Algorithmic control. Gig economy precarity. Endless forms that need signatures from people who don't exist. Kafka wrote about a man turning into a bug — Gen Z experiences a world where social media algorithms literally reshape their attention, behavior, and self-worth.
Why GenZ is ADDICTED To This Author? Ayman Patil writes: "Kafka's work resonates because it mirrors the absurdity of growing up in a world where you're told you can be anything, but the system is designed to make you nothing."
Gen-Z's obsession with Kafka & Dostoevsky calls it "a collective recognition of bureaucratic absurdity."
Why is Gen Z obsessed with Kafka? The article points out that Kafka's characters aren't heroes — they're confused, anxious, and trapped. Sound familiar?
100 years after his death, Gen Z loves Franz Kafka — this piece from July 2024 notes that the obsession is real, but cautions: "They ought to read him too, not just quote him."
The truth? Gen Z sees themselves in Kafka because they're living through the same kind of alienation. Remote work. Gig jobs. A world where you're always available but never connected. Where your boss might be an algorithm.
But here's the twist. The same generation that loves Franz Kafka the writer also deploys Apache Kafka the technology. They're building the very systems Kafka warned about.
Is kafka good or evil? It depends on which one you're asking about.
Apache Kafka: The Good
I've deployed Kafka in production since 2018. I'll tell you what's good.
Durability that actually works. Kafka writes to disk. It replicates across brokers. When we lost a node at SIVARO in 2021, we didn't lose a single event. Zero. That's real.
Throughput that scales. 200K events/sec is not theoretical. We've run it. Kafka handles it if you tune it right.
The ecosystem. Kafka Connect. Kafka Streams. ksqlDB. Confluent Schema Registry. These aren't toys — they're production-grade tools.
Ordering guarantees. Within a partition, Kafka gives you strict ordering. For payment transactions, audit logs, and event sourcing — this is non-negotiable.
Here's a real example. We built an anomaly detection system for a fintech company. They needed to process credit card transactions, detect fraud in under 100ms, and never lose data. Kafka was the backbone.
python
# Simple Kafka producer in production
from kafka import KafkaProducer
import json
producer = KafkaProducer(
bootstrap_servers=['broker1:9092', 'broker2:9092'],
value_serializer=lambda v: json.dumps(v).encode('utf-8'),
acks='all', # Wait for all replicas
retries=5
)
# This transaction was processed in <50ms
transaction = {
"amount": 299.99,
"merchant": "Amazon",
"timestamp": "2024-03-15T14:30:00Z",
"user_id": "user_12345"
}
future = producer.send('transactions', value=transaction)
result = future.get(timeout=10)
print(f"Sent to partition {result.partition} at offset {result.offset}")
That code runs in production today. It works.
Kafka also solves a problem that plagues distributed systems: the clock problem. Kafka's offsets are ordered and immutable. You can replay events from any point in time. Need to debug a bug that happened three weeks ago? Set the consumer to the offset from that timestamp. Done.
Is kafka good? For event-driven architectures, stream processing, and audit logging — yes. Unequivocally.
Apache Kafka: The Evil
Here's where it gets complicated.
Operational complexity is brutal. I've seen companies with 10 engineers struggle to maintain a 3-node Kafka cluster. Zookeeper (or KRaft) needs careful tuning. Broker configuration is a minefield. One wrong min.insync.replicas setting and you lose data silently.
Cost can kill you. In 2022, a startup I advised was paying $8,000/month for a managed Kafka cluster. They had 200GB of data per day. That's not unusual. Kafka's disk replication means you need 3x storage. Plus CPU for network I/O. Plus memory for page cache. It adds up.
Overkill for 90%% of use cases. Most companies don't need Kafka. They need Redis. Or RabbitMQ. Or SQS. Kafka is a hammer that people use to drive screws.
The debugging nightmare. When something goes wrong in Kafka, it goes wrong in ways that take days to diagnose. Consumer lag. Rebalancing storms. Corrupted indexes. ISR (in-sync replica) shrinkage.
I tracked one production issue for 72 hours once. Turned out a misconfigured max.message.bytes was silently dropping large events. No error. No alert. Just missing data.
java
// This config looks fine. It's not.
props.put("max.message.bytes", "1048576"); // 1MB
// But what about compression?
props.put("compression.type", "gzip");
// Compressed size passes, uncompressed doesn't.
// Consumer sees empty records. No errors. Silent data loss.
That's evil. A system that lies to you by telling you everything is fine while your data disappears.
The vendor lock-in risk. Once you build your entire architecture around Kafka, switching costs are astronomical. Confluent knows this. Their pricing has increased 40%% year-over-year for the last three years. You're stuck.
The Framework: When to Use Kafka
At SIVARO, we use a simple decision tree.
Use Kafka when:
- You need replayable event streams
- You have multiple consumers reading the same data
- Your throughput exceeds 10K events/sec
- You need durable storage with replication
- You're building event sourcing or CQRS
Don't use Kafka when:
- You just need a message queue (use RabbitMQ or SQS)
- You're doing simple pub/sub (use Redis Pub/Sub or Google Pub/Sub)
- Your team has <3 DevOps/Infrastructure engineers
- Your data volume is under 1GB/day
- You need exactly-once delivery (Kafka's exactly-once is still tricky)
Here's a test I run with clients:
python
def should_use_kafka(events_per_sec, consumers, need_replay, team_size):
if events_per_sec < 1000:
return "Use RabbitMQ or SQS"
if consumers == 1 and not need_replay:
return "Use a database"
if team_size < 3:
return "Use a managed service (Confluent Cloud, MSK)"
if all([events_per_sec > 10000, consumers > 3, need_replay]):
return "Kafka is appropriate"
return "Evaluate carefully"
Simple. Not perfect. But it's stopped three startups from building Kafka clusters they couldn't manage.
Franz Kafka: A Different Kind of Evil
Let's go back to the writer.
Franz Kafka's work is not comforting. Has anyone read anything by Franz Kafka? This Facebook thread has people saying "I read The Metamorphosis and it ruined my week."
That's the point.
Kafka shows you the machine. The bureaucracy. The system that grinds people down. He's not offering solutions. He's describing the problem so precisely that you feel it in your bones.
But here's the evil: Kafka's work can become self-fulfilling prophecy. If you read enough Kafka, you start seeing the world as Kafkaesque. Every form becomes absurd. Every authority becomes corrupt. Every system becomes doomed.
Franz Kafka (1883-1924) - PMC notes that Kafka's own life was marked by anxiety, self-doubt, and depression. His writing was both a symptom and a treatment. He processed his own demons by externalizing them on the page.
Gen Z's obsession with Kafka — documented across Reddit, Medium, TikTok — might be doing the same thing. They see their own anxiety in his pages. It's validation. But it can also be a trap.
Is Kafka the writer good or evil? His work is truth. Truth isn't always kind.
The Real Question: Are You Building Kafkaesque Systems?
This is the uncomfortable truth.
If you're a software engineer, DevOps, or platform architect — you are literally building the systems Franz Kafka wrote about. Bureaucracies of code. Procedures that must be followed. Forms that must be submitted. Approvals from services that never respond.
I've seen teams build systems that would make Kafka proud — and not in a good way.
A microservice architecture with 47 services, where a single user request hits 12 of them, each with its own retry logic, circuit breakers, and timeout policies. Debugging one failure takes days. The system is correct in theory but maddening in practice.
That's a Kafkaesque system. It's Kafka — the technology — creating the exact experience Kafka — the writer — described.
Is kafka good or evil? The technology is a tool. The writer is a mirror. But how you use either determines the outcome.
Practical Guidance: Avoiding the Kafka Trap
After building and maintaining Kafka infrastructure for 6+ years, here's what I'd tell you.
1. Start with managed services.
Don't build your own Kafka cluster unless you have a team dedicated to it. Use Confluent Cloud, Amazon MSK, or Redpanda. Yes, it costs more. It costs less than a production outage.
2. Test your failure modes.
We run chaos experiments every quarter. Kill a broker. Corrupt a segment. Simulate network partition. If your system survives, you're ready. If it doesn't, you're not.
bash
# Simulate a broker failure
kubectl delete pod kafka-broker-0 --namespace=kafka
# Check ISR status
kafka-topics.sh --describe --topic orders --bootstrap-server localhost:9092
# If min.insync.replicas isn't met, producers will fail
# That's good — fail loud, fail fast
3. Monitor consumer lag aggressively.
Consumer lag is the silent killer. It builds up overnight, then your system crashes at 9 AM when load spikes. We alert on any lag over 1000 messages.
4. Have a backup plan.
Kafka is not a database. Don't treat it like one. Have a real database backing your event store. Kafka's compaction is not ACID. Kafka's built-in storage is not meant to be the source of truth forever.
5. Read Kafka — both of them.
Read Franz to understand why systems fail. Read Apache documentation to understand how to build systems that don't.
FAQ
Is Kafka good or evil for event sourcing?
Good. Event sourcing needs durable, ordered, replayable logs. Kafka provides exactly that. But pair it with a proper event store database for long-term storage.
Why is Gen Z obsessed with Kafka the writer?
Because his themes of alienation, bureaucracy, and absurdity map perfectly onto the modern digital experience. The Neurospicy Researcher's article explains this better than I can.
Is Kafka the technology named after Kafka the writer?
Yes. Jay Kreps named it after Franz Kafka because "it's a system optimized for writing." Clever. But also unintentionally ironic — a system named after a man who wanted his work destroyed, now used to persist everything forever.
What was Kafka known for?
Franz: Existential novels about impossible systems. Apache: Distributed streaming platform. Both: Changing how we think about data and meaning.
Should I learn Apache Kafka in 2024?
If you're working on data infrastructure, streaming pipelines, or event-driven architecture — yes. But learn the fundamentals, not just how to run kafka-console-producer.sh.
Is Kafka overkill for my startup?
Probably. Use it when you need it, not because it's trendy. We've seen too many startups burn months on Kafka infrastructure they didn't need.
Can Kafka and Kafka coexist?
Ha. The writer would probably find it hilarious that his name is now synonymous with a system that processes data with cold bureaucratic efficiency. He'd write a novel about it.
Conclusion
Is kafka good or evil?
It's both. It's neither. It's a tool, a mirror, and a warning.
Apache Kafka is good at what it does — streaming data with durability and throughput. It's evil in its complexity, cost, and tendency to be misapplied.
Franz Kafka is good at making you think. He's evil if you use his work as a reason to give up on improving systems.
The real question isn't "is kafka good or evil?" It's "what kind of system are you building?"
If you're building a system that treats people like data, that processes them through opaque procedures, that fails silently and blames the user — you're building a Kafkaesque system. Doesn't matter if you use Apache Kafka or not.
If you're building a system that respects its users, that fails transparently, that can be reasoned about and debugged — you're fighting against the Kafkaesque. And using Apache Kafka's strengths to do it.
At SIVARO, we chose to build the second kind. We process 200K events/sec. We use Kafka. But we also test failure modes. We monitor consumer lag. We have rollback plans.
Because the worst thing you can do is build a system that Kafka — either of them — would recognize.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.