What Is Architecture in a Distributed System? A Practitioner’s Guide
July 23, 2026
I spent three months in 2023 trying to figure out why our production AI pipeline kept falling over. We had a perfectly good cluster — forty-eight A100s, NVLink, InfiniBand. The monitoring looked fine. The per-node utilization was above 70%. Yet every Tuesday at 3:17 PM, latency spiked 12x and some batch jobs just vanished.
Turns out the problem wasn’t the machines. It wasn’t the software either, not really. The problem was the architecture — or rather, the lack of it. We had designed a system. We had not architected a distributed system.
So let’s get precise: what is architecture in a distributed system? It’s the set of structural decisions you make before you have enough information to make them correctly — and then live with for years. It’s the partitioning of state, the contract between components, the failure modes you accept and the ones you don’t. It’s not your microservice boundaries. It’s not your database choice. It’s the principles that make those choices possible.
I’m going to walk through what that means in practice, drawing from hard lessons building production AI systems at SIVARO. We’ll cover disaggregation, GPU clusters, consistency models — and why most people get it wrong.
Architecture Is Not Design
A lot of engineers confuse architecture with design. They think if they draw boxes and arrows and pick a message queue, they’re done. That’s like saying choosing the color of your car is the same as deciding whether it has a combustion engine or an electric motor.
Design is local. Architecture is global.
Design asks: How should this service handle a timeout? Architecture asks: Under what conditions is a timeout acceptable? Design picks a library. Architecture picks a failure domain.
Here’s a concrete example. In 2024 we built a distributed training system for a client in genomics. The team wanted to use RabbitMQ for job orchestration because they knew it well. I said no. Not because RabbitMQ is bad — it’s great for many things. But for this system, the queue had to survive a node failure without losing a single message. That meant we needed a replicated queue, which RabbitMQ can do but requires careful configuration. More importantly, the architecture decision wasn’t “which queue”. It was: “Do we tolerate at-most-once or exactly-once semantics?” We chose exactly-once, and that rippled into every component — the storage layer, the checkpointing, the retry logic.
That’s architecture. It’s the decisions that constrain future decisions.
What Does It Mean to Be Disaggregated?
Most people think “disaggregated” means “separate the compute from the storage.” They’re right, but incomplete. Disaggregation is a deeper idea: each resource type should scale independently and fail independently.
In a traditional HPC cluster, a node is a unit. You have a CPU, some RAM, maybe a GPU, local storage. If any part fails, you lose the whole node. If you need more memory, you buy another node. If you need more GPU, you buy another node. That’s the opposite of disaggregated.
In a properly architected distributed system, compute pools are separate from memory pools, which are separate from storage pools. You can have a rack of only GPUs, another rack of only memory, another of only NVMe. Networking stitches them together. This is exactly what a GPU Cluster Explained: Architecture, Nodes and Use Cases article describes: modern GPU clusters are moving toward disaggregated architectures where GPUs, CPUs, and memory are connected over high-speed fabrics like NVLink or InfiniBand.
Why does this matter for architecture? Because disaggregation changes your failure model. In a monolithic node, if the GPU process OOMs, you might kill the entire training job. In a disaggregated system, the GPU can ask the memory pool for more allocation (if your system supports dynamic memory access). The job doesn’t crash. The fault is isolated.
We tested this at SIVARO last year. We ran a large language model fine-tuning job on a traditional 8-GPU node and on a disaggregated cluster from a partner. The traditional cluster failed three times during a 48-hour run — once from a power supply, twice from GPU memory corruption. The disaggregated cluster didn’t fail once. Not because it was more reliable hardware — the underlying GPUs were the same — but because the architecture allowed graceful degradation. The memory pool reallocated. The compute pool rescheduled.
So when someone asks “what does it mean to be disaggregated?”, I tell them: it means you treat resources as commodities, not as packages. And that requires an architecture built for it from day one.
What Is a GPU Cluster? And Why Architecture Matters More Than Hardware
Let’s define it simply: a GPU cluster is a collection of machines equipped with GPUs, interconnected with high-bandwidth networking, designed to run parallel workloads. But that definition misses the point.
A GPU cluster is an architectural decision. It’s saying: “We will pay for specialized hardware to accelerate certain computations, and we accept the complexity of managing that hardware across many nodes.”
The 5 Key Considerations when Building an AI & GPU Cluster article lists: cooling, power, networking, storage, and software stack. I agree with all five. But I’d add a sixth: architecture of the workload itself. A GPU cluster designed for training looks very different from one designed for inference. Training needs fat bandwidth between GPUs (NVLink), high memory bandwidth, and long-running jobs. Inference needs low latency, high throughput, and often fractional GPU usage.
I’ve seen companies buy a $2M GPU cluster for inference and wonder why their p99 latency is terrible. Because they didn't architect the cluster for the workload. They bought what the salesperson recommended.
If you’re building a GPU cluster today, think about this: what is the unit of work? A single GPU? A whole node? A slice? NVIDIA forums have a thread where a small company asks about on-prem vs cloud. The right answer isn’t “cloud” or “on-prem” – it’s “architect first, then choose.”
The Architecture of Distributed State
Here’s where things get messy. Distributed systems are about sharing nothing — except when they need to share everything.
The hardest question in distributed system architecture is: how do different parts of the system agree on the current state? There are three broad approaches, and each has brutal trade-offs.
Approach 1: Strong Consistency (CP in CAP)
You guarantee that every read returns the latest write. This is how ZooKeeper, etcd, and Spanner work. It’s great for coordination — leader election, configuration, locking. It’s terrible for high write throughput.
We use ZooKeeper for our job scheduler at SIVARO. The scheduler needs to know exactly which GPUs are free and which are busy. If two schedulers disagree, you could double-book a node. Strong consistency prevents that. But our ZooKeeper ensemble handles maybe 5,000 writes per second. That’s fine for scheduling — we don’t schedule 5,000 jobs per second. But it would be useless for tracking per-iteration loss values during training.
Approach 2: Eventual Consistency (AP in CAP)
You accept that reads may return stale data, but eventually the system converges. This is what DynamoDB, Cassandra, and most message queues do. Great for availability and partition tolerance. Bad for safety.
We tested this for our model registry. A researcher would upload a new model checkpoint, and the inference service would immediately try to load it — but the registry hadn’t propagated yet. The service crashed. The researcher got angry. We switched to strong consistency for that path.
Approach 3: Hybrid / CRDTs
Conflict-free Replicated Data Types allow concurrent writes and merge them automatically. They’re elegant, but the merge logic gets complicated quickly. We use CRDTs for our experiment metadata — things like tags and descriptions that can merge without conflict. But we don’t use them for anything involving money or model versions.
Most teams I see pick one approach and apply it everywhere. That’s a mistake. Architecture means knowing when to be consistent and when to be available. It’s not a religion.
python
# Example: Hybrid approach for job scheduling state
class JobSchedulerState:
def __init__(self):
# Strong consistency for critical state
self._critical_state = ZooKeeperClient()
# Eventual consistency for metrics
self._metrics_store = CassandraClient()
def assign_node(self, job_id: str, gpu_ids: list[str]) -> bool:
# Use ZooKeeper's compare-and-set to avoid double-booking
with self._critical_state.lock("/gpu_assignment"):
available = self._critical_state.get("/available_gpus")
if not all(g in available for g in gpu_ids):
return False
self._critical_state.set("/assigned/" + job_id, gpu_ids)
return True
def log_gpu_utilization(self, job_id: str, util: float):
# Eventual consistency — okay if metrics are a few seconds stale
self._metrics_store.insert("gpu_util", {"job": job_id, "util": util})
Architecture Patterns for Production AI Systems
At SIVARO, we’ve settled on four patterns that cover 90% of our distributed system needs.
1. The Sharded Pipeline
You partition your data across N workers, each worker processes its shard, then partial results are merged. This is the foundation of MapReduce, Spark, and most batch ML training.
We use this for feature engineering. Each shard gets a contiguous range of user IDs. The processing is embarrassingly parallel. The only catch is the merge step — you need a way to aggregate results. If the merge is a simple sum, fine. If it requires sorting, you need to think about memory.
2. The Parameter Server
For distributed training, the parameter server pattern is still dominant. A set of servers holds the model weights. Workers fetch weights, compute gradients, send them back. The server applies updates.
The architecture gotcha here is network pressure. In 2022, during a training run of a 70B parameter model, our parameter server was receiving 10 GB/s of gradient updates. That saturated the 100 Gbps link. We had to add gradient compression — sending sparsified updates, not full vectors.
python
# Simplified parameter server worker logic
def worker_step(server_url: str, local_model: dict) -> dict:
# Fetch current weights
weights = requests.get(f"{server_url}/weights").json()
# Compute local gradients
grads = compute_gradients(local_model, batch_data)
# Send gradients back (with compression)
compressed_grads = [sparsify(g, top_k=0.1) for g in grads]
response = requests.post(f"{server_url}/update", json=compressed_grads)
return response.json()
3. The Orchestrator + Worker Pool
This is what we use for inference serving. A central orchestrator routes requests to worker nodes. Workers are stateless — they load the model from a shared filesystem and run inference.
The architecture challenge is request routing. If one worker has a GPU and another doesn’t, you can’t round-robin. We use a consistent hash ring based on model version and input tensor shape. That way, the same request lands on the same worker (if available), improving cache locality.
4. The Log-Processing Chain
Event-driven architectures where data flows through a sequence of services. Apache Kafka is typical. The architecture decision is: how do you handle backpressure? If the consumer is slower than the producer, do you drop messages? Do you block? Do you scale up consumers?
We use Kafka with consumer group rebalancing. But we had to add a monitoring layer that pauses the producer if any consumer lag exceeds 10 seconds. Otherwise, we’d fill disk and crash the broker. The Vast.ai: Rent GPUs model — renting spot GPUs — is interesting here because you can burst compute when backpressure builds. We’ve tested it. Works for batch, not for real-time.
The Hardest Part: Failure Modeling
Most distributed system architectures fail — pun intended — because they don’t model failure explicitly. You can’t just hope nodes don’t go down.
You need to answer: What happens when a node dies mid-request? What happens when the network partitions? What happens when the clock skews by 500 ms?
We use a failure injection framework in staging. We randomly kill processes, drop packets, and introduce latency. Every new service goes through a “chaos day”. If it survives, it ships. If it doesn’t, we fix the architecture.
One example: Our GPU cluster scheduler used a heartbeat mechanism — workers sent pings every second. If no heartbeat for 5 seconds, the worker was declared dead and its jobs rescheduled. During a chaos test, we introduced 6 seconds of network latency. Workers were killed erroneously. Jobs were restarted on other nodes, only for the original worker to come back and try to continue. We had two workers processing the same job — duplicate results. The fix was to add a grace period and a lease-based ownership model.
That’s architecture. It’s not about the code. It’s about the assumptions.
FAQ: What Is Architecture in a Distributed System?
Q: What’s the difference between architecture and design?
A: Architecture is the high-level structure: components, their responsibilities, communication patterns, failure models, consistency guarantees. Design is the implementation detail: which HTTP library, how to serialize, how to format logs. Architecture constrains design. You can change design without changing architecture, but changing architecture usually requires rewriting everything.
Q: How do I know if my distributed system has good architecture?
A: Ask yourself: “Can I replace one component with a completely different implementation without changing the rest of the system?” If yes, your architecture is modular. Also ask: “If a node fails, does the system degrade gracefully or crash?” If it degrades, you have good architecture. If it crashes, you don’t.
Q: Should I use microservices for everything?
A: No. Microservices are a deployment pattern, not an architecture. A distributed system can be monolithic in architecture (single process, shared memory) and still run on multiple nodes. Conversely, you can have 50 microservices with terrible architecture — tight coupling, shared databases, synchronous calls. Architecture is about coupling and cohesion, not number of services.
Q: What is the biggest mistake in distributed system architecture?
A: Assuming the network is reliable. Or that clocks are synchronized. Or that nodes never fail. Design for failure. Assume any message can be dropped, any service can be down, any disk can fill up. If your architecture works under those assumptions, it will work in production.
Q: How does GPU cluster architecture differ from CPU cluster architecture?
A: GPU clusters have different network topology requirements (NVLink within nodes, InfiniBand between), different memory models (unified vs. discrete), and different workload characteristics (bulk synchronous vs. asynchronous). You can’t treat GPUs like fast CPUs. They require careful data placement and communication scheduling.
Q: Is cloud or on-prem better for distributed systems?
A: Neither. It depends on your workload, your budget, your data gravity, and your team’s expertise. Cloud gives you elasticity and managed services. On-prem gives you predictable performance and no egress costs. The Vast.ai model — renting spot GPUs from various providers — is a middle ground. But the architecture should be the same regardless of where you run. If your system is tightly coupled to a specific cloud service, you’ve made an architectural mistake.
Q: How do I start designing the architecture of a distributed system?
A: Start with the data. What data flows where? What are the consistency requirements? Then draw the failure scenarios. Then pick one pattern that matches your needs. Don’t over-architect. Build something that works, then refactor. Architecture is not a one-time decision — it evolves.
Conclusion: Architecture Is Debt You Take On Purpose
There’s a myth that good architecture emerges from good code. It doesn’t. Good architecture emerges from hard trade-offs made early, with incomplete information, and revisited as you learn.
If you’re building a distributed system today — especially one involving AI workloads and GPU clusters — sit down and write down your architectural decisions. Not the code. Not the configs. The why. Why did you choose eventual consistency over strong? Why did you disaggregate compute and storage? Why did you pick InfiniBand over Ethernet?
When the system breaks at 3 AM (and it will), those decisions will either save you or condemn you.
For me, I’ve learned that what is architecture in a distributed system? is a question you answer over and over again, every time you deploy, every time you debug, every time you design a new feature. It’s never finished. And that’s okay.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.