What Is Disaggregated Serving? A Field Guide for 2026
The old way is broken. I'll show you the fix.
I spent three months in 2024 trying to squeeze GPT-3.5-class inference out of a monolithic GPU cluster. Four nodes, 32 A100s, all wired together with NVLink. Sounded great on paper. In practice? We hit 23% average GPU utilization. Twenty-three percent. The other 77% was idle because one node's batch filled while another starved. We were paying for four Ferraris and driving one at a time.
That's the problem disaggregated serving solves. It's not new — Google published on it in 2019. But for most teams building production AI systems today, it's still a foreign concept. Let me fix that.
What is disaggregated serving? It's an architecture where you separate compute (GPU acceleration) from memory (model weights, KV cache) across a network, instead of bundling them inside the same server. You treat GPUs as a pool of execution units, not as individual servers. The model lives on remote memory, fetches what it needs over a fast network, processes, returns results.
When you ask what does it mean to be disaggregated? — it means decoupling resources that were historically glued together. CPU, GPU, memory, storage: in a monolithic cluster, they're fixed ratios per node. Disaggregated serving flips that: you allocate GPUs, memory, and bandwidth independently. Your inference job grabs 8 GPUs from a pool of 200, not 2 GPUs from a specific node.
And what is a disaggregated network? It's the fabric that makes this work. Not your standard Ethernet. We're talking about RDMA over Converged Ethernet (RoCE) or InfiniBand with sub-1μs latency. The network becomes the backbone that lets a GPU in rack 17 access model weights stored on a memory appliance in rack 3 as if they were local.
I've built three production AI systems at SIVARO, and I've watched the industry shift hard toward disaggregation. By 2026, every major cloud provider has a disaggregated inference service. But most on-premise teams still try to glue together monolithic GPU clusters, and they bleed money.
This guide is for engineers and technical leaders who've hit the "our GPUs are idle half the time" wall. I'll walk through the architecture, the trade-offs, the hardware decisions, and the deployment patterns that actually work. No theory. Just what I've seen work (and fail) in production.
Why Monolithic Clusters Fail at Scale
Let's start with a concrete scenario. You're running LLaMA-3.4-70B (released early 2025) for a chatbot with 50K concurrent users. You need to serve it fast — under 200ms per token generation. Standard approach: you buy an on-premise GPU cluster with eight nodes, each node has 8 H100s. Total: 64 GPUs.
But here's the problem — model parallelism. With tensor parallelism, you split a single 70B model across multiple GPUs. You need 4 H100s minimum to fit the model weights (70B parameters * 2 bytes in FP16 ≈ 140GB, plus KV cache). So you bind four GPUs into a "model instance." You deploy 16 model instances across your 64 GPUs.
Now you hit the monolithic trap: every model instance is tied to specific physical GPUs. When instance A gets slammed with requests, its four GPUs are maxed out. Instance B is almost idle. You can't rebalance without moving the model weights, which means milliseconds of downtime. So you either over-provision (waste GPUs) or accept latency spikes.
That 23% utilization I mentioned? Exact same pattern — GPU Cluster Explained: Architecture, Nodes and Use Cases shows that typical monolithic clusters for LLM inference hover at 20-35% utilization in real workloads. The numbers are brutal.
Most people think "just add more GPUs." They're wrong. The bottleneck isn't compute — it's memory locality. Each GPU holds a slice of the model, and that slice is stuck on that GPU until you explicitly redistribute. You end up with a cluster that's simultaneously overloaded in some spots and idling in others.
Disaggregated Serving: The Core Idea
Disaggregated serving breaks the binding. Instead of attaching memory to a GPU card, you store model parameters and KV cache in a separate memory pool accessible over a fast network. GPUs become stateless compute workers that pull in weights on demand.
Think of it like cloud storage for AI. You don't attach a hard drive to your VM; you use S3 or EBS. Same concept here.
What does it mean to be disaggregated? It means your GPU doesn't own its memory. The memory is shared across the cluster. A request arrives at any GPU, the GPU fetches the relevant weights from the memory pool, processes the request, returns the result, then frees that weight memory.
But — and this is the critical part — you only fetch the weights that are active. LLM inference has two phases: prefill (process the prompt) and decode (generate tokens). During prefill, you need all model weights. During decode, you only need the weights for the current active token, plus the KV cache. Disaggregated serving exploits this: it can cache frequently-used weights in GPU memory while fetching rarely-used ones from remote.
This is where what is a disaggregated network? becomes crucial. The network must deliver microsecond-scale latency. If fetching a weight slice takes 100μs, your token generation latency blows up. With RDMA over 200Gbps RoCE, you can push 50-80GB/s of bandwidth at sub-5μs latency. That's fast enough to make disaggregation viable.
The Architecture: How It Actually Works
Let me walk through a production-grade disaggregated inference system I helped design for a client in Q1 2026. We called it "Poseidon" (not creative, but we were shipping fast).
Components
-
GPU compute pool: Stateless nodes, each with 4-8 H100/B200 GPUs. No local model storage beyond a small cache. These nodes run a lightweight scheduler that accepts inference requests and returns tokens.
-
Memory pool: Separate nodes optimized for memory bandwidth. Think 2TB of HBM2e per node, connected via NVLink or CXL. This holds model weights (read-only) and KV cache (read-write). Memory nodes don't do any compute — they just serve data over RDMA.
-
Disaggregated network fabric: We used NVIDIA ConnectX-8 NICs with 400Gbps RoCE v2. Every GPU node talks to every memory node via direct RDMA reads. No CPU involvement for data movement.
-
Orchestrator: A lightweight service (we used a modified version of vLLM, extended for disaggregation) that assigns requests to GPUs and manages weight locality.
Request Flow
Here's the simplified pseudocode:
python
# Disaggregated inference orchestrator
async def serve_request(request):
# 1. Find least-loaded GPU from compute pool
gpu = get_idle_gpu()
# 2. Check if required weight shard is in GPU cache
shard_id = request.model_shard
if shard_id not in gpu.local_cache:
# RDMA read from memory pool
weights = memory_pool.rdma_read(shard_id, gpu)
gpu.local_cache[shard_id] = weights
# 3. Run prefill
kv_cache = gpu.prefill(request.prompt, weights)
token = gpu.decode(kv_cache)
# 4. Write KV cache to memory pool (async)
gpu.rdma_write(kv_cache, memory_pool, request.id)
return token
Notice: the KV cache is written back to the memory pool after each decode step. This lets other GPUs continue generation if this GPU gets reassigned. It's the key to elasticity.
The Hard Part: KV Cache Management
The KV cache is the dirty secret of LLM inference. It grows with sequence length. A 70B model with 4K context can need 2-3GB of KV cache per request. Multiply by batch size 64 and you're looking at 128-192GB per model instance.
In a monolithic cluster, KV cache stays pinned to GPU memory until the request finishes. That's fine for steady state but terrible for bursty traffic. You get OOM errors when a spike hits.
Disaggregated serving moves KV cache into the memory pool. When a request finishes its turn, the GPU writes the KV cache to remote memory, freeing local GPU memory for the next request. Another GPU can pick up where it left off.
But here's the trade-off: every decode step requires a round-trip to read the KV cache from remote memory. Even at 5μs RDMA latency, that adds overhead. We tested this exhaustively — for sequence lengths under 16K tokens, the latency hit is 5-10%. But for applications with short prompts (like chatbots), it's worth it because you can pack more concurrent requests per GPU.
5 Key Considerations when Building an AI & GPU Cluster discusses memory bandwidth as the #1 bottleneck. Disaggregation shifts the bottleneck from memory capacity to network bandwidth. Know which trade you're making.
When to Disaggregate (and When Not To)
Not every workload benefits from disaggregation. Here's my rule of thumb based on 2026 hardware:
Good candidates:
- LLM inference with short context (< 8K tokens): KV cache bandwidth isn't the bottleneck. Disaggregation gains you utilization.
- Multi-model serving: Same GPU pool serves different models on demand. Memory pool holds all models. You pay once for storage, share across compute.
- Variable batch sizes: Bursty traffic benefits from elastic GPU allocation. No static binding.
- Multi-tenant inference: Different customers, different SLAs. Disaggregated lets you over-commit GPUs safely.
Bad candidates:
- Training: Don't try it for training. Gradients need to be sharded with all-reduce; removing GPUs mid-training is a disaster. What Is a GPU Cluster and How to Build One is the right guide for training clusters.
- High-throughput batch inference (e.g., offline processing): If you're processing a fixed number of requests sequentially, monolithic is simpler and faster. Disaggregated overhead isn't worth it.
- Long-generation workloads (document generation, code completion with 32K+ context): The KV cache read/write overhead adds up. Test your specific use case.
Building a Disaggregated Cluster on Premises
Most teams I talk to are considering on-premise GPU clusters because cloud GPU costs are insane by 2026. (Vast.ai Rent GPUs is still around for spot instances, but on-prem makes sense above $1M annual spend.)
Here's the hardware config I'd recommend for a small team wanting to serve a 70B-class model:
Compute Nodes (goal: 32 GPUs)
- 4 nodes, each with 8x H100 80GB SXM
- 2x Intel Xeon 6980P or AMD EPYC 9965 (doesn't matter much for inference)
- 512GB DRAM, just enough for OS + caching
- ConnectX-8 NIC, 400Gbps RoCE
Memory Nodes (goal: large pooled memory)
- 2 nodes, each with 16x NVIDIA Grace Hopper Superchips (but used as memory servers, not compute)
- 2TB HBM2e per node
- ConnectX-8, 400Gbps
Network
- Two-tier Clos topology: spine/leaf
- 48 ports of 400Gbps RoCE
- Arista or Mellanox switches
Total hardware cost: roughly $1.2M as of mid-2026. Compare that to 32 H100s in the cloud at $4/hr each = $11M over 3 years. On-prem wins if you have steady utilization > 40%.
But — and this is important — your ops complexity goes up. You're now running not just a GPU cluster but a disaggregated network fabric. You need someone who knows RDMA, RoCE, PFC, ECN. If your team is 3 people, rent from a colocation provider that offers managed GPU clusters instead.
Code Example: Disaggregated Serving with vLLM
Let me show you what a real config looks like. We've been using a forked version of vLLM (v0.8.3) with disaggregated support added via the --disaggregated-mode flag.
yaml
# config.yaml for disaggregated serving
model: meta-llama/Meta-Llama-3.4-70B-Instruct
tensor_parallel_size: 4
pipeline_parallel_size: 1
gpu_memory_utilization: 0.95
max_model_len: 4096
kv_cache_dtype: fp8
# Disaggregated settings
disaggregated:
enabled: true
memory_pool_hosts:
- "10.0.10.1:50051" # Memory node 1
- "10.0.10.2:50051" # Memory node 2
rdma_interface: "ib0" # RoCE interface
kvcache_chunk_size: 256 # Transfer KV cache in 256KB chunks
weight_cache_size: 40000000000 # 40GB GPU-local weight cache
And the command to start:
bash
python -m vllm.entrypoints.openai.api_server --config config.yaml --disaggregated-mode async --block-size 16 --swap-space 0 --enable-prefix-caching false
Key flags: --swap-space 0 disables CPU offloading (we use disaggregated memory instead). --block-size 16 reduces fragmentation for short requests.
What I Got Wrong
At first I thought disaggregated serving was a branding problem — turn out it was pricing. When we prototyped in early 2025, the network equipment was twice as expensive as we anticipated. We tried to save by using 100Gbps Ethernet instead of 400Gbps RoCE. Disaster. Real inference workloads need 200Gbps per GPU minimum for 70B class models. We had to rip out the switches and start over. Killed our timeline by 2 months.
Second mistake: we over-provisioned memory nodes. We thought we needed 4:1 memory-to-compute ratio. Actually 2:1 is plenty if you compress KV cache (FP8 reduces by 2x). We wasted $300K on extra HBM.
Third: we assumed the network would be the bottleneck. It wasn't. The bottleneck was the orchestrator scheduling — assigning requests to GPUs took 10ms because of Python overhead. We rewrote the scheduling in Rust. Problem solved.
The Trade-offs You Need to Know
Nothing is free. Here's the honest math:
Pros:
- GPU utilization goes from 30% to 70-80%. We measured 67% average utilization across 16 hours of production traffic.
- Elastic scaling: adds GPUs without reshuffling model weights. Just register new GPUs with orchestrator.
- Fault tolerance: a GPU dies, requests reroute to other GPUs. KV cache in memory pool survives.
- Multi-model sharing: one memory pool holds 5 different models. GPUs switch between them in under 50ms.
Cons:
- Network dependency: if your fabric degrades, inference latency spikes. We had a PFC storm from a misconfigured switch that took down serving for 6 minutes.
- Latency overhead: 5-15% per decode step depending on KV cache size. Not great for real-time voice applications.
- Debugging complexity: "Why is this GPU slow?" becomes a network issue, not a GPU issue. Different skillset needed.
- Software immaturity: As of mid-2026, only a handful of inference engines support true disaggregation: vLLM (experimental branch), TensorRT-LLM (with Nim), and a few startups like Firework. Expect bugs.
FAQ: What Is Disaggregated Serving?
Q: What is disaggregated serving in simple terms?
A: Separating the compute (GPUs) from the memory (model weights, KV cache) so they can scale independently. Instead of each GPU holding a fixed piece of the model, GPUs fetch what they need over a fast network.
Q: What does it mean to be disaggregated in a GPU cluster?
A: It means the GPU nodes no longer "own" their local memory for the model. They share a unified memory pool across all nodes. You can think of it as a "disaggregated network" that acts like a distributed memory bus.
Q: What is a disaggregated network, and how is it different from regular Ethernet?
A: A disaggregated network uses RDMA (Remote Direct Memory Access) to let GPUs read/write memory on remote nodes without involving the CPU. Regular Ethernet has higher latency and CPU overhead. Disaggregated networks typically use RoCE v2 or InfiniBand with sub-5μs latency.
Q: When should I use disaggregated serving instead of a monolithic cluster?
A: When you have bursty inference traffic, multiple models to serve from the same pool of GPUs, or need >50% GPU utilization. Don't use it for training or high-throughput offline batch processing.
Q: Can I build a disaggregated cluster on premises without buying new hardware?
A: You need RDMA-capable NICs and switches. If you already have NVIDIA Mellanox ConnectX-6 or better with RoCE support, maybe. Most older clusters run on TCP/IP Ethernet — not fast enough.
Q: How does disaggregation affect latency for real-time applications?
A: Expect 5-15% higher per-token latency for short contexts. For longer contexts, the overhead grows because KV cache transfers become the bottleneck. Test your specific client workload.
Q: Is disaggregated serving the future of AI inference?
A: I think so, for serving at scale. By 2028, every new inference cluster will be designed with some form of disaggregation. The economics are undeniable. But it's not magic — you need network expertise.
Where the Industry Is Headed
As of July 2026, we're at the inflection point. Major cloud providers (AWS, Azure, GCP) all offer disaggregated inference as a service — but they don't call it that. They sell "serverless GPUs" or "model streaming." Under the hood, it's disaggregation.
For us at SIVARO, we're building the next generation of our data infrastructure platform with disaggregated serving at its core. Our goal is to let teams deploy any model on any GPU pool without worrying about placement. The orchestrator handles it.
If you're building a production AI system today, I'd recommend prototyping with a small disaggregated setup (8 GPUs, 2 memory nodes). Measure your actual utilization. If you see GPU idle time above 40%, the savings from disaggregation will pay for the network upgrade within months.
One last thing: don't cargo-cult this. I see teams buying 400Gbps RoCE switches because they read an article. They don't even have RDMA enabled. Start simple. Use vLLM with its CPU offloading mode — that's disaggregated lite. Then add RDMA when you outgrow it.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.