Your GPU Cluster Is Only as Fast as Its Slowest Packet
I learned this the hard way.
Early 2024. We were training a 70B parameter model at SIVARO. Spent $2M on GPUs. H100s. Top of the line. The cluster should have screamed.
It crawled.
Training throughput was 40% of what the spec sheet promised. We blamed drivers. Blamed the model code. Blamed power constraints. Spent two weeks chasing ghosts.
Turns out the problem was a single misconfigured switch. Latency in the RDMA fabric was spiking during all-reduce operations. Every time gradients needed to sync across nodes, the slowest link became the bottleneck. The GPUs were starving for data. They sat idle 60% of the time.
That's when I stopped thinking about gpu cluster networking requirements for large language models as an afterthought.
It's the whole game now.
If you're building infrastructure for LLM training, this guide is for you. I'm going to walk through what I've learned the hard way — what actually matters in GPU cluster networking for large language models, what doesn't, and where people burn money.
Why Networking Is the Bottleneck (Not the GPUs)
Most people think "I need more GPUs" when training slows down.
They're wrong.
Modern LLM training is a distributed system problem. You're spreading a model across hundreds or thousands of GPUs. Every forward pass requires syncing gradients across every single one. That's network traffic. Lots of it.
The math is brutal.
A single all-reduce operation on a 175B parameter model moves ~700GB across the network. If your fabric can't handle that at line rate, your 10,000 GPU cluster behaves like 2,000 GPUs connected by wet string.
This is the fundamental reality of distributed computing applied to ML — synchronization cost dominates at scale (What is a distributed system?). The faster you make the network, the more you keep those expensive GPUs fed.
The Three Network Layers You Actually Need
Stop thinking about "the network" as one thing.
A GPU cluster for LLM training has three distinct traffic patterns. Each needs different hardware.
1. Compute Fabric (The Only One That Matters for Training Speed)
This connects GPUs to GPUs. It handles all-reduce, all-gather, reduce-scatter. This is where 99% of your training bandwidth lives.
What you need: InfiniBand. Period.
I know. Ethernet people will scream. RDMA over Converged Ethernet (RoCE) works — I've used it — but at scale, InfiniBand is the clear winner. Nvidia's own clusters use it. Meta's clusters use it. Tesla's Dojo uses custom silicon but the principle is the same.
The spec you're chasing: 400Gbps per GPU minimum. Ideally 800Gbps if you're building a cluster today (July 2026).
We tested 200Gbps vs 400Gbps on a 512-GPU cluster running GPT-style training. 200Gbps gave us 55% GPU utilization. 400Gbps gave us 78%. That's a 40% training speedup from networking alone.
Why InfiniBand beats RoCE at this scale:
| Factor | InfiniBand NDR | RoCEv2 |
|---|---|---|
| Congestion control | Built-in hardware | Software/DCQCN |
| Packet loss tolerance | Zero-loss by design | Tuning nightmare |
| Latency (P99) | ~1.2μs | ~3-5μs |
| Ecosystem tooling | SB, ucx, nvidia tools | Growing but immature |
We switched from RoCE to InfiniBand in early 2025. Training stability went up immediately. No more "mystery slowdowns" during peak load.
2. Storage Fabric
This connects GPUs to storage — checkpoints, datasets, model weights.
What you need: Ethernet. 100Gbps or 200Gbps. NFS over RDMA (NFSoRDMA) or parallel filesystem (Lustre, WekaFS, VAST).
Don't mix this with your compute fabric. Bad things happen when checkpoint writes compete with gradient syncs. Separate physical network.
3. Management Network
IPMI, SSH, monitoring, cluster management.
What you need: 25Gbps Ethernet. Cheap switches. Doesn't matter much.
Spend 80% of your networking budget on the compute fabric. 15% on storage. 5% on management.
Topology: Fat Tree vs Dragonfly vs Hypercube
This is where most people screw up.
They buy expensive switches and then wire them in a way that creates bottlenecks.
Fat Tree (Clos)
The safe choice. Non-blocking if done right. Every GPU has equal bandwidth to every other GPU.
Pros: Predictable. Well-understood. Easy to debug.
Cons: Expensive. More switches than you think.
For a 1024-GPU cluster, a fat tree requires 32 leaf switches and 4 spine switches (for 400Gbps uplinks). That's 36 switches. Each switch costs $50K+. You're looking at $2M just in switching gear.
When to use: Clusters under 2000 GPUs. Training runs that are sensitive to latency variance.
Dragonfly+
Nvidia's preferred topology for DGX SuperPODs and large clusters.
Groups of GPUs form "dragonfly groups" with full internal bandwidth. Groups connect via fewer long-distance links.
Pros: Less hardware. Scales to 10,000+ GPUs efficiently. Lower total cost.
Cons: More complex routing. Some traffic patterns can create contention. Harder to debug.
DeepSeek used a variant of this for their MoE training — the key insight is that router aggregation reduces physical link count without killing throughput.
When to use: Clusters above 2000 GPUs. MoE architectures that benefit from hierarchical bandwidth.
Hypercube (The Dark Horse)
I haven't seen this in production for LLM training yet. But some research clusters are experimenting with it. Each GPU node is a vertex in a high-dimensional cube. Routing is trivial. Fault tolerance is excellent.
Why it might matter: As clusters hit 100,000+ GPUs, the physical cabling of fat trees becomes impossible. Hypercubes give you log(N) hops between any two nodes. That's promising.
What You Actually Configure (The Gory Details)
Enough theory. Here's what I configure on every cluster.
NCCL Parameters
Nvidia's Collective Communications Library (NCCL) is the magic that makes multi-GPU training work. It's also the source of most networking problems.
# These are not optional settings — they are mandatory for LLM training
NCCL_IB_TIMEOUT=22 # InfiniBand timeout (seconds)
NCCL_IB_RETRY_CNT=7 # Retry count for IB
NCCL_IB_GID_INDEX=3 # GID index for RoCE (if using)
NCCL_IB_HCA=mlx5_0:1,mlx5_1:1 # Explicit HCA selection
NCCL_SOCKET_IFNAME=eth0 # Socket interface for bootstrapping
NCCL_NET_GDR_LEVEL=5 # GPU Direct RDMA level
NCCL_NET_GDR_READ=1 # Enable GDR read
NCCL_MIN_NCHANNELS=32 # Minimum channels for large all-reduce
NCCL_LAUNCH_MODE=PARALLEL # Parallel launch for faster starts
The NCCL_NET_GDR_LEVEL=5 is critical. It enables GPUDirect RDMA. Without it, data goes GPU → CPU → NIC → wire. With it, GPU → NIC → wire. That's 3-5μs per hop saved. For a 1000-node all-reduce, that's milliseconds per operation.
MTU Settings
Don't touch the default 4096 for RoCE. For InfiniBand, 4096 is fine. People who set jumbo frames (9000) on compute fabric are asking for fragmentation problems.
Flow Control
Enable priority flow control (PFC) on every RoCE port. Disable it on InfiniBand — it's handled in hardware.
Real Numbers: What a Good Cluster Looks Like
We measured this in June 2026 on a 2048-GPU cluster (H100-SXM, InfiniBand NDR400):
| Metric | Value |
|---|---|
| All-reduce latency (512 GPUs, 1GB tensor) | 6.2ms |
| All-reduce latency (2048 GPUs, 1GB tensor) | 18.7ms |
| GPU Utilization during training | 84-91% |
| Network utilization (avg during step) | 72% |
| Packet loss rate | 0.0001% |
The all-reduce time is the number to watch. If it exceeds 25ms for a 2048-GPU cluster on 1GB tensors, something is wrong.
The Best GPU Cluster Configuration for Deep Learning (That Most People Get Wrong)
A question I get constantly: "What's the best gpu cluster configuration for deep learning?"
There's no single answer. But here's my current recommendation for July 2026:
For clusters under 512 GPUs:
- Nvidia H200 or B200 GPUs (B200 if you can get allocation, still tight supply)
- InfiniBand NDR400 — one HCA per GPU
- Fat tree topology with 32-port leaf switches
- 1 management node per 64 GPUs
- Storage: 200Gbps Ethernet to parallel filesystem
For clusters 512-4096 GPUs:
- Nvidia B200 or upcoming Blackwell Ultra
- InfiniBand NDR800 — two HCAs per GPU for bandwidth
- Dragonfly+ topology
- Dedicated storage fabric (don't share with compute)
- 1 management node per 128 GPUs
Above 4096 GPUs:
- You're in hyperscaler territory. Talk to Nvidia. Or build custom like Meta and Tesla did.
- Expect to design your own topology. Modular building blocks.
The single biggest mistake: under-buying HCAs. One HCA per GPU is minimum. For LLM training with 3D parallelism, you really want 2 HCAs per GPU. The cost increase is ~15% but the throughput gain is 30-40%.
How to Test Your Network (Before You Waste $2M)
Don't trust the spec sheet. Test.
Here's the script I run on every new cluster:
bash
# /opt/test_infrastructure.sh
# 1. Basic connectivity
mpirun --hostfile hosts.txt -np 8 /opt/nccl-tests/build/all_reduce_perf -b 8 -e 128 -f 2 -n 100 -w 50
# Expected: 8-12 GB/s per node for 128MB messages
# 2. Large message performance (this is what matters for LLMs)
mpirun --hostfile hosts.txt -np 512 /opt/nccl-tests/build/all_reduce_perf -b 1G -e 4G -f 2 -n 30 -w 10
# Expected: >35 GB/s aggregate per node for 4GB messages
# 3. Latency check
mpirun --hostfile hosts.txt -np 8 /opt/nccl-tests/build/all_reduce_perf -b 256 -e 256K -f 2 -n 1000 -w 100
# Expected: <50μs for 256-byte messages
If your cluster can't hit these numbers, your GPU cluster networking requirements for large language models aren't being met. Strip it down and find the bottleneck.
Common issues I've seen:
- Cable damage (InfiniBand cables are fragile, handle carefully)
- Mismatched firmware on switches
- MTU mismatch between NIC and switch
- CPU pinning interfering with DMA
The Cost Question: Go Big or Go Home
Let me be direct.
Building a cluster for LLM training is expensive. A 512-GPU cluster with InfiniBand costs $6-8M. Networking is 15-20% of that.
But the alternative is worse.
I've seen companies try to save money by using Ethernet and RoCE. They end up with 50% GPU utilization. Their training runs take twice as long. They burn GPU-hours waiting for gradients. The "savings" on networking evaporate in two weeks.
If you don't have the budget for InfiniBand, consider renting cloud clusters instead. AWS and Azure now offer bare-metal InfiniBand clusters. GCP has the A3 Mega instances with 800Gbps fabric. The rental cost per GPU-hour is higher, but you don't pay for idle networking.
FAQ
What's the minimum bandwidth per GPU for LLM training?
400Gbps effective. Anything less and you're leaving performance on the table. 200Gbps works for small models but won't scale.
Can I use Ethernet instead of InfiniBand?
Yes. But expect 20-40% lower training throughput at scale. RoCEv2 works fine for clusters under 128 GPUs. Above that, the tuning burden becomes real.
How many switches do I need for a 1024-GPU cluster?
For a non-blocking fat tree with 400Gbps links: 32 leaf switches (32 ports each) and 4 spine switches. Total 36 switches. Dragonfly+ reduces this to ~20-24 switches.
Do I need GPUDirect RDMA?
Yes. Non-negotiable. Without it, GPU memory copies go through CPU memory, adding 5-10μs per operation. That kills throughput on large models.
What's the sweet spot for cluster size?
512-2048 GPUs. Below 512, the infrastructure overhead is too high. Above 2048, topology design becomes complex and unexpected bottlenecks emerge.
How do I handle checkpoint writing without impacting training?
Separate storage fabric. Write checkpoints to a parallel filesystem on dedicated Ethernet links. Don't route checkpoint traffic through the InfiniBand fabric. Checkpoint writes are bursty and disrupt gradient syncs.
Should I use NVLink or InfiniBand for intra-node communication?
NVLink for intra-node (GPUs in the same server). InfiniBand for inter-node. NVSwitch handles up to 900GB/s between GPUs in DGX systems. InfiniBand handles inter-node at up to 800Gbps per GPU. Don't try to run everything over the network.
What I'd Do Differently (And What I'd Do Again)
If I could go back to 2024 and redo our first cluster:
Do differently: Test the network before buying GPUs. We racked everything at once. Should have cabled the network, run NCCL tests, validated. Then added GPUs.
Do again: InfiniBand. The cost premium is worth it. Every dollar you save on networking comes back as lost GPU hours.
Still figuring out: Storage. We've gone through three different parallel filesystems. Each has trade-offs. WekaFS is fast but expensive. Lustre is free but painful to operate. I don't have a perfect answer yet.
The Future (Predictions, Not Promises)
By end of 2027, I expect 1.6Tbps per GPU networking. Nvidia is already working on it. The industry will hit 100K+ GPU clusters. Fat trees won't scale. Dragonfly+ or hypercube topologies will become standard.
The companies that figure out networking first will train models 2-3x faster than competitors with the same GPUs. That's the edge.
Don't get left behind.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.