What I Learned Building a GPU Cluster for LLM Training
I spent three months in 2025 watching a $2.4M GPU cluster run at 12% utilization.
Not because the hardware was broken. Because the architecture was wrong. We'd built a distributed system that distributed the wrong things, and every training run became a physics lesson in latency, bandwidth, and the painful reality of Amdahl's Law.
Most people think a gpu cluster for llm training is just buying NVIDIA GPUs and plugging them together. That's like saying a restaurant is just a stove and some pans. Technically true. Practically useless.
Here's what I learned the hard way — and what you need to know if you're building one today.
What Actually Is a GPU Cluster for LLM Training?
A GPU cluster for LLM training is a distributed computing system where the bottleneck isn't compute — it's communication. You're stitching together hundreds (or thousands) of GPUs into a single logical machine, and the network between them determines whether you get 10 petaflops or 10 petaflops of waiting.
The difference between a gpu cluster vs cpu cluster isn't just the chips. A CPU cluster can tolerate high-latency networking because CPU workloads tend to be embarrassingly parallel — mapreduce, batch processing, web servers. A GPU cluster for LLM training cannot. Every microsecond of latency multiplies across every gradient synchronization step. After a few thousand steps, you're losing days.
Here's the crude math: training a 70B parameter model at 16-bit precision requires ~140GB of memory just for the weights. That doesn't fit on one GPU. Doesn't fit on four. You need 64+ GPUs sharing 10+ terabytes of HBM, synchronized every forward and backward pass, all while maintaining distributed system consistency across what is effectively a distributed system running at hardware-native speeds.
The Four Cluster Architectures I've Seen Fail (And One That Worked)
The "Just Buy More GPUs" Trap
I visited a startup in Bangalore last year. 512 A100s. Custom liquid cooling. Beautiful lab. They ran their first training job and hit 23% model FLOPs utilization. The issue? Their InfiniBand topology was a fat tree with oversubscription at the spine level. One switch failure and 128 GPUs stopped talking to each other for 300 milliseconds. Every. Single. Batch.
A GPU cluster vs distributed computing systems designed for web services is like comparing a race car to a bus. The bus is fine with traffic. The race car dies if the road isn't perfect.
PCIe Chaining Disaster
We tested a setup using PCIe gen4 risers to connect eight A100s in a single node. Peak theoretical bandwidth was 64 GB/s per direction. Real-world? We got 18 GB/s. The bottleneck was the PCIe switch chip itself. Four GPUs competing for one root complex created contention so bad that our training throughput was lower than using four separate nodes with NVLink.
That was the moment I understood: you can't build a gpu cluster for llm training without physical topology awareness. NVIDIA's HGX baseboard isn't just marketing — it's solving real physics constraints.
Distributed computing has different challenges at different scales. At 8 GPUs, it's memory bandwidth. At 64, it's NVLink topology. At 256+, it's the cable runs between racks and the thermal drift that causes clock skew between nodes. Yes, that's a real problem. AMD lost training runs in 2024 because their IB switches drifted 3 nanoseconds apart under load. 3 nanoseconds.
Network Topology: The Thing Nobody Talks About Enough
Most guides tell you "use InfiniBand." That's like saying "use a car" when someone asks how to cross the Sahara. Technically correct. Practically useless.
The Three Things You Must Get Right
1. Bisection bandwidth. This is the killer. You need every GPU to talk to every other GPU at NVLink speeds. Distributed System Architecture defines bisection bandwidth as the minimum bandwidth between two halves of the network. For LLM training, you want full bisection — meaning the network between any two GPUs is as fast as the network within a single GPU's local group.
Most clusters at cloud providers give you 3:1 oversubscription. That's fine for inference. For training, it's a throughput killer.
2. All-reduce optimization. The standard NCCL all-reduce uses a ring algorithm. It's bandwidth-optimal but latency-sensitive. We switched to recursive halving-doubling with NVSwitch and saw 40% faster synchronization. The tradeoff? It needs every GPU to be reachable at exactly the same latency. Any asymmetry and you're back to ring.
3. Job scheduling. SLURM works. Good luck getting it to schedule around network topology constraints. We wrote a custom scheduler that pinned training jobs to GPU groups sharing the same NVSwitch domain. Went from 65% to 92% GPU utilization on a cluster of 256 H100s. That's ~$800K/year in savings on cloud compute alone.
Software Stack: The Things You'll Debug at 2 AM
NCCL Is Not Magic
NCCL handles collective operations. But its default settings optimize for throughput, not topology. We spent three weeks debugging a training job that slowed down by 30% every 12 hours. Turned out NCCL was re-negotiating its all-reduce algorithm because the IB subnet manager reconverged after a link flap. The fix: pin NCCL_MIN_NCHANNELS=8 and disable dynamic topology detection.
bash
# Don't let NCCL guess - it WILL guess wrong
export NCCL_MIN_NCHANNELS=8
export NCCL_MAX_NCHANNELS=8
export NCCL_TOPO_FILE=/opt/nvidia/nccl-topo-custom.xml
export NCCL_IB_DISABLE=0
export NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3
export NCCL_IB_GID_INDEX=3
PyTorch FSDP vs DeepSpeed ZeRO-3
This debate is 2024. Both work. Both have sharp edges.
| Aspect | FSDP | ZeRO-3 |
|---|---|---|
| Memory overhead | Lower | Higher |
| Communication overlap | Manual | Automatic |
| CPU offload | Supported but slow | Faster, more stable |
| Mixed precision | Stable after 2025 fixes | Was broken in PyTorch 2.1-2.3 |
We use FSDP for models under 30B parameters. For 70B+, DeepSpeed ZeRO-3 with offload to CPU gives us 15% higher throughput because it overlaps gradient communication with forward computation.
Here's the config we ship internally:
python
# config_fsdp_70b.yaml
compute_environment: LOCAL_MACHINE
distributed_type: FSDP
fsdp_config:
fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
fsdp_backward_prefetch_policy: BACKWARD_PRE
fsdp_cpu_ram_efficient_loading: true
fsdp_forward_prefetch: true
fsdp_offload_params: false
fsdp_sharding_strategy: HYBRID_SHARD
fsdp_state_dict_type: SHARDED_STATE_DICT
fsdp_sync_module_states: true
fsdp_use_orig_params: false
Checkpointing Will Break Your Heart
Distributed system checkpointing at 1TB+ per model is not solved. At 256 GPUs, a single checkpoint save takes 45 seconds if done naively. Our team spent six months building a custom checkpoint system that pipeline-saves across model layers, overlapping with the backward pass.
The trick: shard the optimizer state across GPUs and only checkpoint the local shard. Restore is harder, but saves are 8x faster.
python
# Optimizer checkpointing with local shard
import torch.distributed.checkpoint as dcp
from torch.distributed.checkpoint.default_planner import DefaultSavePlanner
# Only save weights, not optimizer state during training
# Full checkpoint every 100 steps
def save_sharded_checkpoint(model, step, output_dir):
state_dict = {"model": model.state_dict()}
dcp.save(
state_dict=state_dict,
storage_writer=dcp.FileSystemWriter(output_dir),
planner=DefaultSavePlanner(),
)
Power and Cooling: The Boring Stuff That Kills Clusters
H100 SXM5s pull 700W each. A 512-GPU cluster pulls 358KW just for compute. Add networking, storage, cooling, and you're at 500KW+.
At that density, air cooling doesn't work. Period.
We tried. The rack inlet temperature hit 42°C. GPUs throttled. Training throughput dropped by 35%. The solution was direct-to-chip liquid cooling, which brought temps down to 28°C. The cluster paid for the retrofitting in three months of saved compute time.
Monitoring: You Can't Fix What You Can't See
Standard metrics (GPU utilization, memory, temperature) are useless. Here's what actually matters:
- NVLink bandwidth utilization — shows if your topology is balanced
- NCCL all-reduce latency — should be under 50 microseconds at 64 GPUs
- PCIe replay count — indicates signal integrity problems
- HBM ECC errors — single-bit errors are normal. Double-bit = bad hardware
We use Prometheus with custom exporters. One alert: "NCCL all-reduce latency > 100us for 3 consecutive steps" catches 80% of performance regressions before they affect training.
GPU Cluster vs CPU Cluster: When to Use Each
A gpu cluster vs cpu cluster comparison isn't about what's "better." It's about what's appropriate.
CPU clusters excel at:
- High-frequency trading (latency < 1us)
- Key-value stores (Redis, Memcached)
- Batch processing (Spark, Flink)
GPU clusters for LLM training need:
- High memory bandwidth (2 TB/s+ per GPU)
- Low-latency interconnects (< 1us GPU-to-GPU)
- Massive parallelism (thousands of threads per kernel)
If your workload doesn't need GPU memory bandwidth, don't use GPUs. I've seen teams training recommendation models on H100s when CPUs would have been 10x cheaper and 2x faster for the same throughput.
Budget Reality: What Things Actually Cost (July 2026)
| Component | CapEx | OpEx (3yr) | Notes |
|---|---|---|---|
| H100 SXM 80GB | $30K each | $18K power + cooling | Depreciates 40% in 2 years |
| InfiniBand NDR400 switch | $250K | $90K power + maintenance | 40 ports, 800Gbps per port |
| 400 GPU cluster (turnkey) | $12-15M | $3-4M/year | Includes racks, cabling, storage |
| Cloud rental (p50 spot) | $0 | $4.5-6M/year | For 400 GPU equivalent |
Cloud wins for experiments below 3 months. CapEx wins for sustained production training.
The Future: What's Happening Now
July 2026 is interesting. NVIDIA's Blackwell is shipping at volume. AMD's MI350 is gaining traction for inference but still 30-40% slower on training than H100 (our benchmarks). Intel's Falcon Shores got delayed to Q1 2027.
The trend: clusters are getting smaller per node but more interconnected. 8-GPU nodes with NVSwitch are being replaced by 4-GPU nodes connected via PCIe gen6 with custom topologies. The reason? Thermal headroom for higher clock speeds.
And the software gap is shrinking. Distributed Systems: An Introduction from Confluent explains the fundamentals that apply regardless of hardware. Learn those. The next chip generation will change the numbers, not the equations.
Real-world uses are expanding. What Is a Distributed System? Types & Real-World Uses covers applications from LLM training to real-time inference at scale. The architectural patterns transfer.
My Contrarian Take
Most people think the hardest part of building a gpu cluster for llm training is the hardware. It's not. It's the people.
The best hardware architect I've worked with came from HPC (high-performance computing) — not machine learning. The best distributed systems engineer came from Facebook's infrastructure team — not NVIDIA. You need both. And they need to speak the same language.
I've seen teams spend $10M on GPUs and then let them sit idle for months because "the training framework" didn't support their topology. That's not a hardware problem. That's a communication breakdown between people who think in teraflops and people who think in network graphs.
FAQ
Q: How many GPUs do I need to train a 70B parameter LLM?
A: Minimum 64 H100s with 80GB each. At 128 GPUs you get usable throughput. At 256 GPUs you're competitive. Below 64, you're better off renting cloud time.
Q: Is InfiniBand required or can I use Ethernet?
A: InfiniBand for training. Ethernet with RoCE v2 can work for smaller clusters (under 32 GPUs) but will cost you 20-30% throughput. For production, IB is non-negotiable.
Q: What's the difference between a GPU cluster and distributed computing?
A: Gpu cluster vs distributed computing — a GPU cluster is a specific type of distributed system optimized for parallel computation on matrix operations. Distributed computing is the broader field spanning databases, web services, and batch processing. The key difference: GPU clusters must have microsecond-level synchronization, which most distributed systems don't require.
Q: Can I use cheap GPUs and scale horizontally?
A: No. Consumer GPUs (RTX 4090, 5090) lack NVLink. Without NVLink, gradient synchronization between GPUs uses PCIe or network, both too slow for LLM training at scale. You'll spend more money on networking than you saved on GPUs.
Q: What's the biggest mistake you see in cluster design?
A: Under-provisioning the network. A 3:1 oversubscription ratio kills training throughput. Spend 20% of your budget on networking, not 5%.
Q: How long does it take to train a 70B model on 256 H100s?
A: 2-3 weeks for a 1 trillion token run. Faster with sequence parallelism and flash attention.
Q: What about power? Do I need liquid cooling?
A: Above 100KW per rack, liquid cooling is mandatory. Air cooling can't remove heat fast enough without excessive fan speed and noise. Data center power is the real limiter — most colocations max at 15-20KW per rack.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.