How to Make Your Own GPU Cluster

Free Technical Audit

Expert Review

I ran my first real training job on a homemade GPU cluster in 2019. Four RTX 2080 Ti's duct-taped to a mining frame, connected with a cheap switch, and powered by a PSU that definitely violated some electrical code. It worked. Barely. But that janky setup taught me more about distributed computing than any cloud certification ever could.

Today, I run SIVARO. We build data infrastructure and production AI systems. And I still believe most teams should think twice before renting 8xA100s from a cloud provider. Building your own cluster isn't for everyone. But if you know what you're doing — or are willing to learn the hard way — it can save you thousands a month and give you control that cloud vendors will never offer.

Here's everything I wish someone told me before I started.

What Even Is a GPU Cluster?

A GPU cluster is just multiple computers with GPUs, networked together to work on the same problem. That's it. No magic. No special hardware required (mostly).

Each machine is a "node." Each node has one or more GPUs. You connect them with high-speed networking, add some software to orchestrate work, and suddenly you can train models that wouldn't fit on a single card (What Is a GPU Cluster and How to Build One).

The architecture breaks down into a few layers:

Compute nodes — the machines with GPUs
Networking layer — connecting nodes (InfiniBand, Ethernet, or NVLink)
Storage layer — shared access to datasets and checkpoints
Orchestration layer — Kubernetes, SLURM, or something custom
Monitoring — because things will break

Most people think a GPU cluster is some exotic datacenter thing. It's not. I've seen production clusters built on (https://www.whitefiber.com/blog/how-to-build-and-manage-gpu-clusters) repurposed gaming PCs that outperformed cloud setups for specific workloads. The difference? Knowing your bottlenecks.

Why Build Instead of Rent?

Let's be direct: cloud GPU pricing is absurd right now. An A100 80GB runs about $3-4/hour on most providers. For a training run lasting two weeks? That's over $13,000. Per card.

I priced out an 8x A100 cluster in early 2026. Building it cost ~$180,000. Renting the equivalent for 18 months? $250,000+.

But cost isn't the only reason. Latency matters. Data sovereignty matters. And honestly? Having physical access to your hardware when debugging a distributed training hang saves days.

But here's the catch — you need to know what you're optimizing for.

If your workloads are bursty (train for 3 days, idle for 2 weeks), rent. If you're running continuous training pipelines 24/7, build. Simple threshold (Building Edge GPU Clusters - Edge Computing Guide).

Step 1: Pick Your GPUs — This Is Where People Waste Money

Everyone rushes to buy the newest, shiniest card. I get it. I've done it.

Don't.

Here's the real breakdown of what works:

For Training (FP32/FP16/BF16)

NVIDIA H100 — $30K+ per card. Amazing. Overkill for 90% of teams.
NVIDIA A100 (80GB) — $15-20K used. Sweet spot for training large models.
AMD MI250X — $8-12K. Good perf/watt but awful software stack. Only if you've got dedicated time.
RTX 4090 — $1.6K. No NVLink. No ECC memory. But for small model training or inference? Ridiculous value. We run 16x 4090s for internal R&D.

For Inference

RTX 6000 Ada — $6.8K. ECC memory, better FP64, NVLink support.
A40 — Used for $4K. Still excellent for inference.
H100 NVL — If you're running an LLM serving tier and have the budget.

What I'd Actually Do

If I were starting today with a modest budget: 4x used A100 80GB ($15K each max) on (https://www.scalecomputing.com/resources/what-is-a-gpu-cluster) proper server hardware. Add nodes as needed. Don't buy H100s unless you absolutely need FP8 or the transformer engine.

Step 2: Hardware That Won't Kill Your Performance

Here's the mistake almost everyone makes: they spend $50K on GPUs and $500 on networking. Then they wonder why distributed training is slower than single-card.

Networking is your bottleneck.

For multi-node training, you need at least:

InfiniBand (HDR 200Gbps) — Best. Expensive. Worth it for 4+ node clusters.
NVLink + NVSwitch — If you're staying within one node with multiple GPUs.
25GbE with RoCE (RDMA over Converged Ethernet) — Cheaper. Works well for 2-4 nodes. Don't try 10GbE for training — you'll bottleneck at the first all-reduce.

I learned this the hard way. First cluster used 10GbE. Training throughput dropped 40% as soon as I scaled past 2 nodes. Upgraded to 100GbE InfiniBand. Problem solved.

CPU and RAM matter less than you think.

For training, your CPU just feeds data to GPUs. A 32-core EPYC or Xeon is fine. 256GB RAM per node is safe. 512GB if you're doing huge dataset preprocessing on the node itself.

Storage: don't ignore this.

Shared NVMe storage over NFS kills performance at scale. Use:

Lustre — Industry standard. Complex to set up.
GPFS / Spectrum Scale — IBM's offering. Works.
MinIO (object store) — Simpler. Pair with local NVMe cache on each node.

We use MinIO + local NVMe cache. 3GB/s read throughput. Cost us $12K per petabyte.

Step 3: The Software Stack — Don't Reinvent This

You do not need to build your own Kubernetes operator from scratch. Stop.

Here's what a production cluster runs:

Operating System: Ubuntu 22.04 LTS (or Rocky Linux 9)
Container Runtime: Docker + nvidia-container-toolkit
Orchestration: Kubernetes + Volcano scheduler (for batch jobs)
Or: SLURM + Pyxis/Enroot (for HPC-style workloads)
Networking: Mellanox OFED drivers + NCCL
Monitoring: Prometheus + Grafana + DCGM
Job Management: Run:ai or Apache Airflow

SLURM vs Kubernetes for GPU workloads?

Most people say Kubernetes. I used to, too. Then I actually benchmarked.

For training jobs that need to run for days, SLURM handles preemption, job dependencies, and resource accounting better than anything K8s has. K8s wins for inference serving and microservices.

We run both. SLURM for training, K8s for inference. Two clusters, purpose-built.

Here's a minimal SLURM configuration for GPU nodes:

bash
# /etc/slurm/slurm.conf
ClusterName=gpu-cluster
SlurmctldHost=master-node
AuthType=auth/munge
ProctrackType=proctrack/linux
TaskPlugin=task/affinity

# Partition for GPU nodes
PartitionName=gpu Nodes=gpu-[01-04] Default=YES 
    MaxTime=INFINITE State=UP 
    GresTypes=gpu

# Node definitions
NodeName=gpu-[01-04] CPUs=64 RealMemory=256000 
    Sockets=2 CoresPerSocket=32 ThreadsPerCore=2 
    Gres=gpu:4

And the GPU-specific job submission:

bash
#!/bin/bash
#SBATCH --job-name=llm-train
#SBATCH --partition=gpu
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=16
#SBATCH --gres=gpu:4
#SBATCH --mem=128GB
#SBATCH --time=14-00:00:00

srun --ntasks=$SLURM_NNODES --tasks-per-node=1     torchrun     --nnodes=$SLURM_NNODES     --nproc_per_node=4     --rdzv_id=$SLURM_JOB_ID     --rdzv_backend=c10d     --rdzv_endpoint=$(scontrol show hostname $SLURM_NODELIST | head -n1):29500     train.py

Step 4: Network Config That Won't Make You Cry

Setting up InfiniBand for the first time is a rite of passage. You will spend three days debugging why NCCL sees only one GPU. You will question your career choices.

Here's the cheat sheet:

Install Mellanox OFED (not the inbox driver)
Set mlx4_core or mlx5_core parameters for RoCE
Verify with ibstatus
Set NCCL environment variables

bash
# /etc/modprobe.d/mlnx.conf
options mlx4_core num_vfs=4 probe_vf=4
options mlx5_core num_vfs=4

# NCCL environment variables for optimal performance
export NCCL_IB_DISABLE=0
export NCCL_IB_HCA=mlx5_0:1,mlx5_1:1
export NCCL_NET_GDR_LEVEL=2
export NCCL_DEBUG=INFO  # Only during debugging

Test your setup with NCCL's built-in benchmark:

bash
# Install NCCL tests
git clone https://github.com/NVIDIA/nccl-tests.git
cd nccl-tests
make

# Run all-reduce benchmark on 2 nodes, 4 GPUs each
mpirun -np 8 -host node1:4,node2:4     ./build/all_reduce_perf -b 8 -e 128M -f 2 -g 1

If you're seeing less than 90% of theoretical bandwidth, something's wrong. Fix it before running real jobs.

Step 5: Storage That Doesn't Suck

I mentioned MinIO + local NVMe cache. Let me explain why.

NFS is simple. It's also terrible for GPU training. The first time 8 GPUs try to read 100MB/s each from the same NFS share, your training IOPS collapse. I've seen training stall for 30 seconds while waiting on data reads.

Our setup:

MinIO cluster on 3 nodes with NVMe drives (7TB each)
Local NVMe cache on each GPU node (4TB, RAID0)
Pre-fetch pipeline that stages next batch of data to local cache while current batch trains

Here's the caching layer in Python:

python
import asyncio
from minio import Minio
import torch

class GPUCache:
    def __init__(self, cache_dir="/mnt/nvme/cache"):
        self.cache_dir = Path(cache_dir)
        self.client = Minio(
            "minio-cluster:9000",
            access_key="your-key",
            secret_key="your-secret",
            secure=False
        )
        
    async def prefetch(self, batch_keys, device="cuda:0"):
        """Prefetch next batch while GPU is computing"""
        futures = []
        for key in batch_keys:
            local_path = self.cache_dir / key
            if not local_path.exists():
                self.client.fget_object("dataset", key, str(local_path))
            # Load to GPU asynchronously
            futures.append(
                asyncio.to_thread(torch.load, str(local_path), map_location=device)
            )
        return await asyncio.gather(*futures)

This pattern eliminated IO bottlenecks entirely. Training throughput went from 60% GPU utilization to 94%.

Step 6: Power and Cooling — The Hidden Costs

Nobody talks about this. Every GPU cluster builder discovers it the hard way.

An H100 draws 700W under load. Four of them? 2.8kW. Plus CPU, RAM, networking — call it 3.5kW per node. For an 8-node cluster, that's 28kW.

At $0.12/kWh running 24/7: $2,900/month. Just for power.

Cooling is worse. Air cooling 28kW requires serious airflow. You'll need:

Server-grade AC (not residential)
Proper rack layout (hot aisle/cold aisle if possible)
Fans rated for continuous operation

I've seen startups bolt GPU clusters into open racks in standard offices. It works briefly. Then summer hits. Then you get thermal throttling. Then the GPUs start failing.

We use immersion cooling for our dense cluster. Mineral oil, dielectric fluid, the works. Initial cost was painful ($15K per node). But our failure rate dropped to zero over 18 months, compared to 15% failure rate on air-cooled nodes in the previous setup.

Step 7: Monitoring — Build This Before You Need It

You will have problems. GPU memory leaks. Network partitioning. Driver crashes. Temperature spikes.

Build monitoring before your first training run.

yaml
# Prometheus + DCGM exporter for GPU metrics
scrape_configs:
  - job_name: 'dcgm'
    static_configs:
      - targets: ['gpu-node-1:9400', 'gpu-node-2:9400']
    metrics_path: '/metrics'
    scrape_interval: 5s

Alert on:

GPU temperature > 85°C
GPU memory utilization > 95%
NCCL errors in logs
InfiniBand link down
Storage latency > 100ms

I use Grafana dashboards from NVIDIA's DCGM examples (https://antmicro.com/blog/2024/03/customizable-and-scalable-gpu-cluster). Modified for our setup. Took a day to set up. Saved us three weeks of debugging time in the first month alone.

Common Mistakes I've Made So You Don't Have To

Mistake 1: Mixing GPU generations in the same cluster.
We tried running A100s and V100s together. NCCL all-reduce downgrades to the slowest card's speed. Performance sucked. Don't do it.

Mistake 2: Not oversubscribing network bandwidth.
InfiniBand costs money. But undersizing it means your GPUs idle waiting for gradients. Calculate: 8x GPUs each generating ~2GB gradient updates per step. That's 16GB to transfer per node. On 100GbE? ~1.3 seconds. On 200Gbps InfiniBand? ~0.65 seconds. Over 50K training steps, that's 9 hours difference. For $2K more.

Mistake 3: Ignoring the orchestrator.
You can manually SSH into nodes and run torchrun. For a week. Then you'll forget which node failed, which checkpoint is latest, and why training stopped at 3 AM.

Use SLURM or K8s from day one. The setup cost is a day. The debugging time it saves is weeks.

When Should You Actually Do This?

Build your own cluster if:

You need >4 GPUs for >3 months continuously
You have a dedicated team member who can handle hardware
Your data can't leave your premises (healthcare, defense, finance)
You care about latency predictability

Don't build if:

Your workloads are short and bursty
You don't have someone who can SSH into a node at 2 AM
You're experimenting (rent cloud credits)

I'm clear on this because I've seen teams waste money both ways. Building a cluster that sits idle is stupid. Renting cloud when you need 24/7 compute is equally stupid.

FAQ

Q: Can I use consumer GPUs for a cluster?
Yes. RTX 4090s work fine for inference and small model training. But you lose ECC memory and NVLink. For anything that needs multi-GPU synchronized training, you'll want NVIDIA datacenter cards.

Q: What's the minimum budget for a usable cluster?
$40-50K gets you 4x A100 80GB cards, a server node, basic networking, and a rack. $15K for a 4x RTX 4090 setup on consumer hardware.

Q: How do I handle job scheduling?
SLURM for batch training jobs. Kubernetes for inference serving and model deployment. Or Run:ai if you want a managed layer on top.

Q: Do I need InfiniBand?
For 2 nodes, no — 100GbE with RoCE works. For 4+ nodes doing distributed training, yes, get InfiniBand. The cost difference is <10% of total cluster cost and doubles training throughput.

Q: What about AMD GPUs?
The hardware is good. The software stack isn't there yet for production. If you have dedicated engineering time to build custom tooling, go for it. Otherwise, stick with NVIDIA.

Q: How much power will I need?
3.5-4kW per node with 4x datacenter GPUs. Plus cooling overhead. Plan for 5kW per node total power draw.

Q: Can I build a cluster for deep learning inference?
Absolutely. Inference clusters are simpler — less inter-node communication, so 25GbE is often enough. Consumer GPUs excel here.

The Bottom Line

Building a GPU cluster isn't magic. It's system design, with real tradeoffs. Spend your money on networking and storage, not on the latest GPUs. Monitor everything from day one. And admit when you should just rent.

The cluster I run at SIVARO today processes training jobs across 32 H100s, 8 nodes, connected by 400Gbps InfiniBand. It cost a lot. But it's ours. We control the upgrade cycle, the software stack, and the data flow.

And when something breaks at 3 AM, I can walk into the server room and touch the hardware. That's worth something cloud vendors will never sell.

Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.