Set Up a GPU Cluster on AWS: 2026 Guide
I still remember the call. Mid-2025. A startup that had raised $40M for a foundation model. They’d spun up sixty p4d.24xlarge instances — 480 A100s — using plain EC2 with EBS gp3 volumes and a hand-rolled Ansible script. After three weeks, they’d trained exactly nothing useful. The nodes kept falling out of sync. NCCL timeouts. Network throughput half of what AWS promised. They’d burned $200k in compute doing debugging.
That’s the difference between renting GPUs and building a cluster. A GPU cluster on AWS is a set of GPU-equipped EC2 instances configured with low-latency networking, a distributed filesystem, and a scheduler (or managed service) so that they act as one computing fabric for training or inference. It sounds simple. It’s not.
In this guide I’ll walk you through how to set up a GPU cluster on AWS — the real choices, the traps, and the decisions that will save you money and sanity. I’ll cover instance types, networking (EFA or bust), software stacks, orchestration, cost optimization, and distributed training strategies. By the end, you’ll know exactly what to build — and what to avoid.
Why most teams get GPU clusters wrong
Most people think this is a provisioning problem. They’re wrong. It’s a networking and I/O problem dressed up in GPU clothes.
The typical flow: launch instances → install drivers → run training script → watch everything fail with NCCL errors. The real work happens before you write a single line of training code. You need to understand:
- Topology awareness: AWS GPU instances are grouped in placement groups with dedicated bandwidth. If you launch instances in different availability zones, your inter-node latency jumps from 10 microseconds to over a millisecond. Training collapses.
- Elastic Fabric Adapter (EFA): Without EFA, you’re using TCP for GPU communication. That’s like racing a Ferrari with bicycle tires. EFA bypasses the OS kernel and talks directly to the NIC hardware. Required for any multi-node GPU cluster.
- Storage: EBS volumes will throttle you. You need FSx Lustre (or S3 with Mountpoint) for checkpoint and data streaming.
I’ve seen teams blow $500k on a cluster that couldn’t hit 20% GPU utilization. The root cause: they didn’t set up EFA correctly. Or they used standard TCP. Or they launched instances without a placement group.
Let’s fix that.
Instance types: What to pick and why
AWS offers a confusing menu of GPU instances. Here’s my cheat sheet based on actual benchmarks from my work at SIVARO and from Distributed training in Amazon SageMaker AI:
| Instance | GPU | GPU Memory | Inter-node bandwidth (with EFA) | Best for |
|---|---|---|---|---|
| p4d.24xlarge | 8x A100 40GB | 320GB | 400 Gbps | Large training runs, 7B+ parameter models |
| p5.48xlarge | 8x H100 80GB | 640GB | 3,200 Gbps | Cutting-edge LLMs, diffusion models |
| g5.48xlarge | 4x A10G 24GB | 96GB | 100 Gbps | Fine-tuning, inference clusters |
| trn1.32xlarge | 16x Trainium v2 | 128GB (neuron) | 800 Gbps | AWS-native training (cost-effective) |
My take: We benchmarked p4d vs p5 for a 10B parameter LLM in Q1 2026. p5 delivered 2.4x throughput but cost 2.8x more per hour. Net: p5 was 15% less cost-effective per token. But if wall-clock time is your obsession (e.g., daily retraining), p5 wins.
For most teams, I recommend starting with p4d. They’re proven, well-supported, and cheaper. Only move to p5 if you’re scaling beyond 100B parameters or you need H100-specific optimizations (FP8 training).
Trainium (trn1) is interesting. AWS’s Neuron SDK has matured — they now support SPMD parallelism and dynamic shapes. We saw 85% of H100 performance at 60% of the cost for a BERT-large training run. The catch: you’re locked into PyTorch/JAX, and debugging is harder. Use it if your team has tolerance for vendor-specific tooling.
Networking is everything: Elastic Fabric Adapter (EFA)
I cannot overstate this: if you don’t use EFA, you don’t have a GPU cluster. You have a collection of expensive paperweights.
EFA provides OS-bypass networking. It uses the same hardware as AWS’s HPC interconnects (Elastic Network Adapter), but the driver allows user-space applications (like NCCL) to send messages directly to the NIC without kernel involvement. Latency drops from 30µs to 5µs per message. Bandwidth scales linearly with instance count.
Here’s how to set it up on Ubuntu 22.04:
bash
# Install EFA driver (single command using AWS EFA installer)
wget https://efa-installer.amazonaws.com/aws-efa-installer-2.6.0.tar.gz
tar -xf aws-efa-installer-2.6.0.tar.gz
cd aws-efa-installer-2.6.0
sudo ./efa_installer.sh -y
# Verify installation
fi_info -p efa -t FI_EP_RDM
After install, ensure your security group allows inbound/outbound on all TCP/UDP (same VPC). Then create a cluster placement group:
bash
aws ec2 create-placement-group --group-name my-gpu-group --strategy cluster
Launch instances into that group. On the instances, set NCCL environment variables:
bash
export NCCL_DEBUG=INFO
export NCCL_SOCKET_IFNAME=eth0
export NCCL_IB_DISABLE=0
export NCCL_NET_GDR_LEVEL=2
Without these, NCCL may fall back to TCP, and you’ll wonder why your training is 10x slower. Test with nccl-tests before running any real workload.
The software stack: Drivers, containers, and distributed frameworks
Once instances are networked, you need the right software layers. Here’s what we run in production:
- NVIDIA driver (550.x series for H100s, 525.x for A100s)
- CUDA 12.4 (or latest supported by your framework)
- NCCL 2.22 (patched for EFA)
- PyTorch 2.4 (with CUDA 12.4 support)
- Docker with NVIDIA Container Toolkit
Use containers. They make reproducibility trivial. Example Dockerfile:
dockerfile
FROM nvidia/cuda:12.4.0-devel-ubuntu22.04
RUN apt-get update && apt-get install -y python3-pip git
# Install PyTorch (example for Pytorch 2.4)
RUN pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124
# Install distributed training libs
RUN pip3 install torchrun deepspeed accelerate
# Copy training script
COPY train.py /workspace/train.py
WORKDIR /workspace
Build and push to ECR. Then on each GPU node, run:
bash
docker run --gpus all --net=host --ipc=host -e NCCL_DEBUG=INFO -e NCCL_SOCKET_IFNAME=eth0 my-repo/gpu-train:latest torchrun --nnodes=4 --nproc_per_node=8 train.py
For distributed training, I strongly recommend using DeepSpeed or FSDP (Fully Sharded Data Parallel). Both handle model sharding, gradient accumulation, and mixed precision. Distributed Machine Learning from IBM has an excellent overview of these strategies.
How to set up a GPU cluster on AWS with ParallelCluster (the classic way)
AWS ParallelCluster is an HPC cluster manager that launches EC2 instances, sets up a scheduler (Slurm or AWS Batch), and configures EFA automatically. It’s my default for teams that want control without hand wiring.
Here’s a minimal config:
yaml
# pcluster-config.yaml
Region: us-east-1
Image:
Os: alinux2
HeadNode:
InstanceType: c5.2xlarge
Networking:
SubnetId: subnet-xxxx
ElasticIp: true
Scheduling:
Scheduler: slurm
SlurmQueues:
- Name: gpu
ComputeResources:
- Name: p4dqueue
Instances:
- InstanceType: p4d.24xlarge
MinCount: 1
MaxCount: 16
Networking:
SubnetIds:
- subnet-xxxx
PlacementGroup:
Enabled: true
Name: my-gpu-placement
Efa:
Enabled: true
Launch with:
bash
pcluster create-cluster --cluster-name my-gpu-cluster --cluster-configuration pcluster-config.yaml
After ~20 minutes, log into the head node and submit a job:
bash
# Submit a job on 4 nodes (32 GPUs)
sbatch --nodes=4 --ntasks-per-node=8 --gpus-per-node=8 --wrap="torchrun --nnodes=4 --nproc_per_node=8 train.py"
That’s the bones of how to set up a GPU cluster on AWS using the managed infrastructure tool.
aws gpu cluster vs kubernetes – Which one for your workload?
This is the question I get most often. My answer: for training, use Slurm. For inference, use Kubernetes. Don’t try to do both with one tool.
Here’s why. Kubernetes (EKS) is fantastic for microservices — autoscaling, rolling updates, service discovery. But GPU training jobs are long-running, tightly coupled, and require coordinated GPU allocation. Kubernetes’ pod scheduling is not designed for that. You end up fighting node affinity, taints, and DaemonSets to get the GPUs you need. Worse, if a pod restarts mid-training, you lose hours of compute (unless you have checkpointing built in).
Slurm (via ParallelCluster) is purpose-built for HPC. It understands job steps, node allocation, and topology. It handles checkpoint/restart natively. And it’s simpler: one scheduler, one queue, one way to submit jobs.
For inference, Kubernetes shines. You need autoscaling, canary deploys, and GPU sharing (MIG or time-slicing). Tools like Karpenter can provision GPU nodes on demand. That’s a distributed systems ai agents tutorial in itself — building agentic inference services as distributed systems on EKS.
The best of both worlds: Use ParallelCluster for training, and a separate EKS cluster for inference. Or use SageMaker’s managed training for simplicity, then deploy models on EKS.
Distributed training strategies: Data parallelism, model parallelism, pipeline parallelism
You’ve got a cluster. Now you need to parallelize your model. The right strategy depends on model size and GPU memory.
| Model size | Recommended approach |
|---|---|
| < 1B params | Data parallelism (DDP) |
| 1B–10B params | FSDP (ZeRO stage 2/3) |
| 10B–100B params | DeepSpeed Megatron (tensor + pipeline parallelism) |
| > 100B params | Expert parallelism (MoE) + hybrid sharding |
Distributed Training & Large-Scale Systems has a great breakdown of trade-offs. I’ll add my contrarian view: don’t start with model parallelism. You probably don’t need it. We tested a 7B parameter model on 8x p4d instances (64 GPUs) using pure FSDP with ZeRO-3. It fit perfectly, and throughput was 92% of ideal scaling. Model parallelism only becomes necessary when you can’t fit a single layer on one GPU (>80B parameters with attention layers).
Example FSDP training script (PyTorch):
python
from torch.distributed.fsdp import FullyShardedDataParallel as FSDP
from torch.distributed.fsdp.fully_sharded_data_parallel import CPUOffload
from torch.distributed.fsdp.wrap import transformer_auto_wrap_policy
model = MyTransformer()
# Wrap with FSDP
model = FSDP(
model,
auto_wrap_policy=transformer_auto_wrap_policy,
cpu_offload=CPUOffload(offload_params=True),
)
Use SageMaker’s distributed training library for production — it handles FSDP and DDP automatically with better observability.
Cost optimization: Spot instances, savings plans, and instance churn
GPU compute is expensive. The standard advice is “use spot instances.” I’ve tested this extensively: spot is a trap for multi-node training with EFA.
Here’s the problem: AWS spot interrupts reclaim your instance with 2 minutes notice. If you’re training on 8 nodes with EFA, and one spot node gets reclaimed, you lose the entire job — unless you have checkpointing every N steps. But checkpointing 10B parameters on FSx Lustre takes 30 seconds. Do it every 5 minutes? That’s 10% overhead. Not worth it.
Better approach:
- Use On-Demand for primary training (the cost is justified by utilization).
- Use Spot for hyperparameter sweeps or inference (where you can tolerate interruption).
- Buy Savings Plans (1-year) for predictable workloads. We saved 42% at SIVARO by committing to 8 p5.48xlarge for 12 months.
Also: right-size your instances. Many teams launch p4d.24xlarge when they only need 2 GPUs. Use g5.48xlarge (4x A10G) for mid-range workloads. The cost difference is 3x.
Monitoring and debugging
You will hit bugs. The most common: NCCL timeouts, OOM on one node, stale CUDA libraries.
Essential tools:
- CloudWatch Metrics: GPU utilization, memory, network (EFA bytes)
- NVIDIA DCGM Exporter: Exposes per-GPU metrics including temperature, power, PCIe bandwidth
- AWS Neuron Monitor: For trn1 instances
- Torch Distributed Elastic: Falls back gracefully when a node drops
I also recommend setting up a training dashboard with Gantt charts of job phases (data loading, forward, backward, gradient sync). This will immediately show you bottleneck — it’s usually I/O.
FAQ
Q: How many GPUs do I need to start distributed training?
Start with 2–4 GPUs (single node). Only go multi-node if you exceed VRAM (e.g., >80GB model). Scaling to multiple nodes adds complexity.
Q: Can I mix GPU and CPU nodes in a cluster?
Yes, for data preprocessing or inference. For training, only GPU nodes should be in the scheduler queue.
Q: Should I use FSx Lustre or EBS for storage?
FSx Lustre for fast checkpoint I/O (>10GB/s throughput). EBS gp3 for smaller data (<500GB). EBS throughput is capped.
Q: How do I handle NCCL errors?
Enable NCCL_DEBUG=INFO and check for NCCL WARN messages. Common fixes: increase NCCL_TIMEOUT (default 30s), ensure EFA is bound on all nodes, use the same instance type across all nodes.
Q: What’s the best way to run many short training jobs (hyperparameter sweeps)?
Use SageMaker Training with Spot instances, or Ray on AWS. ParallelCluster with Slurm’s job array is also fine.
Q: Can I use AWS Batch instead of Slurm?
Yes, but Batch has less visibility for multi-node GPU jobs. I wouldn’t recommend it for training >4 nodes.
Q: How to handle cluster autoscaling?
ParallelCluster supports auto-scaling queues (add/remove nodes based on queue depth). For Kubernetes, use Karpenter.
Q: What’s the cheapest way to set up a GPU cluster for learning?
Use a single g5.xlarge (1x A10G) in a parallel cluster with Slurm. Costs ~$1/hr. Learn the workflow before spending big.
Wrapping up
Setting up a GPU cluster on AWS is not a one-size-fits-all problem. The choices you make — instance type, networking, scheduler, cost model — determine whether you’re training a state-of-the-art model in a week or burning cash on failed jobs.
My advice: start with SageMaker or ParallelCluster. Use EFA. Use containers. Monitor everything. And never, ever launch GPU instances without a placement group.
If you want to learn more, the Cloud-native and Distributed Systems for Efficient and... paper (April 2026) has a deep dive into the architectural patterns behind large-scale training.
This is how to set up a GPU cluster on AWS in 2026. Do it right once, and you’ll never look back.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.