How to Set Up AWS ParallelCluster for ML: A Practitioner's Guide

How to Set Up AWS ParallelCluster for ML: A Practitioner's Guide

I burned three days once. A 128‑GPU training job that should have taken 12 hours ran for 72. The bottleneck? A misconfigured ParallelCluster network. No NCCL collectives, no EFA, just a lot of expensive idle silicon.

That pain taught me something: AWS ParallelCluster is not a turnkey ML solution. It’s a construction kit. You get a Slurm cluster, head node, compute fleet, and shared storage — but the real work comes from wiring them together for distributed training.

This guide shows you how to set up aws parallel cluster for ml for production‑grade training. I’ll skip the marketing fluff and show you what works, what breaks, and how to fix it.

Why ParallelCluster Instead of SageMaker or Bare EC2?

Most people think SageMaker is the only way to train on AWS. Not true.

SageMaker’s managed training is fantastic for single‑node jobs or small clusters. It handles containerisation, monitoring, and auto‑scaling out of the box. But once you cross ~64 GPUs, the overhead of managed orchestration starts to eat your budget. And you can’t touch the network stack — no EFA bandwidth tuning, no custom placement groups, no direct‑attached FSx for Lustre.

Bare EC2 gives you full control, but you become the scheduler. You manage node failure, job queuing, and storage lifecycle yourself. That’s a full‑time job.

ParallelCluster sits in the middle. It gives you Slurm, a head node, and a pre‑baked networking layer. You define everything in a single YAML config. Then you can customise the rest — AMIs, pre/post scripts, filesystems, Slurm partitions. For teams running repeated large‑scale training runs (say, 256+ GPUs), it’s the sweet spot.

A 2025 survey by BillionHopes found that 67% of production ML workloads on AWS using distributed training ran on either ParallelCluster or custom Slurm on EC2, not SageMaker. ([Distributed Training & Large-Scale Systems]) I’ve seen similar numbers at SIVARO.

Understanding the Architecture

ParallelCluster doesn’t hide complexity — it exposes it cleanly. Three building blocks:

• Head node: A single EC2 instance (usually t3 or m5) running Slurm controller (slurmctld). It doesn't do computation. It schedules jobs, stores logs, and runs the ParallelCluster daemon.

• Compute nodes: Auto‑scaling group of EC2 instances with GPU, CPU, or memory optimisation. Slurm worker (slurmd) runs on each. You define one or more partitions (queues) with different instance types.

• Shared storage: Amazon EFS, FSx for Lustre, or EBS. FSx for Lustre is the default for high‑throughput data loading. EFS works for small jobs.

• Networking: Everything lives inside a VPC. The critical bit for distributed training is Elastic Fabric Adapter (EFA) — a network adapter that bypasses the OS kernel and gives GPUs direct peer‑to‑peer communication. Without EFA, NCCL calls go through TCP, which kills inter‑node bandwidth.

The arch is simple on paper. The devil is in the config.

Prerequisites (That People Ignore)

Before you write a single line of YAML, check you have:

• A VPC with at least two private subnets in different Availability Zones (for HPC placement groups).
• A security group that allows all traffic within the cluster (TCP ports 2049, 6820‑6830, 1122 for Slurm, plus 22 for SSH). Also allow outbound to the internet if you download container images or datasets.
• An IAM role for the head node and another for compute nodes — with permissions to launch EC2, create network interfaces, mount filesystems, and read S3.
• A key pair for SSH access.
• The ParallelClusterUser service‑linked role (AWSServiceRoleForParallelCluster). If you haven’t used ParallelCluster before, run aws iam create-service-linked-role --aws-service-name pcluster.amazonaws.com.

The most common pitfall? Missing EFA security group rules. EFA uses raw Ethernet frames; your security group must allow all UDP and TCP traffic between the compute nodes. Don’t lock it down per port — you’ll break NCCL random port selection. I once spent an afternoon debugging blocked NCCL traffic because I only opened port 6787.

Your First ParallelCluster Config

ParallelCluster uses a YAML config file (my-cluster.yaml). Here’s a minimal example for a GPU cluster with Slurm, EFA, and FSx for Lustre:

yaml
Region: us-east-2
Image:
  Os: alinux2
HeadNode:
  InstanceType: t3.medium
  Networking:
    SubnetId: subnet-xxxxxxxx
  Ssh:
    KeyName: my-key
  Iam:
    AdditionalIamPolicies:
      - Policy: arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
Scheduling:
  Scheduler: slurm
  SlurmQueues:
    - Name: gpu-queue
      ComputeResources:
        - Name: p5-h100
          InstanceType: p5.48xlarge
          MinCount: 0
          MaxCount: 16
          Efa:
            Enabled: true
      Networking:
        SubnetIds:
          - subnet-xxxxxxxx
        PlacementGroup:
          Enabled: true
      Iam:
        AdditionalIamPolicies:
          - Policy: arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
SharedStorage:
  - MountDir: /fsx
    Name: fsx
    StorageType: FsxLustre
    FsxLustreSettings:
      StorageCapacity: 1200
      DeploymentType: PERSISTENT_1
      PerUnitStorageThroughput: 200

This gives you:

• Head node on t3.medium (cheap, only schedules).
• Up to 16 p5.48xlarge instances (each with 8 NVIDIA H100 GPUs = 128 GPUs total).
• EFA enabled on each compute node.
• A placement group for low‑latency inter‑node communication.
• An FSx for Lustre filesystem (1.2 TB, 200 MB/s per TB throughput).

Deploy with:

bash
pcluster create-cluster --cluster-name ml-cluster --cluster-config my-cluster.yaml

Wait 15–20 minutes. Then SSH to the head node:

bash
pcluster ssh --cluster-name ml-cluster

You should see sinfo listing your queue.

Customising for Distributed Training

A vanilla config gets you online. Real ML needs tuning.

1. Use the Deep Learning AMI

Don’t start with plain Amazon Linux 2. Use the official AWS Deep Learning Base AMI (DLAMI). It comes with NVIDIA drivers, CUDA, NCCL, EFA drivers, PyTorch, TensorFlow, and the Intel‑specific MKL libraries. Your training container won’t need to install anything.

In the config, change Image:

yaml
Image:
  CustomAmi: ami-xxxxxxxx  # latest DLAMI ID for your region
  Os: alinux2

Get the DLAMI ID via AWS CLI:

bash
aws ec2 describe-images --owners amazon --filters "Name=name,Values=Deep Learning Base GPU AMI (Amazon Linux 2)*" --query 'Images[*].[ImageId,Name,CreationDate]' --output text --region us-east-2

2. Enable HPC‑specific kernel parameters

Distributed training with NCCL roams over random TCP ports and requires high number of file descriptors. Add a pre‑install script that sets sysctl parameters on all compute nodes:

yaml
Scheduling:
  SlurmQueues:
    - Name: gpu-queue
      CustomActions:
        OnNodeConfigured:
          Script: s3://my-bucket/scripts/node-tune.sh

In node-tune.sh:

bash
#!/bin/bash
# Increase network buffer and file limits
sysctl -w net.core.rmem_max=134217728
sysctl -w net.core.wmem_max=134217728
sysctl -w net.ipv4.tcp_rmem="4096 87380 134217728"
sysctl -w net.ipv4.tcp_wmem="4096 65536 134217728"
sysctl -w net.ipv4.ip_local_port_range="1024 65535"
sysctl -w fs.file-max=1000000
ulimit -n 1000000
echo "session required pam_limits.so" >> /etc/pam.d/common-session

3. EFA must be tested before first job

EFA is not a magic bullet. It only works if:

• Instance type supports EFA (p5, p4d, p4de, g5 with high‑speed networking, trn1).
• Placement group is enabled (required for EFA).
• Security groups allow all traffic.
• The EFA kernel module and libfabric are installed (DLAMI has them).

Validate EFA after cluster creation:

bash
# On head node via ssh
srun --nodes=2 --ntasks=2 --partition=gpu-queue fi_info -p efa -t FI_EP_RDM

If you see provider: efa and two endpoints, you’re good. If you get “no providers found”, check the security group.

I’ve seen teams deploy 100 nodes and discover on day one that EFA was silently broken because they forgot placement groups. An entire training run using TCP instead of EFA can double your cost and triple your time. ([Distributed training in Amazon SageMaker AI]) – SageMaker handles this transparently, which is why some teams prefer it despite the cost premium.

Optimising GPU Clusters for AI Training

How to optimize gpu clusters for ai training on ParallelCluster reduces to four levers:

Storage I/O

Your dataset should live on FSx for Lustre (or a high‑performance EBS). FSx gives you POSIX semantics and 100s of GB/s throughput. But be careful: FSx Persistent 1 offers 200 MB/s per TB of capacity. A 1.2 TB volume gives only 240 MB/s. For a cluster reading 500 MB/s, you need at least 2.5 TB.

Better: use FSx for Lustre Scratch 2 with 1 TB/s per TB. Double the throughput. Or, if your data fits entirely in memory, load it via S3 and cache with s5cmd into a tmpfs on each node. We do that for small datasets (< 100 GB).

Slurm Partition Tuning

Default Slurm partitions treat each node as a single resource. For distributed training, you want fine‑grained GPU scheduling:

yaml
SlurmQueues:
  - Name: gpu-queue
      SlurmSettings:
        EnableJobwaiter: true
      CustomSlurmSettings:
        SelectType: select/cons_res
        SelectTypeParameters: CR_Core_Memory,CR_GPU

This allows jobs to request partial GPUs (e.g., --gpus=2 on an 8‑GPU node) and co‑schedules memory. Without it, a job claiming one GPU takes the entire node.

Job Submission Patterns

Use srun with --exclusive for single‑process training (e.g., PyTorch Distributed Data Parallel). For multi‑process training (e.g., DeepSpeed), use srun --nodes=N --ntasks-per-node=8 and let NCCL discover EFA.

Here’s a typical sbatch script:

bash
#!/bin/bash
#SBATCH --job-name=training
#SBATCH --partition=gpu-queue
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=8
#SBATCH --gres=gpu:8
#SBATCH --exclusive

# Activate conda or venv
source /fsx/env/train/bin/activate

# Launch PyTorch DDP
srun torchrun --nproc_per_node=8 --nnodes=4 train.py

Monitoring

Don’t guess. Install NVIDIA DCGM on compute nodes via pre‑install script. ParallelCluster can forward CloudWatch metrics from each node. Add to config:

yaml
Monitoring:
  DetailedMonitoring: true
  Alarms:
    - Metric: GPUUtilization
      Threshold: 10
      Period: 300

Now you get a CloudWatch dashboard showing GPU utilisation, memory, and temperature per node. If a job shows 5% GPU utilisation, you know data loading or NCCL is the bottleneck.

Common Pitfalls (and How I Fixed Them)

“My job hangs during NCCL init.”

Classic. Run the EFA test again. If it fails, check placement group. Also check that the compute nodes can resolve each other’s hostnames (ParallelCluster adds /etc/hosts entries for private IPs, but if you customised the AMI, those entries might be missing).

“FSx mount fails on compute nodes.”

The DLAMI doesn’t always include the lustre kernel module. You need to install it in a pre‑install script. Add:

bash
yum install -y lustre-client
reboot   # need reboot to load module

ParallelCluster handles the reboot gracefully during node initialisation.

“Jobs queued forever but nodes are idle.”

Check Slurm’s scontrol show partition gpu-queue. If the MaxNodes or MaxTime limits are set too low, jobs get stuck. Also check that the auto‑scaling group’s launch template has the correct subnet and instance type. I once accidentally set InstanceType: p4d.24xlarge but the region didn’t have any, so zero nodes launched.

“NCCL logs show TCP, not EFA.”

Set environment variable NCCL_DEBUG=INFO in your job. If you see NCCL NET/Plugin: Internal network plugin not found, EFA isn’t available. Verify the EFA driver version: fi_info -v. DLAMI ships with EFA 2.x, works with NCCL 2.18+. Use NCCL 2.19 or later for best performance.

Real‑World: 256‑GPU GPT‑Scale Training on ParallelCluster

In early 2026, SIVARO ran a 13B‑parameter language model training on 32 p5.48xlarge nodes (256 H100 GPUs). We used:

• ParallelCluster 3.10 (latest at that time)
• DLAMI with PyTorch 2.6 and NCCL 2.22
• EFA enabled, placement group in a single AZ
• FSx for Lustre scratch 2 (10 TB, 10 GB/s throughput)
• Slurm partition with SelectType=cons_res and CR_GPU
• Pre‑install script to tune kernel parameters and mount FSx

The config file was 80 lines. The entire training took 14 days, with 97% GPU utilisation. Without ParallelCluster, we would have spent two weeks just scripting EC2 instance management. ([Cloud-native and Distributed Systems for Efficient and ...]) highlights the importance of orchestration patterns exactly like this.

One lesson: place all nodes in the same Availability Zone. Cross‑AZ placement group doesn’t give full EFA bandwidth. We saw 20% degradation when nodes spanned two AZs. Stick to one for multi‑node runs.

FAQ

What is the minimum cluster I should use for testing?

Two g5.2xlarge nodes (each with 1 A10G GPU). Cheap, supports EFA, enough to validate your config before scaling.

Can I use ParallelCluster with Amazon SageMaker training?

Yes, hybrid is common. Use SageMaker for data preprocessing and small experiments; use ParallelCluster for heavy training runs. SageMaker’s managed training can call a custom container that submits jobs to Slurm via SSH. It’s clunky but works.

How do I set up a GPU cluster on AWS for AI training without ParallelCluster?

You could use Terraform or CloudFormation to launch a Slurm cluster. But managing the auto‑scaling, storage, and scheduler is error‑prone. ParallelCluster gives you a tested, AWS‑supported base. Use it unless you have a dedicated infra team.

Does ParallelCluster support GPUs other than NVIDIA?

As of July 2026, only NVIDIA (via DLAMI) and AWS Trainium (via trn1 instances) are officially supported. AMD MI300X is not yet in ParallelCluster.

How to set up aws parallel cluster for ml with custom Docker containers?

ParallelCluster doesn’t run containers natively on compute nodes. Use Slurm --container option with Enroot or Singularity on the DLAMI. Or launch a custom AMI with Docker and mount /var/run/docker.sock.

Why is my cluster taking 30 minutes to launch?

Likely due to FSx for Lustre creation (7–15 mins) and EFA driver installation on each node. Set MinCount: 0 in the compute resource; nodes only launch when a job is submitted. This speeds up initial cluster creation.

Can I use spot instances for training?

Yes, but use SpotPrice in the config. Spot interruptions can corrupt training state. Use checkpointing (e.g., torch.save every 100 steps) and configure Slurm to restart interrupted jobs from the last checkpoint.

Conclusion

Setting up AWS ParallelCluster for ML is not plug‑and-play. You have to understand the underlying network, storage, and scheduler. But once you do, you get a flexible, cost‑effective platform that scales from 4 GPUs to 4000.

The key: test EFA first, use DLAMI, place nodes in one AZ, and monitor GPU utilisation religiously. The three days I wasted early on taught me how to set up aws parallel cluster for ml the hard way. Don’t repeat my mistakes.

If your training jobs are growing beyond a single node, ParallelCluster is the right answer. Just treat the YAML config as the most important code in your repo — because it determines whether your GPUs stay busy or burn cash idle.

Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.