What Are the Basics of Distributed Training? A Practitioner’s Guide

What Are the Basics of Distributed Training? A Practitioner’s Guide

You’ve got a model that takes two weeks to train on a single GPU. You need it in two days. The obvious answer: throw more GPUs at it. But if you just stack cards and press “go,” you’ll discover the hard way that distributed training is less about adding hardware and more about not wasting it.

I’m Nishaant Dixit, founder of SIVARO. We build production AI and data infrastructure. Over the last six years, I’ve watched teams burn millions on underutilized clusters. I’ve also seen a two-person startup train a 7B parameter model in a weekend using rented H100s and good parallelism strategy.

This article is the guide I wish I had in 2022 when I was debugging all-reduce timeouts at 3 AM.

What are the basics of distributed training? At its core, distributed training is taking one training job and splitting it across multiple devices (GPUs) so they work on the same model simultaneously, then combine their work fast enough that the speedup is real. The “basics” cover three things: how you split the work (parallelism strategy), how you connect the devices (networking), and how you avoid idleness (scheduling and scaling). By the end of this guide, you’ll understand the trade-offs between data parallelism and model parallelism, what networking actually matters, and when you should just rent instead of build.

The Core Question: Where Does Your Bottleneck Live?

Every distributed training setup has a bottleneck. Most people think it’s compute. In my experience, it’s almost always communication. You can have the fastest GPUs in the world—but if your cluster’s interconnect can’t keep up, you’ll spend most of your time waiting on gradients.

Let’s back up. The simplest form of distributed training is data parallelism. You copy the entire model onto each GPU. You give each GPU a different slice of the batch. Forward pass happens independently. Then, at the end of the backward pass, every GPU computes gradients and needs to share them so the optimizer applies the same update. That sharing step is technically called a gradient all-reduce. If your network is slow, that synchronization kills your throughput.

But data parallelism only works when your model fits on one GPU. Once your model exceeds GPU memory—and with 70B+ parameter LLMs, that’s today’s reality—you need model parallelism. You split the model itself across GPUs. Now each GPU holds a slice of the layers. The forward and backward passes require device-to-device communication at every layer boundary.

If you’re thinking “this sounds like a networking problem” — you’re right. That’s why a guide to distributed training must start with gpu cluster networking requirements. Not because networking is sexy, but because it’s the part most people underestimate. GPU Cluster Explained: Architecture, Nodes and Use Cases is a solid primer on node design. I’ll return to networking in detail later.

Data Parallelism: Simple, but Don’t Ignore Its Nuances

Most tutorials show you a snippet like this and call it a day:

python
import torch
import torch.distributed as dist
from torch.nn.parallel import DistributedDataParallel

dist.init_process_group("nccl")
model = MyModel().cuda()
model = DistributedDataParallel(model)

for batch in dataloader:
    outputs = model(batch)
    loss = compute_loss(outputs)
    loss.backward()
    optimizer.step()

That’s it, right? Not quite. That code works. But it hides three problems that will kill your throughput on real hardware.

First, batch size scaling. If you used batch size 32 on one GPU, using 8 GPUs with the same per-GPU batch size gives you a global batch of 256. That changes optimizer behavior (learning rate should typically scale linearly). In 2024, I worked with a team that doubled GPUs but forgot to adjust LR. Training diverged after 20 steps. They blamed hardware. It was math.

Second, gradient accumulation overlap. DistributedDataParallel by default synchronizes gradients after every backward call. You can overlap computation and communication by setting bucket_cap_mb and using gradient accumulation with no_sync contexts. Here’s a pattern we use at SIVARO:

python
dataloader = iter(train_loader)
accumulation_steps = 4
model.require_backward_grad_sync = False

for micro_step in range(accumulation_steps):
    batch = next(dataloader)
    loss = model(batch)
    loss = loss / accumulation_steps
    loss.backward()

# Sync once after accumulation
model.require_backward_grad_sync = True
with torch.no_grad():
    # trivial forward to trigger all-reduce
    model(batch)
optimizer.step()

This pattern reduces communication frequency, but only helps if your GPUs are fast enough to overlap communication with compute. On clusters with InfiniBand, the overhead of frequent small all-reduces is lower than with 25GbE. We’ll benchmark that shortly.

Third, memory fragmentation. DDP creates gradient buckets and copies. On older GPUs (V100, A100 40GB), memory was tight. We’ve seen memory spikes of 1-2GB from DDP wrappers alone. If you’re pushing model size limits, consider using torch.distributed.fsdp (Fully Sharded Data Parallel) instead—it shards optimizer states too.

Model Parallelism: When One GPU Isn’t Enough

Data parallelism fails when a single GPU can’t hold the model. Enter model parallelism. There are three main flavors, and each has a painful trade-off.

Pipeline Parallelism

You split layers across GPUs. GPU 0 holds layers 1-4, GPU 1 holds 5-8, etc. Input passes through like an assembly line. The problem: GPU 1 is idle while GPU 0 processes the first micro-batch. To fix that, you inject multiple micro-batches and let them flow in a staggered fashion. This is the “pipeline” in GPipe or PyTorch’s torch.distributed.pipeline.sync.

The throughput gain is roughly number of GPUs * bubble-efficiency. Micro-batch scheduling determines efficiency. In 2025, I saw a team get 80% of linear scaling using 16 GPUs with pipeline parallelism—but only after tuning micro-batch size and number of pipelining stages. Don’t expect 100%; the bubble is real. What Is a GPU Cluster and How to Build One gives a good overview of why intra-node interconnect matters more in pipeline-parallel designs: every micro-batch transfer crosses the bus.

Tensor Parallelism

Instead of splitting layers depth-wise, you split individual operations across GPUs. Each GPU holds a chunk of the weight matrix. During forward, each GPU does a partial matrix multiply, then all-reduces the partial result. This is how Megatron-LM and later LLaMA-2 style training scaled to thousands of GPUs. Tensor parallelism eats communication bandwidth because every transformer layer ends with an all-reduce. That’s why modern clusters use NVLink NVSwitch (NVIDIA) or Infinity Fabric (AMD) for intra-node tensor parallelism—they provide 600-900 GB/s per GPU.

We benchmarked A100 80GB nodes with both NVLink inter-node (DGX A100 style) versus standard PCIe connections. The difference in tensor parallelism performance was 35% in favor of NVLink. That’s not subtle. If you’re planning to train models >10B parameters, you need NVLink or similar.

Fully Sharded Data Parallel (FSDP)

This is the newer hybrid: shard optimizer states, gradients, and optionally parameters across GPUs, but reconstruct full parameters on the fly for the forward/backward pass. FSDP is popular in open-source because it blends data parallelism’s simplicity with model parallelism’s memory savings. In practice, FSDP introduces communication overhead for every forward and backward pass to gather and unshard parameters. That overhead is worth it when model memory would otherwise prevent training at all.

At SIVARO, we’ve been using FSDP since 2023 for our internal 13B experiment. The key tuning knobs are sharding_strategy (FULL_SHARD vs HYBRID_SHARD) and cpu_offload. HYBRID_SHARD, which shards within a node but replicates across nodes, gave us the best balance for 8-node clusters.

GPU Cluster Networking Requirements — the Basics You Can’t Ignore

Here’s the contrarian take: most people think “lots of GPUs” solves everything. It doesn’t. Without the right network, adding more GPUs reduces per-GPU throughput so much that total training time barely drops. I’ve seen a 64-GPU cluster with 10GbE actually train slower than a 4-GPU box with NVLink.

GPU cluster networking requirements break down into two tiers: intra-node and inter-node.

Intra-Node

Inside a single server, GPUs communicate via PCIe lanes or direct GPU-to-GPU links (NVLink, Infinity Fabric). For data parallelism, PCIe Gen4 x16 gives about 32 GB/s per direction—enough for small models. For tensor parallelism, you need NVLink or similar. 5 Key Considerations when Building an AI & GPU Cluster lists NVLink as a top requirement for any cluster targeting models >7B. I agree. If you’re buying new hardware today (July 2026), do not skimp on NVLink. H100 GPUs deliver 900 GB/s per GPU in NVSwitch topologies. That’s not fast—that’s absurdly fast.

Inter-Node

Between servers, you generally have Ethernet or InfiniBand. InfiniBand (HDR200, NDR400) delivers 200-400 Gb/s per link with RDMA, reducing CPU overhead. But it’s expensive. Ethernet at 100GbE/200GbE with RoCE (RDMA over Converged Ethernet) is a cheaper alternative.

Our benchmark: on a 4-node cluster (each node 8x A100 80GB, NVLink intra-node), we tested DDP training of a BERT-large variant. With 100GbE RoCE, we got 220 samples/sec. With HDR200 InfiniBand, we got 285 samples/sec — a 30% improvement. For large models with frequent all-reduces, InfiniBand is worth every cent. GPU Cluster Benchmark Comparison (note: I’m referencing the benchmarking section within that article) shows similar numbers.

But what if you’re renting? Then the network is fixed. Vast.ai: Rent GPUs offers both InfiniBand and Ethernet instances. You pay more for InfiniBand, but for distributed training jobs, the throughput gain often justifies it. We’ve run cost analyses: for a 7B parameter model trained for one week, the InfiniBand premium adds ~$800 but saves 2 days of training time. That’s $800 to get results two days earlier. Usually worth it.

Benchmark Comparison: Practical Numbers

Let’s make this concrete. Below is a GPU cluster benchmark comparison using real data from a project we ran in early 2026 (training a LLaMA-3-styled 7B model with 4K sequence length, using FSDP hybrid shard, micro-batch size 8 per GPU, gradient accumulation 4).

Cluster Configuration	Nodes	GPUs/node	Intra-node Interconnect	Inter-node Network	Training Speed (tokens/sec)	Scaling Efficiency vs Single Node
Single node, 8x A100	1	8	NVLink	N/A	18,000	1.0 (baseline)
2 nodes, 8x A100 each	2	16	NVLink	100GbE RoCE	30,500	0.85
2 nodes, 8x A100 each	2	16	NVLink	HDR200 InfiniBand	34,200	0.95
4 nodes, 8x A100 each	4	32	NVLink	HDR200 InfiniBand	68,500	0.95
4 nodes, 8x H100 each	4	32	NVLink/NVSwitch	HDR400 InfiniBand	110,000	~0.93 (higher base throughput)

Notice: Ethernet scales to 2 nodes okay (85% efficiency), but InfiniBand keeps efficiency above 90%. With 4 nodes, Ethernet would likely dip below 70% for this model size. Also note that H100 gives a massive raw throughput bump, but scaling efficiency is slightly lower because the inter-node bandwidth (even HDR400) doesn’t keep up with the intra-node speeds. This is the current frontier: models are so big that even InfiniBand is a bottleneck for tensor parallelism across nodes.

Building vs Renting: The Decision Framework

You’re probably thinking: “Should I build an on-premise GPU cluster or rent cloud/third-party GPUs?”

I’ve done both. What is the best option to setup on premise GPU cluster for a small company? is a thread I read when we were evaluating. My take: unless you plan to train 24/7 for at least 18 months, rent.

On-prem means paying for power, cooling, networking cables, and downtime. In 2025, we lost a week because a single InfiniBand cable mis-seated. The rental market (Lambda, Vast.ai, CoreWeave) has commoditized H100/A100 access. Vast.ai even lets you rent individual nodes and connect them via their internal network. Vast.ai: Rent GPUs is where I send startups now.

But there’s a catch: spot instances and preemption. If you’re running a distributed job that takes 24 hours, a single node preemption kills the entire job unless you’ve built checkpointing into your training loop. Here’s a robust checkpointing pattern using PyTorch’s distributed checkpoint (available since 2.1):

python
from torch.distributed.checkpoint import (
    save, load, FileSystemWriter, FileSystemReader
)
from torch.distributed.fsdp import FullyShardedDataParallel as FSDP

def save_checkpoint(model, optimizer, step, dir_path):
    state = {
        "model": model.state_dict(),
        "optimizer": optimizer.state_dict(),
        "step": torch.tensor(step),
    }
    writer = FileSystemWriter(dir_path)
    save(state, writer)

def load_checkpoint(model, optimizer, dir_path):
    reader = FileSystemReader(dir_path)
    state = {
        "model": model.state_dict(),
        "optimizer": optimizer.state_dict(),
    }
    load(state, reader)
    model.load_state_dict(state["model"])
    optimizer.load_state_dict(state["optimizer"])

We run checkpoints every N steps to a shared filesystem (NFS or object store). This makes spot preemption a nuisance, not a disaster.

Common Mistakes and Hard Lessons

I said I’d write like a practitioner talking to a peer. Here are the mistakes I see most often in distributed training setups, in order of pain.

1. Not profiling communication. You add a new parallelism strategy, it runs, it’s slow. You guess it’s the model. We profiled: 60% of time was spent in all-to-all for tensor parallelism. Switch to all-reduce with fused kernels, cut time by 40%.

2. Wrong batch size scaling. Scaling GPUs from 4 to 16? Increase your learning rate by 2x. No, not 4x. To really tune, use the scaling rule: LR_new = LR_old * sqrt(new_batch / old_batch). I’ve seen convergence fail with linear scaling for huge batch sizes (>1024). Use the square root rule if your batch size > 1024.

3. Ignoring NCCL tuning. NVIDIA’s NCCL library has environment variables like NCCL_IB_DISABLE=0 or NCCL_SOCKET_IFNAME=eth0. If your inter-node network uses RoCE but you don’t set NCCL_IB_DISABLE=1, NCCL will try to use InfiniBand verbs and fail silently. We debugged a 2-day performance issue that boiled down to one env var.

4. Using GPU-bound CPUs. Training nodes need fast CPUs for data loading. If your dataloader is using CPU workers on thread-starved CPUs, your GPUs will idle. We’ve seen this on older Xeon Gold 5120 CPUs. Swap them. Or preprocess data to compressed tensors on disk.

Practical Code: Launching a Distributed Job

Here’s how we launch a DDP job on a multi-node cluster at SIVARO. We use torchrun with environment-based configuration.

bash
# Launch script for 4 nodes, 8 GPUs per node
torchrun   --nnodes=4   --nproc_per_node=8   --rdzv_endpoint=10.0.0.1:29400   --rdzv_backend=c10d   train_ddp.py   --model vit_large   --batch_size 32

And inside train_ddp.py:

python
import os
import torch
import torch.distributed as dist

def setup():
    local_rank = int(os.environ["LOCAL_RANK"])
    global_rank = int(os.environ["RANK"])
    world_size = int(os.environ["WORLD_SIZE"])
    torch.cuda.set_device(local_rank)
    dist.init_process_group("nccl")
    return local_rank, global_rank, world_size

That’s the skeleton. The real tuning comes in the dataloader settings and gradient accumulation logic.

FAQ: Straight Answers to What People Actually Ask

Q: What are the basics of distributed training if I only have two GPUs?
Use DDP. Set batch size per GPU to what fits, then double your global batch. Adjust learning rate. Ensure both GPUs are connected via PCIe or NVLink. You’ll get near 2x speedup.

Q: Do I really need InfiniBand?
For models >1B parameters and >4 GPUs, yes. For smaller models, 100GbE RoCE is sufficient. Test with your workload using a cloud trial.

Q: How do I choose between data parallelism and FSDP?
If your model fits on one GPU with spare memory, use DDP. If it barely fits, use FSDP with FULL_SHARD. If it doesn’t fit at all, you need tensor + pipeline parallelism.

Q: Can I mix data and model parallelism?
Yes, modern frameworks (Megatron-LM, DeepSpeed, FSDP) support hybrid approaches. Typically data parallelism across nodes, tensor/pipeline parallelism inside nodes.

Q: What’s the most underrated performance factor?
CPU memory bandwidth for data loading. If your dataloader can’t feed the GPUs fast enough, you’ll see GPU utilization drop below 50%. Use memory-mapped files or pre-fetch in separate processes.

Q: How do I benchmark my cluster’s networking speed?
Use NCCL’s all_reduce_bench. Install nccl-tests and run:

bash
mpirun -np 8 ./build/all_reduce_perf -b 128M -e 8G -f 2 -g 1 -n 100

That tests intra-node all-reduce performance. For inter-node, use -host flags.

Q: Should I care about CUDA graphs?
If your model is stable in compute graph (no dynamic shapes), yes. CUDA graphs reduce kernel launch overhead by 15-25% for small models. For large models, the gain is smaller but still free lunch. Enable with torch.cuda.CUDAGraph.

Q: Is renting still viable in mid-2026?
Yes, more than ever. Spot prices have dropped ~20% since 2024. Vast.ai: Rent GPUs offers on-demand H100 clusters at competitive rates. Just add checkpointing.

Conclusion

The basics of distributed training aren’t complicated in theory, but brutal in practice if you ignore the networking, parallelism strategy, and scaling laws.

What are the basics of distributed training? They are: choose the right parallelism for your model size (data for small, FSDP for medium, hybrid for large), ensure your cluster’s intra-node and inter-node bandwidth doesn’t bottleneck all-reduce, and tune hyperparameters systematically. Start small—two GPUs and a profiling tool—then scale up. And for the love of GPUs, benchmark your networking before training a 30-day job.

At SIVARO, we’ve helped teams move from a single-GPU prototype to a 64-GPU production pipeline in two weeks. It’s not magic. It’s understanding where the latency lives and cutting it out.

Stop guessing. Profile. Then scale.

—

Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.