SIVARO
GPU Cluster Capacity Planning with Queueing Theory
I spent three weeks in early 2025 watching GPUs idle while users screamed. We had 512 H100s, a waiting list a mile long, and yet the cluster ran at 40% utilization. The queueing metrics said we were fine. The users said we were terrible. Both were right.
Here's the thing nobody tells you about GPU cluster capacity planning with queueing theory: the math works perfectly until humans enter the picture. Then it gets interesting.
This guide covers what queueing theory actually predicts about GPU clusters, where it breaks down, and how to build a capacity plan that survives contact with real workloads. You'll learn the specific equations I use, the metrics that matter, and the hard trade-offs between latency and throughput that no vendor slides deck will show you.
The Core Problem: GPUs Aren't Servers
Most capacity planning advice comes from the web server world. That world is stateless, ephemeral, and forgiving. GPU clusters are none of those things.
A web request lives for milliseconds. A GPU job lives for hours. A web request can be retried infinitely. A GPU job that gets preempted might have burned 30 hours of checkpoint progress. A web server can handle 10,000 concurrent requests. An H100 can handle one model training run, maybe two if you're clever with MIG.
That changes everything about how you apply queueing theory.
The standard model — M/M/c queues — assumes Poisson arrivals and exponential service times. GPU workloads are the opposite. Training jobs arrive in bursts (someone's deadline, someone's paper submission) and service times are deterministic (a 7-day training run is a 7-day training run). The variance is in the arrival process, not the service process.
I've found Kendall's notation still works if you're honest about what you're modeling. Don't call it M/M/c. Call it G/G/c with high variance arrivals and near-zero variance service. The equations get uglier but they get useful.
Why Your Queue Metrics Are Lying to You
Most people look at average queue length and average wait time. Those numbers hide the real problem.
Here's what I mean. Your cluster has a 200-job queue. Average wait time is 4 hours. Sounds fine, right? But the distribution is bimodal. 180 of those jobs are short inference tasks waiting 20 minutes. The other 20 are 30-hour training runs that will wait 8 hours because the scheduler keeps squeezing them behind "urgent" short jobs.
The average tells you nothing. The percentile distribution tells you everything.
I learned this the hard way. In March 2025, our team at SIVARO was helping a financial services client (I can't name them, but they trade a lot of derivatives) size a cluster for their quant research team. The vendor's capacity plan said 128 H100s would give them a 95th percentile wait time of under 2 hours. Three weeks after deployment, the quant team was threatening to leave because their 95th percentile wait was 9 hours.
What happened? The vendor modeled homogeneous jobs. The reality was 90% short backtesting jobs and 10% massive simulation runs that pinned entire nodes. The short jobs stacked up behind the long ones, and the percentile curve looked like a hockey stick.
The fix wasn't more GPUs. It was smarter scheduling. But we'll get to that.
The Heavy Traffic Approximation: Your Real Capacity Ceiling
Here's the equation that actually matters for GPU cluster capacity planning with queueing theory. It's called the heavy traffic approximation, and it works when your utilization is high — which is when you actually care about queueing:
Lq ≈ (ρ² / (2(1-ρ))) × (Ca² + Cs²) / 2
Where:
- Lq = expected queue length
- ρ = utilization (arrival rate × service time / number of servers)
- Ca = coefficient of variation for arrivals (std dev / mean)
- Cs = coefficient of variation for service times
For GPU workloads, Cs is usually close to 0 (deterministic service). Ca is where the pain lives.
Let me give you concrete numbers. Say you have 64 GPUs, each job takes 4 hours, and you receive 15 jobs per hour. Your utilization is 15 × 4 / 64 = 0.9375. That's 93.75% utilization.
If arrivals were perfectly regular (Ca = 0), your queue would be:
Lq = (0.9375² / (2 × 0.0625)) × (0 + 0) / 2 = 0
Zero queue! Perfectly smooth arrives, no waiting. Never happens.
If arrivals are Poisson (Ca = 1):
Lq = (0.9375² / (2 × 0.0625)) × (1 + 0) / 2 = 7.03
Seven jobs waiting on average. Each job is 4 hours, so average wait is 28 hours. That's brutal.
But real GPU workloads are worse than Poisson. Arrivals cluster. Your Ca might be 2 or 3. At Ca = 2:
Lq = (0.9375² / (2 × 0.0625)) × (4 + 0) / 2 = 28.1
Twenty-eight jobs waiting. Average wait of 112 hours. The cluster is 93.75% utilized and your users want to riot.
This is the core insight of GPU cluster capacity planning with queueing theory: throughput and latency pull in opposite directions, and the curve is nonlinear as hell.
The Throughput-Latency Tradeoff: Pick Your Poison
Most people think they want both. They're wrong.
In October 2025, I was on a call with a team from a robotics company (they build warehouse automation, they're profitable, they're growing fast). They had 32 H100s and they wanted to add 32 more because "queue times were unacceptable."
I asked one question: "What's your average GPU utilization right now?"
Pause. "About 95%."
"Then adding GPUs won't help. You're not bandwidth-constrained, you're scheduling-constrained."
They were running every job as a dedicated single-GPU job with a priority based on submission time. Long jobs starved short jobs. Short jobs starved in absolute terms, even though they'd finish in 5 minutes if they ever got a GPU.
We moved them to a preemptive scheduling model with gang scheduling for their multi-GPU jobs. Three weeks later, 95th percentile wait time dropped from 4 hours to 22 minutes. Same hardware. Utilization stayed at 94%.
Here's what I tell every team now: you have three knobs — utilization, wait time, and fairness. Pick two. You can't have all three. If you optimize for utilization, wait times balloon. If you optimize for wait time, you'll under-utilize (packing slack). If you optimize for fairness, both metrics suffer equally.
The trick isn't finding the magic equilibrium. It's deciding which failure mode hurts less.
For most production AI teams, I recommend this: cap utilization at 85-90% and let queueing theory compute your expected wait times from there. Under 85%, you're wasting money. Over 90%, you're gambling that your arrival process stays calm.
Where Queueing Theory Sizes Your Cluster: A Practical Recipe
Let me give you the actual steps I use when I sit down with a client to do GPU cluster capacity planning with queueing theory. I've done this at SIVARO for 14 clients since 2023. The recipe hasn't changed.
Step 1: Measure Your Actual Arrival Process
Don't guess. Pull three months of scheduler logs and compute the coefficient of variation for arrivals by hour of day, day of week, and by job class.
Here's Python code I use to compute this:
python
import pandas as pd
import numpy as np
def compute_arrival_cv(job_log_path):
"""Compute coefficient of variation for job arrivals."""
df = pd.read_csv(job_log_path)
df['submitted_at'] = pd.to_datetime(df['submitted_at'])
# Bucket arrivals by hour
hourly = df.set_index('submitted_at').resample('1H').size()
# Remove empty hours (scheduler down, etc.)
hourly = hourly[hourly > 0]
cv = hourly.std() / hourly.mean()
mean = hourly.mean()
p95 = np.percentile(hourly, 95)
return {
'mean_arrivals_per_hour': mean,
'cv_arrivals': cv,
'p95_arrivals_per_hour': p95,
'peak_to_mean_ratio': p95 / mean if mean > 0 else float('inf')
}
If your CV is above 1.5, you have bursty arrivals. If it's above 2, your schedule assumes a Poisson process and your whole capacity plan is fiction.
Step 2: Model Service Times Separately From Wait Times
Service time is the job's actual runtime once it gets a GPU. Wait time is how long it sits in the queue. These are different random variables and people conflate them constantly.
python
def capacity_planning(mean_runtime_hours, arrivals_per_hour, num_gpus,
target_p95_wait_hours, cv_arrivals=1.0):
"""
Compute required GPU count given target p95 wait time.
Uses M/G/c approximation for deterministic service times.
"""
rho = arrivals_per_hour * mean_runtime_hours / num_gpus
# Kingman's approximation for G/G/c
ca_sq = cv_arrivals ** 2
cs_sq = 0.1 # Approximate for deterministic service with some noise
# Expected queue length
lq = (rho ** 2 / (2 * (1 - rho))) * ((ca_sq + cs_sq) / 2)
# Expected wait time (hours)
wq = lq / arrivals_per_hour
# p95 wait time approximation: exponential tail assumption
p95_wait = -wq * np.log(0.05)
return {
'utilization': rho,
'avg_wait_hours': wq,
'p95_wait_hours': p95_wait,
'meets_target': p95_wait <= target_p95_wait_hours
}
# Example: 10 jobs/hour, 4 hour average runtime, target 2 hour p95 wait
result = capacity_planning(10, 4, 40, 2.0, cv_arrivals=1.5)
print(result)
This isn't exact — nothing in capacity planning is — but it's close enough to make decisions.
Step 3: Simulate Before You Buy
Queueing theory gives you the shape. Simulation gives you the details. You should always do both.
In June 2025, we built a trace-driven simulator for a healthcare AI company (they do medical imaging analysis, they needed HIPAA-compliant infrastructure, which adds its own challenges). We fed their last 6 months of job traces into a discrete-event simulator — we used SimPy because it's Python and easy to extend — and tested different GPU counts from 16 to 128.
The queueing theory said 64 GPUs would give a 95th percentile wait of 1.5 hours, assuming CV of 1.2. The simulation said 84 GPUs for the same target, because the real arrival process had weekly seasonality the CV didn't capture.
We went with 96. Six weeks later, the 95th percentile wait was 1.8 hours. Close enough.
The Scheduling Side: Queue Management Best Practices
Capacity planning with queueing theory only works if your scheduler behaves like the model assumes. Most don't. Here's what I've learned about GPU cluster queue management best practices the hard way.
Preemption Changes Everything
The biggest single change you can make isn't buying GPUs. It's implementing preemptive scheduling.
If you use Slurm (which most of my clients do), here's the key configuration:
bash
# slurm.conf — enable preemption
PreemptMode=CANCEL
PreemptType=preempt/partition_prio
PreemptExemptTime=60
The PreemptExemptTime=60 means a job can run uninterruptibly for the first 60 seconds. That prevents preemption storms on tiny jobs.
Here's the catch: preemption only helps if you have checkpoints. A preempted job that loses 20 hours of work is worse than a job that waits 20 hours in the queue. You need to enforce checkpointing as a cluster policy.
We set up checkpoint intervals of 15 minutes for all training jobs at every client we've worked with. The overhead is about 2-3% of training time. The benefit is that preemption becomes nearly free.
Priority Should Come From SLAs, Not Users
The most controversial recommendation I make: user-claimed priority is a lie.
Engineers claim their job is urgent all the time. If everyone is urgent, no one is. Instead, use a simple tiered system:
- Production serving (uninterruptible, highest priority)
- Time-bounded training (deadlines from the business)
- Best-effort training (everything else)
- Idle-filling (testing, experimentation)
Map each job to a tier, and only override based on hard SLAs, not someone's mood on a Friday.
Gang Scheduling When You Need It
Multi-GPU training jobs are the bane of cluster utilization. A job that needs 8 GPUs but can only get 2 immediately sits and blocks those 2 GPUs for hours.
With gang scheduling, the job doesn't start until all 8 GPUs are available. This looks wasteful — those 2 GPUs could be doing useful work — but it prevents fragmentation from killing your effective throughput.
The math here is brutal. If your cluster is 90% utilized and jobs need 8 GPUs on average, the probability of finding 8 free GPUs simultaneously is roughly 0.1^8 = 10^-8. The job waits forever.
This is why I tell clients to run two logical clusters: one for small jobs (<4 GPUs) and one for large jobs with gang scheduling. Mixing them kills both.
The Cost Question: GPUs Are Expensive, Idle GPUs Are Fatal
Let's talk money, because that's what capacity planning ultimately comes down to.
As of August 2026, an H100 costs about $2.50-$3.00 per hour on cloud providers like AWS. An 8-GPU node is $20-24 per hour. That's $175,000 to $210,000 per node per year.
If you're running at 90% utilization, you're spending roughly $19,000 per node per year on idle capacity. That's the cost of having headroom. It's not wasted — it's insurance against arrival spikes. But you should buy exactly enough insurance, not more.
I've developed a heuristic that works well across industries: size for the 95th percentile arrival rate, not the mean. That means you'll have idle GPUs about 5% of the time, but your 95th percentile wait time will stay within budget.
Here's the code I use to size from historical data:
python
def size_cluster_from_history(job_log_path, target_p95_wait,
max_utilization=0.90):
"""Size cluster based on historical peak demand."""
import pandas as pd
import numpy as np
df = pd.read_csv(job_log_path)
df['submitted_at'] = pd.to_datetime(df['submitted_at'])
df['runtime_hours'] = df['runtime_seconds'] / 3600
# Compute GPU-hours demand per hour
gpu_demand = df.set_index('submitted_at').resample('1H')['gpus_needed'].sum()
# 95th percentile of demand
p95_demand = np.percentile(gpu_demand, 95)
# Number of GPUs needed
num_gpus = int(np.ceil(p95_demand / max_utilization))
return num_gpus, p95_demand
This gives you a starting number. Then you simulate, then you adjust for your actual arrival variability.
When Queueing Theory Breaks Down
I've been singing the praises of queueing theory, but it has limits. Know them.
The biggest one: interactive workloads. If you're running Jupyter notebooks, inference endpoints, or anything where a human waits synchronously for a response, queueing theory doesn't apply cleanly. Those need autoscaling infrastructure, not capacity planning.
Another breakdown: long-tailed service times. If your workloads are a mix of 2-minute jobs and 2-week jobs, the heavy traffic approximation falls apart. You're better off running separate queues by job class — which I recommend anyway, regardless of the math.
Finally, hardware failures. Queueing theory assumes servers don't die. GPUs die. I've seen failure rates of 1-2% per month in some clusters. That means 2-4% of your capacity needs to be spare just for repopulating nodes when a GPU dies. Don't plan for exactly N GPUs. Plan for N×1.05.
The Political Dimension No One Talks About
Here's the truth about capacity planning that I wish someone told me in 2023: it's 40% math and 60% organizational politics.
Math says you need 64 GPUs. Your research leads say they need 128. Your finance team says they'll authorize 32. The real number ends up being whatever the most powerful person in the organization decides.
I've started publishing the queueing theory modeling results as a one-page memo before every capacity decision. Not the technical appendix — the one-pager. It shows the utilization vs. wait time curve, the probability of exceeding the SLA, and the cost of each option. It doesn't eliminate the politics, but it gives the reasonable people ammunition to fight the unreasonable ones.
In 2024, I watched a cluster team burn $4 million on 128 extra GPUs they didn't need because a VP wanted to "stimulate research velocity." The utilization data said they were at 60%, the queue metrics were fine, and the extra GPUs just sat there. The VP never looked at the data. Four months later, they were quietly repurposed. Millions gone.
Your Capacity Plan Should Be a Living Document
The biggest mistake I see: teams do one capacity planning exercise, write a report, and never revisit it. Then the business changes — new models, new workloads, new data sources — and the plan is stale before anyone notices.
I recommend a quarterly review cycle. Every 90 days:
- Pull the last 3 months of scheduler logs
- Recompute arrival CVs and utilization by job class
- Compare actual wait times against queueing theory predictions
- Adjust capacity targets accordingly
This catches drift early. It's the difference between a proactive plan and a reactive firefight.
Here's a checklist I've refined over the years:
- [ ] Arrival process measured, not assumed
- [ ] Service times modeled separately from wait times
- [ ] Heavy traffic approximation computed for your utilization regime
- [ ] Discrete-event simulation run on real traces
- [ ] Preemption enabled with checkpointing enforced
- [ ] Job-tiered priorities based on SLAs
- [ ] Gang scheduling for multi-GPU jobs
- [ ] 5% spare capacity for hardware failures
- [ ] Quarterly review process established
FAQ: GPU Cluster Capacity Planning with Queueing Theory
How much GPU utilization should I target?
For production AI workloads, target 85-90% sustained utilization with bursts above 95% during peak demand. Below 85%, you're wasting money on idle GPUs. Above 90%, wait times explode nonlinearly due to the (1-ρ) term in the equations. I've seen clusters at 95% utilization with 20-hour average wait times. It's not worth the GPUs you save.
Why does my average wait time look fine but my users are unhappy?
Because you're looking at the mean instead of the distribution. GPU job wait times are heavily skewed. A 95th percentile wait time that's 5-10x your mean is normal. If users care about their worst-case experience, track and report P95 or P99, not the average.
Is queueing theory even useful for GPU clusters?
Yes, but you have to adapt it. Standard M/M/c assumes Poisson arrivals (CV=1) and exponential service times (CV=1). GPU workloads have CV>1 for arrivals and CV≈0 for service. Use Kingman's approximation or G/G/c models. It's not as tidy, but it's far more accurate than pretending your workload is M/M/c.
How often should I revisit my capacity plan?
Quarterly, minimum. The AI landscape changes fast — Llama took six months to go from 7B to 405B parameters, and every jump changes your compute ratio. Also revisit after any major infrastructure change: new GPU generation, new scheduler, new job types.
What's the best scheduler for a heterogeneous GPU cluster?
I like Slurm for HPC-style workloads and Kubernetes with the Kueue controller for cloud-native workloads. For pure training clusters, Slurm is hard to beat. For mixed training/inference, Kubernetes gets you autoscaling and better isolation. There's no universal answer, but I lean toward Slurm for dedicated GPU clusters because it handles preemption, gang scheduling, and accounting better out of the box.
Does adding more GPUs always reduce queue times?
No. If your problem is scheduling fragmentation — long jobs blocking short ones, severe gang scheduling splits — adding GPUs masks the problem temporarily but doesn't fix it. I've seen a client double their cluster and still have 3-hour queues for 10-minute jobs because the scheduler never reorders the queue. Fix scheduling first, then buy hardware.
What's the role of spot instances in capacity planning?
Buy spot instances only for fault-tolerant, preemptible workloads. Training jobs with checkpointing can use spot instances for the transient phase. But you can't plan a cluster around spot capacity — it's a supplement, not a foundation.
The Bottom Line
GPU cluster capacity planning with queueing theory isn't academic math. It's the difference between a $4 million hardware purchase you regret and one that pays for itself in six months.
Here's what I want you to take away:
The equation is simple, the application is not. Measure your actual workload, model it honestly (G/G/c, not M/M/c), simulate before buying, and build a scheduling system that matches your assumptions. Then revisit the plan quarterly like it's a security audit.
The GPUs are expensive. But the wasted GPU-hours from bad planning are worse.
At SIVARO, we've done this for 14 clients since 2023. Every single one came to us with the same problem: "our queue times are too long." None of them had actually measured their arrival process. All of them had bought expensive GPUs that didn't fix the real issue.
Don't be that team.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.