SIVARO
Queue Theoretic Admission Control GPU Cluster Example
You've got a $2 million GPU cluster idling at 40% utilization while your ML engineers scream for more capacity. Sound familiar?
I've watched this exact scenario play out at three different companies in the last two years. The knee-jerk reaction is always the same: buy more GPUs. But that's throwing money at a math problem.
Let me show you what I mean by queue theoretic admission control GPU cluster example — and why it's the difference between a cluster that feels infinite and one that's perpetually congested.
What Is Admission Control in GPU Scheduling?
Admission control is the gatekeeper between "job submitted" and "job runs." It's the decision function that asks: should we let this workload onto the cluster right now, or should it wait?
Most teams conflate admission control with scheduling. They're not the same thing.
A scheduler (like Kubernetes' default kube-scheduler or Slurm's backfill algorithm) decides where a job runs once it's accepted. Admission control decides whether it runs at all, given current cluster state and predicted future demand. Kubernetes has an --admission-control flag, but most people never touch it beyond enabling NamespaceLifecycle and LimitRanger. That's a missed opportunity.
Here's the thing nobody tells you: if you admit every job the moment it arrives, you're building a system that degrades gracefully under load. Except it doesn't degrade gracefully. It degrades like a server hitting a thundering herd — everything slows down, timeouts cascade, and your training jobs start failing with NCCL timeouts because stragglers can't keep up.
The fix isn't better hardware. It's better admission decisions, backed by queueing theory.
Why Queueing Theory? Why Now?
I know — queueing theory sounds like something from a 1970s operations research textbook. But the math is dead simple, and the payoff is enormous when you're dealing with expensive, finite resources like GPUs.
Here's the core insight: a GPU cluster is a queueing system. Jobs arrive, wait for resources, get served, and depart. The same math that predicts wait times at a bank teller predicts GPU allocation delays. The difference? A bank teller costs $15/hour. An A100 costs $30/hour to keep idle in depreciation alone.
The most useful model for admission control is the M/M/c queue — Poisson arrivals, exponential service times, c servers (GPUs). Yes, real workloads aren't perfectly Poisson. But the model gives you a baseline that's surprisingly robust for capacity planning and admission thresholds.
The key formula? Erlang-C. It tells you the probability that a job will wait longer than some threshold, given arrival rate, service rate, and number of servers.
P(wait > t) = C(c, ρ) * e^(-c * μ * (1 - ρ) * t)
Where:
c= number of GPUsρ= utilization (arrival rate / service rate * c)μ= 1 / average job durationC(c, ρ)= Erlang-C probability that all GPUs are busy
I'm not going to make you do this by hand. But here's what the math tells you that surprises everyone:
At 50% utilization, wait times start climbing non-linearly. At 80%, they explode. This isn't opinion. It's the math of M/M/c queues. The "knee" of the curve sits around 70-75% utilization for most cluster configurations.
That means if you're running your cluster at 90% utilization with no admission control, your users are experiencing queueing delays that are 10-20x worse than they'd be at 75% utilization. And the work they're submitting isn't completing faster — it's just sitting in a queue that's growling like an angry dog.
Here's where the contrarian take comes in: most Kubernetes GPU admins think their problem is "not enough GPUs." It's actually "admission control that ignores queueing theory."
The Setup: How I Frame This Problem
At SIVARO, we built an admission controller for a client's internal ML platform. The cluster: 128 NVIDIA A100s (80GB), used by ~40 data scientists and ML engineers. The symptoms were textbook:
- Average queueing delay: 45 minutes (users noticed)
- P99 queueing delay: 6+ hours (users revolted)
- Cluster utilization: 84% (the ops team was proud)
- Jobs killed due to preemption after already waiting: 20% (users quit)
The ops team wanted more GPUs. The CFO wanted to know why $1.2M of hardware wasn't enough. My team wanted to measure before we bought anything.
The first thing we did was profile actual arrival rates and service times. We pulled data from the last 90 days of scheduling events and looked at the empirical distributions.
What we found:
- Arrivals were NOT Poisson. They were bursty — peaking at 9:30 AM and 2:00 PM (people starting work after meetings).
- Service times were NOT exponential. They were closer to log-normal — most jobs were 10-20 minutes, but a few training runs took 10+ hours.
The textbook models don't perfectly apply. But they get you in the right ballpark, and that's the point. You're not building a proof for a math journal. You're building a heuristic that keeps your cluster from falling over.
Queue Theoretic Admission Control GPU Cluster Example: The Implementation
Here's the pattern we landed on. It's called threshold-based admission control with workload classification.
The idea is simple: you don't admit every job unconditionally. You classify jobs by their characteristics (duration, priority, GPU count) and apply different admission thresholds based on current cluster state and predicted queueing behavior.
Let me show you the code structure.
Step 1: Characterize Workloads
First, classify incoming jobs. We created a simple config that let the platform team define job classes.
yaml
admission_policy:
job_classes:
- name: "interactive"
max_wait_minutes: 5
priority: 100
gpu_count_limit: 4
preemptible: false
- name: "training"
max_wait_minutes: 60
priority: 50
gpu_count_limit: 64
preemptible: true
- name: "batch"
max_wait_minutes: 240
priority: 20
gpu_count_limit: 128
preemptible: true
This isn't just labels — these classes drive the admission decision. Interactive jobs get admitted only if the predicted wait is under 5 minutes. Batch jobs can tolerate a 4-hour wait, so we're willing to queue them much longer.
Step 2: Estimate Current Cluster State and Predicted Wait
The admission controller needs to know: if I admit this job right now, what's the expected wait time?
We built a simple estimator that tracks the queue length for each GPU pool, estimates service rate from historical data (running average of job completions per minute), and applies Erlang-C.
python
import math
from dataclasses import dataclass
@dataclass
class ClusterState:
total_gpus: int
busy_gpus: int
queued_jobs: int # total GPUs requested by queued jobs
avg_job_duration_min: float = 30.0
completion_rate_per_min: float = 0.8 # historical completions per minute
def estimated_wait_time(state: ClusterState) -> float:
"""Estimate wait time (minutes) using M/M/c approximation."""
utilized = state.busy_gpus / state.total_gpus
# If there's free capacity and nothing queued, no wait
free_gpus = state.total_gpus - state.busy_gpus
if free_gpus > 0 and state.queued_jobs == 0:
return 0.0
# Erlang-C probability of waiting
rho = state.busy_gpus / state.total_gpus
# Simplified: if all GPUs busy, queue drains at completion_rate
if state.busy_gpus >= state.total_gpus:
# Little's Law estimate: queue length / completion rate
estimated_wait = state.queued_jobs / (state.completion_rate_per_min * state.total_gpus)
return estimated_wait * avg_job_duration_min(state)
return 0.0
def avg_job_duration_min(state: ClusterState) -> float:
return state.avg_job_duration_min
This isn't a perfect predictor. But it's good enough to sort jobs into "safe to admit" vs "will blow up the cluster."
Step 3: The Actual Admission Decision
Here's where the queueing theory meets the pragmatism. We set utilization targets per job class.
python
def admission_decision(job, cluster_state):
"""Return True if job should be admitted now."""
# Hard constraint: don't admit jobs that will push us past 85% utilization
# if they're non-preemptible
projected_utilization = (cluster_state.busy_gpus + job.gpu_count) / cluster_state.total_gpus
if not job.preemptible and projected_utilization > 0.85:
return False, "Cluster at 85% non-preemptible utilization"
# For preemptible jobs, we allow higher utilization but enforce wait limits
wait = estimated_wait_time(cluster_state)
if job.max_wait_minutes is not None and wait > job.max_wait_minutes:
return False, f"Predicted wait {wait:.1f}m exceeds limit {job.max_wait_minutes}m"
# If we're above 90% utilization, only admit jobs that can be preempted
if cluster_state.busy_gpus / cluster_state.total_gpus > 0.9 and not job.preemptible:
return False, "Cluster over 90% utilized, only admitting preemptible jobs"
return True, "Admitted"
That 85% threshold is doing the heavy lifting. It's not arbitrary. For an M/M/128 queue with average service time of 30 minutes, staying under 85% utilization keeps the probability of waiting more than 15 minutes below 5%. Go to 90%, and that probability triples.
The Hard Lessons We Learned
The theory works. But we hit real-world complications that forced us to adapt.
Lesson 1: Your Service Time Distribution Matters More Than You Think
We initially assumed exponential service times (the "M" in M/M/c). Real data showed a bimodal distribution: many short jobs (<5 min) and a few long training runs (>2 hours).
We switched to an empirical distribution for estimating wait times. The formula changed, but the admission logic stayed the same. The key surprise: short jobs massively inflate queuing delays for long jobs unless you separate their queues.
That means separating "interactive" jobs from "training" jobs at the scheduling level. Not just in admission policy. We created two virtual GPU pools: one for interactive/short jobs, one for training. Each had its own admission controller. It was like turning one congested airport into two separate terminals — total capacity went down slightly (some GPUs sat idle in the interactive pool), but perceived performance went up dramatically.
Lesson 2: Preemption Changes the Calculus
We spent a month optimizing admission thresholds. Then the client said, "What about preemption?" They wanted to run a few very long (3-day) training runs that could be checkpointed and resumed.
Preemption is a cheat code — if you can checkpoint jobs frequently and preempt them when higher-priority work arrives, you can run at way higher utilization. We tested this and hit 92% utilization with no perceived queueing for interactive jobs. But checkpointing adds overhead, and failed preemptions eat into job completion times.
Trade-off honestly: preemption complexity was worth it for this client, but I've seen startups burn months building preemption machinery that could've been solved by just buying 8 more GPUs.
Lesson 3: Arrival Rates Are Not Stationary
The 9:30 AM spike is real. So is the post-lunch dip. Our admission controller had a single fixed threshold, which meant at 2 PM we were under-admitting (conservative) and at 10 AM we were over-admitting (causing queues).
We implemented a simple time-of-day model:
python
def predicted_arrival_rate(hour, day_of_week):
"""Return predicted jobs/minute based on historical patterns."""
if day_of_week >= 5: # weekend
return 0.1
rate_per_hour = {
8: 0.2, 9: 0.8, 10: 1.2, 11: 1.0, 12: 0.5,
13: 0.6, 14: 1.1, 15: 1.0, 16: 0.8, 17: 0.4,
}
return rate_per_hour.get(hour, 0.15) / 60.0 # per minute
We tuned admission thresholds dynamically based on hour of day. At 9 AM, we'd cap utilization at 80%. At 2 PM, we'd allow 92%. This single change cut perceived queueing delay by 30% without any additional hardware.
The Implementation Skeleton
If you're building this on Kubernetes, here's what the admission webhook skeleton looks like.
python
from flask import Flask, request, jsonify
import json
app = Flask(__name__)
@app.route('/admit', methods=['POST'])
def admit():
"""Admission webhook handler for Kubernetes."""
admission_review = request.json
pod = admission_review['request']['object']
# Extract GPU request count
gpu_count = 0
containers = pod.get('spec', {}).get('containers', [])
for container in containers:
resources = container.get('resources', {}).get('limits', {})
if 'nvidia.com/gpu' in resources:
gpu_count += int(resources['nvidia.com/gpu'])
# Get current cluster state (simplified — in practice, query a metrics store)
cluster_state = query_cluster_state() # defined elsewhere
job_class = classify_job(pod) # read labels/annotations
allowed, message = admission_decision(
job_class,
cluster_state,
gpu_count
)
response = {
"apiVersion": "admission.k8s.io/v1",
"kind": "AdmissionReview",
"response": {
"uid": admission_review['request']['uid'],
"allowed": allowed,
"status": {
"message": message,
"code": 200 if allowed else 429
}
}
}
return jsonify(response)
if __name__ == '__main__':
app.run(port=8443, ssl_context='adhoc')
The webhook itself is trivial. The hard part is the query_cluster_state() function and the patience to tune thresholds.
The Fast-Track Analysis: Do You Even Need This?
Before you spend weeks building an admission controller, do this back-of-envelope check.
If your cluster is under 60% average utilization, you don't have an admission control problem. You have a user adoption problem — people aren't submitting jobs because they don't know how, or they're afraid of the platform. Fix that first.
If your cluster is between 60-85% utilization, admission control with fixed thresholds will get you 80% of the value with 20% of the effort. Start there.
If you're above 85% utilization, you've got three options: buy more GPUs, implement preemption, or make admission control dynamic.
I'll say something heretical: sometimes buying more GPUs is the right answer. GPU prices have been volatile through 2024-2026. But in late 2025, we saw A100 prices drop 40% on the secondary market as more companies shifted to H100s and H200s. If you catch that wave, buying hardware beats building software.
At SIVARO, we always do an ROI analysis comparing the engineering cost of admission control against the straightforward cost of buying more capacity. The math is never as clean as vendors suggest — you're trading your engineers' time against hardware that's depreciating.
What to Measure First
Before you write a single line of admission control code, measure these three things:
1. Arrival rate. Jobs per hour, by job class. Track time-of-day patterns.
2. Service time distribution. Job duration from pod start to completion. Separate by job class and GPU count.
3. Current queueing delay. Time between job submission and pod scheduling. The gap between these is the pain your users feel.
Run this measurement for two weeks. You'll have enough data to build an appropriate model, and you'll probably discover that your scheduling bottleneck isn't where you think it is.
FAQ
Q: Is queueing theory still relevant with modern GPU orchestration platforms like Kubernetes and Slurm?
A: The mechanics are relevant, the implementation changes. Kubernetes already has basic admission plugins, but they don't do queueing-theoretic prediction. You're extending the platform, not replacing it.
Q: What's the difference between admission control and autoscaling?
A: Autoscaling adds resources when demand grows (if you're using Karpenter or cluster-autoscaler with dynamic GPU pools). Admission control decides whether to accept work given current resources. They're complementary — autoscaling handles slow-moving demand changes, admission control handles instantaneous bursts.
Q: Does admission control help with non-GPU workloads?
A: Yes. The same logic applies to CPU, memory, or any resource with contention. But GPUs are the clearest use case because they're (a) expensive, (b) indivisible, and (c) hard to multiplex.
Q: Everyone talks about utilization numbers. What should I aim for?
A: It depends on your tolerance for wait times. Our rule of thumb: non-preemptible workloads should aim for 75-80% utilization. With preemption and good checkpointing, 90% is achievable. Above 95% is where everything breaks down.
Q: What if I use GPUs for inference, not training?
A: Different beast. Inference workloads have strict service-level objectives — you care about p99 latency per request, not job admission delays. Admission control for inference is closer to traditional load balancing. Use a different model.
Q: Does this work on a single-GPU cluster?
A: The queueing math is simpler with one server. You're aiming for the M/M/1 model. It works, but you'll be conservative. Better to batch jobs and let one GPU churn through them.
Q: How do I handle jobs that request more GPUs than my admission controller thinks is safe?
A: In the Kubernetes example above, set a hard cap on requests per job. Anything above 64 GPUs in a single job on a 128-GPU cluster should be rejected unless it's explicitly approved. Large GPU-spanning jobs induce expensive topology-aware scheduling and NCCL communication overhead that degrades everyone else.
Q: I don't have historical data. Can I still use admission control?
A: Start with conservative estimates. Set arrival rate at 0.5 jobs/min, duration at 30 minutes, and utilization threshold at 75%. Tune upward as you observe actual behavior. Even a rough model beats blind scheduling.
The Bottom Line
Admission control built on queueing theory isn't just a "performance optimization." It's the difference between a platform your users trust and one they quietly abandon for shadow IT (renting spot instances on AWS, running jobs locally, or worse — emailing requests for someone to manually schedule their job).
We got this client from 45-minute average queueing delays to under 3 minutes for interactive jobs, without buying a single new GPU. The ops team's initial instinct to purchase 32 more A100s would've been a $300,000+ mistake.
Try this today: Write down your cluster's utilization, your average job queue time, and your job arrival rate. If your utilization's over 80% and your queue times are over 15 minutes, you've got an admission control problem. The math exists. The code patterns exist. Stop buying GPUs and start setting boundaries.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.