SIVARO
GPU Oversubscription Admission Control Risks and Mitigation
Two years ago I watched a fintech client burn $47,000 in a single weekend because their GPU nodes kept spinning up while requests piled up behind a broken queue. Classic case of GPU oversubscription admission control risks and mitigation being an afterthought. They'd autoscaled aggressively, assumed Kubernetes would sort the rest, and learned the hard way that GPUs aren't CPU.
Quick definition before we go deeper. GPU oversubscription means you're scheduling more work onto a GPU (or a pool of GPUs) than the hardware can physically execute concurrently, betting on statistical multiplexing to fill the gaps. Admission control is the gate that decides which requests get in. Get the gate wrong and you've either got idle silicon or OOM-killed pods at 3 AM. This piece covers what breaks, why it breaks, and the specific patterns I've used at SIVARO to keep it from breaking.
Why oversubscription exists in the first place
GPUs are expensive. An H200 runs about $30-40K per node today; an 8xH100 box on-demand on AWS is north of $100/hour in most regions. If you're running LLM inference, utilization on a per-request basis is brutal. A single chat completion might occupy a GPU for 200ms of actual compute and 400ms waiting on KV cache growth, token streaming, and network. Idle time is money going nowhere.
Most teams respond by oversubscribing. Put three inference replicas on one A100. Share MIG slices across tenants. Cram batch jobs alongside real-time traffic. It works — until it doesn't.
The "until it doesn't" is where admission control comes in. And this is the part most teams get wrong.
How GPU oversubscription admission control risks and mitigation actually connect
Most people think admission control is just a rate limiter. It's not. It's the mechanism that decides whether the system accepts more work than it can serve, and it has to be aware of GPU-specific constraints that don't exist on CPU.
Here's what I mean. On CPU, if you oversubscribe 4:1, the OS scheduler time-slices. Latency degrades gracefully. On GPU, you get three outcomes and none of them are graceful:
- VRAM exhaustion. New request comes in, model weights are already loaded, KV cache needs 8GB more, and there's nothing left. CUDA OOM. The whole process dies, not just the request.
- Compute starvation. Scheduler accepts the request, but SM occupancy is already at 97%. Latency for the new request goes from 180ms to 4 seconds. P99 explodes.
- Memory bandwidth contention. Both of the above are fine until you're sharing HBM. Two workloads pulling 2TB/s each on an H100 with 3.35TB/s available means both get half. Chat latency doubles.
Admission control has to model all three. A naive concurrency limiter doesn't.
The autoscaling trap
Here's the contrarian take: gpu node autoscaling vs queue admission control cost is not a real tradeoff. Most teams think they need to pick one. They don't. Autoscaling without admission control is how you get a $47K weekend. Admission control without autoscaling is how you get 40% average utilization and angry customers when the queue backs up.
I tested this in early 2025 with a customer running Llama 3.3 70B inference. Their setup: HPA on GPU utilization, scale-up when >70%, scale-down when <40%. Twenty-minute cold start because the model had to reload from S3 each time.
What actually happened: traffic spiked at 9 AM Pacific, HPA fired, nodes started spinning up. Twenty minutes later, they were ready — but the spike had passed. Nodes idled, HPA scaled down. New spike at 9:45, same dance. They paid for 38 node-hours that day and served maybe 12 hours of real traffic.
The fix wasn't a better autoscaler. It was a queue with admission control in front of it, so requests waited in a bounded buffer while a much smaller, stabler GPU fleet served them. Utilization hit 71%. Latency P50 went from 900ms to 1.4s. P99 dropped because we stopped thrashing. Yearly cost dropped 41%.
Queue theory GPU scheduling for LLM inference
You can't talk about admission control seriously without queue theory. The M/M/c model — Poisson arrivals, exponential service, c servers — is the starting point, but LLM inference breaks the M/M/c assumptions in specific ways.
Token generation has variable service time that correlates with output length. It's not exponential. It's more like a heavy-tailed distribution. Prefill and decode have different cost profiles. Batch size affects throughput nonlinearly.
What this means practically: you can't just set concurrency to (GPU memory / model size) and call it done. You have to measure the actual service time distribution at your concurrency level and back out the queueing behavior.
Something like:
python
import numpy as np
from scipy import stats
# Measured on A100 80GB, Llama 3.1 8B, batch=32, 512-token outputs
# Service time samples in seconds
service_times = np.array([1.2, 1.4, 1.1, 2.8, 1.3, 4.1, 1.2, 1.5, 1.9, 1.1])
# Don't use mean. Use P95 for admission decisions.
p50 = np.percentile(service_times, 50) # 1.4
p95 = np.percentile(service_times, 95) # 3.7
p99 = np.percentile(service_times, 99) # ~4.1
# Little's Law: L = λW
# If you want P95 latency under 5s and P95 service is 3.7s
# max_in_flight = target_latency / p95_service_time
target_p95_latency = 5.0
max_in_flight = int(target_p95_latency / p95)
print(f"Concurrency cap: {max_in_flight}") # 1 — brutal, but honest
That output — concurrency cap of 1 — is why people oversubscribe. It's also why they get burned. The right answer is usually to reduce service time variance via batching, or accept higher P95, not to pretend the math away.
Failure modes I've actually seen
Let me get specific. These are real incidents, sanitized.
The MIG surprise. A media company ran 7 MIG slices on an A100 80GB. Each slice had ~10GB. Their model needed 11GB at load time. Every seventh request, done. This wasn't oversubscription in the classic sense — it was mis-sized slices. Admission control caught it when we added a pre-flight memory estimator that rejected requests the slice couldn't serve. Should have caught it earlier.
The shared cache stampede. Multi-tenant setup, shared KV cache across requests with prefix caching. Tenant A sent a huge prefix-shared batch, evicted Tenant B's cache. Tenant B's next request had to re-prefill, spiked GPU memory, took down the node. Admission control per-tenant would have prevented it. We now enforce per-tenant VRAM quotas in the admission layer.
The autoscaler death spiral. Karpenter was scaling on nvidia_gpu_duty_cycle from DCGM. Metric had a 30-second scrape interval. Traffic pattern was 20-second bursts. Autoscaler saw the spike after it was over, scaled up, then scaled down two minutes later. Cost per request was 6x normal. Fix: switch scaling decision to a queue-depth metric with smoothing.
The warm pool that never warmed. Team pre-loaded models on a warm pool of nodes to avoid cold starts. Warm pool had no admission control in front. Guess what happened. Everyone hit the warm pool, it exhausted, and the "cold" path was now the primary path. Warm pool became just an expensive cold pool.
Building admission control that works
The admission controller I build for clients sits in front of the GPU fleet and answers one question: should this request be accepted now, queued, or rejected?
The decision needs input from three places: current GPU state (memory, SM occupancy, HBM bandwidth), queue state (depth, arrival rate, service rate), and request metadata (priority, tenant, expected resource profile).
python
from dataclasses import dataclass
import time
from collections import deque
@dataclass
class GPUState:
free_vram_gb: float
sm_utilization: float
hbm_bandwidth_used_pct: float
@dataclass
class RequestProfile:
tenant: str
priority: int
est_vram_gb: float
est_duration_p95_s: float
class AdmissionController:
def __init__(self, queue_window_s: int = 30):
self.recent_completions = deque(maxlen=200)
self.recent_arrivals = deque(maxlen=200)
self.queue_window_s = queue_window_s
def _arrival_rate(self) -> float:
now = time.time()
recent = [t for t in self.recent_arrivals if now - t < self.queue_window_s]
return len(recent) / self.queue_window_s if recent else 0.0
def _service_rate(self) -> float:
# Completions per second across the fleet
now = time.time()
recent = [t for t in self.recent_completions if now - t < self.queue_window_s]
return len(recent) / self.queue_window_s if recent else 0.001
def decide(self, req: RequestProfile, state: GPUState) -> str:
arrival = self._arrival_rate()
service = self._service_rate()
rho = arrival / service # utilization factor
# Reject early if this request can't fit anywhere
if req.est_vram_gb > state.free_vram_gb:
# Allow high priority to preempt queued lower-priority work
if req.priority >= 8:
return "preempt"
return "reject"
# If we're past 85% utilization, don't accept normal traffic
if rho > 0.85 and req.priority < 5:
return "queue"
# Reject outright if we're past 95% — even priority traffic queues
if rho > 0.95:
return "queue"
return "accept"
Note the utilization thresholds. 85% is where I usually set the "start queueing normal traffic" line. Real systems degrade nonlinearly past about 80% utilization in a queue. Go read any M/M/c table — the tail latency curve is basically flat until it isn't, then it's a wall.
The 85% rule
I said 85% above. Let me justify it — this is the number I've converged on across maybe 20 productions systems since 2023.
At 70% utilization, an M/M/c queue with c=8 servers has a P99 latency of about 3x the P50. At 85%, it's roughly 8x. At 95%, it's 30x or worse. At 100%, it's infinite, because you're in the unstable region where arrivals exceed service capacity.
LLM inference is worse than M/M/c because service times are heavy-tailed. You want to sit under 80% if you care about tail latency. You flash above it, you don't live there.
The counterargument is utilization. Yes, at 80% you're burning 20% of capacity on headroom. Yes, that costs money. But run the math on what a P99 violation costs you in customer churn or SLA penalties, and the headroom pays for itself.
Multi-tenant fairness is where this gets hard
Single-tenant admission control is a math problem. Multi-tenant is a politics problem dressed up as math.
If Tenant A sends 10x the traffic of Tenant B, and you run a single global queue, Tenant B is going to be sitting behind Tenant A's queue depth forever. Their P99 will be awful. And when they complain, the answer "well, you sent less traffic" isn't going to fly.
The fix is weighted fair queueing (WFQ) or a variant. Each tenant gets a share of GPU capacity, admission control enforces a per-tenant concurrency cap, and unused capacity is borrowed from lower-priority tenants dynamically.
python
# Simplified WFQ admission for multi-tenant GPU sharing
TENANT_WEIGHTS = {
"prod-tier-1": 0.5, # 50% reserved
"prod-tier-2": 0.3,
"batch": 0.2,
}
def tenant_concurrency_cap(tenant: str, fleet_gpus: int, base_concurrency: int) -> int:
weight = TENANT_WEIGHTS.get(tenant, 0.05)
reserved = int(fleet_gpus * base_concurrency * weight)
# Borrowing: allow up to 1.5x reserved if other tenants are idle
return max(1, int(reserved * 1.5))
This isn't perfect. Borrowing is exploitable — a tenant can burst at the exact moment another tenant is quiet and grab more than their share. Guardrails: cap borrowing at 1.5x, apply a short decay, and be prepared to preempt on the next higher-priority arrival.
When admission control is the wrong answer
I'll be honest — sometimes the right move is to fix the underlying system instead of papering over it with admission control.
If your P95 service time is 8 seconds for a 200-token completion, admission control is going to be brutal. Your options aren't accept-queue-reject; they're "your inference is too slow, go fix prefill batching, quantization, or speculative decoding."
If your autoscaler takes 20 minutes to add a node, admission control is going to be doing all the work. Fix the cold start — persistent model cache, pre-warmed images, faster weight loading from local NVMe.
If your model is 70GB and you're running it on an 80GB card, you have 10GB for KV cache and admission control just decides how fast you fail. Get a bigger card or use a smaller model.
Admission control is a tool for handling the regime where your system fundamentally works but traffic is bursty. It's not a band-aid for a system that's already broken.
Observability that actually helps
You can't tune what you can't see. The metrics I insist on for any GPU fleet with admission control:
Queue depth by priority class. Not aggregate. Per-class. If your priority-1 queue is growing and priority-5 is flat, you've got a priority inversion somewhere.
Admission decision histogram. Accept / queue / reject, tagged by tenant and priority. A 30% reject rate is a signal. So is a 95% accept rate — you're underutilizing.
Time-in-queue percentiles. P50, P95, P99. Not average. Means lie about queues.
GPU memory headroom. Min across the fleet, not average. The average is fine right up until the busiest node OOMs.
Service time P95 by model and batch size. This is your model input for admission thresholds. If service time drifts, your thresholds are wrong.
We pipe these into a small controller that auto-tunes the admission threshold. If queue depth is high but GPU utilization is under 70%, the controller knows we're leaving money on the table and raises the concurrency cap. If P99 latency crosses a threshold, it drops it. At SIVARO we've been running this pattern since mid-2025 and it's saved clients from a lot of manual tuning.
What I'd tell a team starting today
Don't build this from scratch. Start with vLLM's built-in admission controls and Ray Serve's queueing layer. Both have gotten dramatically better in 2026. vLLM's scheduler handles PagedAttention-backed memory accounting for you; Ray Serve has decent fair queueing.
What you'll still need to add: cross-fleet state (they're both single-node-ish in their defaults), multi-tenant weights, and a rejection policy that doesn't surprise your clients. Also, a way for the autoscaler to see queue depth, not just GPU utilization.
Instrument first. Measure your P95 service time at realistic batch sizes. Figure out what 85% utilization actually looks like in your system. Then set thresholds. Then — and only then — write the admission controller.
FAQ
Is GPU oversubscription always bad?
No. It's how you get utilization above 40% on most inference workloads. It's oversubscription without admission control that's bad. The two go together.
Does MIG solve oversubscription?
Partially. MIG gives you hard memory and compute isolation, which removes the "one tenant takes down the node" failure mode. But it also fixes slice sizes, which means you can't dynamically reallocate. MIG is a good floor, not a complete solution.
How does queue theory GPU scheduling for LLM inference differ from classic M/M/c?
Service time isn't exponential — it's heavy-tailed, correlated with output length, and depends on batch composition. You can't use standard M/M/c results directly. Measure actual distributions and use simulation or empirical thresholds.
What's a good rejection rate?
Under 5% for interactive traffic. Under 1% for premium tier. Batch traffic can tolerate 20%+ rejection if it auto-retries.
Can I just use a rate limiter instead of full admission control?
You can, but you'll be leaving utilization on the table and you won't handle variable-cost requests. A rate limiter says "100 req/s." Admission control says "accept if this specific request fits the current state."
How does admission control interact with autoscaling?
Autoscaling changes capacity. Admission control changes the acceptance decision. They should share the same signal — queue depth relative to service rate — or they'll fight each other.
Is this different for training vs inference?
Very. Training jobs are long-running and typically hand-scheduled. Admission control matters most for inference, especially interactive.
What about KV cache as the bottleneck, not VRAM or compute?
Yes, this happens constantly with long-context models. KV cache is the real capacity limit. Admission control has to model KV cache growth per request, not just peak memory. vLLM's automatic prefix caching gives you hooks into this.
Where this is heading
Through 2026 we're seeing more hardware-level isolation — Blackwell and beyond have better MPS-style partitioning, and NVIDIA's Multi-Instance GPU is getting more granular. That helps, but doesn't eliminate the need for admission control. The fundamental problem — that you're multiplexing requests of variable cost onto expensive accelerators with hard failure modes — is going to exist as long as GPUs exist.
On the software side, expect to see more built-in admission control in serving frameworks. vLLM and TensorRT-LLM are already moving this direction. Ray Serve has fair queueing. The piece that's still missing is cross-fleet, tenant-aware admission that talks to the autoscaler. That's where I'm spending my engineering time in 2026, and where I expect most teams building serious inference platforms to spend theirs.
The TL;DR of GPU oversubscription admission control risks and mitigation: oversubscribe, but accept requests through a gate that knows your memory, your queue, and your tenants. Measure service time, don't assume it. Set your concurrency cap from P95, not mean. And remember that the 15% headroom you're giving up on utilization is buying you tail latency, and tail latency is what your customers actually feel.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.