SIVARO
Admission Control vs Autoscaling LLM Inference: The 2026 Buying Guide
You've deployed your model. The p50 latency looks great. Then one rogue customer starts sending 50 concurrent requests with 8K-token prompts, and suddenly your entire GPU fleet looks like a parking lot at rush hour.
I've seen this exact failure mode at SIVARO more times than I can count. In 2025, we helped a fintech client who'd burned three months trying to "optimize" their way out of this problem. They'd layered on autoscaling, thinking more GPUs would fix their queue buildup. It didn't. Because the issue wasn't capacity — it was admission control vs autoscaling llm inference governance.
Here's the uncomfortable truth most vendors won't tell you: autoscaling and admission control solve different problems. Autoscaling handles sustained load growth. Admission control handles bursty, unpredictable traffic that would otherwise collapse your system. If you conflate the two, you'll either overspend on idle GPUs or watch your service fall over during a traffic spike.
By the end of this guide, you'll know exactly which mechanism to deploy, when, and why — plus the operational gotchas that only show up after you've wired both into production.
Why Everyone Gets This Wrong
The default approach in 2024 was autoscaling everything. Add more replicas, scale to zero, let the cloud handle it. That worked for stateless CRUD APIs. LLM inference is anything but stateless.
Each request consumes variable compute — a 200-token prompt costs a fraction of a 32K-token prompt. Input lengths are unpredictable. Output generation is autoregressive, meaning latency compounds with every token. And unlike traditional services, you can't just reject excess load with a 503 and expect the client to retry gracefully. LLM inference clients often retry aggressively, creating a thundering herd that makes everything worse.
The admission control circuit breaker difference llm systems need isn't a binary "on or off." It's a graded response that protects the SLO envelope.
Let's get specific.
Autoscaling: What It Actually Does (and Can't Do)
Autoscaling is reactive. It watches metrics like queue depth, GPU utilization, or request count, then adjusts the number of replicas. The problem? GPU provisioning isn't instant.
Standing up a new inference pod with a 70B model loaded means pulling a container image, downloading weights (unless you've pre-baked them into the image), and loading them into VRAM. Even with a perfect setup, that's two to five minutes between trigger and readiness. Add Kubernetes scheduling delays, and you can easily hit ten minutes before a new replica serves traffic.
During those ten minutes, your existing replicas are drowning.
Here's a real example. A media company running a summarization service had autoscaling configured on p99 latency. When a viral news story hit, traffic tripled in under three minutes. Their autoscaler correctly identified the need for more replicas and started provisioning. But every new pod that came up immediately hit the same overloaded service mesh, failed health checks, and got killed before it could register.
They were paying for GPUs that never served a single request.
The fix wasn't better autoscaling. It was admission control at the front door.
Admission Control: The Unpopular Lesson
Admission control is the mechanism that decides which requests enter the system. It's not about capacity planning. It's about protecting the requests you've already accepted.
Think of it like a bouncer at an exclusive club. The autoscaler is the building manager adding more rooms. But if the bouncer lets 500 people into a 200-person venue, the rooms don't matter — the fire marshal will shut you down.
For llm inference servers, admission control has three primary functions:
- Enforcing concurrency limits — capping how many requests are in-flight per GPU
- Prioritizing requests by SLO — letting interactive traffic jump the queue ahead of batch jobs
- Rejecting excess load gracefully — with meaningful feedback to clients
The tricky part? LLMs make admission control genuinely hard. The compute cost of a request isn't known upfront. A request that generates 100 tokens costs a fraction of one generating 2,000 tokens. Traditional services can estimate cost from request size. LLMs can't.
That's why admission control for vllm inference server is its own discipline. It's not like the admission webhooks you see in Kubernetes, which validate resource requests before scheduling. It's a runtime concern.
The vLLM Admission Control Breakthrough
In late 2024, vLLM introduced request admission based on estimated output tokens from the model's own logits. This was a step forward. By predicting whether a request would generate a long response, the scheduler could make better admission decisions.
But here's what we learned at SIVARO working with clients on 100+ GPU production clusters: you can't rely solely on in-process admission control. The system gets into a degraded state before the admission controller can respond correctly. By the time the metrics say "overloaded," the interconnects are saturated and you're in the cascade.
You need admission control upstream of the inference server.
We tested this directly in 2025. One client was using a custom vLLM admission policy that rejected requests when KV-cache utilization hit 85%. The problem? Cross-node communication via NCCL had already degraded by the time that metric tripped. We moved admission to a sidecar proxy in front of the inference pods — essentially a lightweight circuit breaker with a token bucket — and saw the availability delta immediately.
The admission control circuit breaker difference llm systems need is the ability to fail fast when downstream capacity is consumed, not when it's detected as exhausted. A circuit breaker opens on consecutive failures or timeout patterns, while admission control looks at current capacity. The subtlety? For LLMs, the circuit breaker must understand timeouts, not just 5xx errors. A request that takes 30 seconds to time out is worse than one rejected outright at the admission layer.
python
# A practical admission control policy for LLM inference in 2026
from dataclasses import dataclass
from enum import Enum
class RequestClass(Enum):
INTERACTIVE = "interactive" # SLO: 2s TTFT, 95th percentile
BATCH = "batch" # SLO: 5 min, 99th percentile
BACKGROUND = "background" # No strict SLO, fill capacity
@dataclass
class AdmissionDecision:
admitted: bool
reason: str | None = None
class LLMAdmissionController:
def __init__(self, max_concurrent: int = 32,
max_estimated_decode_tokens: int = 4096,
token_bucket_rate: float = 100.0):
self.max_concurrent = max_concurrent
self.current_inflight = 0
self.max_estimated_decode_tokens = max_estimated_decode_tokens
self.token_bucket = token_bucket_rate # start with full bucket
def examine_request(self, request, model_metadata) -> AdmissionDecision:
# vLLM can estimate expected output length from input tokens
estimated_output = model_metadata.estimate_output_tokens(
request.input_tokens
)
if estimated_output > self.max_estimated_decode_tokens:
return AdmissionDecision(False, "estimated_decode_too_long")
if self.current_inflight >= self.max_concurrent:
return AdmissionDecision(False, "concurrency_limit_reached")
# Token bucket for global rate limiting across replicas
if self.token_bucket < 1.0:
return AdmissionDecision(False, "rate_limit_exceeded")
self.token_bucket -= 1.0
self.current_inflight += 1
return AdmissionDecision(True)
This isn't the exact vLLM API, but it captures the thinking. The key insight is estimating work not just counting requests.
Admission Control vs Autoscaling LLM Inference: Working Together
Here's the framework we now use at SIVARO for every inference deployment — and the one I believe every serious team should adopt:
Autoscaling scales the fleet. Admission control throttles the flow. They operate on different timescales and must have independent control loops.
Autoscaling cares about the macro trend. If your p50 latency drifts above 500ms for five consecutive minutes, add a replica. If utilization drops below 30% for ten minutes, remove one. This handles gradual growth, daily patterns, the ebb and flow of normal traffic.
Admission control cares about the micro burst. When concurrency per GPU exceeds a threshold — say, 8 in-flight requests on an A100 with a 70B model — you start rejecting or prioritizing. Not because the fleet is undersized, but because any more concurrent requests would extend time-to-first-token (TTFT) beyond your SLO.
The two interact more than you might think.
If admission control is too aggressive, your autoscaler never sees the load signal. Queue depth stays low. Latency looks fine. Your fleet stays small. Then a legitimate traffic surge comes, admission rejects 60% of requests, and you're left wondering why your service "can't scale."
If autoscaling is too aggressive, you add replicas before they're needed, driving cost up. Each replica has a minimum power footprint — even idling A100s pull hundreds of watts. Over-provisioning for bursty workloads is exactly the mistake we helped a SaaS analytics company undo in early 2026. They'd spiked GPU spend by 4x while their actual utilization stayed under 20%.
Our rule of thumb: Autoscale based on sustained utilization over a 5-10 minute window. Admission control based on real-time concurrency and estimated-decode-tokens. One is a thermostat. The other is an overpressure valve.
The Failure Mode You Haven't Considered
Here's a scenario that will bite you. Autoscaling kicks in, adds three replicas. The new replicas go through their warmup — loading the model, filling the KV cache, compiling CUDA graphs. For 60 to 90 seconds, they consume network and memory bandwidth without serving traffic. During that warmup, they actually increase contention on the nodes where they boot.
If admission control is tied to cluster-wide concurrency metrics, it sees the new replicas registering and opens the floodgates. But the replicas aren't ready. The requests land in queue. The scheduler tries to process them with cold KV caches and un-optimized weights. TTFT explodes. The circuit breaker upstream sees latency spikes, opens, and starts rejecting requests that could have been served by the healthy replicas.
We hit this with a logistics client in 2025. Their system had admission control set to 80% of total cluster concurrency. When autoscaler added a node, the admission controller calculated 80% of a larger number — automatically admitting more traffic — write before the new node was healthy. Blue Friday. The kind of outage that gets you a call from the CEO.
The fix was simple in retrospect: admission control must only consider ready replicas, not provisioned replicas. In Kubernetes, that means counting pods with the Ready=True condition, not just pods that exist.
yaml
# Kubernetes horizontal pod autoscaler with readiness gate for admission control sync
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: llm-infra-autoscaler
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: llm-inference
minReplicas: 2
maxReplicas: 16
metrics:
- type: Pods
pods:
metric:
name: llm_ute
target:
type: Utilization
averageUtilization: 45
behavior:
scaleDown:
stabilizationWindowSeconds: 300
policies:
- type: Percent
value: 50
periodSeconds: 60
A truly rigorous admission control implementation polls the Kubernetes API for readiness state before recalculating allowed concurrency. It shouldn't be a static number; it should be dynamic based on actual serving capacity.
Admission Control Circuit Breaker Difference LLM: Clarified
People ask me about the admission control circuit breaker difference llm systems expose. The standard circuit breaker opens a circuit after N consecutive failures — it's reactive to errors. Admission control is preventive — it never lets the system get to the error state.
For LLM workloads, you need both, but they must react to different stimuli:
- Circuit breaker: Trips on TTFT exceeding 3x the p99 SLO for 60 seconds, or on queuing delays exceeding a threshold.
- Admission control: Operates before you know how the system is performing. It examines the current state (inflight count, KV cache headroom) and makes a forward-looking decision.
The subtle but critical difference in LLM inference: failure is silent. A request that generates tokens slowly doesn't fail. It just produces a terrible user experience. Circuit breakers won't help because they only catch failures, not quality degradation.
That's why we set admission control not just on concurrency but on estimated KV cache latency. At SIVARO, we implemented a small ML predictor that takes current cache hit rate, sequence length distribution, and GPU interconnect utilization, then projects TTFT for the incoming request. If the projection exceeds the SLO, we reject or delegate to a lower-priority queue.
It added about 3ms overhead per request at the admission layer. Utterly trivial compared to a 25-second inference failure.
Choosing Your Admission Control Strategy by Deployment Model
Your deployment topology changes the answer. Here's the decision tree I use with clients — and it's genuinely deployment-model dependent.
Single-GPU, single-replica (prototype): You don't need admission control. You need monitoring. If you're running a demo, just let it break.
Multi-GPU, multi-replica (production, lower traffic): Light admission control. Set a generous concurrency cap per GPU. Let vLLM's native scheduler handle most of it. Your autoscaler with a slow stabilization window (5+ minutes) is the primary regulator.
Multi-node, distributed inference (heavy production): Full-stack admission control. This means an ingress-level admission component (generic infrastructure like an Envoy-based rate limiter), a vLLM-aware scheduler that understands decode-phase contention, and a global concurrency governor. This is where the admission control for vllm inference server differentiation really matters.
Geographically distributed edge deployment: Admission control must be local, but autoscaling should be global. The edge node enforces admission based on local GPU state; a central controller aggregates utilization across regions and shifts allocation rules accordingly.
Most of SIVARO's work sits in that third category. And that's where the real money is spent.
Practical Recommendations If You Only Have Time For Three Things
1. Turn on vLLM's max-memory-pool-for-kv-cache parameter and set it to the 75-80% range. This gives vLLM enough KV cache to handle long generations while leaving headroom for CPU-GPU transfers and concurrent sequence execution. Then set --max-num-seqs conservatively — I'd rather keep 16 concurrent sequences with 2,000-token context each than 32 sequences starved into memory thrashing.
2. Build an admission check into your ingress layer. This doesn't need to understand LLMs, but it does need to track concurrency. At SIVARO we use a shared Redis-based counter that tracks in-flight requests per tenant. If Tenant A crosses their concurrency entitlement (for example, 8 concurrent requests when the GPU can handle 64 total), reject their excess requests immediately with a custom 429 and a Retry-After header. Simultaneously, every request from Tenant B is admitted, regardless of how much A is hammering.
This isn't standard in off-the-shelf API gateways. You'll need to write it. But it's a few hundred lines of code and it eliminated the tenant-collision outages we saw early on.
3. Wire autoscaling to your request-level SLO, not just GPU utilization. Here's the thing: you can have 100% GPU utilization while total throughput is garbage. When the KV cache is heavily fragmented by long-context requests, the SM (streaming multiprocessor) utilization stays high but actual decode efficiency collapses. At SIVARO, we track two metrics for autoscaling: average TTFT and standard deviation of TTFT across recent requests. If both rise simultaneously, it signals KV cache pressure, which admission control can fix. If only average TTFT rises but variance is stable, it signals genuine demand exceeding the fleet, which autoscaling should fix.
Teams that wire autoscaling only to compute metrics end up over-provisioned and under-quality. The signal must encode memory state, not just compute state.
Monitoring: What to Watch and Where the Gaps Are
The scoring guides to buying inference infra always mention "observability," and here's a hard truth: current observability tooling for LLM inference is a hot mess. There are 14 vendors claiming to solve it, but most are just logs and traces repurposed for LLM ingress.
What you actually need to instrument for admission control vs autoscaling decisions:
Metric 1: Time-to-first-token per replica. At the gateway level, this metric catches KV cache pressure. But it's noisy, so I derive a rolling p95 over 300 seconds and use that. You need this at the replica level, not just fleet level.
Metric 2: KV cache utilization. These are LLM-specific metrics vLLM and SGLang now expose: kv_cache_usage, block_table_used. This tells you how much decode capacity remains. Set admission warnings at 75%, rejects at 90%.
Metric 3: Real decode token rate. The number of tokens generated per second across all active requests. If this drops while your in-flight count rises, you're in memory contention territory — but the throughput metric alone won't tell you whether adding admission control will help. You need to cross-reference with per-request attributes.
Metric 4: Effective batch size. If your inference engine is running 32 concurrent requests but they all have widely differing context lengths, the effective batch is smaller than 32. Admission control based on raw batch count will be too conservative — you'll reject requests while the GPU has memory headroom because the "batch" is actually compute-efficient.
At SIVARO, we built a small in-house sidecar that exports these custom metrics for each client. The standard Prometheus exporters just don't cut it.
Monitoring gaps are especially bad in hardware. NVIDIA DCGM exposes per-SM utilization, but not per-model concurrent decode utility. You'll need to correlate GPU memory use with your request mix to understand whether a GPU is "overloaded" or just "handling a diverse batch."
What We Learned from Deploying This at Scale
I'll end with a story about a client in the gaming industry (late 2025). They ran a 100M-user companion AI system. Their traffic spiked with every game update and every live-stream event. The variance was extraordinary — weekends were 5x weekdays, and in-game events would create 50x traffic bursts for 2-3 hours.
They initially deployed with autoscaling only. After a major launch event in November 2025, they saw p95 TTFT degrade from 1.8 seconds to 14 seconds over a two-hour window. Their autoscaler was adding 30 GPUs an hour but couldn't keep up with event-driven traffic.
We rewrote their admission to follow this construction:
- Per-replica concurrency cap: 12 concurrent requests.
- Fleet-level token bucket: 600 requests/minute burstable, reservoir of 200 requests/minute sustained.
- A non-AI circuit breaker that blacklists a replica if its STDDEV of TTFT crosses 150% of the fleet median.
On the next event (January 2026), they saw p95 TTFT of 2.4 seconds throughout — with 85% of that being intended burst rejection. That's the admission control circuit breaker difference llm systems show when tuned correctly: healthy rejects, not total failures.
Their GPU spend went up by 60% for that event window — but their supported concurrency went up 3.2x. A very good trade.
The Verdict on Admission Control vs Autoscaling LLM Inference
For the majority of production systems (beyond prototype, under moderate-to-high load), autoscaling alone is insufficient. Adoption control combined with slow-throttle autoscaling is the correct baseline architecture. The balance shifts toward admission control as your traffic unpredictability rises. It shifts toward autoscaling as your cost efficiency demands rise, provided you have provisioned headroom.
Buy a product (or build an internal tool) that gives you both control loops, run them independently, and make sure the admission control loop understands LLM-specific memory state — not just generic request counting.
If you only take one thing from this guide: Autoscaling is a budget conversation. Admission control is a trust conversation. Budget conversations you can have monthly. Trust conversations are about every single request working right now. Order of magnitude more important for production AI.
FAQ
Is admission control worth it for a single-GPU deployment?
No. Just use vLLM's native --max-num-seqs. But eventually you'll exceed a single GPU, and I'd rather you knew admission control basics before that day arrives.
Does admission control hurt throughput?
Only if misconfigured. Good admission control raises throughput by allowing the engine to operate in a healthy state where it can keep multiple decode steps per second per request. An overloaded engine's throughput collapses super-linearly — you don't get 10% less throughput at 110% load; you get 70% less.
What's the best open-source tool for admission control for vllm inference server?
Kubernetes-native admission webhooks don't work for this. Look at Envoy proxy's greedy rate limiting or a shared Redis-based request governor you write yourself. The claim that "you just need vLLM's built-in admission control" is wrong for any distributed deployment. The built-in admission control is the last gate before the engine; you want gates further upstream.
What is the admission control circuit breaker difference llm needs?
A circuit breaker protects you from damage done. Admission control prevents the damage event entirely. LLM systems need both because inference resource consumption is unpredictable — a breaker can't protect you from a slow generation that degrades all concurrent requests. Only admission control can project and reject the bad mix before it runs.
Which should I set up first, autoscaling or admission control?
Admission control. Reason: until you can protect a single healthy replica from being overwhelmed, autoscaling just adds more replicas to get overwhelmed. A system that fails fast when overloaded will keep its healthy replicas alive. That's a far better starting point than planning for optimal capacity.
Will adding admission control remove the need for more GPUs?
No. It will let you see what your real capacity is, by rejecting requests when you hit the envelope. You can then use that utilization data to justify GPU purchases — and to drive autoscaling decisions. Admission control is not an alternative to capacity; it's a lens on where capacity genuinely ends.
What latency overhead does admission control add?
At SIVARO we've seen 2-15ms overhead per request depending on implementation. A token-bucket admission decision (10 lines of code) can be sub-millisecond. A model-aware predictor that estimates decode tokens adds 3-5ms. This is meaningless next to a 200ms-to-15-second inference time. Don't optimize admission latency; optimize the decisions it makes.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.