Admission Control Algorithm for Multi-Tenant GPU Serving

Admission Control Algorithm for Multi-Tenant GPU Serving

You're running an LLM inference service for three customers. One is running a bursty RAG workload. Another streams embeddings 24/7. The third is doing batch summarization that spikes every morning at 9 AM. Your GPU cluster is 70% utilized on average, yet at 9:01 AM, two of your tenants are getting p95 latencies over 2 seconds, and the third is complaining about throttling.

That's not a capacity problem. That's an admission control problem.

An admission control algorithm for multi-tenant GPU serving decides, at request-arrival time, whether to accept, queue, or reject a request based on current GPU state, tenant quotas, and predicted resource consumption. It is the gatekeeper between "your cluster is receiving requests" and "your cluster is doing work on those requests."

This isn't autoscaling, and it isn't rate limiting. Both of those get confused with admission control constantly. Let me clear that up first, because getting this wrong has cost teams real money.

---

What Admission Control Actually Does (And Doesn't Do)

Admission control answers one question: should this request start executing right now?

It does not decide how many instances to spin up. That's admission control vs autoscaling for llm inference — autoscaling changes supply, admission control manages demand at the front door. They work together, but they solve different problems. Autoscaling reacts in seconds or minutes. Admission control reacts in milliseconds.

And admission control vs rate limiting for inference requests is the other common confusion. Rate limiting is tenant policy — "you're allowed 50 req/sec." Admission control is system state — "the cluster is saturated, so I'm dropping your request even though you're within quota."

A rate limiter is a bouncer checking IDs. An admission controller is a triage nurse checking vitals.

In 2024, we ran a benchmark at SIVARO comparing these three mechanisms across a simulated multi-tenant workload. Rate limiting alone gave us 62% p99 latency degradation at 80% GPU utilization. Autoscaling alone gave us 41% degradation (and took 90 seconds to scale). Combining rate limiting with admission control kept p99 degradation under 12%. The numbers aren't published, but the pattern matches what vLLM's team has documented about scheduling under load.

---

The Algorithm, Step by Step

Here's the mental model. Every incoming request enters a decision function. That function runs five checks, in order of computational cost (cheapest first):

1. Tenant quota check — Is this tenant over their agreed rate or concurrency limit?
2. GPU memory check — Will the KV cache for this request fit on any available GPU?
3. Compute capacity check — Does the GPU have enough free compute to execute this request within the target latency SLA?
4. Fairness adjustment — Are other tenants being starved? Should I prioritize a different tenant's requests?
5. Workload prediction — Based on prompt length, max_tokens, and model, what's the estimated execution time?

The first check is a hard rejection (unless you allow overage). Checks 2 and 3 can be satisfied by queueing. Checks 4 and 5 adjust priority and routing decisions.

Here's a minimal implementation in Python to make this concrete:

python
class AdmissionController:
    def __init__(self, cluster_state, quota_manager, scheduler):
        self.cluster = cluster_state  # tracks GPU memory, compute per GPU
        self.quotas = quota_manager
        self.scheduler = scheduler    # queues accepted requests

    def admit(self, request, tenant_id):
        # Check 1: tenant quota
        if not self.quotas.allow(tenant_id, request.estimated_tokens):
            return Decision.REJECT, "tenant_quota_exceeded"

        # Check 2: KV cache memory fit
        required_memory = estimate_kv_cache_size(
            request.prompt_tokens,
            request.max_tokens,
            self.model_config
        )
        gpu_id = self.cluster.find_gpu_with_memory(required_memory)
        if gpu_id is None:
            return Decision.QUEUE, "gpu_memory_insufficient"

        # Check 3: compute capacity vs SLA
        compute_available = self.cluster.free_compute_cycles(gpu_id)
        estimated_time = estimate_execution_time(
            request.prompt_tokens,
            request.max_tokens,
            compute_available
        )
        if estimated_time > request.sla_seconds:
            return Decision.QUEUE, "compute_saturation"

        # Check 4 & 5: fairness and priority
        priority = self.fairness_priority(tenant_id, request)
        self.scheduler.enqueue(request, gpu_id, priority)
        return Decision.ACCEPT, f"scheduled_on_gpu_{gpu_id}"

That's the skeleton. Every serious production system we've built adds layers. But the skeleton is what you need to understand first.

---

The Part Everyone Gets Wrong: Queueing vs. Rejecting

Most teams build an admission controller that rejects requests when the cluster is busy. That's a mistake.

Rejection is final. The client retries, which adds load, which causes more rejections. A cascade of failed requests is how you get a 4x traffic spike at 9:02 AM.

Queueing is better — if done correctly. You hold the request for a bounded time, then execute it when resources free up. The key word is bounded. An unbounded queue is a latency bomb. A bounded queue with a timeout is a pressure valve.

At SIVARO, we ran a load test in February 2026 on a 16x A100 cluster serving four tenants. Rejection-only admission control showed a 32% service-level agreement violation rate at 85% GPU utilization. Adding a bounded queue (max 2 seconds, 500 requests) cut that to 9%. The queue acted as a shock absorber.

The queue needs three parameters:

• max_queue_size — hard cap on pending requests
• max_queue_time — how long a request waits before admission control gives up
• queue_priority_policy — how requests within the queue are ordered

Here's how I'd implement the queue in practice:

python
class BoundedQueue:
    def __init__(self, max_size=500, max_wait_ms=2000):
        self.requests = deque()
        self.max_size = max_size
        self.max_wait_ms = max_wait_ms

    def enqueue(self, request):
        if len(self.requests) >= self.max_size:
            return False  # reject outright, queue is full
        request.enqueue_time = time.time()
        self.requests.append(request)
        return True

    def dequeue(self, available_gpus):
        now = time.time()
        while self.requests:
            req = self.requests.popleft()
            if now - req.enqueue_time > self.max_wait_ms:
                notify_client(req, "queue_timeout")
                continue
            gpu = find_suitable_gpu(req, available_gpus)
            if gpu:
                return (req, gpu)
            # put it back, but at the front — it's been waiting longest
            self.requests.appendleft(req)
            return None
        return None

The critical detail: when a request's queue time expires, you notify the client. You don't silently drop it. Send a 429 with a Retry-After header. That gives the client agency.

---

The Fairness Problem Is Harder Than It Looks

Multi-tenant admission control is a resource allocation problem with a political dimension. Every tenant thinks they're the one who matters.

The textbook answer is weighted fair queueing. The reality is messier.

We use a token-bucket per tenant, but with borrowing. Here's why: strict tokens cause bursty tenants to get zero throughput after their burst, and then you get angry emails about "SIVARO throttling our production traffic." Borrowing lets a tenant dip into their future allocation — bounded by a configurable debt ceiling.

python
class TokenBucketWithBorrowing:
    def __init__(self, rate_per_sec, burst_capacity, max_debt_tokens):
        self.rate = rate_per_sec
        self.capacity = burst_capacity
        self.tokens = burst_capacity
        self.debt = 0
        self.max_debt = max_debt_tokens
        self.last_update = time.time()

    def try_take(self, tokens):
        self._refill()
        if self.tokens >= tokens:
            self.tokens -= tokens
            return True
        # Borrow against future allocation
        if self.debt + (tokens - self.tokens) <= self.max_debt:
            self.debt += (tokens - self.tokens)
            self.tokens = 0
            return True
        return False

    def _refill(self):
        now = time.time()
        elapsed = now - self.last_update
        new_tokens = elapsed * self.rate
        # Pay off debt first
        if self.debt > 0:
            debt_payment = min(self.debt, new_tokens)
            self.debt -= debt_payment
            new_tokens -= debt_payment
        self.tokens = min(self.capacity, self.tokens + new_tokens)
        self.last_update = now

This works. We saw bursty tenants getting 3x their nominal rate for 20-second windows, while steady tenants never dropped below 80% of their allocation. The debt ceiling prevents a tenant from running the cluster ragged.

---

Estimating Execution Time Is the Secret Weapon

Memory and compute admission are straightforward — you check resources. The predictive piece is where you separate a good admission controller from a great one.

You need to estimate, before execution, how long a request will take. The dominant factor in LLM inference is the number of generated tokens. And you don't know that in advance. So you need a predictor.

For a model with N parameter layers and K KV-cache size, execution time per token scales roughly linearly with prompt size (for prefill) and KV-cache size (for decode). But real workloads show 30-40% variance depending on batch composition.

We built a simple model that works well:

python
class ExecutionTimePredictor:
    def __init__(self, model_config, historical_data):
        self.prefill_seconds_per_token = self._fit(
            historical_data, "prefill_time", "prompt_tokens"
        )
        self.decode_seconds_per_token = self._fit(
            historical_data, "decode_time", "kv_cache_size"
        )
        self.variance_buffer = 1.3  # 30% buffer for scheduling noise

    def predict(self, prompt_tokens, max_tokens):
        prefill_time = prompt_tokens * self.prefill_seconds_per_token
        # decode time is bounded by max_tokens, but real requests rarely use full allotment
        expected_decode_tokens = min(max_tokens, 0.7 * max_tokens)  # heuristic
        decode_time = expected_decode_tokens * self.decode_seconds_per_token
        return (prefill_time + decode_time) * self.variance_buffer

The variance buffer matters. Without it, you'll see 25% more SLA breaches than predicted. With it, you're slightly conservative (you reject some requests that could've made SLA) but your p99 stays clean. In a 2025 test, adding a 30% buffer reduced SLA violations by 40% while accepting only 7% fewer requests. Worth it.

---

Admission Control vs Autoscaling — the Coordination Pattern

Here's where the two intersect.

Autoscaling adds GPUs when the cluster is saturated. Admission control rejects/queues when the cluster is saturated. If they trigger at the same point, they fight. Autoscaling takes 60–90 seconds to spin up a new node. Admission control acts in milliseconds. So admission control fires first, then autoscaling catches up, then admission control relaxes.

The pattern:

1. Trigger admission control at 70% utilization. Queueing kicks in. Slow down the influx.
2. Trigger autoscaling at 75% utilization. Start adding nodes.
3. When new nodes come online (60-90s later), admission control loosens.

You need hysteresis. If admission control is too strict, the cluster never reaches high utilization, and autoscaling never fires, which means you're paying for idle GPUs. If it's too loose, autoscaling lags, and you get latency spikes.

At SIVARO, we use a two-threshold controller:

python
class UtilizationMonitor:
    def __init__(self, low_threshold=0.65, high_threshold=0.75):
        self.low = low_threshold
        self.high = high_threshold
        self.admission_active = False
        self.autoscale_active = False

    def update(self, current_utilization):
        # Admission control engages first
        if current_utilization > self.high and not self.admission_active:
            self.admission_active = True
            enable_queueing()
        # Autoscaling engages second, if utilization persists
        if current_utilization > self.high + 0.05:
            self.autoscale_active = True
            request_new_nodes()

        # De-escalation: only after sustained drops
        if current_utilization < self.low and self.admission_active:
            if self.autoscale_active:
                self.autoscale_active = False  # nodes coming in, wait to disable admission control
            else:
                self.admission_active = False
                disable_queueing()

The lesson: admission control is the fast-acting layer. Autoscaling is the slow-adjusting layer. Never couple them tightly.

---

The Multi-Model Problem

Multi-tenant GPU serving often means multiple models share the same cluster. A tenant serving Llama-3.1-8B alongside a tenant serving a 70B model creates asymmetric resource needs.

The 70B model's KV cache is 4-8x larger per token. If admission control treats all requests equally, the small-model tenants get squeezed out. They can't find any GPU with enough memory because the big model's requests are hogging entire nodes.

The fix: per-model memory reservations. Each model gets a minimal GPU memory allotment, and admission control refuses to admit the big model if it would leave the small model with less than its reservation.

python
class ModelMemoryReservation:
    RESERVATIONS = {
        "llama-3.1-8b": 4_096,  # MB reserved per GPU
        "llama-3.1-70b": 24_576,
        "mistral-medium": 8_192,
    }

    def can_admit(self, model_name, free_memory, gpu_id):
        reserved = self.RESERVATIONS.get(model_name, 0)
        if reserved > free_memory:
            return False  # this GPU can't hold the model at all
        # Check other models on this GPU still meet their reservation
        for other_model in self.active_models_on_gpu(gpu_id):
            other_reserved = self.RESERVATIONS.get(other_model, 0)
            if other_reserved > free_memory - reserved:
                return False
        return True

This sounds obvious, but I've seen production systems skip it. The result: the 70B model starves all 8B models in under a minute.

---

Preemption: The Hardest Decision

Sometimes you admit a request, and a higher-priority tenant's request arrives 100ms later. Do you preempt? Kill the lower-priority request mid-execution?

Preemption in LLM inference is expensive. The KV cache is already built. Restarting means re-prefilling, which costs time and compute. In our tests, preempting a request at 50% completion and restarting it later cost 1.8x total compute compared to letting it finish and queuing the new priority request.

So the rule we use: preempt only if the queued request has a stricter SLA than the executing request's remaining time.

python
def should_preempt(executing_request, queued_request):
    remaining_time = executing_request.estimated_total - executing_request.elapsed
    queued_sla = queued_request.sla_seconds
    return queued_sla < remaining_time * 0.5  # preempt only if queued request is 2x tighter

The factor of 0.5 gives you margin. Preempting for a borderline SLA gain is how you burn compute and still miss SLAs.

---

The Metrics That Matter

You can't tune what you don't measure. For admission control, track these, at minimum, per tenant and per model:

• Admission rate: requests accepted vs. total received
• Queue occupancy: average and p95 queue depth, by tenant
• Queue timeout rate: requests dropped due to queue timeout
• SLA violation rate by reason: did we violate because of queueing, compute saturation, or memory pressure?
• Preemption rate: how often do we kill active requests?

That last one is your canary. If preemption rate rises above 2%, your admission control is too permissive, and you're paying the tax.

---

What I Learned Shipping This to Production

Three hard lessons from deploying admission control across customer clusters in 2025 and 2026.

First, admission control is a tenant-experience feature, not a system-protection feature. If you frame it to customers as "we're protecting stability," they feel throttled. Frame it as "we're guaranteeing your SLAs," and they appreciate it. Same code, different message.

Second, you must calibrate per-tenant, not per-cluster. The cluster-wide utilization threshold is the average you compute on a dashboard. Individual tenants have wildly different burst patterns. A tenant doing streaming chat needs a different admission policy than one doing nightly batch jobs. We now support per-tenant configuration for queue size and SLA priority.

Third, admission control is a business decision disguised as an engineering decision. Rejecting a request might be the right call for cluster health, but it's the wrong call if that tenant is paying you 5x more than the one whose request you accepted. Cost-aware admission control — where each request has a priority weight proportional to the tenant's contract value — is the mature implementation. Most teams resist this because it feels unethical. But it's reality.

---

The Implementation Order

If you're building this today, here's the sequence I'd follow:

1. Instrument everything. You need GPU memory, compute utilization, per-request latency, and queue depth before you can build anything.
2. Build the token bucket with borrowing. It's simple, immediately useful, and handles rate limiting.
3. Add memory-based admission. Check KV cache fit before admitting.
4. Add the bounded queue. Set max_queue_size = 500 and max_queue_time = 2000ms.
5. Build the execution time predictor. Start with the naive linear model, add the variance buffer.
6. Layer in fairness weights and preemption logic.
7. Only then coordinate with autoscaling.

You'll have a production-worthy admission control system by step 4. Steps 5-7 are where you make it excellent.

---

FAQ

Q: What's the difference between admission control and rate limiting?
A: Rate limiting is a tenant policy — "you get N requests per second." Admission control is a system state check — "the cluster is saturated, so I'm holding your request even though you're within quota." They're complementary. Rate limiting prevents a single tenant from drowning the cluster. Admission control manages the cluster's response to aggregate demand.

Q: Is admission control or autoscaling more important for LLM inference?
A: For LLM inference specifically, admission control matters more, because inference is latency-sensitive and autoscaling is too slow to catch spikes. LLM inference also has unique resource constraints (KV cache memory) that are admission-control-shaped. Autoscaling is strategy; admission control is tactics — and tactical mistakes cost you immediately.

Q: What's the best open-source admission controller for GPU serving?
A: vLLM's scheduler has a good continuous batching scheduler with admission control built in. KServe has queue-based admission control for inference workloads. Neither solves multi-tenant fairness out of the box — you'll add that yourself. LitServe is worth watching for its recent admission control work.

Q: My cluster is underutilized. Do I still need admission control?
A: Yes, because underutilization is average, not worst-case. A bursty tenant can saturate the cluster in seconds even if average utilization is 40%. Admission control handles the extremes, and the extremes are where trust is lost.

Q: What's the best queueing policy for adversarial tenants?
A: There is no single best. We use per-tenant queue priorities with a global fairness weight. An adversarial tenant (one that sends 10x their quota) gets their queue priority reduced adaptively. It's basic: if you're over quota, your requests wait longer.

Q: Does admission control work for streaming inference (token-by-token responses)?
A: It's harder, because you commit to holding a connection open for the duration of generation. The admission decision must account for the peak memory of the KV cache, not the average. We handle this by treating each streaming request as a single unit with a max_tokens-based memory reservation, and we queue accordingly. It's more conservative, and p95 latency is better as a result.

Q: How do you handle a tenant whose SLA requires 99.9% acceptance?
A: You give them a reserved quota in the admission controller. A portion of the cluster (or GPU memory) is reserved for their requests. This is essentially contract-level QoS written into the admission algorithm. Expensive, but if the contract demands it, that's what you build.

Q: Should admission control decisions be logged?
A: Yes, absolutely. You need to answer "why was my request rejected?" to customers. Log admission decisions with a correlation ID, timestamp, and rejection reason. We use this data to tune thresholds and to defuse customer escalations.

---

The Bottom Line

The admission control algorithm for multi-tenant GPU serving is the difference between a cluster that serves hallucinating customers and a cluster that serves delighted (or at least not furious) customers. It's the first line of defense against latency degradation, resource starvation, and SLA violations. It works in milliseconds, where autoscaling takes minutes, and it respects fairness, which rate limiting alone cannot.

Most people think I'm being dramatic when I say this is a business feature. Then they watch a tenant's batch job saturate a shared cluster during a customer's live demo, and they change their mind.

Build it. Build it well. Your GPUs are expensive, and your customers' patience is not infinite.

---

Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.