GPU Admission Control for Real-Time Inference: The Missing Ingredient

GPU Admission Control for Real-Time Inference: The Missing Ingredient

Here’s a number that should terrify you: P99 latency of 250ms.

That was the number we saw at a fintech client in late 2025 when their fraud-detection model started throttling. The GPU wasn't saturated. Memory wasn't full. But requests were queuing in the worst possible way. The model server was accepting work it shouldn't have, and the GPU was doing the admission control for us — poorly.

Most people think GPU inference performance is a hardware problem. It's not. It's a policy problem. And the solution is something called GPU admission control for real-time inference.

So what is it, exactly?

GPU admission control is the practice of deciding, before a request enters the GPU execution pipeline, whether it should be allowed in based on the current state of the accelerator. It's the gatekeeper. It's the difference between a system that degrades gracefully and one that collapses under load.

In this article, I'll give you the practical playbook we've built at SIVARO over the last three years. I'll cover the algorithms that actually work, the open-source tools you should know about, and share the hard-won lessons from production systems processing 200K+ events per second.

Why Your GPU Is a Terrible Traffic Cop

Here's the thing about GPUs. They're batch processors. They love parallelism. They hate latency variability.

When you send a single inference request to a GPU, the scheduler has to decide: fill the remaining capacity in the current batch, or wait for more requests to arrive to form a bigger batch? This is called dynamic batching, and it's where most systems fail.

In 2024, NVIDIA's own documentation noted that dynamic batching can improve throughput by up to 10x on certain models NVIDIA Triton docs. But here's what they don't tell you: the batching logic has zero awareness of the criticality of individual requests.

A real-time trading signal gets queued behind a batch of image-processing jobs. The GPU thinks it's being efficient. It's not. It's destroying your SLOs.

I've seen this happen at three separate companies in the last 18 months alone. And the fix is never a bigger GPU. It's admission control.

The Core Problem: It's Not About Rejection

Let me be clear about one thing upfront. GPU admission control for real-time inference is not about rejecting requests when the GPU is full.

That's the naive interpretation. And it's wrong.

If you're rejecting requests at the GPU boundary, you've already paid the latency cost of getting there. The network hop. The serialization. The model server overhead. You might as well not have a GPU at all.

Real admission control happens upstream. It happens before the request even reaches the inference server. It's a distributed decision that considers:

• Current GPU queue depths
• Model-specific latency predictions
• Request type (are we serving a trading signal or a batch analytics job?)
• Time-to-deadline for the request

The goal is not to protect the GPU. The GPU can handle more load than you think. The goal is to protect your latency SLOs.

GPU Admission Control Algorithm for Inference Servers: What Actually Works

Everyone asks me for the magic algorithm. It doesn't exist. What works is a combination of techniques applied at the right layers. Let me walk you through what we've validated in production.

Layer 1: Token Bucket with Weighted Priorities

The simplest admission control that actually does something uses a weighted token bucket. Each request type gets a weight. Real-time inference gets high weight. Batch jobs get low weight.

# Simple weighted token bucket for admission control
class WeightedTokenBucket:
    def __init__(self, capacity, refill_rate_s, weights):
        self.capacity = capacity
        self.tokens = capacity
        self.refill_rate = refill_rate_s
        self.weights = weights  # e.g., {'realtime': 1.0, 'batch': 0.2}
        self.last_refill = time.time()
        self.lock = threading.Lock()
    
    def try_acquire(self, request_type):
        with self.lock:
            now = time.time()
            # Refill tokens
            elapsed = now - self.last_refill
            self.tokens = min(self.capacity, 
                              self.tokens + elapsed * self.refill_rate)
            self.last_refill = now
            
            weight = self.weights.get(request_type, 0.1)
            if self.tokens >= weight:
                self.tokens -= weight
                return True
            return False

This works. But it's static. It doesn't account for current GPU state. Which brings us to Layer 2.

Layer 2: Queue Depth Feedback (The Game Changer)

At SIVARO, we build a system in early 2025 that we call "admission control with echo." The inference server sends a heartbeat back to the admission controller every 50ms. That heartbeat contains:

• Current queue depth per model
• Average execution time for the last 100 requests
• GPU utilization

The admission controller uses this feedback to adjust acceptance rates dynamically. The algorithm is embarrassingly simple:

python
def admission_decision(request, gpu_state):
    # Predict execution time based on model + input size
    predicted_latency = predict_latency(request.model, request.input_size)
    
    # Calculate total predicted risk (queue + execution)
    queue_risk = gpu_state.queue_estimate * average_execution_ms / 1000
    total_predicted = queue_risk + predicted_latency
    
    # SLO is 150ms. Add 20% safety margin.
    slo_deadline = 0.150 * 0.8
    
    if total_predicted > slo_deadline:
        # Check if we can backpressure or reject 
        return AdmissionDecision.REJECT_EARLY
    return AdmissionDecision.ADMIT

Note something crucial here. We're not rejecting because the GPU is saturated. We're rejecting because the predicted end-to-end latency will breach the SLO. That's a fundamentally different — and more useful — decision.

This pattern was modeled on work done by the TensorFlow Serving team at Google around request demotion, though they never shipped it as a general-purpose feature.

Layer 3: Request Demotion and Sub-SLO Queues

Here's where things get interesting.

Most people think admission control is binary: admit or reject. I'm telling you to think in three buckets:

1. Admit immediately — queue is short, GPU is fine
2. Admit with demotion — the request can be processed but with lower priority (it'll wait longer)
3. Reject immediately — the SLO is unreachable, don't waste resources

This demotion layer is the most underrated technique in GPU serving. Airbnb published a paper in 2024 on their inference gateway using exactly this approach, reducing SLO violations by 58% Airbnb's ML Inference Gateway.

Layer 4: The Overload Shedding Fallacy

Now let me be contrarian for a moment.

Most overload shedding algorithms in the literature — the ones from the microservices world — rely on response time measurements. If response time exceeds a threshold, shed load. This works fine for CPU-bound microservices.

It fails for GPUs.

Why? Because GPU execution times are non-linear with load. A GPU that's at 70% utilization can still have 5ms execution times. Push it to 80%, and suddenly you're seeing 50ms execution times. The utilization doesn't correspond smoothly to latency. The scheduler does weird things when it has to preempt contexts.

I've measured this on A100 and H100 GPUs with PyTorch and TensorRT models. The jump isn't gradual. It's a cliff. Wait — let me be careful about claiming a universal number. In our tests, the knee point varied between 75-85% depending on the model architecture. But the shape of the curve was consistent: flat, flat, flat, CLIFF.

This means response-time-based admission control will always be too late. By the time you see the latency spike, you're already getting SLO violations.

The only thing that works is predictive admission control using queue depth and known execution times.

GPU Admission Control Open Source: What's Actually Out There

There's a lot of vaporware in this space. Every company claims they have "AI infrastructure." Very few have open-sourced anything useful.

Here's what I've actually used in production:

1. Triton's Built-in Rate Limiter

NVIDIA Triton has a rate limiter that supports priority-based scheduling. It's decent for static prioritization but lacks the feedback loop you need for dynamic conditions. Documentation here.

2. KServe Queue Proxy

Knative's queue proxy (which KServe uses) has basic capacity controls. It can limit concurrency, but it's per-replica and doesn't have global GPU awareness. Useful but incomplete.

3. Ray Serve's Autoscaling + Admission

Ray Serve has some admission control via its autoscaler. In 2025 they added average queue latency as a first-class autoscaling metric. That's a step in the right direction, but it reacts rather than predicts.

4. KEDA with Custom Metrics (Our Approach)

At SIVARO, we built admission control using KEDA scalers that watch GPU queue depth via Prometheus metrics. It's not a ready-made admission controller — it's a framework for building one. That's the honest answer.

The open-source ecosystem still has a massive gap here. Nobody has built the Admission Control as a Service at the GPU layer. We considered open-sourcing our solution but kept finding use-case-specific logic we couldn't generalize. The honest truth is: you'll likely write 300-500 lines of custom code specific to your inference patterns.

Practical Implementation Recipe

Let me give you the implementation order we recommend to clients. Skip steps at your own peril.

Step 1: Instrument Everything

You can't do admission control if you can't see the current state. You need:

• Per-model queue depth (exposed via Triton's metrics endpoint)
• Execution time percentiles (P50, P95, P99)
• GPU utilization per device (via DCGM exporter)

yaml
# Prometheus rules to track inference SLO health
groups:
  - name: inference_slo.rules
    rules:
      - record: inference:queue_depth:avg
        expr: avg(triton_inference_queue_duration_us) / 1000000
      
      - record: inference:exec_time_p99:avg
        expr: histogram_quantile(0.99, 
                 sum(rate(triton_inference_exec_time_bucket[5m])) 
                 by (le, model))
      
      - record: slo:health_score
        expr: (inference:exec_time_p99:avg < 0.150)

Step 2: Build the Telemetry Backbone

You need sub-50ms feedback from the inference server to the admission controller. Long-polling won't cut it. Use a pub/sub model with streaming.

At SIVARO, we use NATS JetStream for this because it has low overhead and excellent throughput. GRPC bidirectional streaming works too.

Step 3: Implement the Admission Decision Service

This is the core. You'll need a service that takes a request descriptor (model name, target SLO, priority, input size) and the latest GPU telemetry, then returns a decision.

go
// Admission decision service (simplified)
func (a *AdmissionController) Decide(ctx context.Context, req *InferenceRequest) (*Decision, error) {
    // Get latest telemetry (cached in memory, updated every 50ms)
    telemetry := a.cache.GetTelemetry(req.ModelID)
    
    // Predict latency using historical model
    predictedLatency := a.latencyPredictor.Predict(req.ModelID, req.InputSize)
    
    // Total expected latency = queue wait + execution
    queueWait := time.Duration(float64(telemetry.QueueDepth) * 
                               float64(telemetry.AvgExecTime)/1000000.0) 
    totalExpected := predictedLatency + queueWait
    
    // SLO check with safety margin
    availableBudget := req.SLO - totalExpected
    
    if availableBudget > 0.030 { // 30ms margin
        return &Decision{Action: ACCEPT}, nil
    }
    
    if availableBudget > 0 && req.Priority >= MEDIUM {
        return &Decision{Action: ACCEPT_BUT_DELAY}, nil
    }
    
    return &Decision{Action: REJECT, Reason: "SLO unreachable"}, nil
}

Step 4: Feedback Loop and Auto-Tuning

Once you have a working admission controller, you'll find that thresholds need tuning. SLO of 150ms doesn't translate directly to an admission threshold of 150ms minus predicted latency. Models are non-deterministic.

We built a simple bandit-style auto-tuner that adjusts the safety margin dynamically based on observed violations.

safety_margin = current_margin
if violation_rate > 0.01:  # more than 1% violated SLO
    safety_margin *= 1.1  # increase margin
elif violation_rate < 0.001:
    safety_margin *= 0.95  # decrease margin slightly

Saturate at a minimum of 10ms margin and a maximum of 80ms. This simple feedback loop eliminated manual threshold tuning for our team.

The "For Real-Time Inference" Requirement: What Makes It Different?

The "real-time inference" qualifier matters. GPU admission control for batch processing is a non-problem. You can queue for seconds and nobody cares.

Real-time inference has hard deadlines. If you don't return the inference result within 150ms, the result is wrong. It doesn't matter if you're 5ms late or 5 seconds late — the decision becomes useless.

This is analogous to packet processing in networking. An old packet that arrives late is worthless.

The 5G network slicing work done by the Open RAN consortium in 2025 had similar constraints — they were dealing with 10ms deadlines for control-plane signaling O-RAN Alliance specifications. They used a hierarchical admission control scheme that's remarkably similar to what I'm describing here.

The key insight: for real-time inference, admission control isn't about throughput maximization. It's about deadline meet rate. This changes the optimization target entirely.

Edge Cases That Will Bite You

After running this in production for clients, here are the edge cases that consistently cause problems:

Model hot-weather requests. When one model receives a spike, its queue depth explodes. This affects other models on the same GPU because they share the SAME hardware. Admission control needs to be per-GPU, not per-model. We added a shared_capacity field to telemetry that tracks aggregate queue depth across models.

Temperature effects. Literal GPU temperature. Thermal throttling changes execution times by up to 40% on H100s. We discovered this after a client's datacenter cooling failed. If your probability model for execution time doesn't account for current thermals, your admission control will fail intermittently.

A/B models that don't fit. If you can't fit the model in GPU memory, all admission control logic is irrelevant. This isn't an admission control problem. Get out of that territory before you start.

A Config That Gets You 80% There

Here's a configuration structure that works across Triton, vLLM, and TensorRT deployments. It covers the basics and gives you knobs to tune:

yaml
admission_control:
  enabled: true
  mode: predictive  # predictive | reactive | hybrid
  safety_margin_ms: 25
  fallback_action: demote_low_priority
  
  metrics_source: 
    type: prometheus
    url: http://metrics.internal:9090
    scrape_interval: 15s
    streaming_feedback: true
    feedback_port: 8081
  
  slo:
    default_ms: 150
    requests:
      "/v2/models/llm/versions/1/infer":
        slo_ms: 200
      "/v2/models/fraud_detect/versions/1/infer":
        slo_ms: 100
  
  scheduling:
    types:
      realtime: 
        weight: 3
        priority_class: high
        max_concurrent: 50
      analytics:
        weight: 1
        priority_class: low
        max_concurrent: 200
    shared_capacity_reserve: 0.1  # Keep 10% headroom for emergencies

This isn't hypothetical — this is the config skeleton we drop into client deployments and iterate on over weeks.

Measuring Success: Metrics That Matter

Admission control isn't a "you have it or you don't" thing. You need to measure:

• SLO compliance rate (the only metric that matters for real-time)
• Admission rate (what percentage of legitimate requests got admitted)
• Premature rejection rate (requests rejected that would have met SLO)
• Avg queue depth under peak (was queuing bounded?)

At SIVARO, we use a composite metric called the "Admission Efficiency Score" which is the ratio of SLO-compliant requests to admitted requests. If this number drops below 0.97, something's wrong.

In a client deployment in early 2026 for a logistics company, we saw this number jump from 0.89 (pre-implementation) to 0.99 (post-implementation). And the utilization went up because we weren't wasting cycles on requests that would fail anyway.

The Costs and Honest Trade-offs

This isn't free.

• Latency overhead: Admission control adds 2-5ms to each request's total latency path. SIVARO deployments at 200K req/s on financial workloads found the overhead closer to 1ms, because the API gateway could cache the streamed state locally.
• Operational complexity: You're now debugging a distributed admission control system in addition to your inference serving.
• False rejection risk: A poorly tuned predictor will reject requests that could have made the SLO. This is worse than not having admission control at all.

I want to be crystal clear about that last point.

Badly tuned admission control can destroy your throughput. We've worked with companies that had 99.9% SLO compliance but were rejecting 40% of requests — worse than not having the system. The GPU utilization was sitting at 50% while the SLO was technically being met.

This is why we don't roll out admission control until we have solid latency prediction.

FAQ: The Answers Nobody Makes You Dig For

Does admission control work for LLM inference over tokens?

Yes, but with modifications. LLMs have variable token generation timing. The queue depth admission control I described works for models with fixed sequence lengths. For LLMs, you need to predict based on input token count because the prefill step and decode step have different GPU resource characteristics. vLLM has internal continuous batching, which makes queue depth near-zero but computation concurrency harder to reason about. Test carefully. SIVARO has found the latency prediction window does not serve LLM workloads. You need real-time telemetry on generation speed.

Can I use admission control with model ensembles?

We don't recommend enforcing admission control at the first step of the ensemble without propagating context across steps. Pipeline imbalance will kill you. Apply admission control at the entry point of the entire pipeline, not individual steps. The execution time of the pipeline is the SUM of the models, and you need per-stage GPU queue accounting.

Windows inference servers?

Admission control is platform-independent, but most of the monitoring tooling (DCGM, NVIDIA GPU operator, Prometheus GPU exporter) is Linux-first. On Windows, you'll need to build custom metrics collection that gathers utilization and queue state. It's not worth the effort unless you have a hard dependency. At SIVARO we've deployed client fleets on Azure with NVIDIA's MIG partitions on Linux containers for isolated GPU slices. Windows gets you in the weeds fast. Stick to Linux for inference nodes.

Can I just use a load balancer?

No. Load balancers distribute equally among replicas. GPU queues are non-equal. And admission control must happen before the load balancer when the GPU cluster is conjoined. We built a routing layer at SIVARO in 2025 that routes requests to the least risky GPU instance, not the least loaded — risk defined by my previous admission control equation. That reduced SLO violations by 30% compared to a bare Layer 7 load balancer.

Is there a "best" open source project for this?

No. As of 2026, live open source projects in this exact area are sparse. Remember to search when you look for the gpu admission control open source keyword on GitHub — new projects ship monthly. Companies like Machine Learning Compiler and Cohere's inference team have talked openly about building internal systems. But nothing has the ubiquity of, say, Envoy for HTTP routing. The GPU admission control ecosystem is 3-4 years behind the microservices ecosystem in maturity.

Does Kubernetes help or hurt here?

Hurt. Kubernetes autoscaling is too slow (seconds to minutes). GPU admission control requires response in milliseconds. K8s doesn't have the right primitives for this and its scheduling model is designed for long-lived workloads. We run admission control as a sidecar with an in-memory cache. It's essential to run admission control outside of the Kubernetes API server loop.

What about serverless inference, like RunPod or Modal?

If you are on a serverless infrastructure, the admission control happens at a provider level. You get less control. I estimate that within 5 years, the major serverless providers will expose the admission control configurations we are describing here as a service parameter. Some already expose them, like Baseten's scale-to-zero policies. For now, if real-time inference SLOs matter to you, you need your own infra layer.

The Final Decision: Build the Gatekeeper

Here's the brutal truth: GPU admission control for real-time inference looks straightforward until you try to run it at massive scale with 99.9% uptime under unpredictable traffic.

But it's necessary.

In 2026, we're entering the era where models are everywhere. Every bank, hospital, retailer, and manufacturer has at least one model in production. And most of them are about to discover that their GPU infrastructure is the bottleneck — not because of hardware, but because of policy.

If you are running real-time inference on GPUs, or are planning it in the next 6 months, start the work now:

1. Instrument your GPUs to expose per-model queue depths
2. Build a latency predictor for your inference server
3. Write a minimal admission controller with the three-tier decision scheme
4. Test it under load with your SLOs, and iterate on margins

Our SIVARO clients who've taken this seriously have consistently seen SLO violations cut by half or more. Those who waited for a magical one-size-fits-all open source tool are still waiting. And burning their budget on oversized GPU fleets.

The technology isn't the hard part. The discipline is.

If you are thinking about doing this now, test with your own metrics. Don't copy configurations from our examples and think they'll work. Get in the trenches with your own telemetry.

And that is the most honest answer I can give you: it's an engineering discipline problem. It's solvable. Just not by installing a single package and walking away.

---

Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.