Admission Control for Multi-Tenant GPU Clusters: The Gatekeeper Your Inference Stack Is Missing

Free Technical Audit

Expert Review

You've built the cluster. You've containerized the models. You've got a scheduler that places pods like a Tetris grandmaster.

And then your Monday-morning traffic spike hits, and one noisy tenant's batch job swamps the L4 cache on every A100, and your p99 latency for a critical production inference path goes from 40ms to 900ms in the span of three minutes.

That's not a scheduling problem. That's an admission control problem.

Admission control for multi-tenant GPU cluster is the policy layer that decides whether a workload gets to enter the cluster at all, and under what conditions. It runs before the scheduler places anything. It's the bouncer at the club door, not the person arranging the furniture inside.

In this guide, I'll show you what admission control actually is, how it differs from scheduling (especially for LLM inference), the algorithm we've settled on at SIVARO after two years of fighting this fight, and the exact code patterns you can steal.

What Admission Control Actually Does

Admission control is the gate. When a pod, job, or inference request arrives, the admission controller evaluates it against a set of policies:

Does this tenant have quota left?
Does this workload's resource request match reality?
Will accepting this workload violate any hard limits on the node or the cluster?
What is the blast radius if this workload misbehaves?

If the answer to any of those is "no" or "unknown", the workload gets rejected. It doesn't get queued. It doesn't get scheduled. It gets a 403 or a FailedAdmission event and the client can decide to retry or degrade.

Most people conflate this with scheduling. They're related but distinct. Scheduling answers "where does this workload go?" Admission control answers "should this workload be allowed to enter at all?"

At SIVARO, we run multi-tenant GPU infrastructure for clients who are serving LLM inference alongside training jobs and batch data processing. The difference between admission control and scheduling for LLM inference is stark. A scheduler looks at a GPU and says "there's 40GB free, place the model." An admission controller looks at the same GPU, sees that six other tenants are on it with bursty traffic patterns, and says "if we place this 8B parameter model here, the shared compute resources and memory bandwidth will cause a cascade of latency violations. Reject."

Scheduling optimizes for utilization. Admission control optimizes for predictability.

Why Your Scheduler Can't Save You

Four years ago I watched a team at a fintech company try to solve this with Kubernetes custom schedulers. They built a beautiful scoring system. It considered GPU memory, tensor core utilization, even thermal headroom. It was genuinely impressive engineering.

It failed within a week.

Why? Because a scheduler's job starts after admission. By the time the scheduler sees the pod, the decision to admit has already been made. The scheduler can place the pod on the best node, but it can't undo the fact that the pod was admitted into a cluster that was already oversubscribed.

Here's the dirty secret of GPU clusters: the resource that actually matters for LLM inference isn't memory, it's memory bandwidth and compute interference. An A100 has 80GB of HBM2e. If you've got one model using 30GB and another using 30GB, the scheduler sees "fine, 20GB left." But if those two models are doing inference concurrently, they're hammering the same memory controllers. Effective bandwidth per model drops by 60-70%. Token generation speed collapses.

Your scheduler cannot see that. It's not a fault of the scheduler — it's that the information needed to make that decision lives perpetually in the future. You can't predict interference patterns from a resource spec.

So you need a gate before the scheduler. A gate that says:

"I don't care what the scheduler thinks it can place. I know this node is running a 70B model with high QPS, and I'm not letting another latency-sensitive workload onto it. Period."

That's admission control for multi-tenant GPU clusters.

The Algorithm We Use (And What We Rejected)

Let me walk you through what we've settled on at SIVARO, after testing several approaches with production workloads in 2025-2026.

Approach 1: Pure Quota-Based (Rejected)

The simplest approach. Each tenant has a quota: "Max 8 GPUs." If a request would exceed it, reject.

What's wrong with it: Quota says nothing about interference. We had a tenant using 6 of their 8 GPUs for batch training jobs that ran at 100% utilization, and another tenant running production inference on the other 2. The inference tenant saw p99 latency spikes of 350% during training checkpoints. Quota-based admission control didn't catch it because both tenants were within quota.

Approach 2: Static Node Isolation (Rejected)

Dedicate GPU nodes per tenant. Admission control just checks if the tenant has a free node.

What's wrong with it: Massive fragmentation. We ran a simulation on 512 H100s with 14 tenants and found we'd need 40% more GPUs to maintain the same throughput. That's a non-starter economically.

Approach 3: Interference-Aware Admission Control with Budget Credits (WINNER)

This is what we run now. The core insight: every workload gets a "noise budget" — a number representing how much it's allowed to degrade performance for co-tenant workloads.

Here's the algorithm:

1. On admission request:
   a. Classify the workload type (inference, training, batch)
   b. Determine the "workload signature" - expected memory touch rate, compute intensity, burst profile
   c. Find candidate nodes based on basic memory/compute fit
   d. For each candidate node, calculate: Current total noise credit remaining
   e. If remaining noise credit >= workload's expected noise contribution, ADMIT
   f. Else, REJECT (or redirect to a different node/queue)

The "noise credit" is the key innovation. Instead of tracking GPU memory and cores, we track interference potential.

Here's what it looks like in practice:

python
# admission_controller.py - Simplified version
class NoiseCreditAdmissionController:
    def __init__(self):
        self.node_state = {}  # node_id -> NodeState
        self.workload_profiles = {
            'llm_inference': {'noise_cost': 35, 'max_burst': 2.0},
            'cv_training': {'noise_cost': 18, 'max_burst': 4.0},
            'batch_processing': {'noise_cost': 12, 'max_burst': 6.0},
            'embedding_cache': {'noise_cost': 4, 'max_burst': 1.0},
        }
        self.max_noise_credit = 100  # per node, calibrated empirically

    def admit(self, tenant_id, workload_type, gpu_requests):
        workload = self.workload_profiles[workload_type]
        
        # Check tenant quota first
        if not self.check_tenant_quota(tenant_id, gpu_requests):
            return AdmissionDecision.REJECT("Quota exceeded")
        
        # Find nodes with enough raw resources
        candidate_nodes = self.find_candidate_nodes(workload_type, gpu_requests)
        
        for node_id in candidate_nodes:
            node = self.node_state[node_id]
            
            # The critical check: will adding this workload blow the noise budget?
            projected_noise = node.current_noise_usage + workload['noise_cost']
            
            # But also: if the workload is bursty, we need headroom
            if projected_noise <= self.max_noise_credit * 0.85:  # 15% safety margin
                # ADMIT - reserve the noise credit
                node.current_noise_usage = projected_noise
                return AdmissionDecision.ADMIT(node_id)
        
        # No node can safely host this workload
        return AdmissionDecision.REJECT("Interference threshold exceeded", 
                                        retry_after=30)

The noise_cost values come from calibration. We spent a month profiling workloads on H100s with NVIDIA Nsight Compute, measuring effective bandwidth and compute throttling under contention.

Here's the brutal truth: these numbers are highly workload and hardware specific. A noise cost of 35 for a 8B LLM on H100 might be 55 on an A100. On a L4 it's a different ballgame. You need to calibrate this for your specific GPU fleet.

Admission Control as a Mechanism for LLM Inference Quality

Let's go deeper on the LLM inference angle, because it's where this matters most.

Last year at SIVARO we ran a stress test with a client. Two tenants. Tenant A: an 7B parameter model serving a customer-facing chatbot. Tenant B: a fine-tuning run on a 13B model. They were on separate GPUs, but the same NVLink-connected node.

The scheduler did its job perfectly. Both pods were placed on different GPUs. GPU memory allocated correctly. No resource contention at the memory level.

The result? The inference tenant's p99 latency went from 42ms to 190ms. A 4.5x degradation.

Why? Because even on "separate" GPUs, both GPUs share the same NVLink domains, the same node's host memory controllers, and crucially, the same PCIe switch. The training job's gradient updates hammered the PCIe bus, which the inference job needs for its KV cache transfers.

Scheduling couldn't prevent this. Admission control can.

The admission controller for LLM inference needs to ask a different set of questions than for batch jobs:

Is this a latency-sensitive workload? If so, it needs a node with low "inference pressure" — not just low GPU utilization, but low co-tenant burstiness.
What model size? Bigger models touch more memory per token, which means more bandwidth demand.
What's the estimated concurrency? A model with 100 concurrent requests has a very different interference profile than one with 10.

Here's a pattern we use for inference-specific admission:

yaml
# inference-admission-policy.yaml
apiVersion: admission.acme.io/v1
kind: InferenceAdmissionPolicy
metadata:
  name: llm-inference-latency-slo
spec:
  workloadTypes: ["llm_inference"]
  # Only admit if the noise credit headroom is enough
  noiseCreditRequirements:
    minRemainingPercentage: 25
  interferenceThresholds:
    # Block admission if co-tenant p99 latency already above SLO
    currentLatencyP99Ms: 80
    # Block admission if node has >3 tenants already
    maxTenantsPerNode: 4
    # Block admission if any co-tenant has a mutating operation type
    # (training is mutating, inference is read-only)
    blockOnMutatingCoTenants: true
  burstProtection:
    # For bursty workloads, require extra headroom
    maxCombinedBurstFactor: 1.5

The maxTenantsPerNode rule was huge for us. We found that even with noise credits, the variance of interference increases non-linearly with tenant count. Four tenants on a node is qualitatively worse than three, even if the aggregate noise credit usage is identical. It's chaotic. The admission controller needs to constrain tenant count, not just resource usage.

The Overcommit Problem and Admission Control Algorithms

Let's talk about the GDP of GPU clusters: overcommitment.

Most GPU clusters are overcommitted by 1.5x to 3x. You have more GPU requests than GPUs. Scheduling handles the placement, but admission control decides who gets in the consideration set.

At a previous engagement with a large e-commerce company, we ran a fine-grained admission control algorithm that pre-computed the "blast radius" of each workload. We modeled it like a credit risk assessment. Each workload gets a risk score based on:

The workload's historical behavior (has this tenant ever caused a node OOM or GPU hang?)
The workload's tolerance for preemption (can it checkpoint and resume?)
The criticality of the tenant (a production customer is riskier than an internal batch job)
The SLO type (latency-SLO workloads are riskier than throughput-SLO workloads)

go
// admission_risk.go - Simplified risk-based admission algorithm
type AdmissionRequest struct {
    TenantID    string
    WorkloadType string
    GPURequest  int
    Duration    string
    Preemptible bool
}

type RiskAssessment struct {
    RiskScore float64
    Reasons   []string
}

func AssessRisk(req AdmissionRequest) RiskAssessment {
    score := 0.0
    reasons := []string{}
    
    // Historical behavior penalty
    if history := getTenantHistory(req.TenantID); history.FaultCount > 5 {
        score += 25
        reasons = append(reasons, "High historical fault count")
    }
    
    // Non-preemptible, latency-SLO workloads are riskier
    if !req.Preemptible && req.WorkloadType == "llm_inference" {
        score += 30
        reasons = append(reasons, "Non-preemptible latency-sensitive")
    }
    
    // Long-running workloads have more blast radius
    if req.Duration == "long" {
        score += 10
        reasons = append(reasons, "Long duration increases collision probability")
    }
    
    return RiskAssessment{RiskScore: score, Reasons: reasons}
}

// Admission logic: admit if risk score below threshold
func ShouldAdmit(req AdmissionRequest) bool {
    assessment := AssessRisk(req)
    
    // Higher risk workloads need higher tenant priority
    effectiveThreshold := 70.0 - float64(getTenantPriority(req.TenantID))
    
    if assessment.RiskScore > effectiveThreshold {
        return false // Reject
    }
    
    // Check credit + interference
    return checkNoiseCredits(req)
}

This worked. We reduced cross-tenant interference events by 76% in the first month. But it came with an operational cost: we had to build a telemetry pipeline to track historical behavior, and we had to manually review the risk model's decisions.

The algorithm is only as good as your data. If you don't have clean logs of GPU faults and tenant behavior, this approach will let you down.

Admission Control vs Scheduling: A Clear Divide

I keep saying these are different, but let me be explicit about the division of labor.

Admission control answers:

Can this workload be admitted anywhere on this cluster?
Does this workload meet the tenants' constraints?
What is the cluster-wide impact of admitting this?

Scheduling answers:

Of the nodes where this workload could go, which is optimal?
How do I pack this workload to minimize fragmentation?
How do I balance load across nodes?

The practical implication: if your admission control is too permissive, your scheduler becomes a bottleneck. If it's too restrictive, your scheduler has nothing to do and your utilization tanks.

In our tests, the sweet spot was an admission control acceptance rate of 70-85%. Above that, the cluster was unstable. Below that, we were wasting 20% of our GPU capacity.

Implementing Admission Control in Kubernetes

The cleanest way to implement admission control for multi-tenant GPU clusters in Kubernetes is via ValidatingAdmissionWebhook. Here's the pattern:

yaml
# webhook-deployment.yaml
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionWebhook
metadata:
  name: gpu-admission-controller
spec:
  webhooks:
    - name: gpu-admission-controller.sivaro.io
      rules:
        - operations: ["CREATE"]
          apiGroups: [""]
          apiVersions: ["v1"]
          resources: ["pods"]
      failurePolicy: Fail
      sideEffects: None
      admissionReviewVersions: ["v1"]
      clientConfig:
        service:
          name: gpu-admission-controller
          namespace: kube-system
          path: "/admit"

And the admission handler:

python
# webhook_handler.py
from flask import Flask, request, jsonify
import base64, json

app = Flask(__name__)

@app.route('/admit', methods=['POST'])
def admit():
    review = request.get_json()
    pod = json.loads(base64.b64decode(review['request']['object']['raw']))
    
    # Extract GPU spec
    gpu_request = extract_gpu_request(pod)
    
    if gpu_request > 0:
        decision = admission_controller.evaluate(pod, gpu_request)
        if decision.admit:
            return jsonify({'response': {
                'uid': review['request']['uid'],
                'allowed': True
            }})
        else:
            return jsonify({'response': {
                'uid': review['request']['uid'],
                'allowed': False,
                'status': {
                    'message': f"GPU admission denied: {decision.reason}"
                }
            }})
    
    return jsonify({'response': {
        'uid': review['request']['uid'],
        'allowed': True  # Non-GPU workloads pass through
    }})

Note the failurePolicy: Fail. I've seen teams set it to Ignore for safety. That's a mistake. If your admission controller goes down and you set Ignore, all GPU requests flood in and you get the exact interference catastrophe you built the controller to prevent. Fail closed.

Advanced Patterns: Queue-Based Admission Control

We discovered early that binary admit/reject isn't enough. When a tenant gets rejected, they retry, and the retry storm creates its own problems.

Now we use a two-tier admission system. First tier is the binary decision; second tier is a delay-based admission control:

go
// delayed_admission.go
type AdmissionQueue struct {
    mu sync.RWMutex
    queue map[string][]AdmissionRequest // tenantID -> pending requests
    quota map[string]int
}

func (q *AdmissionQueue) AdmitOrQueue(req AdmissionRequest) AdmissionResult {
    if q.CanAdmit(req) {
        q.DeductQuota(req.TenantID)
        return AdmissionResult{Action: "admit"}
    }
    
    // Check if we should reject outright or queue
    if len(q.queue[req.TenantID]) > 5 {
        return AdmissionResult{Action: "reject", Reason: "queue full"}
    }
    
    q.queue[req.TenantID] = append(q.queue[req.TenantID], req)
    return AdmissionResult{Action: "queue", RetryAfter: 30}
}

This smoothed out the cluster's behavior significantly. In January 2026, we released this pattern to one of our financial services clients. Their admission rejection rate dropped by 40% while their cluster utilization didn't change. The 30-second retry puts the burden back on the tenant to manage their workload queue.

The Missing Metric: Incumbent Workload Protection

Most admission control algorithms focus on the incoming workload. Here's the thing nobody talks about: admission control should protect the incumbent workloads first.

We have a policy: "New workloads must propose a plan for how they'll coexist with existing latency-critical inference." No plan, no admission.

We call it SLO guardrails. We track the SLO attainment of every latency-critical workload on a node. If a node's SLO attainment is trending below 99.5% over the last 5 minutes, the admission controller marks that node as "overloaded" and rejects any new workloads.

python
def node_slo_health(node_id, slo_threshold=99.5):
    recent_slo = get_slo_attainment(node_id, window_minutes=5)
    if recent_slo < slo_threshold:
        return {"status": "overloaded", "admission": "reject"}
    else:
        return {"status": "healthy", "admission": "allow"}

This is reactive, not proactive. It's a safety net. Combined with the noise credits (proactive) and the risk model (strategic), it creates a layered defense.

Conclusion: Admission Control is a Business Decision

Here's what I've learned after three years of building admission control systems for GPU clusters: the technology is the easy part. The hard part is understanding what your business needs.

A search engine's GPU cluster needs a different admission policy than an AI research lab's cluster. Different users, different SLOs, different tolerance for latency spikes.

The good news? You don't need a PhD to make this work. You need:

Clear workload classification (inference vs training vs batch)
A noise/interference model for your specific hardware
A quota system for your tenants
An SLO monitoring system on incumbent workloads
A webhook that ties it together

We've seen admission control cut interference-induced latency spikes by 90%. It turned an unusable multi-tenant cluster into a predictable one.

But the real win? The business win. When your cluster is stable, when a new workload doesn't cause an outage, when your inference p99 is predictable — your sales team can promise SLOs to customers and actually keep them. That's the outcome users care about.

Start with the noise credit model. Calibrate, observe, adjust. It's an important pattern to get right for your multi-tenant GPU cluster.

FAQ

Q: How is admission control different from scheduling?

A: Admission control runs first. It decides whether a workload is allowed to enter the cluster at all, based on quotas, interference potential, and SLO protection. Scheduling runs second — it decides where on the cluster the admitted workload goes. You can't fix interference problems with a scheduler; the decision to admit was already made.

Q: Does admission control work with any scheduler or just Kubernetes?

A: We've used it with Kubernetes and with Slurm. The pattern is generic: intercept the workload submission, evaluate policies, admit or reject. Kubernetes' webhook mechanism is the cleanest, but Slurm's sacctmgr and custom plugins work too.

Q: What metrics should I track to calibrate admission control?

A: For interference: effective memory bandwidth per GPU, NVLink utilization, PCIe bus utilization, p99 latency of latency-sensitive workloads, and node-level SLO attainment. Track these for at least 30 days to get statistically meaningful baselines.

Q: Can admission control handle bursty LLM inference load?

A: Yes, but you need to explicitly design for burstiness. We allocate a "burst headroom" — a reserved noise credit buffer on each node. When a burst comes, the admission controller allows the workload in as long as it doesn't consume the buffer. Once the buffer is consumed, new workloads get rejected until the burst recedes.

Q: What's the hardest part of implementing this?

A: Calibrating the noise costs. It's tempting to use general numbers from papers, but they'll be wrong for your specific configuration. We spent a month creating a calibration workload suite that runs on your nodes and measures interference under various load levels. It's time-consuming but essential.

Q: Is admission control for multi-tenant GPU cluster only relevant for large companies?

A: No. If you have more than 4 GPUs, you have contention and interference. Even a small team running two fine-tuning jobs alongside a production inference endpoint on the same node will benefit from admission control.

Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.