Admission Control LLM Inference Kubernetes (Stop the Bleed)

Admission Control LLM Inference Kubernetes (Stop the Bleed)

Three A100s. Twelve GPUs. One production LLM serving cluster. And at 2:47 AM on a Tuesday in March, every single GPU was running at 11% utilization while our P99 latency exploded past the 800ms SLO. Not because of a traffic spike. Because someone on the ML platform team deployed a training job with a nvidia.com/gpu: 1 request that grabbed a half-free H100, fragmented the memory topology, and quietly killed inference performance across four pods.

I was on the phone with our VP of Engineering. She said, "We need this fixed by Friday." I said, "We need this fixed before Monday's board demo." She agreed.

What we ended up building — and what I want to walk you through here — is a proper admission control llm inference kubernetes pipeline. Not the toy example from a blog post. The thing that actually holds up when you have 60+ engineers spinning up workloads, three different LLM serving frameworks, and a finance team that wants cost reports by GPU node.

By the end of this article, you'll understand how admission webhooks intercept GPU allocation requests before they hit the scheduler, how to write the actual Go code for a custom validator that enforces topology-aware placement, and where the gaps are that will bite you in production. I'll also cover quota management patterns that keep training and inference from stepping on each other. No fluff. Just what we built, what broke, and what I'd do differently.

What's Actually Happening When Your GPU Cluster Fragments

Here's the thing most platform engineers miss until it's too late: Kubernetes' device plugin model for GPUs is fundamentally dumb. The nvidia.com/gpu resource is a count. One. Two. Four. The scheduler sees "I need 2 GPUs on this node" and hands out any two available. It doesn't care if those two GPUs are on the same NVLink domain or not. It doesn't care if one of them has 40GB of residual memory from a zombie process that the device plugin didn't clean up.

For a 70B-parameter model doing tensor-parallel inference across 8 H100s, that doesn't matter as much. You're using all 8. But the moment you're running a mix — maybe a 7B model on 2 GPUs for low-latency serving, a 70B on 8 for your main product, and a 13B fine-tune job on 4 — the fragmentation becomes catastrophic.

I watched this happen at a Series B health-tech company in February 2026. They had 16 A100s on two nodes. Inference was using 12. Training was using 4. Sounds fine. But the training job grabbed GPUs 5, 6, 13, and 14 — splitting the NVLink topology on node 1. The inference pods on that node saw their all-reduce bandwidth drop by 40% overnight. No one noticed until a customer's API latency went from 200ms to 900ms and the on-call engineer (a third-year SWE, bless him) started kubectl describe pod'ing for four hours before someone said "check the GPU topology."

The fix wasn't a topology-aware scheduler rewrite. It was a 300-line admission webhook that said "no" to any request that would break NVLink domain integrity.

How Admission Control LLM Inference Kubernetes Actually Works

Let's get precise about the mechanism, because the term "admission control" gets thrown around loosely.

In Kubernetes, admission control is the set of controllers and webhooks that intercept a request after API validation but before the object is persisted to etcd. You have two types:

• ValidatingAdmissionWebhook — inspects the request, says yes or no. Can't modify it.
• MutatingAdmissionWebhook — inspects the request, can modify it, then it gets re-validated.

For GPU-aware LLM inference scheduling, you need both. The mutating webhook injects topology hints (which NVLink domain, which NUMA node, which PCIe switch). The validating webhook enforces the hard rules (don't fragment the domain, don't exceed the inference partition's GPU budget, don't schedule a 4-GPU tensor-parallel group across nodes unless you've explicitly opted into multi-node).

The flow looks like this:

Pod creation request (e.g., inference-server with nvidia.com/gpu: 8)
    → API Server validation (basic schema check)
    → MutatingAdmissionWebhook (your topology-injection service)
    → ValidatingAdmissionWebhook (your fragmentation-check service)
    → etcd persist
    → Scheduler picks up the Pod

That last step is critical. The admission webhooks don't schedule. They constrain what the scheduler is allowed to do. The scheduler still runs its scoring function. But by the time the Pod hits the scheduler, the admission layer has already encoded "this Pod must land on node X with GPUs 0-7 on NVLink domain A" as a nodeSelector or podAntiAffinity in the mutated spec.

The GPU Fragmentation Problem (and Why admission control to prevent gpu fragmentation Is Non-Negotiable)

I'll be blunt: if you're running a mixed training/inference cluster with more than 8 GPUs and you don't have admission-level GPU topology enforcement, you're running a time bomb. The question is whether it goes off during a quiet Tuesday or during your biggest launch week.

The fragmentation happens in three ways:

1. NVLink domain splitting. A 4-GPU training job grabs GPUs that "belong" to an 8-GPU inference domain. The inference pods don't get evicted — they just run at degraded bandwidth. Slow leak.

2. PCIe switch sharing. Two Pods on the same PCIe switch, neither of which is the bottleneck alone, but together they saturate the host bridge. Latency jitter appears. No single Pod looks wrong in monitoring.

3. Memory residue. A Pod crashes, the CUDA context doesn't fully tear down, 3-8GB of VRAM stays allocated. The device plugin reports the GPU as "available" because the driver-level count is freed. The next Pod schedules, hits the OOM at the CUDA level, crashes, retries, hits OOM again. You've got a crash loop that nvidia-smi says shouldn't be possible.

At SIVARO, we built a topology map as a ConfigMap that gets updated by a DaemonSet running nvidia-smi topo -m every 30 seconds. The admission webhook reads this map and validates every request against it. It's ugly. It works. We've had it in production since November 2025 with zero GPU fragmentation incidents.

The admission control to prevent gpu fragmentation isn't a "nice to have." It's the difference between a cluster that degrades gracefully and one that silently rots until a customer escalates.

Building the Admission Webhook: The Code That Actually Ships

Here's the core of our validating webhook. I'm stripping out the error handling and metrics for readability, but this is the structure that's running in production:

go
package main

import (
    "context"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"

    "github.com/golang-jwt/jwt/v4"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// TopologyDomain represents a NVLink/PCIe domain from our ConfigMap
type TopologyDomain struct {
    NodeName  string   `json:"nodeName"`
    GPUs      []string `json:"gpus"`      // e.g., ["0","1","2","3","4","5","6","7"]
    UsedByInference []string `json:"usedByInference"` // pod UIDs
    UsedByTraining  []string `json:"usedByTraining"`
}

type AdmissionRequest struct {
    UID       string          `json:"uid"`
    Kind      metav1.GroupVersionKind `json:"kind"`
    Request   json.RawMessage `json:"request"`
    Operation string          `json:"operation"`
}

type AdmissionResponse struct {
    Allowed  bool   `json:"allowed"`
    Result   *metav1.Status `json:"status,omitempty"`
    Patch    string `json:"patch,omitempty"` // for mutating
}

func validateGPURequest(ctx context.Context, r AdmissionRequest) (*AdmissionResponse, error) {
    var pod Pod // unmarshaled from r.Request

    gpuRequest := getGPURequest(pod)
    if gpuRequest == 0 {
        return &AdmissionResponse{Allowed: true}, nil
    }

    // Load topology from ConfigMap (cached, refreshed every 30s)
    domains, err := loadTopologyDomains(ctx)
    if err != nil {
        return nil, fmt.Errorf("topology load failed: %w", err)
    }

    // Rule 1: N-way tensor parallel MUST stay in one NVLink domain
    if isTensorParallel(pod) && gpuRequest > 4 {
        valid := false
        for _, d := range domains {
            free := getGPUsInDomain(d)
            if len(free) >= gpuRequest && !isFragmented(free) {
                valid = true
                break
            }
        }
        if !valid {
            return &AdmissionResponse{
                Allowed: false,
                Result: &metav1.Status{
                    Code:    403,
                    Reason:  "Forbidden",
                    Message: fmt.Sprintf("No single NVLink domain has %d contiguous free GPUs. Rejecting to prevent fragmentation.", gpuRequest),
                },
            }, nil
        }
    }

    // Rule 2: Training jobs must not steal from inference domains
    if isTrainingJob(pod) {
        for _, d := range domains {
            if len(d.UsedByInference) > 0 {
                // This domain is "claimed" by inference.
                // Only allow if training GPU count doesn't split remaining.
                free := getGPUsInDomain(d)
                if len(free) < gpuRequest {
                    // Would need to take from inference. Reject.
                    return &AdmissionResponse{
                        Allowed: false,
                        Result: &metav1.Status{
                            Code: 403,
                            Message: "Training job would fragment an active inference NVLink domain. Use dedicated training partition or wait for domain drain.",
                        },
                    }, nil
                }
            }
        }
    }

    return &AdmissionResponse{Allowed: true}, nil
}

func main() {
    server := &http.Server{
        Addr:    ":8443",
        Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            body, _ := io.ReadAll(r.Body)
            var req AdmissionRequest
            json.Unmarshal(body, &req)

            resp, err := validateGPURequest(r.Context(), req)
            if err != nil {
                w.WriteHeader(500)
                io.WriteString(w, err.Error())
                return
            }

            out, _ := json.Marshal(resp)
            w.Header().Set("Content-Type", "application/json")
            w.Write(out)
        }),
    }

    cert := os.Getenv("TLS_CERT")
    key := os.Getenv("TLS_KEY")
    server.ListenAndServeTLS(cert, key)
}

That's the skeleton. In production, it's wrapped in a service mesh, has retry logic, has a fallback mode (if the webhook is down, fail-open for inference, fail-closed for training — that decision kept our SLO up during a webhook outage in April).

The mutating webhook is simpler. It reads the same topology map, finds the best-fit domain, and injects a nodeAffinity plus a nvidia.com/gpu count that's aligned to the domain boundary. Eight GPUs in, eight GPUs on one domain out. No "I need 5 GPUs but the only free node has 4 free and 3 in use by a zombie."

Quota Management: The Part Everyone Skips

Here's where it gets political, and I won't pretend it's just a technical problem.

You have inference. You have training. You have data engineering pipelines that spin up ephemeral GPU jobs for ETL. You have a research team that wants to "just try a thing" on the cluster at 11 PM. And you have 48 GPUs (or 64, or 32 — pick your poison) that everyone wants.

The naive approach is a single ResourceQuota per namespace. nvidia.com/gpu: 16. Everyone shares it. Works for 3 months. Then the research team runs a 4-week training job, inference SLOs degrade, and you're in a Slack argument that goes nowhere.

What actually works (and this is a refinement of ai training cluster quota management best practices we've iterated on over 14 months):

• Partition at the node level, not the namespace level. Your inference nodes are labeled gpu-partition=inf. Your training nodes are gpu-partition=train. Admission webhook enforces that inference Pods can only schedule on inf nodes, training on train.
• Within a partition, use priority classes + preemption. Inference Pods get PriorityClass: llm-inference-critical (value 1000). Training gets llm-training-batch (value 100). If you run out of GPUs, the scheduler preempts training. The admission webhook doesn't need to do this — the built-in preemption handles it. But the webhook ensures the initial placement respects the partition.
• Ephemeral GPU jobs get a separate ResourceQuota with a nvidia.com/gpu: 4 ceiling and a TTL annotation. The webhook checks the annotation. If TTL is past, it rejects. No more zombie training jobs holding 8 GPUs for three weeks.

yaml
# ResourceQuota for the inference namespace
apiVersion: v1
kind: ResourceQuota
metadata:
  name: inference-gpu-quota
  namespace: llm-serving
spec:
  hard:
    nvidia.com/gpu: "32"
    pods: "48"
---
# Priority class that makes inference preemptible-last
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: llm-inference-critical
value: 1000
globalDefault: false
description: "Production LLM serving. Preempt training, never be preempted."
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: llm-training-batch
value: 100
globalDefault: false
description: "Batch training jobs. Can be preempted by inference."

The webhook's role here is enforcement of placement within those constraints. The quota system handles how many. The webhook handles where and which specific GPUs. They're complementary, not redundant.

What We Got Wrong (And What I'd Do Differently)

Honest section. I learned this the hard way.

We started with a mutating webhook only. For the first two months, we only had the mutator that injected topology hints. No validator. Which meant if the topology ConfigMap was stale (and it was — the DaemonSet that updated it was on a node that had a kernel panic), the webhook would inject wrong topology hints. Pods would schedule based on stale data, land on the wrong domain, and you'd have fragmentation that looked like a scheduler bug. We spent three days debugging "why is the scheduler putting this Pod here" when the answer was "your webhook told it to."

We didn't version the topology format. The ConfigMap schema changed twice in the first quarter. The webhook was deployed to a staging cluster that had the new format while prod had the old. Webhook returned 500s for 47 minutes. Not a huge outage, but enough to scare the infra team.

We treated the webhook as stateless. It reads the ConfigMap on every request. Fine for 50 Pods per day. Not fine for 200. We added an in-memory cache with a 30-second TTL and a background refresher. Latency went from ~40ms per admission to ~2ms. That mattered when we scaled to 64 GPUs and 12 concurrent Pod creations during a deployment.

The TLS/cert rotation. I won't pretend I had this figured out on day one. The webhook needs a TLS cert signed by the cluster CA, and Kubernetes has a 10-minute grace period on cert expiry. We automated rotation with cert-manager. Took me a week to get the ValidatingWebhookConfiguration CABundle field updating correctly. If you're building this, start with cert-manager. Don't hand-roll it.

FAQ

How is admission control different from a scheduler plugin?

Different layer, different timing. A scheduler plugin runs after the object is in etcd and the scheduler is making placement decisions. An admission webhook runs before etcd persist. Practically: the webhook can reject the Pod entirely (user sees the error immediately, doesn't get a "Pod is pending forever" state). The scheduler plugin can only deprioritize or block — the Pod still exists in the cluster, just unschedulable. For GPU topology enforcement, I want the hard "no" before the object exists. You can argue for a scheduler plugin if you need to react to current cluster state in a way the ConfigMap can't capture, but for static topology, the webhook is simpler and faster.

Can I use the built-in ResourceQuota and LimitRange instead of a custom webhook?

For "how many GPUs total," yes. For "which specific GPUs and in what topology," no. The built-in resources are opaque. nvidia.com/gpu: 4 means "four GPUs, any four, on any node." It doesn't encode topology. The moment you need NVLink domain awareness, you need custom logic, and the admission webhook is the right extension point. You could do it with a scheduler plugin, but you lose the early rejection and the user-facing error messages.

What happens if my webhook goes down?

This is the critical design decision. Ours fails open for inference Pods and fails closed for training Pods. Rationale: inference SLOs are customer-facing, sub-second. A 30-second delay in Pod creation (waiting for webhook timeout) is worse than a slightly suboptimal placement. Training can wait. The ValidatingWebhookConfiguration failurePolicy field lets you set this per-webhook. Set inference validator to FailOpen. Set training validator to FailClosed. Different risk profiles, different policies.

Do I need this if I'm only running inference (no training)?

Probably not at first. If every Pod is a 1-GPU or 8-GPU tensor-parallel serving instance and they all fit in one NVLink domain per node, the scheduler's "pick 8 GPUs on this node" is fine. The fragmentation risk emerges when you have mixed sizes (2-GPU, 4-GPU, 8-GPU) or mixed tenants (serving + batch + research). If you're a single-tenant inference cluster with uniform model sizes, you can probably skip this until you hit 16+ GPUs.

How do I handle multi-node tensor parallelism (e.g., a 400B model across 16 H100s on 2 nodes)?

This is where it gets genuinely hard. You need the webhook to verify that the two nodes have RDMA/NVLink connectivity between them (InfiniBand, RoCE). The topology ConfigMap needs to encode not just per-node NVLink domains but also inter-node links. We handle this by requiring the Pod spec to include a topology-group label, and the webhook validates that all nodes in the group have the required interconnect. If the interconnect is degraded (we check link status in the DaemonSet), the webhook rejects. This is the part I'm least happy with — it's brittle, and a single SFP transponder swap in the data center can take down your 400B model serving until someone updates the ConfigMap.

Is there a Kubernetes-native way to do this without a custom webhook?

As of k8s 1.32 (which is what we're running), there's the DRA (Dynamic Resource Allocation) framework. It's been beta since 1.32, GA is targeted for 1.34. The idea is that you declare GPU "resource slices" with topology metadata, and the scheduler natively understands NVLink domains, NUMA topology, etc. In theory, you don't need a custom webhook. In practice, the DRA implementation as of late 2025 is still missing multi-resource group constraints and the "don't fragment" policy is a proposal, not a shipped feature. We're watching it. If it hits GA in the next release cycle, we'll migrate. Until then, the webhook is the pragmatic choice.

What about cost tracking? How do I attribute GPU-hours to teams?

The webhook logs every admission decision (allowed, denied, mutated) to a structured log. We ship those to our observability stack. Each log line has the requesting namespace, the GPU count, the target domain, and a timestamp. From that, we generate per-team GPU-hour reports. It's not perfect (preemptions and reschedules create gaps), but it's good enough for finance. The alternative — per-GPU Prometheus metrics from DCGM — gives you utilization but not attribution. You need both.

The Bigger Picture

Admission control for LLM inference on Kubernetes isn't a single component. It's a policy layer. It sits between "we have GPUs" and "we have a working cluster" and it encodes the operational knowledge that no scheduler algorithm will derive on its own.

At first I thought this was a scheduling problem. "Just build a better scheduler plugin." Turns out it wasn't. It was a governance problem. Who gets the GPUs, in what configuration, with what failure semantics. The webhook is the enforcement mechanism for a policy that you, as the platform team, have to decide on. The code is 80% of the work. The other 20% is the Slack threads with the ML research lead who wants to "just run one experiment" on the inference GPUs at 2 AM.

If you're running more than 8 GPUs with mixed workloads, build this before you need it. The 2:47 AM phone call is a lot easier to avoid than to answer.

The admission control llm inference kubernetes pattern isn't glamorous. It's a Go HTTP server, a ConfigMap, and a bunch of if statements that say "no." But it's the difference between a cluster that behaves like infrastructure and one that behaves like a shared office printer where someone always puts in the wrong paper.

---

Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.