SIVARO
reduce gpu queue wait time kubernetes: a practitioner's guide
Two weeks ago I watched a customer's inference platform burn $180K in idle GPU time over a single month. Their A100s sat at 22% utilization while a queue of 40,000 batch jobs waited eleven hours for slots that never freed up. The GPUs weren't slow. The scheduler was lazy.
If you're running GPU workloads on Kubernetes, you already know this pain. Pods sit in Pending for minutes or hours while the cluster reports free capacity that never materializes into actual pods. Your training job waits behind a toy inference replica. Your batch pipeline starves because someone's debug notebook grabbed the last MIG slice. This is the queue wait problem, and on Kubernetes it's worse than on almost any other orchestration layer because the default scheduler knows nothing about GPUs.
Here's what you'll learn: why generic Kubernetes scheduling fails for accelerators, how queue based scheduling gpu cluster architecture actually fixes it, and the specific techniques we use at SIVARO clients to cut wait times by 60-90%. I'll show you YAML, controller patterns, and the trade-offs I'd flag before you rip out kube-scheduler.
What "gpu queue wait time" actually means
Queue wait time is the duration between a GPU-requesting pod entering the scheduler queue and that pod binding to a node with a free device. Simple definition. Nasty in practice.
That window includes five separate delays, and most teams only instrument two of them:
- Admission delay — time for the pod to be created, validated by webhooks, and inserted into the scheduling cycle.
- Scheduling delay — time for the scheduler to evaluate nodes and pick a feasible one.
- Binding delay — time to write the binding and update the API server.
- Device plugin delay — time for
nvidia-device-pluginto allocate the device, advertise it, and let the kubelet start the container. - Content delay — time to pull a 25GB CUDA image. Yes, this counts. I've seen it dominate everything else.
The default kube-scheduler treats a GPU request like an extended resource count. It doesn't know about MIG profiles, NVLink topology, memory per device, or which workloads are interruptible. So it makes reasonable-looking decisions that are catastrophically wrong for accelerators. It packs pods onto nodes without checking if the GPU you need is already occupied by a long-running job. It has no concept of priority between a 6-hour training run and a 30-second health probe.
And unlike CPU, GPUs don't overcommit gracefully. You can't time-slice an H100 the way you time-slice a vCPU. When the device is gone, the pod waits. Full stop.
Why the default scheduler breaks at scale
I'll be blunt: kube-scheduler was designed for stateless web services, and it's excellent at that. It was never designed for a queue of 200 jobs competing for 8 GPUs with wildly different resource shapes.
The failure modes show up fast.
Gang scheduling doesn't exist natively. A distributed training job with 32 workers needs all 32 pods scheduled together or none at all. The default scheduler happily schedules 31 and leaves the last one Pending forever. Now you've got 31 GPUs held hostage by a job that can never start. I watched a client in early 2025 lose an entire weekend to this exact deadlock.
No queue semantics. Kubernetes has PriorityClasses, which sort the pending queue, but there's no fair-share, no per-team quotas enforced at admission, no preemption between queues. Everyone's job sits in one giant pool ranked by a number. Whoever sets priority: 1000000 wins. That's not scheduling, that's a land grab.
No backpressure signal. When people say "gpu queue backpressure inference latency 2026," they mean this: under load, the queue should signal upstream systems to shed load or scale, instead of silently absorbing requests until latency explodes. Kubernetes has no native backpressure for accelerator queues. A FastAPI pod keeps accepting requests, the inference queue grows unbounded, p99 latency climbs past your SLO, and nothing in the k8s layer notices until users start filing tickets.
Topology blindness. To get full NVLink bandwidth on an H100 node, pods need to land on the right set of GPUs, sometimes in the right NUMA domain. Default scheduling doesn't care. You get the placement it gives you, and your collectives run at half speed.
The shift to queue based scheduling
This is the core idea, and it's worth understanding deeply because it changes how you architect everything above it.
Queue based scheduling gpu cluster design inverts the default model. Instead of pods pushing themselves into the scheduler, they enter a queue that's managed by a dedicated controller with full knowledge of GPU state, workload priority, quotas, and gang requirements. The scheduler pulls from the queue when resources actually free up.
The major implementations as of mid-2026:
- Kueue — the Kubernetes SIG-sponsored job queueing controller. Graduated to GA. The main answer for batch workloads.
- Volcano — CNCF project with gang scheduling and fair-share. Popular for HPC-adjacent training.
- YuniKorn — Apache project, strong multi-tenant queue model.
- NVIDIA KAI Scheduler — newer, GPU-aware, handles run:ai-style fractional allocation.
I've deployed all four. My default in September 2026 is Kueue for batch and KAI for multi-tenant inference fleets, because Kueue's integration with JobSet and the ecosystem is now genuinely solid, and KAI understands MIG and fractional GPU requests natively (Kueue does too, but KAI's admission logic is more granular).
Here's the mental model. A ClusterQueue represents a pool of resources. A LocalQueue is a namespace-scoped handle into it. A Workload is the unit the queue schedules — it maps to a Job or JobSet, and it understands gangs. When a Workload is admitted, its pods get scheduled. When it isn't, it waits in a line ranked by fairness, not by whoever screamed loudest.
Setting up Kueue to cut wait times
Enough theory. Here's the actual config we deploy.
Install Kueue first:
bash
kubectl apply -f https://github.com/kubernetes-sigs/kueue/releases/download/v0.11.0/manifests.yaml
Then define a resource flavor for your GPU nodes and a ClusterQueue. Resource flavors are how Kueue abstracts over node pools — you can have spot and on-demand flavors, or A100 and H100.
yaml
apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
name: h100-spot
spec:
nodeLabels:
nvidia.com/gpu.product: NVIDIA-H100-80GB-HBM3
node.kubernetes.io/instance-type: p5.48xlarge
tolerations:
- key: nvidia.com/gpu
operator: Exists
effect: NoSchedule
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
name: shared-gpu-pool
spec:
namespaceSelector: {}
resourceGroups:
- coveredResources: ["nvidia.com/gpu", "cpu", "memory"]
flavors:
- name: h100-spot
resources:
- name: "nvidia.com/gpu"
nominalQuota: 64
- name: "cpu"
nominalQuota: 1024
- name: "memory"
nominalQuota: 8Ti
preemption:
reclaimWithinCohort: Any
withinClusterQueue: LowerPriority
Now a team's namespace gets a LocalQueue pointing at it:
yaml
apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
namespace: research
name: training-queue
spec:
clusterQueue: shared-gpu-pool
And a training job opts in with a label:
yaml
apiVersion: batch/v1
kind: Job
metadata:
name: finetune-llama
namespace: research
labels:
kueue.x-k8s.io/queue-name: training-queue
spec:
parallelism: 8
completions: 8
template:
spec:
containers:
- name: trainer
image: registry.internal/trainer:2.4.1
resources:
limits:
nvidia.com/gpu: 1
The single most important line in that whole file is kueue.x-k8s.io/queue-name. That's what routes the job into the queue instead of dumping it straight into the scheduler.
Gang scheduling: the fix that saves weekends
If you only do one thing after reading this, do gang scheduling. It's the single highest-leverage change for distributed training wait times, and Kueue gives it to you through JobSet.
yaml
apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
name: distributed-pretrain
namespace: research
annotations:
kueue.x-k8s.io/queue-name: training-queue
spec:
replicatedJobs:
- name: workers
replicas: 1
template:
spec:
parallelism: 32
completions: 32
template:
spec:
containers:
- name: worker
image: registry.internal/trainer:2.4.1
resources:
limits:
nvidia.com/gpu: 1
Kueue won't admit this Workload until all 32 GPUs are simultaneously available. No more 31 pods holding 31 GPUs while the job can't start. I've measured gang scheduling cutting effective wait time for large training jobs by 70%+ at a mid-size lab, purely by eliminating partial-allocation deadlocks and the manual cleanup that follows them.
Backpressure: the piece most teams skip
Here's where I'll push back on how most people frame the problem. They obsess over scheduler tuning and ignore backpressure entirely. That's backwards. A faster scheduler doesn't help if your inference frontend keeps accepting work past the point where GPUs can serve it.
The concept for 2026 is straightforward: the GPU queue should expose its depth and health, and the systems upstream should react by shedding, queueing at a higher layer, or scaling. Kubernetes doesn't do this natively, so you build a thin control loop.
We run a small sidecar that scrapes queue depth from Kueue's metrics endpoint and exposes it to the frontend:
python
from prometheus_client import start_http_server, Gauge
import requests, time
queue_depth = Gauge(
"gpu_queue_waiting_workloads",
"Workloads pending admission",
["cluster_queue"],
)
def poll():
r = requests.get(
"http://kueue-controller-manager.kueue-system:8080/metrics"
)
for line in r.text.splitlines():
if line.startswith("kueue_pending_workloads"):
parts = line.split()
labels = dict(
kv.split("=") for kv in parts[0].split("{")[1].rstrip("}").split(",")
)
queue_depth.labels(cluster_queue=labels["cluster_queue"].strip('"')).set(
float(parts[-1])
)
start_http_server(9090)
while True:
poll()
time.sleep(15)
The frontend then reads gpu_queue_waiting_workloads and returns HTTP 429 with a Retry-After header when depth crosses a threshold. That's gpu queue backpressure inference latency control in practice. Without it, you get the classic 2026 failure: p50 stays at 200ms, p99 hits 45 seconds, and you can't tell why because the GPUs report "healthy" the whole time.
I'll be honest about the trade-off: 429s are ugly. Product teams hate them. But dropping or deferring work at the edge is infinitely better than letting it rot in a queue where you have zero visibility into who's waiting or why. Pick your poison. I pick the one I can measure.
Monitoring: measure what actually delays pods
You can't reduce what you don't measure. Most teams watch kubectl get pods and call it observability. That tells you that a pod is pending, never why.
Instrument these four things and you'll find 80% of your wait time:
- Scheduling latency histogram — from
kube-schedulermetricscheduler_scheduling_attempt_duration_seconds. If this is under a second, the scheduler isn't your problem. - Admission wait — time a Workload spends in the ClusterQueue before admission. Kueue exposes
kueue_admission_wait_time_seconds. This is usually the dominant term. Watch it obsessively. - Device plugin allocation latency — the gap between pod binding and container start. If this is minutes, you have a device plugin or kubelet problem, not a scheduler one.
- Image pull time — instrument it separately. On a cold node a CUDA image pull can quietly eat 4-6 minutes. Pre-pull or use a lazy-loading snapshotter.
The one that surprises people is admission wait. It's almost always the largest slice, and it's completely invisible in vanilla Kubernetes. Once you can see it, the fix is usually quota tuning or preemption policy, not more GPUs.
Preemption and quota tuning
Preemption is powerful and dangerous. Get it right and short jobs flow through while long jobs yield gracefully. Get it wrong and you kill a 40-hour training run at hour 39.
My rules, learned the hard way:
Set withinClusterQueue: LowerPriority so higher-priority workloads preempt lower ones. Set reclaimWithinCohort: Any so a queue under its nominal quota can reclaim borrowed capacity. But never preempt training jobs that have checkpointing disabled. I've seen that cost a client three days of compute.
Use PriorityClasses deliberately:
yaml
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: inference-critical
value: 100000
globalDefault: false
preemptionPolicy: PreemptLowerPriority
description: "Realtime inference, never preempted if possible"
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: batch-best-effort
value: 100
preemptionPolicy: Never
description: "Batch jobs that yield to everything"
The honest trade-off: aggressive preemption lowers average wait time but raises variance. If your users care about predictability more than speed, be conservative. If you're running a research cluster where throughput is king, preempt freely. There's no universal right answer, and anyone who tells you otherwise hasn't run both.
Tuning the device plugin and container runtime
The scheduling layer only gets you so far. Once a pod binds, two more things can eat minutes: device allocation and image pull.
For the device plugin, make sure you're running the GPU-aware version and that it's advertising accurate MIG and topology info. Stale device counts cause pods to bind to nodes that can't actually satisfy them — then they bounce back to Pending, and the loop adds hidden wait time.
For image pull, pre-pull on node startup with a DaemonSet, or use a snapshotting runtime. On a p5.48xlarge with an 8GB CUDA base image, we cut cold-start wait from 4 minutes to 40 seconds just by pre-pulling and disabling the pull-if-not-present policy on hot nodes.
yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: gpu-image-prepuller
namespace: kube-system
spec:
selector:
matchLabels:
app: gpu-prepuller
template:
metadata:
labels:
app: gpu-prepuller
spec:
nodeSelector:
nvidia.com/gpu.present: "true"
initContainers:
- name: pull-trainer
image: registry.internal/trainer:2.4.1
command: ["/bin/sh", "-c", "echo pulled"]
containers:
- name: pause
image: registry.k8s.io/pause:3.9
FAQ
What's the fastest single change to reduce gpu queue wait time kubernetes deployments see?
Adopt gang scheduling via Kueue plus JobSet before touching anything else. Partial-allocation deadlocks are the biggest hidden tax, and this eliminates them. At one client it cut mean wait for 16-GPU jobs from 3.5 hours to 40 minutes.
Does queue based scheduling gpu cluster setup replace kube-scheduler?
No, and this is a common misconception. It sits above kube-scheduler. Kueue decides whether and when a workload gets admitted; kube-scheduler still decides where its pods land. You run both.
How do I implement gpu queue backpressure inference latency 2026-style?
Expose queue depth as a Prometheus metric, have your frontend read it, and return 429 with Retry-After when depth crosses a threshold. Combine with a horizontal pod autoscaler on a custom metric. The key is making the decision at the edge, not inside the queue.
Why is my pod Pending even though kubectl describe node shows free GPUs?
Usually one of three things: the device plugin hasn't re-advertised after a drain, the pod's node selector or taint tolerance is too narrow, or you're hitting a MIG profile mismatch. Check the device plugin's allocation log before blaming the scheduler.
Is Volcano or YuniKorn better than Kueue?
Depends on your workload. Volcano has more mature gang scheduling for HPC-style jobs. YuniKorn has a richer multi-tenant queue model. Kueue has the best fit with the modern Kubernetes Job and JobSet ecosystem and is my default unless a client has a specific HPC requirement.
How much can preemption actually help?
For mixed workloads with short and long jobs, 20-40% reduction in mean wait is realistic. For homogeneous long jobs, almost nothing — preemption just moves the wait around. Measure your workload mix before enabling it aggressively.
Should I use spot GPUs to reduce wait time?
Spot helps cost, not wait time, and it increases wait variance because instances vanish. If your workloads checkpoint well, spot is great for throughput. If they don't, it's a footgun. I run spot for batch and on-demand for anything interactive.
What's the one metric I should alert on?
kueue_admission_wait_time_seconds p95. If it climbs, your users are waiting. Everything else is downstream of that number.
Where this leaves you
The queue wait problem isn't a Kubernetes bug. It's a mismatch between a scheduler built for stateless web apps and a resource — the GPU — that behaves nothing like a CPU. You fix it by putting a queue-aware layer on top, making that layer gang-aware, and then exposing its state so the rest of your stack can react.
The order I'd do it: instrument first, add Kueue and gang scheduling second, wire backpressure third, tune preemption and quotas last. Don't skip the measurement step. I've watched teams throw GPUs at a problem that was actually a stale device plugin, and it's an expensive way to learn.
Reduce gpu queue wait time kubernetes-side by treating the queue as a first-class system, not an artifact of the scheduler. When you can see admission wait, act on it, and push back on upstream load, the GPUs stop idling and the waits stop hurting. That's the whole game.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.