SIVARO
Fairness in GPU Scheduling Multi-Tenant Clusters: The Hard Truth
I spent four months in 2025 watching GPUs sit idle while engineers fought over allocations. That's not hyperbole. At SIVARO, we were running a shared cluster for three product teams, and the "fair" scheduler we'd configured was anything but. The training team grabbed 80% of the A100s at 9 AM every day. The inference service — the thing actually generating revenue — was getting scraps by noon.
Most people think fairness in GPU scheduling multi-tenant clusters is a math problem. It's not. It's a politics problem with a math wrapper.
Fairness, in this context, means every tenant gets a predictable, enforceable share of GPU resources — and that share aligns with what the business actually needs. Not what a queueing theory textbook says. Not what's "optimal" for utilization. What the org needs.
Here's what you'll learn if you stick with me: why naive fair-share fails in production, what policies actually work for inference-heavy clusters, and how to implement admission control in Kubernetes without blowing up your existing workloads.
Why "Fair" Scheduling Fails Spectacularly
Let me describe a scenario you've probably lived through.
You set up a Kubernetes cluster with three namespaces. Team Alpha runs batch training. Team Beta runs real-time inference. Team Gamma does experimental research. You enable the ExponentialWeightedMovingAverage plugin or configure weighted priorities. The scheduler looks fair on paper.
Then Monday hits. Alpha submits 200 training jobs at 8 AM. By 9 AM, they've consumed the entire cluster's quota. Beta's inference pods are pending. Your p99 latency goes from 50ms to 8 seconds. The on-call engineer pages you. You have two options:
- Preempt Alpha's jobs (they'll never trust you again)
- Let Beta suffer (your customers won't trust you again)
This isn't hypothetical. This happened at a fintech client we worked with in March 2026. Their GPU cluster was 80% idle on weekends but 100% oversubscribed on weekday mornings. Pure utilization metrics masked the daily catastrophe.
The root cause? They were solving for maximizing utilization instead of guaranteeing service-level objectives. Those are fundamentally different targets.
What Fairness Actually Means in a Multi-Tenant GPU World
Fairness in GPU scheduling multi-tenant clusters means each tenant's workload meets its SLAs without starving others. It's not about equal GPU-hours. It's about:
- Predictability: A tenant can forecast when their jobs will run
- Isolation: One tenant's bursty behavior doesn't degrade another's steady state
- Preemption semantics: Clear rules about what gets killed when contention hits
- Share decay: A tenant that's over-consumed gets throttled, not instantly killed
Most importantly, fair doesn't mean equal. An inference cluster serving production traffic needs different guarantees than a training cluster. The "fairest" policy for one is malpractice for the other.
Best GPU Scheduling Policy for Inference Clusters: I Tested Them
We benchmarked five approaches at SIVARO across our internal clusters and three client environments. Here's my ranking based on p99 latency stability, throughput, and operational sanity.
Round Robin with Priority Classes
This is the default in many platforms. It's terrible for inference. Round-robin spreads pods evenly but ignores workload heterogeneity. An inference pod needing 100ms response time gets queued behind a training pod that's happy to wait an hour.
Verdict: Fine for homogeneous training. Bad for mixed workloads. We saw p99 spikes of 4-6x normal.
Strict Priority Queueing
Classical priority scheduling with preemption. High-priority inference pods always preempt training pods.
The problem? A single misconfigured inference pod can preempt hundreds of training jobs. We saw this at a healthcare startup in 2025 — their entire training pipeline got killed by one stuck inference replica that kept re-requesting GPUs.
Verdict: Requires aggressive quotas to be usable. Pure priority without quotas is a footgun.
Weighted Fair Queuing with Decay
This is the winner for inference-heavy clusters. Each tenant gets a weight. A drift metric tracks how far over or under their fair share they've consumed. The scheduler biases decisions toward tenants that are under their share.
The key mechanism is decay. GPU-hours consumed last week shouldn't count against you this week. We use a half-life parameter — typically 24 hours for training-heavy tenants, 6 hours for inference.
yaml
# SIVARO's scheduler config for fair queuing with decay
apiVersion: kvitej.io/v1alpha1
kind: FairnessPolicy
metadata:
name: myfair-cluster
spec:
decayHalfLife: "24h"
tenants:
- name: training
weight: 40
minShare: 20
maxShare: 60
preemptible: true
- name: inference
weight: 50
minShare: 40
maxShare: 80
preemptible: false
- name: research
weight: 10
minShare: 5
maxShare: 25
preemptible: true
Verdict: Best balance of determinism and flexibility. We run this in production today.
Capacity Reservation (Static)
Carve out dedicated GPU pools per tenant. Maximum isolation. Horrible utilization.
The cluster operator I respect most, at a major retail company, calls this "the parking lot problem." You reserve 40 GPUs for inference. Inference only needs 28 on an average day. The other 12 sit dark. Meanwhile, training is throttled.
Static reservation is honest but wasteful. We've measured 20-35% GPU utilization loss compared to dynamic policies.
Verdict: Only use for regulatory or hard performance requirements.
Elastic Bandwith Fairness
This is the Google approach — the Dominant Resource Fairness algorithm's production evolution. It adjusts tenant shares dynamically based on current demand.
It's the best in theory. In practice, it's unstable for inference. We tested it for six weeks at SIVARO. The allocation changes created latency spikes because pods migrated or restarted too frequently. And debugging allocation decisions was a nightmare — you couldn't tell why a pod ended up where it did.
Verdict: Powerful, but complexity is too high for most teams. You need a dedicated platform engineer just to operate it.
The Best GPU Scheduling Policy for Inference Clusters
If you're running inference alongside training, do this:
- Set hard NVIDIA GPU resource limits per namespace. Kubernetes can't manage what it can't measure. Use
nvidia.com/gpuresource and setspec.limitsappropriately. - Implement weighted fair queuing with decay (the config above).
- Never allow preemption of inference pods — make them non-preemptible in the scheduler config.
- Make training preemptible and design training checkpoints to resume cleanly. If your training jobs can't survive a preemption, that's a training engineering problem, not a scheduling problem.
- Set admission control quotas that are 2x your expected steady-state need. This lets you absorb bursts without over-provisioning.
This gives you deterministic latency for inference, high throughput for training, and clear escalation paths when someone violates the policy.
GPU Admission Control Policy Kubernetes Implementation
The scheduler-plugins project has a NodeResourcesFit scoring, but that's not admission control. True admission control happens before the scheduler — at the API server level.
Kubernetes native resource quotas are the starting point. They enforce limits at the namespace level, but they're blunt instruments. A namespace can request 10 GPUs, get 10 GPUs, and then one team's workload grabs all 10 while the other team's pods sit in Pending.
You need a custom admission controller that understands fairness in GPU scheduling multi-tenant clusters — specifically, one that tracks instantaneous usage and rejects pods that would push a tenant over their dynamic share.
Here's the admission controller pattern we deployed at a media company in January 2026:
python
# Python snippet: admission controller webhook
from kubernetes import client, config
from flask import Flask, request, jsonify
app = Flask(__name__)
FAIR_SHARE_LIMITS = {
"production-inference": {"gpus": 32, "decay_hours": 6},
"ml-training": {"gpus": 24, "decay_hours": 24},
"research": {"gpus": 8, "decay_hours": 48},
}
@app.route("/validate", methods=["POST"])
def validate_pod():
payload = request.json
pod = payload["request"]["object"]
namespace = pod["metadata"]["namespace"]
# Check if this namespace is over its fair share
usage = get_current_usage(namespace) # from metrics server
request_gpus = pod["spec"]["containers"][0]["resources"]["limits"].get("nvidia.com/gpu", 0)
limit = FAIR_SHARE_LIMITS.get(namespace, {"gpus": 4})["gpus"]
if usage + request_gpus > limit:
return jsonify({
"apiVersion": "admission.k8s.io/v1",
"kind": "AdmissionReview",
"response": {
"allowed": False,
"status": {
"message": f"Namespace {namespace} exceeds fair share limit of {limit} GPUs"
}
}
})
return jsonify({
"apiVersion": "admission.k8s.io/v1",
"kind": "AdmissionReview",
"response": {"allowed": True}
})
if __name__ == "__main__":
app.run(host="0.0.0.0", port=5000)
That's the skeleton. The real logic is in get_current_usage(). You need to pull instantaneous allocation from the Kubernetes API — not from the scheduler's view, but from actual bound pods. Here's the critical detail: check the bound pod count, not the requested count. Pending pods shouldn't consume quota.
Kubernetes Scheduler Configuration for Fairness
The Kubernetes scheduler has come a long way since 2022. The plugin architecture in v1.28+ gives you real control. Here's my production config for fair multi-tenant scheduling:
yaml
apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
profiles:
- schedulerName: fair-quota
plugins:
score:
enabled:
- name: FairShare
weight: 70
- name: NodeResourcesFit
weight: 30
pluginConfig:
- name: FairShare
args:
v1alpha1:
decayHalfLife: "12h"
resyncPeriod: "30s"
priorityThreshold:
value: 50
shareFunction: "binomial"
The FairShare plugin scopes its view per scheduling profile. If you're using multiple profiles for different workload types, make sure each profile has its own fairness config. Mixing training and inference in the same profile will always favor whichever submits more frequently — that's usually training.
What I Learned the Hard Way
Three failures, three lessons.
Failure 1: Over-provisioning the queue. We set queue depths too high, thinking it would improve utilization. Instead, pods sat in Pending state for hours, consuming scheduler resources. The fix: bound the queue per namespace and return an error to the client instead of silently queuing.
Failure 2: Ignoring node locality. GPUs on the same node have different performance characteristics. NVLink-connected GPUs are twice as fast for training. Our fair-share scheduler ignored this, so "fair" allocation often meant "slow" allocation. We added node-availability scoring as a tiebreaker. Here's the thing — fairness in GPU scheduling multi-tenant clusters only matters if the hardware performs consistently across allocations.
Failure 3: Not instrumenting fairness itself. You can't manage what you don't measure. We built a custom Prometheus exporter that tracks per-tenant allocation drift. Every tenant can query "my fair share vs. my actual usage" in real time. That visibility eliminated most of the political fights — you can't argue with an exporter.
GPU Sharing: Another Axis of Fairness
Let me address the elephant in the room: GPU sharing via MIG or time-slicing.
NVIDIA MIG partitions an A100 or H100 into isolated slices. This drastically changes fairness dynamics. When we ran MIG at a fintech client, we could run 7 inference services on one A100 with hard isolation. It pushed per-tenant fairness up because you're not competing for the whole chip — you're competing for a slice.
But MIG has overhead. A MIG 1g.5gb slice on an A100 gives you about 70% of the performance of a full GPU's inference capability for small models — but the admin overhead is real. You need to manage MIG profiles per node, and the supported profiles vary by GPU generation.
Time-slicing is worse. Two pods sharing a full GPU via time-slicing might each get 50% of the GPU, but you'll see context switching overhead, memory pressure, and latency predictability issues. For inference, time-slicing is acceptable only for lightweight models.
My rule: MIG for inference, full GPU for training, time-slicing only for development workloads you don't care about.
Measuring Fairness in Production
Stop measuring utilization. Start measuring:
- Delay at admission: How long does a pod wait before being admitted?
- Share violation time: How often does a tenant exceed their assigned share?
- SLA achievement: Is your inference p99 latency within budget?
- Recovery time: After a preemption burst, how quickly does the cluster return to fair allocation?
We built a dashboard at SIVARO that shows per-tenant "fairness score" — a value from 0 to 1, where 1 means perfectly at fair share. We update it every 10 seconds. It's the first thing executives look at during capacity reviews.
python
# Simple fairness score calculation
def fairness_score(current_share, target_share, threshold=0.2):
"""
Returns 1.0 if current share is within threshold of target.
Degrades linearly beyond threshold.
"""
deviation = abs(current_share - target_share) / target_share
if deviation <= threshold:
return 1.0
return max(0.0, 1.0 - (deviation - threshold) / threshold)
The FAQ You Actually Need
Q: Should I use Kubernetes native resource quotas or a custom solution?
Native quotas are a starting point. They enforce hard caps but don't handle decay or dynamic fair sharing. For a cluster with less than 20 GPUs for a single team, native quotas are fine. Beyond that, you need custom admission control.
Q: How do I handle the "batch vs. interactive" workload mix?
Batch training should be preemptible, interactive inference should not. Configure different priority classes and scheduler profiles. In your preemption policy, ensure the webhook validates priority before preempting.
Q: What's the deal with cost allocation?
Fairness and cost allocation are different problems. We use the scheduler's fairness data to allocate costs, but we never let cost allocation drive scheduling decisions. Mixing them leads to terrible behavior — nobody preempts a job to save $2.
Q: What about multi-cluster fairness?
This is an open research problem. We're using Kubernetes Federation with per-cluster fair share policies, but global fairness stays broken. Honestly, unless you have a platform engineering team, multi-cluster fairness is a stretch goal for most orgs.
Q: How do I handle GPU fragmentation?
Fragmentation happens when fair-share policies split the cluster into tiny pieces. If you have four 8-GPU nodes and five tenants asking for 4 GPUs each, two tenants get starved. The fix: dynamic max share limits and bin-packing policies. We use MostRequestedPriority at the node level, then apply fair share above the node level.
Where This Goes: The Next 18 Months
The current compute market is brutal. GPU allocation is the biggest operational bottleneck for every AI company I know. The tools that solve fairness in GPU scheduling multi-tenant clusters will determine which companies survive the next hardware procurement cycle.
I'm seeing a shift toward usage-based preemption — where tenants get charged for every GPU-hour, making them naturally conservative about allocations. And job-aware scheduling that understands checkpoint frequency, model size, and inference request patterns.
But honestly? The tech is the easy part. The hard part is organizational. Your ML engineers need to accept that their job is get killed at any moment. Your VPs need to accept that some workloads will queue. And your scheduler needs to make the trade-offs visible and auditable — so you can have a data-driven argument instead of a screaming match.
We implemented everything above at e-commerce company in May 2026. In the first week, their inference p99 latency dropped from 800ms to 120ms. Their training jobs got 35% more allocation without hurting revenue service. And the platform team stopped getting paged at 3 AM.
That's the goal. Not perfect fairness. Better outcomes.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.