How to Optimize Priority Derivation for Osprey
I spent three weeks in early 2026 staring at a dashboard that showed 40% GPU utilization. We had 256 NVIDIA H100s in a single cluster, running a mix of training jobs and model inference pipelines. The hardware was fine. The networking was fine. The problem was Osprey — our custom scheduler — and specifically how it derived priority for each task.
Most people think priority derivation is a simple ranking problem. Give each job a number, sort, execute. That's wrong. Priority derivation is a decision-making system that must adapt to real-time cluster state, cost constraints, and business urgency. Get it wrong and you're either starving critical jobs or wasting GPU cycles on low-value experiments.
Here's what I learned. Hard way.
The Problem: Why Priority Derivation Matters Now
Distributed training at scale isn't new. IBM's primer on distributed machine learning points out that the shift from single GPU to multi-GPU clusters requires entirely different scheduling logic. But the jump from, say, 8 GPUs to 1024 GPUs introduces a complexity curve that most teams underestimate. In a single GPU workload, priority is trivial — run one job, finish, run next. In a gpu cluster vs single gpu for ai workloads scenario, you're juggling preemptible jobs, long-running training, bursty inference requests, and data staging. Osprey's priority derivation needed to capture all of that.
The core insight: priority isn't just about job A vs job B. It's about deriving a scalar value from multiple, often conflicting signals — queue age, estimated remaining time, owner, SLA tier, current cluster fragmentation, and even carbon intensity. Optimize that derivation, and you can double throughput without buying more hardware.
What Most People Get Wrong About Priority
They treat priority as a static attribute. You assign a number when the job arrives, and it stays fixed until the job finishes. That's the first mistake.
The second mistake: priority is treated as a single global ordering. In a distributed system, that's not how physics works. Agentic systems are distributed systems, and priority must be localized — what's urgent on Node A may be irrelevant on Node B. Osprey's early design used a single priority queue across all nodes. It caused constant head-of-line blocking. Inference requests for a latency-sensitive model would sit behind a 12-hour training job that happened to have a slightly higher static priority.
I'm not saying static priority is useless. It's a baseline. But the derivation must be dynamic — recalculated at each scheduling decision based on current cluster state.
Key Metrics for Priority Derivation
We settled on four primary factors. Each gets a weight, and the total is normalized to a 0-1 score. The trick is tuning those weights — and making some of them contextual.
1. Urgency (business SLA)
Measured in minutes until deadline. A model update for a customer-facing product with a 1-hour SLA gets 0.8; an internal research experiment with no deadline gets 0.1.
2. Resource efficiency
How well does the job fit the current cluster fragmentation? If you have 4 free GPUs on a node and the job needs exactly 4, it gets a boost. If it needs 16 and will stall waiting for more nodes, it drops. This is the part that saved us most.
3. Queue age
Exponential decay: older jobs get higher priority, but we cap it to avoid starvation. After 24 hours, age contribution maxes out.
4. Owner fairness
Each team gets a share of total compute time. If Team Alpha has used 200 GPU-hours in the last week and Team Beta only 50, Beta jobs get a priority bump. This prevents resource hoarding.
We also experimented with carbon intensity (lower when grid is greener) but found it didn't affect throughput meaningfully for our workloads. Maybe in European data centers it matters more.
How to Optimize Priority Derivation for Osprey: The Core Algorithm
Here's the function we ended up with after 12 iterations. It's not fancy — it's a weighted sum with a normalization layer and a dynamic weight recalibration.
python
def derive_priority(job, cluster_state):
urgency = min(1.0, (job.deadline - now()) / (24 * 3600)) # normalized to 0-1
efficiency = job.gpu_count / cluster_state.free_gpus()
age = min(1.0, (now() - job.arrival_time) / (24 * 3600))
fairness = 1 - (team_usage[job.team] / target_share[job.team])
# Dynamic weights based on cluster utilization
if cluster_state.utilization < 0.5:
w_efficiency = 0.2
w_urgency = 0.5
w_age = 0.2
w_fairness = 0.1
else:
w_efficiency = 0.5
w_urgency = 0.2
w_age = 0.1
w_fairness = 0.2
priority = (w_urgency * urgency +
w_efficiency * efficiency +
w_age * age +
w_fairness * fairness)
return min(1.0, max(0, priority))
Yes, the weights change with utilization. That made a bigger difference than any other single change. When the cluster is underloaded, urgency dominates — get the critical work done. When it's overloaded, efficiency rules — fit more jobs into each allocation.
But that's still coarse. We then added a second pass that adjusts priority for inter-job dependencies. If job C depends on job D's output, job D gets a boost. This is classic distributed training scheduling wisdom. The billionhopes.ai article mentions this as "priority chaining." We implemented it as a graph traversal before each scheduling tick.
python
def propagate_dependency_priorities(jobs):
dep_graph = build_dependency_graph(jobs)
for job in topo_order(dep_graph):
for dependency in dep_graph.predecessors(job):
if dependency.priority < job.priority * 0.9:
dependency.priority = job.priority * 0.9
Simple. It nearly eliminated pipeline stalls.
Real-World Example: Scaling from 8 to 1024 GPUs
In April 2026, we moved a customer — let's call them Finova — from 8 GPUs to 1024 on Amazon SageMaker AI using distributed training with PyTorch DDP. The naive priority derivation (static, global) failed immediately. Jobs with high static priority but large GPU counts would occupy 256 GPUs for hours, blocking dozens of small inference jobs. Utilization dropped to 30%.
We rolled out the dynamic derivation above. Within two hours, utilization hit 85%. The inference jobs got through, the large training jobs still finished on time (just later in the cycle). The customer's cost per training run actually decreased because the cluster was no longer sitting idle waiting for large jobs to release resources.
Key insight: Not all GPU time is equal. A job that uses 256 GPUs for 10 hours consumes the same GPU-hours as 256 jobs using 1 GPU for 10 hours each. But the flexibility to interleave small jobs during large job provisioning slots is huge. Our efficiency weight captured that.
When to Ignore Priority
Contrarian take: sometimes you should run a lower-priority job first. If that low-priority job is short and its output unblocks a high-priority job, the end-to-end latency improves. This is equivalent to shortest-job-first with dependency awareness.
We had a case where a policy model training (priority 0.9) was waiting for a data preprocessing job (priority 0.3) that was queued behind three other medium-priority jobs. We temporarily boosted the preprocessing job to 0.95, ran it in 4 minutes, and the policy training launched immediately. Total wall clock time dropped from 3 hours to 45 minutes.
You need to trust your dependency propagation. We automated this with a rule: "if job A depends on job B, and B can finish in < 5% of A's remaining time after B finishes, then boost B above A." The logic is in the code above — the threshold check if dependency.priority < job.priority * 0.9 is a simplified version.
Tools and Techniques
You don't need a bespoke scheduler to implement these ideas. If you're on AWS, SageMaker's built-in prioritization can be customized via lifecycle hooks, but it's limited. Cloud-native and distributed systems for efficient AI workloads (the 2026 arXiv paper) shows how to embed Osprey-like logic into Kubernetes schedulers. We actually started with a Kubernetes mutating admission webhook that rewrote priorityClassName based on similar rules. It worked, but the overhead of calling an external service for every pod creation became a bottleneck beyond 5,000 pods/hour.
So we moved to a dedicated scheduler running alongside the cluster. Osprey is now a gRPC service that the cluster's training orchestrator queries before starting a job. It responds in under 1ms per request. The priority derivation runs in a background thread that updates the priority table every 10 seconds (or on state change events). This pattern is exactly what Agentic Systems Are Distributed Systems describes — a reactive actor model that maintains cluster state and responds to queries.
Historical Context: AWS and Cloud Computing
Side note on aws meaning cloud computing history: When AWS launched EC2 in 2006, nobody thought about GPU scheduling. The first GPU instances (EC2 G2) came in 2012. Back then, you spun up a single instance, ran your workload, terminated. Priority was a file in a queue script. As cloud computing evolved, so did the complexity of resource management. Today, in 2026, a typical cluster might have spot instances, reserved instances, savings plans, and different GPU types (H100, B200, etc.) all in the same pool. Priority derivation must account for cost differences, not just raw GPU count. We recently added a cost factor: if a job can run on cheaper spot capacity, we lower its priority on on-demand nodes, saving money. That's another layer of derivation.
FAQ
Q: What is Osprey?
Osprey is the internal scheduler SIVARO built for managing distributed AI training and inference workloads across heterogeneous GPU clusters. It derives priority for each task based on multiple dynamic signals.
Q: How often should priority be recalculated?
We recalculate every 10 seconds or on any cluster state change (node join/leave, job completion, job submission). Faster than that adds overhead; slower leads to stale decisions. For very volatile workloads (inference bursts), we push recalculation to every second.
Q: Can I use this approach with existing schedulers like Slurm or Kubernetes?
Yes. Slurm's job priority plugin can be replaced or extended. For Kubernetes, you can write a custom scheduler (like Volcano) or use a mutating webhook to adjust PriorityClass before pods are created. We have open-sourced a version for Kubernetes at github.com/sivaro/osprey-k8s.
Q: What's the biggest mistake teams make when implementing priority derivation?
Overfitting to one metric. I've seen teams make priority = queue age * 100, which just makes it FIFO. Or priority = GPU request size, which punishes large jobs. You need at least 3 factors, with dynamic weights.
Q: How do you handle fairness across teams without slowing critical work?
We use a "fairness budget" — each team gets a GPU-hour allocation per week. Urgency can override fairness temporarily, but a team that exceeds its budget gets a penalty multiplier. The fairness factor in the derivation algorithm handles this.
Q: Does priority derivation affect inference latency for real-time models?
Yes, and that's a separate concern. For inference, we use a preemptible priority scheme: inference jobs get a static high priority but can be preempted by training jobs if the cluster is full. Preemption triggers a failover to a different node or a fallback model. We wrote about this in Distributed Training & Large-Scale Systems.
Q: How do you test priority derivation changes without breaking production?
We have a simulator that replays historical cluster traces — job submissions, node failures, durations. We compare the total throughput and wait times of different derivation configurations. We run it nightly. Any proposed change must beat the baseline. That's how we found that dynamic weights gave 15% better throughput than static weights on our historical data.
Q: Is there a one-size-fits-all priority formula?
No. Our current algorithm is version 12. Each customer's workload mix is different. A cluster running only large training jobs needs different weights than one mixing inference and training. The framework — dynamic weights, dependency propagation, contextual factors — is universal. The exact numbers are not. Measure, iterate, measure again.
Final Advice
Optimizing priority derivation for Osprey isn't a one-time project. It's a living system. We update our weights monthly based on cluster telemetry. The team first thought this was a scheduling problem — turns out it was a measurement and feedback problem. You can't optimize what you don't observe.
Start with a baseline: static priority + FIFO. Add one factor at a time. Measure utilization and job completion times. When you see a regression, revert. I've seen teams try to build the perfect algorithm from day one and end up with a system nobody understands.
Keep it simple. Our final algorithm is 40 lines of Python. It runs in a few milliseconds. The complexity is in the data — the cluster state, the dependency graph, the team usage history. Feed it good data, and the derivation takes care of itself.
One more thing: don't forget human override. Sometimes a VP needs a job to run right now because a customer demo is in 2 hours. We have an API endpoint that accepts a manual priority boost with a 1-hour TTL and a mandatory audit log. It's used maybe once a month. But it prevents people from gaming the system.
Now go look at your own cluster dashboard. If utilization is below 70%, priority derivation is probably the culprit.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.