HomeFind your caseOur GPUs sit idle while the bill keeps running
Case · Enterprise
A common situation · not a client storyOur GPUs sit idle while the bill keeps running.
Teams training or serving models on rented GPUs, paying for capacity that spends much of the day waiting.
What you're probably seeing
If this sounds familiar, you're in the right place.
- GPU use is low most of the day, but the instances stay on.
- Training jobs wait on data loading while the GPUs sit idle.
- Each team holds its own GPUs, and nobody shares.
- Inference servers are sized for peak, and peak is rare.
- Nobody can say what one training run or one answer costs.
The usual ways out
What teams usually try. Each works somewhere. None works everywhere.
| The way | Works when | The catch |
|---|---|---|
| Buy reserved capacity | Use is steady and high | Locks in today's idle time |
| Switch to spot or preemptible GPUs | Jobs can checkpoint and resume | Interruptions, and lost work without checkpoints |
| A shared scheduler and queue | Several teams need GPUs at different times | Needs rules everyone accepts, and jobs that can checkpoint |
| Fix data loading | GPUs wait on input | The fix depends on where the data lives |
| Right-size or scale inference to zero | Traffic is uneven | Cold starts, and some latency to manage |
| Smaller or quantised models | Quality holds at a smaller size | Needs an evaluation to prove it |
What other companies did. Their published results, not ours.
We read each company's own write-up and asked one question: would our diagnostic's tests have led to the same call? We judge the approach, not the result.
| Company | What they did | What they published | Source |
|---|---|---|---|
| RunwayVideo-generation model company. GPU clusters are shared by several research and production teams. Cluster size is not stated. | Put every team's GPUs behind Kueue. Each team keeps a guaranteed quota, idle quota is lent to a shared pool of interruptible jobs, and those jobs are preempted when the owning team needs the capacity back. | GPU utilization up 'over 20 percentage points' with team guarantees kept. Baseline and timeframe not stated. Preemption takes 20-30 minutes from start to the reserved job running.They noted: Kueue sometimes admitted jobs the cluster could not place. Preemption was slow. Borrowing required checkpointing, restart infrastructure and acceptance that default jobs can be interrupted. A CI check that blocks reservations exceeding physical capacity prevented more incidents than any other check. | No Idle GPUs: Managing Research Compute at RunwayApr 2026 |
| UberModel training on Ray clusters with Horovod and Petastorm. Datasets of tens of terabytes, larger than a worker's local disk. | Ran tests to find where training time went, then moved data transformation into worker threads and cached transformed data on local disk. They also made the data order reproducible. | GPU utilization from 10-15% to 60%+. Training time from about 22 hours to about 3 hours per run (6x speedup for this model).They noted: The first caching fix showed no gain because a second bottleneck was hidden behind the first: a CPU conversion step on the main thread. Eviction-based caching worked poorly because training reads every row group once per epoch. | Accelerating Deep Learning: How Uber Optimized Petastorm for High-Throughput and Reproducible GPU TrainingUndated |
| LunitAI company training deep learning models. Burst usage sometimes uses over a thousand GPUs at once, with multi-node distributed training. | Made spot instances the default for training and kept on-demand for urgent jobs. Built automatic preemption detection, resume on a new instance, and restart of multi-node runs. | Not stated. The post cites only general provider pricing: spot is 60-91% cheaper than on-demand.They noted: Retrying on another spot instance, as officially advised, sometimes caused an endless preemption loop. Each new instance took over 10 minutes to set up. Providers do not publish preemption rates. Losing the master node restarts the whole multi-node run. | Intelligent Cloud — Part 3: Optimizing GPU Costs by Leveraging Spot InstancesFeb 2024 |
| SalesforceEinstein AI Platform model-serving team. LLMs from a few GB to 20-30 GB on multi-GPU AWS instances (such as P4d and P5, including H200-based P5e/P5en). Larger, lower-traffic models often underused multi-GPU instances. | Worked with AWS so several models share the GPUs of one multi-GPU instance, each scaling within its own GPU limit, instead of one model per endpoint. | Cost savings of 'up to 8X' from hosting several LLMs on large instances. Separately, 'up to 80%' lower inference cost compared with single-model deployments, described as particularly useful for moderate-to-low traffic LLMs. No baseline or period stated.They noted: Shared hosting gives up some isolation and predictable performance, so they kept single-model endpoints as an option. | How AWS SageMaker Inference Components Save AI Inference Costs By Up to 8XMar 2025 |
Our read
Same call. Each company picked the fix that matched where its GPU time was going. Uber measured first and found a hidden second bottleneck.
Uber ran tests to see where training time went before changing code, and found GPUs at 10-15% waiting on data. That is our training-time test, and fixing data loading was the right call. Runway's problem was reserved quotas left idle across teams, so a shared queue with borrowing and preemption fits, and its write-up says this required checkpointing and restart infrastructure. Salesforce's large low-traffic models underused whole multi-GPU instances, so packing models and scaling each one fits uneven traffic. Lunit is the weakest match: it moved to spot to cut the unit price, but its post does not discuss idle time or report a measured saving, so we would still check hourly GPU use and cost per training run before and after.
What we would test first: Measure real GPU use on each instance, hour by hour, over a week.
What the diagnostic would test
The tests that rule the wrong ways out.
On your own systems and data, read-only first. The number of tests sets the diagnostic's price, quoted before it starts.
- 1
Real GPU use per instance over a week, hour by hour.
- 2
Where training time goes: compute, data loading or waiting.
- 3
Who uses which GPUs, and when.
- 4
Inference traffic against capacity, and the latency you need.
- 5
Cost per training run and per answer, today.
This week, without us
Three things you can do yourself.
- 1
Log GPU use every minute for a week on your busiest instances.
- 2
List every GPU instance, its owner, and what it ran last week.
- 3
Check whether your training jobs checkpoint often enough to survive an interruption.
Sound like you? Bring it to a 30-minute call. We'll tell you which tests your case needs, and what they cost.
This page describes what teams in this situation commonly face. It is not a client story.