Bidirectional Resource Scheduling Post Training LLM
I remember the moment it clicked. Late February this year. We were bleeding compute on a Llama 3.5 fine-tuning run. Our cluster looked busy, but loss was flatlining. The GPU utilization graphs told a different story: 30% average, peaking in weird bursts. Everyone focuses on model architecture or dataset quality. They miss the real bottleneck—how you schedule resources after training has started.
This isn't about training from scratch. It's about the messy, iterative post-training phase: fine-tuning, RLHF, alignment, domain adaptation. The part that eats 70% of your budget if you're doing custom models.
Bidirectional resource scheduling is the practice of treating both forward (compute → data) and backward (data → compute) signals as first-class citizens in your resource allocation. Forward means provisioning hardware for the training step. Backward means reacting to training dynamics—changing dataset mix, adjusting batch sizes, reshuffling shards—based on real-time loss signals and gradient patterns.
Most teams build one-way pipelines. Data flows to GPUs. That's it. Then they wonder why fine-tuning stalls or overfits.
I'll show you what we've learned building production AI systems at SIVARO. The scheduling patterns that saved us 40% in compute. The tools we actually use. And why open source fine-tuning isn't cheaper unless you do this right.
Why Post-Training Scheduling Became a Bottleneck
Fine-tuning an LLM isn't a static job. You're trying to adjust a 70B-parameter model on a custom dataset. The first pass might look fine—then on epoch three, the model starts regurgitating. Or the loss spikes on a specific data shard. Or you realize you need more synthetic data.
Standard job schedulers (SLURM, Kubernetes with simple batch jobs) treat all steps uniformly. They allocate nodes, run the script, collect results. That's for training from scratch. Post-training is different: you need to adjust resources dynamically based on what the model is telling you.
In June 2026, we saw a client burning $80K/month on fine-tuning runs that didn't converge. Their scheduler was static. We added bidirectional feedback loops. Cost dropped to $32K. The Best 5 LLM Fine-Tuning Tools of 2026 list several tools now supporting dynamic resource adjustment—but adoption is still low.
The problem is cultural. Teams think "just throw more GPUs" or "use a bigger dataset." They don't schedule for the actual training dynamics.
What Is Bidirectional Resource Scheduling?
Let me define it tightly. Bidirectional resource scheduling means your compute allocation and your data pipeline are connected by a feedback loop. The training process can request more compute, change data composition, or pause for re-processing. At the same time, the scheduler can override training parameters based on system state—node failure, thermal throttling, job priority.
Two directions:
- Forward scheduling: Provision GPUs, allocate memory, set parallelism. This is what everyone does.
- Backward scheduling: The training loop emits signals (loss per step, gradient norms, sample hardness) that trigger resource changes. Example: loss on a specific shard goes up → scheduler pauses that shard, fetches augmented version, resumes with higher priority.
This isn't just adaptive training. It's scheduling that treats data and compute as co-equal peers, not master and slave.
We implemented this at SIVARO using a custom orchestration layer over Ray and Kubernetes. The key insight: the scheduler must understand both resource topology and data provenance. If shard 47 from your custom dataset causes gradient explosion, the scheduler needs to know not just "shard 47" but "shard 47 came from chunk B of the legal corpus, which was synthetically generated with temperature 0.8." Then it can route that shard for reprocessing.
The Three Layers You Must Schedule
Compute
GPUs are scarce and expensive. In 2026, H100s still dominate, but B200s are trickling in. Cost per hour varies wildly: $2.50 on reserved spot vs $15 on on-demand. Bidirectional scheduling means you can preempt lower-priority runs when higher-priority jobs arrive. But you need the backward signal: "this fine-tuning job is 2% from convergence, don't kill it, allocate more nodes from the spot pool."
We use a priority queue with two signals: estimated completion time and current loss improvement rate. If a job is improving fast, we boost its priority. If it's flatlining, we demote and eventually preempt.
Data Pipeline
This is the layer most teams ignore. Data preprocessing for post-training is often done once at the start. Bad idea. In February 2026, we fine-tuned a 13B model on legal documents. Halfway through, we noticed the model couldn't handle ambiguous clauses. We needed to augment the dataset with contrastive examples—but our pipeline was static. Rewind, reprocess everything, re-launch. Two days wasted.
Now we schedule data transformations as dynamic jobs. The scheduler monitors training loss by category (legal categories, in that case). When loss on "contract ambiguity" exceeds a threshold, it triggers a pipeline to generate more ambiguity examples, mix them into the training queue, and accelerate their training frequency.
Tools like SuperAnnotate's 2026 guide cover auto-labeling and active learning loops—but they don't tie it to resource scheduling. You have to build that bridge.
Model Synchronization
Distributed training has its own scheduling nightmare. When fine-tuning across 64 GPUs with FSDP, you need to sync gradients at precise intervals. But if one node lags (network contention, memory pressure), the whole job slows down. Bidirectional scheduling can detect a lagging node and either redistribute its shards or spawn a replacement before the lag causes a burst barrier sync.
We learned this the hard way. A training run with 128 A100s was spending 40% of time waiting for all-reduce. We added a backward signal: each worker reports its current step time. If variance exceeds 10%, the scheduler reduces the number of workers for that batch (dynamic world size). Throughput jumped 2x.
How We Solved the Stale Gradient Problem at SIVARO
Stale gradients kill fine-tuning. When you adjust resources mid-run, you might interrupt gradient accumulation. Workers that were on different micro-batches produce gradients that don't align. Standard approach: sync after every intervention. That's slow.
We developed what we call "gradient snapshot scheduling." Before any resource change, the scheduler forces a collective checkpoint of gradients. Then the resource change happens. After, the scheduler reloads the gradient states and continues. Sounds expensive. It's not. Checkpointing takes ~2 seconds. The alternative (dropping gradients and restarting from save) costs 2 minutes per intervention.
We open-sourced the core mechanism in April 2026. You can find it on our GitHub (linked from SIVARO's site). It's a simple Python wrapper around torch.distributed.checkpoint. The scheduling logic checks every 50 steps whether a resource change is needed. If yes, it triggers the gradient snapshot.
The result: we can scale down from 256 GPUs to 64 mid-run without losing gradient fidelity. Perfect for spot preemption.
Open Source vs Closed Source Fine-Tuning: Cost Reality 2026
Everyone asks: "Should I fine-tune Llama 3.5 or use GPT-4o?" The answer depends on your scheduling maturity. This 2026 comparison shows open source can be 3x cheaper—but only if you manage compute efficiently. Without bidirectional scheduling, open source fine-tuning often costs more because you waste GPU hours on inefficient runs.
Let me give numbers from our client work:
- Closed source fine-tuning (OpenAI, Anthropic, Google): $0.05–0.15 per 1K tokens processed. For a 10M token fine-tuning job, that's $500–$1,500. No infrastructure cost. But you lose control over the scheduling. You can't dynamically adjust data mix. You're stuck with their pipeline.
- Open source fine-tuning (Llama 3.5, Mistral 7B, Qwen 2.5): Compute cost ~$200–$600 for the same job on spot H100s. But you need a scheduler. Without one, add 40% overhead for failed runs, wasted idle GPUs, and rework. This ScienceDirect paper covers training efficiency metrics—they found that poorly scheduled fine-tuning wastes 28% of compute on average.
Our recommendation: if your dataset is small (< 1M tokens) and stable, closed source is cheaper total cost. If you're doing iterative post-training (multiple rounds of RLHF, data augmentation, domain expansion), open source with bidirectional scheduling wins—by 5x or more.
The key metric is not just cost per training hour, but cost per acceptable model. Our bidirectional scheduler reduces the number of failed runs from ~30% to ~5%. That's the real savings.
A Real-World Example: Fine-Tuning Llama 3.5 on Custom Dataset
Let me walk through how to fine-tune llama 3.5 on custom dataset with bidirectional scheduling. I'll assume you have a dataset of 500K instruction-response pairs for a medical QA system.
Step 1: Prepare with dynamic sharding
Don't create a static train/val split. Instead, define shards with metadata:
python
# shard_config.yaml
shards:
- id: "cardio_001"
source: "cardiology_books_2025"
type: "factual"
size_mb: 45
hardness_score: 0.7 # initial estimate
- id: "neuro_002"
source: "neurology_questions_synthetic"
type: "reasoning"
size_mb: 32
hardness_score: 0.9
The hardness_score will be updated during training based on loss per shard.
Step 2: Launch with scheduling hooks
We use a launcher script that communicates with the scheduler via REST:
bash
python launch_finetune.py --model meta-llama/llama-3.5-70b --shard_config shard_config.yaml --scheduler_endpoint http://scheduler:8080 --gradient_snapshot_dir /checkpoints/grads --max_gpus 128 --spot_threshold 0.7 # tolerate up to 30% preemption
Step 3: Training loop with backward signals
Inside the trainer, we emit events every N steps:
python
class BidirectionalTrainer(transformers.Trainer):
def training_step(self, model, inputs):
loss = super().training_step(model, inputs)
step = self.state.global_step
if step % 50 == 0:
# Send loss per shard to scheduler
scheduler_client.report_shard_stats({
"shard_id": inputs["shard_id"],
"loss": loss.item(),
"gradient_norm": compute_grad_norm(model),
"throughput": self.state.gpu_throughput
})
return loss
The scheduler aggregates these stats and decides:
- Shard "neuro_002" has loss 2.3 (high) → replicate it with data augmentation at higher temperature.
- Node 12 is 15% slower than average → drain it, redistribute its micro-batches to other nodes.
- Run is converging well → increase batch size to speed up.
Step 4: Dynamic resource adjustment
The scheduler sends commands back:
python
# Scheduler decision
{
"action": "augment_shard",
"shard_id": "neuro_002",
"method": "back_translation",
"count": 200,
"priority": "high"
}
This triggers a data pipeline job that runs on CPU nodes, generates new examples, and pushes them to a priority queue. Within two minutes, the training loop picks up the new samples.
For how to fine-tune llama 3.5 on custom dataset properly, these three steps ensure you're not just running a script but managing resources intelligently.
Tools That Actually Work
Don't build from scratch. Use these:
- Deepchecks – Their fine-tuning toolkit (linked earlier) includes a bidirectional scheduling module since their v2025.2 release. We tested it. It handles basic shard reprioritization well, but lacks advanced gradient snapshotting. The Best 5 LLM Fine-Tuning Tools of 2026 ranks it top for monitoring.
- TechSy's fine-tuning solution – We evaluated their 2026 platform. It's cheaper than Deepchecks but the scheduling isn't truly bidirectional—it's adaptive at the data level only. Their comparison shows cost advantages for small runs.
- SuperAnnotate – Great for data pipeline management. Their active learning loop feeds into a scheduler, but you need to write the glue code. Their 2026 guide covers the concept but not the implementation.
- Ray + Kubernetes – This combination gives you the flexibility. We built our custom scheduler on Ray's distributed actor model. Kubernetes handles node allocation, Ray handles training and data ops. Not turnkey, but you need customization anyway.
If you're just starting, use Deepchecks for visibility, then layer on your own backward signals.
Bidirectional Scheduling in Production AI Systems
At SIVARO, we run production AI systems that need continuous fine-tuning. A customer's model starts degrading as new data arrives. We can't take it offline to retrain. So we schedule fine-tuning jobs that coexist with inference traffic.
Bidirectional scheduling here means: inference latency monitoring feeds into training resource allocation. If inference queue size spikes, the scheduler preempts fine-tuning jobs. If training loss plateaus, the scheduler pauses training, runs a batch of inference requests through the model to gather new hard examples, then resumes training with those examples.
This is the frontier. Most companies separate training and inference completely. They shouldn't. The feedback loop between them is bidirectional resource scheduling at the system level.
We've built this into our product. It's not hypothetical. It's running on clusters processing 200K events per second.
FAQ
Q: What's the difference between bidirectional scheduling and dynamic batch sizing?
A: Dynamic batch sizing is one small part of bidirectional scheduling. The full concept includes data pipeline changes, compute reallocation based on loss signals, and gradient state management.
Q: Do I need bidirectional scheduling for all fine-tuning?
A: No. If you're doing one-shot fine-tuning on a small, clean dataset (< 100K examples, no iteration), static scheduling is fine. You need it when you're doing iterative post-training—RLHF, domain adaptation, multi-task fine-tuning.
Q: How does this compare to hyperparameter optimization (HPO) tools?
A: HPO tunes hyperparameters before the run. Bidirectional scheduling adjusts resources during the run. They complement each other.
Q: What's the overhead of the scheduling loop?
A: Our implementation adds < 2% compute overhead for the backward signals. The gradient snapshot adds ~2 seconds per resource change. Total overhead is less than 3% in production.
Q: Can I use it with closed-source APIs (OpenAI fine-tuning)?
A: No. The APIs don't expose enough control. This is only for self-hosted fine-tuning.
Q: How long to implement a basic bidirectional scheduler?
A: A team with Kubernetes and Ray experience can do a prototype in 2 weeks. Production-ready takes 2-3 months. We ship a turnkey version at SIVARO if you don't want to build.
Q: What metrics should I monitor to know if scheduling is working?
A: GPU utilization per run, failed run rate, convergence speed per compute dollar, and time-to-convergence. If these improve, your scheduling is working.
Conclusion
Bidirectional resource scheduling post training llm isn't a nice-to-have. It's the difference between a model that works and a budget that bleeds. The field has moved beyond "just fine-tune and pray." Tools are maturing. AI Agents Plus Best Practices Guide now includes scheduling recommendations. But most practitioners still think statically.
Start with one feedback loop: loss signal → data reprioritization. Then add gradient snapshotting for compute changes. Then tie it to inference monitoring.
You'll spend less. Your models will converge faster. And you'll stop asking "how to fine tune llama 3.5 on custom dataset" without considering the scheduling that makes it efficient.
The fine tuning cost comparison open source vs closed source llm only tells part of the story. The rest is how you schedule.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.