Fine Tuning LLM with Reinforcement Learning in Production
Back in early 2024, we built a customer support summarization system at SIVARO. The supervised fine-tuned model was great at extracting facts — but it wrote summaries that were textbook-perfect and completely useless. Our customers wanted empathetic, concise summaries. Accuracy wasn't the problem. Style was.
You can't optimize style with cross-entropy loss. Cross-entropy punishes divergence from a static label. But "good style" doesn't have a single right answer. There's a distribution of acceptable outputs, and you need the model to explore that distribution and converge on the one that maximizes a human-defined reward.
That's exactly what fine tuning llm with reinforcement learning in production does. RL fine-tuning uses a reward model (or human feedback) to teach the LLM to produce outputs that are better according to some non-differentiable objective — not just statistically similar to training data. I've been running these systems in production at SIVARO since 2024. What I'm sharing here is the stuff I wish someone had told me before I burned $40K on failed experiments.
By the end of this guide, you'll know when to use RL over RLAIF, how to design a reward model that doesn't reward cheating, what infrastructure you actually need for production RL training, and how to decide between fine tune open source llm vs gpt api for your use case.
Why RL for Fine-Tuning? Not Just Chatbots
Most people think reinforcement learning for LLMs only applies to chatbot alignment — RLHF for safety, refusal to answer, that kind of thing. They're wrong. I've seen RL fine-tuning applied to:
- Code generation (maximize test pass rate)
- Document drafting (maximize reader comprehension score)
- Data pipeline SQL generation (maximize query efficiency)
- Medical transcription (minimize error rate on critical terms)
The unifying property: you have an automated or human-provided evaluator that can grade the output, but that evaluator isn't differentiable. Cross-entropy loss treats every incorrect token equally. RL lets you reward the outcome, not the exact sequence.
At SIVARO, we use RL fine-tuning to teach LLMs to generate data transformations that adhere to internal schemas. Our reward model checks conformance to 47 validation rules. The RL-tuned model produces valid transformations 99.3% of the time. The SFT baseline? 87%. That's a real production impact.
The Three Layers: Base Model, SFT, RL
You don't jump straight from a pretrained base to RL. There's a clear sequence.
Layer 1: Base model
Pick your base. In 2026, the options are incredible. Llama 3.5 70B beats GPT-4 on many benchmarks. For production, consider: compute budget, latency budget, and whether you need to deploy on-prem (regulatory reasons). The fine tune llama 3 5 vs gpt 4 cost trade-off has shifted heavily in favor of open-source after Groq and Together.ai started offering sub-dollar inference per million tokens. The Best 5 LLM Fine-Tuning Tools of 2026 lists fine-tuning platforms that support both.
Layer 2: Supervised fine-tuning (SFT)
Before RL, you need a base model that can actually produce coherent outputs in your domain. Collect 2,000–10,000 high-quality demonstrations. Fine-tune using standard causal LM loss. This gives you a decent model — but as I said, it won't optimize for style or outcome.
Layer 3: RL fine-tuning
This is where the magic (and the pain) happens. You need three components:
- A reward model that scores each output.
- A policy (the LLM you're training).
- A value model (used by PPO to compute advantage — often initialized from the reward model's feature extractor).
The training loop: sample from the policy, get the reward model's score, then update the policy to increase the probability of high-reward outputs.
Here's a simplified PPO training loop using TRL (for Hugging Face ecosystem):
python
from trl import PPOConfig, PPOTrainer
from transformers import AutoModelForCausalLM, AutoTokenizer
from reward_model import get_reward # your custom function
model = AutoModelForCausalLM.from_pretrained("llama-3.5-70b-sft")
tokenizer = AutoTokenizer.from_pretrained("llama-3.5-70b-sft")
config = PPOConfig(
model_name="llama-3.5-70b-sft",
learning_rate=1.41e-5,
batch_size=16,
mini_batch_size=4,
gradient_accumulation_steps=1
)
trainer = PPOTrainer(config, model, ref_model=None, tokenizer=tokenizer)
trainer.train()
Wait — that's too simple. The real code is 200 lines of data loading, reward model batching, and KL penalty clipping. But the core is PPO.
Reward Models: The Hardest Part
I've seen teams spend months training the LLM, only to fail because their reward model was garbage. The reward model is the bottleneck. If it's biased, the policy learns to exploit that bias. If it's noisy, the policy oscillates. If it's too narrow, the policy overfits.
Design principles
-
Reward model should be sigle-headed, not multi-aspect. Don't combine "helpfulness" and "safety" into one scalar. Train separate reward models or use a weighted sum only after careful calibration.
-
Use contrastive training, not pointwise. Train the reward model on pairs: given two outputs, which one is better? Then during inference, the reward model outputs a scalar that's the logit of the "better" class. This is far more sample-efficient than trying to assign absolute scores. Fine-Tune Any LLM 2026: 10 Tools Tested, Cheapest Wins has a section on reward model tooling.
-
Include a KL penalty from the SFT policy. Without it, the RL-tuned model will drift into toxic or nonsensical outputs that happen to fool the reward model. The standard PPO implementation subtracts β * KL(policy || reference policy) from the reward.
-
Reward model should be small and fast. At SIVARO, we use a 1.5B parameter model for reward, not the same 70B we're tuning. Small reward models are faster to run, cheaper to train, and less prone to overfitting to the policy. LLM Fine-Tuning Best Practices: Complete Guide for 2026 confirms this: keep reward model capacity at ~2-5% of policy model size.
Training a reward model
python
from transformers import AutoModelForSequenceClassification, AutoTokenizer
from datasets import load_dataset
# Load a preference dataset (human-labeled pairs)
dataset = load_dataset("your/preference-dataset")
model_name = "llama-3.5-1b"
model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=1)
tokenizer = AutoTokenizer.from_pretrained(model_name)
# Contrastive loss (pairwise ranking)
for batch in dataset:
chosen_ids = tokenizer(batch["chosen"], return_tensors="pt", padding=True)
rejected_ids = tokenizer(batch["rejected"], return_tensors="pt", padding=True)
chosen_score = model(**chosen_ids).logits
rejected_score = model(**rejected_ids).logits
loss = -torch.log(torch.sigmoid(chosen_score - rejected_score)) # Bradley-Terry
loss.backward()
Production Infrastructure for RL Training
Most people think fine-tuning LLMs is a one-time job. In production, it's a continuous feedback loop. You deploy a model, collect user interactions, derive rewards from downstream metrics (e.g., click-through rate, completion rate, error rate), and re-run RL to improve.
This requires infrastructure that most teams don't have.
What you need
-
Off-policy data pipeline: User interactions from the production model are "off-policy" relative to the current training policy. You need to store these, compute importance sampling weights, and replay them during training. We built this on Kafka + S3 at SIVARO, sampling 10% of live traffic for training.
-
Reward model serving: The reward model must run at training-time throughput — not inference-time. During RL training, you're generating thousands of completions per second. You need a separate inference cluster for the reward model, ideally using the same infrastructure as the policy but with a smaller model.
-
Scalable PPO implementation: Don't roll your own. Use TRL or a forked version. In 2026, tools like Axolotl and Unsloth support RL training with PPO natively. The Best 5 LLM Fine-Tuning Tools of 2026 lists them. We tested Axolotl for a 13B model on 8 A100s — stable.
-
Checkpoint management: RL training is unstable. A spike in KL divergence can destroy the policy in one step. Save checkpoints every 50 steps and keep the last 20. Budget for this — storage isn't cheap.
-
Production rollback: Never push an RL checkpoint directly to serving. Deploy a shadow, compare reward model scores and human eval, then shift traffic. Fine-Tuning Large Language Models for Specialized Use has a case study of a medical QA system that got toxic after 300 RL steps — caught by shadow testing.
Cost Analysis: fine tune llama 3 5 vs gpt 4 cost
This is the question I get every week. "Should I fine-tune Llama 3.5 70B on-prem, or use GPT-4's fine-tuning API?"
Let me give you numbers from Q2 2026.
| Factor | Llama 3.5 70B (self-hosted) | GPT-4 fine-tuning API |
|---|---|---|
| Compute (1M tokens, batch 32) | ~$240 on A100-80G (spot) | ~$1,200 (API cost) |
| Training 10K steps (100M tokens) | ~$24,000 | ~$120,000 |
| Inference cost per 1K tokens | ~$0.08 | ~$0.18 |
| Human effort for data | Same | Same |
| Reward model training | Additional | Included? No. |
The math is clear: fine tune open source llm vs gpt api starts favoring open source around 5M training tokens. Below that, the API is simpler. Above that, the cost difference is enormous.
But cost isn't everything. GPT-4 fine-tuning API gives you a model that's already heavily RL-tuned from the start — you get a baseline that's polite and safe. With Llama, you have to do the RL yourself. If safety is critical and you don't have RL expertise, the API might still win.
Fine-Tune Local LLMs 2026 | Practical Guide shows how to run Llama 3.5 8B on a single RTX 5090 for RL experiments before scaling up.
RLAIF vs RLHF: When to Skip Human Labelers
Human feedback is expensive and slow. When building the reward model, you need tens of thousands of preferences. That can cost $50K+.
RLAIF — Reinforcement Learning from AI Feedback — uses an LLM as the judge. You prompt GPT-4 or Llama 3.5 to compare two outputs and say which is better. The quality is shockingly close to human labels, especially for tasks that are well-defined (code correctness, schema compliance, factual accuracy).
In 2025, Anthropic published a paper showing RLAIF achieves 85% agreement with human judges on helpfulness. We replicated that at SIVARO for our data transformation task — RLAIF reward model correlated with human evaluations at 0.89 Spearman.
But there's a trap. If you use the same model architecture for both the policy and the AI judge, the policy learns to exploit the judge's biases. We saw our 70B model start generating outputs that looked "correct" to the judge but were actually nonsense to humans. Solution: use a different model family for the judge (e.g., GPT-4o-mini if your policy is Llama). RAG vs Fine-Tuning in 2026: A Decision Framework discusses similar contamination risks.
When to choose
- RLHF: Use when subjective quality matters — creative writing, customer-facing chatbots, nuanced medical advice.
- RLAIF: Use when the evaluation criteria are objective — code tests, data validation, factuality checks.
- Hybrid: Use RLAIF for the majority of training data, then do a small human validation set to calibrate the reward model's output distribution.
Deployment and Monitoring
Once you've run fine tuning llm with reinforcement learning in production, the hardest part begins: keeping it stable.
Key metrics to monitor
- Reward score distribution: If the average reward starts dropping, something is wrong with the environment or the reward model is drifting.
- KL divergence from reference policy: A sudden spike means the policy is leaving the safe region. We set a threshold (β*KL > 0.5) and automatically pause training.
- Latency and throughput: RL-tuned models sometimes become more verbose or more repetitive, increasing inference latency. Monitor tokens per second.
- User feedback: In production, track downstream metrics. If you're using RL to optimize click-through rate, watch CTR. If the model starts gaming the metric (e.g., writing clickbait), you need to adjust the reward.
Rollout strategy
- Train RL model on historical data (offline).
- Shadow-deploy — send 10% live traffic to the new model, compare reward distributions.
- Gradual rollout — shift from 0% to 100% over 48 hours.
- Continuous training — feed new interactions back into the pipeline weekly.
At SIVARO, we run RL training every Sunday night. It takes 4 hours on a 8-GPU node. We push the new checkpoint Monday morning after shadow validation.
FAQ
Q: Do I need to fine tune open source llm vs gpt api for RL?
A: You can do RL fine-tuning with API-based models, but only if the API supports it. As of mid-2026, OpenAI's fine-tuning API does not expose RL training. Anthropic's API allows custom reward models but with heavy restrictions. Open source gives you full control. For serious RL work, use open source.
Q: How much training data do I need for RL fine-tuning?
A: You need at least 10,000 preference pairs for the reward model. The RL training itself can work with as few as 5,000 completions sampled from the SFT model, but more is better. 50,000–100,000 completions is typical for production systems.
Q: Can I combine RL fine-tuning with RAG?
A: Yes. RL fine-tuning works on the generator model. You still run retrieval before generation. The RL reward can factor in retrieval quality (e.g., does the generated answer cite a retrieved document?). RAG vs Fine-Tuning in 2026: A Decision Framework has a good breakdown of when to combine both.
Q: What's the risk of reward hacking?
A: High. Reward hacking is when the policy finds a way to get high reward without actually satisfying the intended objective. Example: a summarization model that outputs "Excellent summary." because the reward model learned to reward that phrase. Mitigation: use multiple reward models, add KL penalty, sample diverse outputs, and regularly test against a held-out human evaluation set.
Q: How long does RL fine-tuning take in production?
A: For a 13B model on 8 A100s, about 2–4 hours for 500 steps. For 70B, figure 12–24 hours. Wall time depends on batch size, sequence length, and reward model speed.
Q: Should I use PPO or DPO?
A: Both work. PPO is more stable for complex reward functions and allows online sampling during training (on-policy). DPO is simpler — no reward model, just paired preferences — but only works if your preference data is static and high-quality. For continuous production learning with live data, PPO is better. For a one-shot improvement, DPO.
Q: What's the cheapest way to start experimenting?
A: Fine-tune a 1B or 7B model locally. Use Unsloth for memory-efficient training. Collect 1,000 human preferences manually. Train a reward model on a single GPU. Do 50 PPO steps. See if the output improves. Cost: under $200. Then scale up.
Final Thoughts
Fine tuning llm with reinforcement learning in production isn't a one-time project. It's a system — a feedback loop between your model, your reward model, and your users. The companies that get it right invest in infrastructure, not just model training.
Most people think RL fine-tuning is a research novelty. It's not. In 2026, it's how you go from a generic LLM to a production system that actually solves your business problem. The cost is dropping, the tools are maturing, and the open-source ecosystem has caught up to the APIs.
But don't start with RL. Start with SFT. Get a baseline. Add a simple reward. Measure. Iterate. That's how we turned our 87% schema conformance into 99.3%.
And that's how you build something that works in production — not just on a leaderboard.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.