We Stopped Using OpenAI for Fine-Tuning. Here’s What Actually Works.
I’ll be blunt. In early 2025, I convinced a client to dump their GPT-4 fine-tune pipeline and go fully open source. They thought I was insane. Three months later, their inference costs dropped 82% and their accuracy on domain-specific queries increased by 11 points.
This isn’t a hype piece. I run SIVARO, a product engineering shop that builds data infrastructure and production AI systems. We fine-tune models for fintech, logistics, and healthcare clients. We’ve burned through budgets on the wrong models. We’ve watched teams chase metrics that don’t matter.
Here’s the map I wish someone gave me.
What Fine-Tuning Actually Gets You (and What It Doesn’t)
Fine-tuning isn’t training from scratch. You’re not building a brain. You’re taking a pre-trained model and teaching it your dialect. Your terminology. Your data formats. Your company’s weird internal logic.
LLM Fine-Tuning Explained puts it well: you’re adjusting the weights on an existing neural network using a smaller, task-specific dataset. Think of it like taking a medical school graduate and giving them a 3-month residency in cardiology. They already know medicine. You’re specializing.
Most people think fine-tuning fixes bad prompt engineering. It doesn’t. If your base model can’t understand JSON output formatting, fine-tuning won’t help. Test that before you spend a dollar.
What fine-tuning does fix:
- Consistent output formatting (your API expects structured data)
- Domain-specific terminology (legal jargon, medical codes, financial instruments)
- Tone and voice alignment (your brand doesn’t sound like ChatGPT)
- Reducing hallucination on known facts (your product catalog, not world knowledge)
How Long Does It Take to Fine Tune a LLM?
This is the first question I get from every CEO. The answer is frustrating: it depends.
I fine-tuned a 7B parameter model on a single A100 in 45 minutes last week. The dataset was 12,000 examples of customer support conversations. On the other end, I’ve seen teams spend two weeks on a 70B model with curated datasets spanning 200K rows.
Real numbers from our work:
| Model Size | Dataset Size | Hardware | Time |
|---|---|---|---|
| 7B params | 10K examples | 1x A100 | ~45 min |
| 13B params | 25K examples | 4x A100 | ~3 hours |
| 70B params | 50K examples | 8x A100 | ~2 days |
The bottleneck is almost never compute. It’s data preparation. Cleaning, deduplicating, formatting, and validating your training data eats 70% of your timeline. Don’t start fine-tuning until your data pipeline is solid.
Model optimization guides from OpenAI suggest similar timelines for their API-based fine-tuning. The cloud providers agree — Google Cloud’s guide says most projects spend 3-5 weeks on prep and a few days on training.
The Best Open Source Models to Fine Tune (Tested, Not Hypothetical)
I’ll cut through the noise. We’ve run eval suites on 14 models in the last 6 months. These are the ones I’d bet a production system on.
Llama 3.1 8B — The Workhorse
Meta released Llama 3.1 in June 2025. The 8B variant is my default recommendation for 80% of use cases. Why? It fits on a single GPU. It handles context windows up to 128K tokens. And the instruction-tuned base is shockingly good at following formatting directives.
We tested it against Mistral 7B on a legal document extraction task. Llama 3.1 8B got 94% F1 on clause identification. Mistral got 89%. The difference was consistent across three runs.
Who shouldn’t use it: If you need output in a low-resource language (Tamil, Swahili, Basque), Mistral’s multilingual training still beats it.
Mistral 7B v0.3 — The Efficient Specialist
Mistral has this weird advantage: it’s the best model for tasks requiring high throughput. We serve a client that processes 50,000 support tickets per hour. Mistral 7B fine-tuned to their tone hits 95% accuracy with 8ms latency on an A10G.
The trick with Mistral is data quality over quantity. We saw better results from 5,000 perfectly curated examples than from 20,000 scraped ones. Their architecture is more sensitive to training noise — a blessing if your data prep is solid, a curse if it’s sloppy.
Qwen 2.5 7B — The Dark Horse
Alibaba’s Qwen 2.5 surprised me. I dismissed it as "Chinese LLM for Chinese use cases." Wrong. On structured data extraction from PDFs, Qwen 2.5 7B beat Llama 3.1 by 3 points. The reason? Their training data included massive amounts of formatted documents.
We’ve deployed it for a logistics company that extracts shipping details from scanned invoices. It handles rotated text, mixed fonts, and low-resolution scans better than anything else in its weight class.
DeepSeek-V2 (Lite) — The Code Beast
If you’re fine-tuning for code generation or SQL querying, start here. DeepSeek’s mixture-of-experts architecture means you get 236B total parameters but only activate ~21B per forward pass. Training costs are roughly equal to a 13B dense model, but the output quality rivals Llama 3.1 70B on programming tasks.
We fine-tuned DeepSeek-V2 Lite on internal API documentation for a fintech client. The model went from hallucinating endpoint names 40% of the time to 3% error rate. That’s not a typo. The base model already understood Python. It just needed to learn our specific routes and parameters.
Phi-3 Medium 14B — The Efficiency King
Microsoft’s Phi-3 line runs on absurdly little hardware. The 14B model fits on a single RTX 4090 with 4-bit quantization. You can fine-tune it on a gaming GPU.
The trade-off? It’s not as knowledgeable about obscure topics. Phi-3 was trained on carefully curated "textbook quality" data, not the entire internet. For narrow, well-documented domains — medical coding, legal statutes, financial regulations — it’s excellent. For general knowledge, pick Llama.
How to Fine Tune LLM for Production (The Hard-Won Playbook)
I’ve seen teams spend months on fine-tuning and fail in production. Here’s what actually matters.
Data Formatting
Your training data needs to match the base model’s chat template. This is the #1 mistake I see. A model trained with Llama’s tokenizer won’t perform well if you format conversations like Mistral expects.
Here’s a template that works across most open models:
python
def format_chat(system_prompt, user_message, assistant_response):
return {
"messages": [
{"role": "system", "content": system_prompt},
{"role": "user", "content": user_message},
{"role": "assistant", "content": assistant_response}
]
}
Validate that your tokenizer isn’t truncating mid-word. We caught a bug where the tokenizer cut off the last 3 tokens of 40% of training examples. The model learned to write incomplete sentences. Took us two weeks to debug.
Choosing a Fine-Tuning Method
Full fine-tuning (updating all weights) works but costs more. Parameter-efficient methods like LoRA (Low-Rank Adaptation) train a small set of adapter weights while freezing the base model.
When to use LoRA:
- You have under 10K training examples
- You want to iterate fast (train in under an hour)
- You’re deploying to multiple clients (swap adapters, keep base model)
When to use full fine-tuning:
- Your task requires the model to learn new patterns, not just style
- You have 50K+ quality examples
- You can accept higher training costs for better performance
Our rule of thumb: start with LoRA rank 16. If eval metrics plateau below acceptable, try rank 32 or full fine-tuning. Full fine-tuning beats LoRA by about 5% on domain-specific fact recall, according to Coursera’s advanced fine-tuning course.
Training Hyperparameters
Don’t overthink this. Use the base model’s recommended settings from their paper. Change one thing at a time.
python
from transformers import TrainingArguments
training_args = TrainingArguments(
output_dir="./fine-tuned-model",
per_device_train_batch_size=4,
gradient_accumulation_steps=4,
learning_rate=2e-4, # For LoRA. Use 1e-5 for full fine-tuning
num_train_epochs=3,
logging_steps=25,
save_strategy="epoch",
evaluation_strategy="epoch",
warmup_ratio=0.03,
lr_scheduler_type="cosine",
bf16=True # Use if your GPU supports bfloat16
)
Start with 3 epochs. If training loss keeps dropping but eval loss starts rising, you’re overfitting. Early stopping is your friend.
Infrastructure Choices
Run fine-tuning at 4-bit quantization for prototyping, 16-bit for production. The difference in final model quality is minimal if you use LoRA (maybe 1-2% accuracy), but 4-bit lets you train on cheaper hardware.
We use Unsloth for most training runs. It’s 2x faster than vanilla Hugging Face and uses 50% less memory. Not sponsored — just honest. It works.
The Business Side: Cost, ROI, and When to Walk Away
The business guide from Stratagem Systems breaks down costs well. Here’s my version with real numbers from our projects.
Cost breakdown for a typical 7B model fine-tune:
| Item | Cost |
|---|---|
| Data preparation (100 hours of engineer time) | $15,000 |
| Compute (single A100 for 2 hours) | $6 |
| Evaluation and iteration (40 hours) | $6,000 |
| Production inference (12M tokens/month) | $900/month |
| Total first month | ~$22,000 |
Compare that to OpenAI’s fine-tuning API, which charges roughly $25 per million training tokens plus inference costs. For a 50K-token dataset trained for 3 epochs, that’s $3,750 in training + ~$1,500/month in inference.
Open source wins on inference costs. Period. But it wins only if you have the engineering team to manage infrastructure. If you’re a solo founder, pay the API premium until you have product-market fit.
When to NOT fine-tune:
- Your use case is general question answering
- You haven’t tried prompt engineering + RAG
- You have fewer than 500 high-quality examples
- Your team can’t evaluate model outputs systematically
Evaluating Fine-Tuned Models
Here’s where most teams screw up. They look at perplexity or loss curves. Those metrics don’t predict production performance.
Build a task-specific eval set. For a summarization model, use ROUGE-L and human preference judgment. For a classification model, measure precision and recall at your operating threshold.
python
def evaluate_finetuned_model(model, tokenizer, eval_dataset):
from tqdm import tqdm
import numpy as np
predictions = []
references = []
for example in tqdm(eval_dataset):
inputs = tokenizer(
example["input"],
return_tensors="pt",
truncation=True,
max_length=2048
)
with torch.no_grad():
outputs = model.generate(
**inputs,
max_new_tokens=128,
temperature=0.0 # Greedy decoding for evaluation
)
prediction = tokenizer.decode(outputs[0], skip_special_tokens=True)
predictions.append(prediction)
references.append(example["target"])
# Compute your specific metric here
return predictions, references
We use a three-stage evaluation:
- Unit tests — verify specific behaviors (does it always output JSON? Does it refuse PII?).
- Accuracy tests — measure against labeled ground truth.
- A/B tests in production — serve 10% of traffic on new model, compare business metrics.
NVIDIA’s NeMo framework has decent evaluation tools, but we’ve built our own. The open source tooling for model evaluation is still immature. Be prepared to write custom scripts.
Common Pitfalls (I’ve Made All of These)
Catastrophic forgetting. Your model was good at general English. After fine-tuning on legal documents, it might forget how to do basic reasoning. Solution: mix 10-20% of general instruction data into your training set.
Format mismatch. You fine-tuned with system prompts formatted one way, but your production calls use a different template. The model breaks silently. Solution: freeze your chat template before training and never change it.
Overfitting to noise. Your dataset has typos or inconsistent annotations. The model learns them. Solution: run an outlier detection script on your training data. Remove examples where the target is more than 3 standard deviations from the mean length.
Data leakage. Your evaluation set contains near-duplicates of training examples. Accuracy looks 98% but real performance is 72%. Solution: deduplicate across train/eval/test splits using exact or fuzzy matching.
The Future (Mid-2026 Perspective)
In the last six months, the gap between open source and closed-source fine-tuning has narrowed dramatically. OpenAI’s model optimization API is getting cheaper and faster. But open source models like Llama 3.1 are closing the quality gap while keeping inference costs 10x lower.
The real shift I’m seeing: organizations are moving away from fine-tuning a single monolithic model. Instead, they’re fine-tuning small specialist models for each task. One model for customer support. Another for document extraction. A third for code review. The total compute budget is lower, and each model performs better.
I’m also seeing job market demand spike. ZipRecruiter lists dozens of LLM fine-tuning roles as of this month. The skills gap is real — most ML engineers know training but not deployment. If you can fine-tune a model and serve it in production, you’re worth a lot.
FAQ
Q: How long does it take to fine tune a LLM for production?
A: First project? Plan 4-6 weeks. Data prep takes 2-3 weeks, training and evaluation takes 1-2 weeks, deployment takes 1 week. Subsequent projects take 1-2 weeks total.
Q: What are the best open source models to fine tune in 2026?
A: Llama 3.1 8B for general use. Mistral 7B for high-throughput production. Qwen 2.5 7B for document extraction. DeepSeek-V2 Lite for code. Phi-3 Medium for resource-constrained environments.
Q: Do I need multiple GPUs for fine-tuning?
A: Only for models above 13B parameters. A single A100 (or RTX 4090 with quantization) handles 7B and 8B models fine.
Q: How many examples do I need?
A: Minimum 500 high-quality examples for noticeable improvement. Sweet spot is 5,000-10,000. Diminishing returns after 50,000.
Q: Can I fine-tune a model on a laptop?
A: Yes, with limitations. You can fine-tune Phi-3 Mini (3.8B) on a MacBook Pro with M3 Max. For anything larger, rent cloud GPUs.
Q: How do I prevent overfitting?
A: Use LoRA (lower rank means less capacity to overfit). Add 10-20% general data. Use early stopping. Keep your training set clean.
Q: What’s the difference between fine-tuning and RAG?
A: RAG retrieves information and injects it into the prompt. Fine-tuning changes the model’s weights. Use RAG for facts that change (pricing, inventory). Use fine-tuning for style, format, and stable knowledge (product documentation, legal terms).
Q: Should I fine-tune or use an API?
A: Use API if you have no ML team. Use open source if you have infrastructure skills and inference volume justifies the upfront cost. Breakeven is typically at 1-2 million inference tokens per month.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.