How to Fine Tune LLM for Production

Free Technical Audit

Expert Review

You've got a base model. It knows Shakespeare and SQL. It can write a poem about Kubernetes. But ask it to classify customer support tickets by urgency? It guesses. Ask it to generate responses in your brand voice? It sounds like a generic bot from 2023.

That's why you're here. Fine-tuning isn't magic — it's targeted surgery. And most people mess it up.

I'm Nishaant Dixit. At SIVARO, we've fine-tuned over 40 models for production systems since 2023. We've burned thousands of dollars on experiments that went nowhere. We've also built systems that cut inference costs by 60% while improving accuracy by 30 points.

This guide is what I wish someone had handed me three years ago.

Let's get into it.

What Fine-Tuning Actually Is (And Isn't)

Fine-tuning takes a pre-trained LLM and continues training it on a smaller, specific dataset. The model already knows language — you're teaching it your language. Your domain. Your format.

It's not training from scratch. That costs millions. Fine-tuning costs anywhere from $50 to $5,000 depending on model size and data volume. (LLM Fine-Tuning Explained)

It's not prompt engineering. Prompts are instructions. Fine-tuning changes the model's weights. Prompts are like giving someone a map. Fine-tuning is like rewiring their GPS.

It's not RAG. Retrieval-Augmented Generation pulls in external data at inference time. Fine-tuning bakes behavior into the model. You might need both. We often do.

Here's the mental model I use: base models are generalists. Fine-tuned models are specialists. Don't turn a surgeon into a general practitioner — hire the surgeon, keep the GP.

When You Should (And Shouldn't) Fine-Tune

You should fine-tune when:

Your task has a consistent structure. Think: "classify email as urgent/not urgent" or "convert natural language to SQL queries."
You need the model to follow a specific format reliably. JSON output. Bullet points. Whatever.
You're hitting token limits with prompt engineering. If your prompt is 2,000 tokens and your output is 50, you're wasting money.
Your domain uses specialized terminology. Legal documents. Medical records. Internal product names.

You shouldn't fine-tune when:

You need up-to-date information. That's what RAG is for. Fine-tuning doesn't add new facts — it adjusts behavior.
You have fewer than 100 examples. I've seen teams try with 20. It doesn't work. You'll just degrade the model.
You haven't exhausted prompt engineering. Simple things first. Fine-tuning adds complexity to your ML pipeline.
You can't measure success. If you don't have evaluation metrics, you'll ship a worse model and not even know it.

Most people think they need fine-tuning. They're wrong about 60% of the time. (Generative AI Advanced Fine-Tuning for LLMs covers this distinction well.)

Choosing Your Base Model

This decision matters more than your training data. Pick wrong and you'll fight the model the entire way.

The contenders as of July 2026:

Model	Size	Best For	Cost to Fine-Tune
Llama 3.3 70B	70B params	Complex reasoning, code	High (GPU cluster)
Mistral Small 3	24B params	General purpose, good cost/performance	Medium
Phi-4	14B params	High accuracy on small data	Low
Gemma 3	27B params	Multilingual, instruction following	Medium
Qwen 2.5	32B params	Long context, structured output	Medium

For production, the best open source models to fine tune right now are Mistral Small 3 for most tasks and Phi-4 when you have small datasets. (Model optimization | OpenAI API has numbers too.)

We tested all of them at SIVARO. Phi-4 with 500 examples outperformed a heavily prompted GPT-4 on a ticket classification task. Cost per inference? 8 cents vs 3 cents. But training was a pain because the 14B parameter version needs careful learning rate tuning.

For code generation, Llama 3.3 70B is still king. For everything else, Mistral Small 3 is the sweet spot.

The Data: Your Make-or-Break

90% of fine-tuning success is data quality. I'll say that louder: 90% is data.

I've seen teams spend two weeks on hyperparameter tuning and two hours on data. That's backwards.

How Many Examples Do You Need?

Depends on the task complexity:

Simple classification (sentiment, urgency): 300-500 examples
Structured output (JSON, SQL): 500-1,000 examples
Complex reasoning (summarization, analysis): 1,000-3,000 examples
Full domain adaptation (medical, legal): 5,000+ examples

But quantity isn't the point. We got better results with 200 hand-curated examples than with 2,000 scraped from Zendesk. Garbage in, garbage out still applies.

Data Format

Most fine-tuning frameworks expect a conversation format:

json
{
  "messages": [
    {"role": "system", "content": "You are a support classifier. Classify tickets as urgent, high, medium, or low."},
    {"role": "user", "content": "Our production database crashed. All customers are down."},
    {"role": "assistant", "content": "urgent"}
  ]
}

Structure matters. My team spent a month training models that were inconsistent because our data had different system prompts. Standardize early.

Data Quality Heuristics

Diversity over quantity. 500 examples covering edge cases beats 5,000 repetitive ones.
Correct labels. Ours had a 12% error rate on the first pass. Human review caught it.
Balanced classes. If 95% of your tickets are "low priority," the model learns to say "low" and get 95% accuracy. Useless.
Realistic examples. Don't write synthetic data that's cleaner than production. The model will break on real input.

We wrote a validation script that checks for duplicates, label imbalance, and format inconsistencies. Run it before you train.

Training: The Practical Steps

Environment Setup

You need GPUs. A lot depends on model size:

7B params: 1x A100 (40GB) or 2x RTX 4090
14B params: 4x A100 or cloud TPU
70B params: 8x A100 or H100

We use RunPod and Lambda Labs for spot instances. Cost is about 60% less than on-demand.

LoRA and QLoRA

Full fine-tuning adjusts all parameters. For most production use cases, you don't need that. LoRA (Low-Rank Adaptation) trains a small set of adapter weights while keeping the base model frozen.

LoRA saves you:

90% of memory requirements
70% of training time
50% of storage (adapters are 10-50MB vs 14GB full model)

QLoRA adds 4-bit quantization. You can fine-tune a 70B model on a single A100. Not kidding.

Here's our standard training script:

python
from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments
from peft import LoraConfig, get_peft_model
from trl import SFTTrainer

model_name = "mistralai/Mistral-Small-3"
tokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="bfloat16",
    device_map="auto",
    use_cache=False,
)

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)

training_args = TrainingArguments(
    output_dir="./ft-model",
    per_device_train_batch_size=4,
    gradient_accumulation_steps=8,
    learning_rate=2e-4,
    num_train_epochs=3,
    logging_steps=10,
    save_strategy="epoch",
    evaluation_strategy="epoch",
    fp16=True,
    warmup_ratio=0.03,
    lr_scheduler_type="cosine",
)

trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    tokenizer=tokenizer,
    peft_config=lora_config,
    max_seq_length=2048,
)

trainer.train()

How Long Does It Take to Fine Tune a LLM?

With LoRA, a 14B parameter model on 1,000 examples with 3 epochs takes about 2-3 hours on an A100. Full fine-tuning takes 8-12 hours.

A 70B model with LoRA? 6-8 hours. Full? 24-48 hours.

That's training time. The total timeline — including data prep, evaluation, and iteration — is usually 2-3 weeks for production readiness. Anyone who tells you different hasn't shipped to production. (LLM Fine Tune Model Jobs on ZipRecruiter will show you teams hiring for this exact skill set.)

Hyperparameters That Actually Matter

Learning rate: Start at 2e-4 for LoRA, 1e-5 for full fine-tuning. We've seen ranges from 5e-5 to 2e-4 work for LoRA. (Fine-tuning LLMs: overview and guide has good defaults.)
Batch size: Max out your GPU memory. We use gradient accumulation to simulate larger batches without OOM errors.
Epochs: 2-4 is the sweet spot. More epochs cause overfitting. We've detected overfitting when eval loss starts climbing while train loss drops.
Sequence length: Use the shortest length that covers your data. Longer sequences cost more memory and time.

Most teams evaluate by gut feeling. "Looks good to me."

That's how you ship a model that says "urgent" on everything.

Automated Evaluation

Build a test set that mirrors production distribution. Run every epoch:

python
def evaluate_model(model, tokenizer, test_dataset):
    correct = 0
    total = 0
    for example in test_dataset:
        prompt = tokenizer.apply_chat_template(
            example["messages"][:-1],
            tokenize=False
        )
        inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
        outputs = model.generate(**inputs, max_new_tokens=10)
        prediction = tokenizer.decode(outputs[0], skip_special_tokens=True)
        # Extract predicted label
        actual = example["messages"][-1]["content"]
        if prediction.strip() == actual.strip():
            correct += 1
        total += 1
    return correct / total

Track accuracy, precision, recall. For generation tasks, use BLEU or ROUGE. For classification, confusion matrices tell you more than accuracy alone.

Human Evaluation

Automated metrics lie. Humans catch nuance. We do:

50 examples per iteration, blind tested against baseline
Two evaluators per example, third resolves disagreement
Track inter-rater reliability (Cohen's kappa above 0.7)

Human evaluation caught that our model was great at urgency classification but terrible at maintaining tone in long responses. We wouldn't have found that from metrics.

Deployment Gotchas

Serving Infrastructure

Don't deploy a fine-tuned model the same way as a base model. You need different infrastructure.

Adapter merging: For production, merge LoRA adapters into the base model. It eliminates an inference step and reduces latency by 15-30%.
Batch inference: If serving real-time requests, use vLLM or TensorRT-LLM. They handle batching and KV-cache management.
Cold start: Loading a 70B model takes 60-90 seconds. We pre-warm with dummy requests.

We learned this the hard way. First deployment of our customer support classifier timed out on every cold start for 2 minutes. Users saw errors. Not good.

Monitoring

You need:

Prediction drift: Is the model suddenly classifying everything as "high"? Monitor output distribution.
Input drift: Has customer language changed? Monitor embedding similarity.
Latency p99: Not average. P99 tells you the worst case.

We use Arize AI for monitoring. Costs $200/month. Saved us from shipping a broken model twice.

Fallback Logic

Fine-tuned models fail. Have a fallback.

Our production pipeline routes through the fine-tuned model first. If confidence is below 0.7, it falls back to GPT-4 with a prompt. Costs more but prevents catastrophic failures.

Real Costs (The Honest Numbers)

Here's what we spent on our last production fine-tuning project for a financial services client:

Item	Cost
Data labeling (2,000 examples)	$4,000 (outsourced with QA)
GPU compute (LoRA on 70B model)	$1,200
Evaluation (human raters)	$800
Production serving (2x A100, 24/7)	$3,500/month
Monitoring and tooling	$600/month

Total setup: ~$6,000. Monthly run: ~$4,100.

This replaced a solution that cost $12,000/month in API calls to GPT-4. Payback period: 2 months. (LLM Fine-Tuning Business Guide has a good ROI framework.)

But here's the thing nobody tells you: the first fine-tuning project almost always loses money. You're building infrastructure, learning pipelines, fixing data issues. The second one breaks even. The third one saves money.

Common Failures (And How to Avoid Them)

Model collapses to single output. This happens when your data is too homogeneous. Add more diversity or reduce epochs.

Model forgets how to be a general LLM. Called "catastrophic forgetting." Mix in 10-20% general instruction data to preserve capability.

Training loss goes down, eval loss goes up. Classic overfitting. Use fewer epochs, add dropout, or get more data.

Model hallucinates more after fine-tuning. Usually because your dataset has errors or inconsistencies. Audit your labels. We found our dataset had 15% wrong labels — fixing that reduced hallucinations by half.

Fine-tuned model performs worse than prompt engineering. Your task might not need fine-tuning. Or your data is bad. Or your base model is wrong for the task. Go back to first principles.

When Fine-Tuning Doesn't Pay

I need to be honest here.

For every model we've shipped, we've abandoned two.

A sentiment classifier for a retailer? Prompt engineering on GPT-4o mini was 92% accurate. Fine-tuning got us to 94%. Not worth the complexity.
A legal document summarizer? The data was too diverse. Fine-tuning made it worse on edge cases. We stuck with RAG.

Don't fine-tune because it's trendy. Fine-tune because it solves a problem nothing else can.

The Future (Mid-2026 Edition)

The landscape is shifting fast. Three things I'm watching:

Smaller models getting better. Phi-4 and Gemma 3 are approaching GPT-4 quality on specialized tasks. This means you can fine-tune on cheaper hardware.
Fine-tuning as a service. OpenAI, Anthropic, and Google all offer managed fine-tuning now. Trade-off: you lose control but gain convenience.
Automated fine-tuning. Tools like Weights & Biases' sweeps and Optuna are automating hyperparameter search. We're experimenting with this for our next project.

But the fundamentals don't change: good data beats good code. Always.

FAQ

Q: How to fine tune llm for production without a big GPU budget?

Use LoRA or QLoRA with a small model like Phi-4 (14B) or Mistral Small 3 (24B). You can run these on a single RTX 4090 (24GB VRAM) or rent a cloud instance for $1-3/hour. Keep your dataset under 2,000 examples. You don't need a cluster.

Q: How long does it take to fine tune a llm from start to production?

Training: 2-12 hours depending on model size. Total project timeline: 2-3 weeks. Data preparation takes the longest — usually 1-2 weeks. Evaluation and iteration takes another 3-5 days. Deployment is 1-2 days if your infrastructure is ready.

Q: What are the best open source models to fine tune in 2026?

Mistral Small 3 (24B) for general tasks, Phi-4 (14B) for small datasets, Llama 3.3 70B for complex reasoning. Qwen 2.5 (32B) is excellent for tasks requiring long context. Avoid fine-tuning models under 7B parameters for production — they lack the reasoning capability.

Q: Can I fine-tune on a single GPU?

Yes. Use QLoRA with 4-bit quantization. A 14B model fits in 16GB VRAM. A 70B model in 48GB VRAM. We've fine-tuned a 34B model on a single RTX 4090. Takes longer but works.

Q: How do I know if my fine-tuned model is ready for production?

Three gates: (1) Automated metrics beat baseline by at least 10%. (2) Human evaluation shows consistent improvement over baseline. (3) Stress test with real production data (not curated test set). If it passes all three, ship it.

Q: Should I use OpenAI's fine-tuning API or open source?

OpenAI's API is easier — no GPU management, built-in evaluation, simple API calls. But you lose control over data, pay per token, and get locked into their ecosystem. Open source gives you ownership and lower per-inference costs. We use both depending on client requirements. (Fine-Tuning a Chat GPT AI Model LLM compares the options.)

Q: What's the minimum dataset size for fine-tuning?

100 examples minimum. 300-500 recommended. Below that, you're better off with prompt engineering.

Q: How do I prevent my model from overfitting?

Use fewer epochs (2-3 max), add dropout (0.05-0.1), use LoRA with low rank (r=8-16), and include 10-20% general instruction data. Monitor eval loss — if it increases while training loss decreases, stop training.

Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.

How to Fine Tune LLM for Production