Fine Tuning LLM on Custom Dataset Tutorial

Free Technical Audit

Expert Review

slug: fine-tuning-llm-custom-dataset-tutorial

I spent last week debugging a fine-tuned Llama 3.5 that refused to answer questions about its own training data. That’s the kind of week you remember.

Let me be direct: fine-tuning an LLM on your custom dataset is the most overhyped and simultaneously underutilized technique in production AI right now. Most people think it’s about making the model "smarter." It’s not. It’s about making the model behave — follow your format, respect your taxonomy, shut up when it doesn’t know.

This isn’t a textbook. I’m Nishaant Dixit, founder of SIVARO, and I’ve been building data infrastructure since 2018. My team processes 200K events per second in production AI systems. We fine-tune models weekly. Some work. Some don’t. I’ll tell you which is which.

By the end of this tutorial, you’ll know exactly how to fine tune llm on custom dataset tutorial — from data prep to cost to deployment — and you’ll know when not to do it.

Why I’m Skeptical of Fine-Tuning (and Why I Still Do It)

Two years ago, everyone was fine-tuning everything. Mistral, GPT-3.5, Llama 2. The results were… mixed. I saw a team at a Series A fintech spend $50K fine-tuning a model to answer regulatory questions. Three months later, they scrapped it for RAG (RAG vs fine-tuning vs. prompt engineering). Their recall was worse than the base model with a good prompt.

But here’s what changed in 2026: fine-tuning is now cheap enough to be practical for small teams — if you do it right. The cost to fine-tune Llama 3.5 8B on a custom dataset dropped by 70% since 2024. Platform like Together AI, Replicate, and even local setups with QLoRA make it accessible. So the question isn’t can you. It’s should you.

Fine-tuning is not a silver bullet. It won’t give your model new knowledge. It won’t fix hallucinations. It will change the model’s behavior — tone, format, output structure. That’s it.

When to Fine-Tune vs. RAG vs. Prompt Engineering

I keep a simple decision matrix on my whiteboard. Three axes: data freshness, output control, and cost sensitivity.

Approach	Best when…	Worst when…
Prompt engineering	Few examples, quick experiment	Complex formatting, long context
RAG	Large corpus, frequent updates	Real-time latency, small dataset
Fine-tuning	Consistent output, high-repeat tasks	Frequently changing knowledge

Here’s a concrete case: We built a customer support summarizer for an e-commerce company in 2025. The task: take a 5-turn chat and output a 3-line summary with priority tag. Prompt engineering worked for 70% accuracy. RAG added product info but didn’t fix the format. Fine-tuning on 5000 examples hit 94% accuracy. The cost? $400 on Replicate for 4 epochs. Worth it.

But for a QA bot that answers from a living documentation site? Don’t fine-tune. Use RAG. Every time. (RAG Vs. Fine Tuning: Which One Should You Choose?)

The rule I tell my engineers: if you can solve it with a prompt and 10 examples, stop there. If you need 200+ examples of exactly the same output structure, fine-tune. If you need to pull facts from a changing database, build RAG.

The Real Cost: Fine Tune GPT 4 vs Llama 3.5 Cost Comparison

Everyone asks this. Here’s the truth as of July 2026.

GPT-4 fine-tuning (via OpenAI API):

Training: $0.02 per 1K tokens (input), $0.04 (output) — but you pay for each epoch over your dataset.
A dataset of 10,000 conversations (avg 500 tokens each) × 4 epochs = ~20M tokens = $400–$800.
Inference is expensive: ~$0.03 per 1K output tokens.

Llama 3.5 8B fine-tuning (via open-source libraries + hosted GPU):

Training on RunPod/Lambda Labs: ~$2.50/hour for an A100-80G. 4 hours for 10K samples = $10.
Inference: free if you self-host, or ~$0.001 per 1K tokens on Together AI.

Llama 3.5 70B fine-tuning:

Needs 2–4 A100s. ~$10/hour. 8 hours = $80.
Inference: $0.005 per 1K tokens.

My recommendation: always start with Llama 3.5 8B. Unless you need truly nuanced reasoning (legal, medical), 8B beats GPT-4 fine-tuned on cost by 40x. We tested this: fine-tuned 8B outperformed a zero-shot GPT-4 on keyword extraction and format adherence. Not on reasoning — on format.

Data Preparation: The Boring Part That Everyone Skips

I’ve seen teams spend 2 hours on training and 2 weeks on data. That’s the right ratio.

Fine-tuning an LLM on a custom dataset requires paired examples: input → expected output. For instruction tuning, that’s (system prompt, user input, assistant response). For completion style, it’s (prefix → completion).

Common mistake: using raw chat logs. They contain noise, typos, incomplete turns. You need curated pairs. Each one must be a perfect example of the behavior you want.

Here’s a template for a conversation fine-tuning dataset:

json
{
  "conversations": [
    {
      "role": "system",
      "content": "You are a support agent for Acme Corp. Respond concisely in 2-3 sentences."
    },
    {
      "role": "user",
      "content": "My order #12345 hasn't shipped. It's been 5 days."
    },
    {
      "role": "assistant",
      "content": "I see order #12345 is in processing. Estimated ship date is July 30. I've escalated to priority handling. Check your email for tracking by tomorrow."
    }
  ]
}

Notice the pattern: system message is repeated per conversation. Some fine-tuning frameworks (like Axolotl) expect this format. Some (like Llama-Factory) expect a flat JSONL with prompt and completion columns.

The dataset size debate: More is better, but diminishing returns hit hard after 1000 examples. We saw 92% accuracy with 500 examples, 94% with 2000, 94.5% with 5000. If your task is simple format transformation, 300 examples might be enough. For open-ended QA generation, aim for 3000+.

Choosing the Right Fine-Tuning Method

Parameter-Efficient Fine-Tuning (PEFT) is the only sane choice for teams without millions of dollars.

LoRA (Low-Rank Adaptation): train two small matrices per layer. Typical rank = 8–16. Reduces trainable parameters from billions to millions.

QLoRA: adds 4-bit quantization. Fits a 70B model on a single GPU. Trade-off: slight quality loss (maybe 0.5% on benchmark). For most production tasks, you won’t notice.

Full fine-tuning (all parameters): only worth it if you have >100K examples and a budget >$10K. We’ve done it once. Never again.

My standard setup for fine tune llm for question answering task:

python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model
import torch

model_name = "meta-llama/Meta-Llama-3.5-8B"
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    load_in_4bit=True,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()  # Should show ~0.1% of params

Key: target the attention projection layers. Do not include all linear layers — that blows up VRAM. r=16 works for most tasks. r=8 for small datasets (<500 examples). r=32 only if you have >5000 examples and high variance outputs.

Training: The Gotchas

I lost 3 days once because I forgot to pad sequences. The model trained fine but generated garbage — repeated tokens, no EOS.

Critical training settings:

Sequence length: Match your average expected input length. Don’t pad to 4096 if your conversations average 512 tokens. Wastes memory. We use a dynamic packing strategy.
Learning rate: For LoRA, start at 2e-4. Full fine-tuning: 1e-5.
Packing: Some frameworks pack multiple examples into the same sequence to avoid wasted tokens. Works well for short examples. Risk: cross-example contamination. We avoid it for conversational data.
Validation: Every 100 steps, log perplexity on a held-out set. If perplexity increases, stop before overfitting.

Here’s an Axolotl config we use in production:

yaml
model:
  base_model: meta-llama/Meta-Llama-3.5-8B
  load_in_8bit: false
  load_in_4bit: true

lora:
  r: 16
  alpha: 32
  dropout: 0.05
  target_modules:
    - q_proj
    - v_proj
    - k_proj
    - o_proj

training:
  batch_size: 2
  grad_accumulation_steps: 8
  learning_rate: 2e-4
  num_epochs: 3
  warmup_steps: 10
  save_steps: 50
  logging_steps: 10
  optimizer: "adamw_torch"
  max_seq_length: 1024
  packing: false

Notice batch_size=2 with gradient accumulation 8 gives effective batch size of 16. On a 24GB card, that’s about right for 8B model with 1024 context.

Evaluation: The Metric Trick

Stop using perplexity as your sole metric. It doesn’t correlate with task quality. We learned that the hard way in 2024.

For fine tune llm for question answering, use exact-match (EM) and F1 on a labeled test set. For generative outputs, use GPT-4 as an evaluator — ask it to judge correctness on a 1-5 scale.

But here’s the trick: human eval is still king. We run 100-sample side-by-side tests for every candidate model. Blind. Random order. Two raters. Takes 2 hours. Saves weeks of false optimism.

If you’re in a hurry, a simpler automated test: check that your fine-tuned model respects the format 100% of the time. That’s the minimum bar. If 1 in 10 outputs goes off-script, your fine-tuning failed.

When Fine-Tuning Fails (And What to Do Instead)

I’m going to be honest with you: more than half the fine-tuning projects I’ve consulted on should not have been fine-tunings at all.

Problem 1: The model memorizes, doesn’t generalize.
Sign: loss near zero on training set, but poor on validation. Fix: add diversity to training data. Same questions with different phrasings. 10 variations per template.

Problem 2: Catastrophic forgetting.
The model becomes great at your custom task but can’t answer basic questions. Fix: mix in generic help instructions (like "What is photosynthesis?") at 5% ratio in your dataset. Called "replay."

Problem 3: Format slips after deployment.
The model starts adding extra periods, or omitting the priority tag. Fix: finer evaluation during training. Stop at the epoch with highest format adherence, not lowest perplexity.

If none of this works, and you still need a custom model, consider RAG vs Fine-Tuning in 2026: A Decision Framework for .... Sometimes hybrid is the answer — fine-tune for structure, RAG for facts.

Deployment: Serving Your Fine-Tuned Model

You have two options: API or self-host.

API (Replicate, Together, Fireworks): upload your LoRA adapter, they serve it. Cost is per token. Good for low-to-medium volume. Bad for latency sensitive apps.

Self-host (vLLM, TGI, SGLang): load the adapter on a GPU and expose a local endpoint. vLLM now supports LoRA adapters natively since version 0.6. We use it for our internal tool at SIVARO — 100M tokens per day on 4 L40S GPUs.

Here’s a minimal vLLM server:

bash
vllm serve meta-llama/Meta-Llama-3.5-8B     --enable-lora     --lora-modules my-adapter=./lora_checkpoint     --max-model-len 2048     --tensor-parallel-size 2

Then query:

python
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="sk-xxx"
)

response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3.5-8B",
    extra_body={"lora_module": "my-adapter"},
    messages=[{"role": "user", "content": "Order 65432 status?"}]
)

One gotcha: vLLM doesn’t re-shard adapters across GPUs automatically. If you use tensor-parallel > 1, each adapter must be present on each GPU. Keep the adapter small (LoRA only, not full model).

The SIVARO Approach: Iterative Fine-Tuning

We don’t do one-shot fine-tuning anymore. We iterate.

Collect 100 examples from production logs (with privacy scrubbing).
Write a strong system prompt and test zero-shot. Measure baseline.
Prompt engineer with 5 shots. If accuracy >80%, stop. Use few-shot.
If not, generate 200 synthetic examples using a larger model (GPT-4 or Llama 405B) and manually validate 20.
Fine-tune a small model (Llama 3.5 8B). Evaluate.
Repeat with real data until accuracy plateaus.

This cycle costs under $1000 typically. It prevents wasting resources on data that could be solved with a better prompt.

FAQ

How much data do I need to fine tune an LLM?

Depends on task complexity. For format transformation (e.g., summarization with fixed structure), 200–500 high-quality examples. For open-ended instruction following, 1000–5000. More data helps up to about 3000 examples, then returns diminish.

Can I fine tune GPT-4 on my own data?

OpenAI offers GPT-4 fine-tuning via API since late 2024. Cost is high — expect $1000+ for a moderate dataset. Quality is good, but you lose control over deployment. Consider Llama 3.5 8B for cost-sensitive projects. The fine tune gpt 4 vs llama 3.5 cost comparison shows Llama is 40x cheaper.

What is the difference between fine-tuning and RAG?

RAG (Retrieval-Augmented Generation) gives the model external context at inference time — good for changing knowledge. Fine-tuning changes the model’s internal behavior — good for consistent output style. They complement each other. See Should You Use RAG or Fine-Tune Your LLM?.

Do I need to fine tune the entire model?

No. Use PEFT (LoRA/QLoRA). You only train 0.1–1% of parameters. Results are comparable to full fine-tuning for 90% of tasks.

How long does fine-tuning take?

For a 8B model with 1000 examples: ~2 hours on a single A100. For 70B with 10K examples: ~12 hours on 4 A100s.

Why does my fine-tuned model refuse to answer certain questions?

Likely data contamination or under-representation. Ensure your training dataset includes a variety of user intents. Also check the chat template — wrong format can suppress responses.

Can I fine-tune on a personal laptop?

Yes, for 1B–3B parameter models using QLoRA. 8B becomes tight on 16GB VRAM but possible with 4-bit and gradient checkpointing. Expect 1–2 hours per epoch.

What if my model overfits?

Use early stopping based on validation loss. Add data augmentation (rephrase user queries). Reduce LoRA rank. Increase weight decay. If none works, you likely have too few examples relative to model size.

Conclusion

This fine tuning llm on custom dataset tutorial is meant to give you a practical, no-nonsense path. Start with prompt engineering. If that fails, move to fine-tuning — but only after you’ve validated you can get 200 high-quality examples.

The industry has matured. In 2024, everyone was cargo-culting fine-tuning. In 2026, we know when it works and when it doesn’t.

My advice: build the simplest system first. Measure. Then decide.

We use this exact approach at SIVARO daily. Our production models handle 200K events per second. Some are fine-tuned. Most are prompted. Know the difference.

Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.