How to Fine Tune Llama 3.5 on Custom Dataset

How to Fine Tune Llama 3.5 on Custom Dataset

We shipped four Llama 3.5 fine-tunes at SIVARO this quarter alone. Two worked. Two ended up as expensive parlor tricks. The difference wasn't the model. It was how we prepped the data, chose the method, and dodged the landmines.

Let me save you the tuition I paid.

Fine-tuning Llama 3.5 on a custom dataset means taking Meta’s base model (released April 2026, 8B and 70B variants) and training it further on your specific examples – emails, customer logs, medical records, whatever. You're not building from scratch. You're bending a giant brain toward your corner of the world.

This guide covers everything I wish someone had told me six months ago. Data preparation that doesn't suck. The right fine-tuning technique for your budget. How to avoid catastrophic forgetting when fine tuning. Hardware choices that work (and ones that waste money). Testing that actually catches regressions.

I'll show code. I'll name tools. I'll tell you where we screwed up so you don't have to.

Let's get into it.

Why Llama 3.5? And Why Fine-Tune at All?

Llama 3.5 isn't flashy. It's reliable. That's the point.

Meta shipped it in April 2026 with a 128K context window, grouped-query attention improvements, and significantly less hallucination on factual retrieval compared to Llama 3.1. The 8B model runs on a single consumer GPU. The 70B still fits on two A6000s with proper quantization.

But out of the box, it answers like a Wikipedia summary. It doesn't know your product. It can't follow your company's tone. It has no idea what "P0 incident" means in your org.

Fine-tuning fixes that. You show it 500 examples of how your support team resolves escalations. It learns the pattern. Next time a user says "my invoice is wrong," the model doesn't explain what an invoice is – it says "I see you're on the Enterprise plan, let me pull up your last three billing cycles and check the tax exemption."

That's the difference between generic and useful.

The Best 5 LLM Fine-Tuning Tools of 2026 lists platforms that automate parts of this. We use a mix. But I want you to understand the mechanics first, not just click a button and pray.

What You Actually Need to Know Before Starting

Data is everything

Here's the hard truth: most fine-tuning failures are data failures, not model failures.

I've seen teams spend $3,000 on GPU credits for a fine-tune that produced a model that couldn't follow basic instructions. They blamed the technique. I looked at their dataset – 200 examples where half the "completions" were copy-pasted from GPT-4. Garbage in, garbage out.

You need three things:

1. High-quality pairs – Each example should be a clear (instruction, response) or (prompt, completion). The response must be something you'd show a customer. Not "eh, good enough."
2. Coverage – Your dataset must represent the distribution of real queries. If 80% of your users ask about pricing, your fine-tuning data better have 80% pricing examples.
3. Diversity – Include edge cases. Angry users. Misspellings. Half sentences. The model needs to handle real-world noise.

At SIVARO we built a pipeline that pulls production logs, filters for high-quality human responses, and deduplicates. That's the gold standard. If you don't have logs, you can generate synthetic data using a stronger model (GPT-4o or Claude 3.5 Sonnet) and then manually review. But don't skip the review.

How to avoid catastrophic forgetting when fine tuning

Most people think catastrophic forgetting is when the model loses its general knowledge. That's part of it. But the real problem is subtler: the model overfits to the fine-tuning distribution and starts refusing anything that doesn't look like your examples.

We tested this at SIVARO. A client wanted to fine-tune a medical diagnosis assistant. They trained on 10,000 radiology reports. The resulting model was fantastic at chest X-rays. But ask it "What's the capital of France?" and it would respond with a differential diagnosis.

That's catastrophic forgetting in the wild.

How to avoid it:

• Use LoRA (Low-Rank Adaptation), not full fine-tuning. LoRA adds small trainable weights to attention layers while freezing the rest. It's like a surgical implant rather than brain surgery. Fine-Tune Local LLMs 2026 | Practical Guide shows that LoRA with rank 16-32 preserves general knowledge while adapting to new tasks.
• Include general instruction data in your mix. We add 10-15% generic Q&A (like from OpenOrca or Dolly) alongside our domain data. This keeps the model's broad capabilities alive.
• Early stopping. Monitor loss on a held-out validation set. Stop training when validation loss flattens. Don't just run for N epochs because a blog post said so.
• Learning rate warmup. Start low. We typically use 1e-4 for LoRA on 8B, 5e-5 on 70B.

What hardware do you actually need?

This is where I get asked "Can I fine-tune on my gaming PC?"

Short answer: yes, for 8B. No, for 70B.

Best hardware for fine tuning llama 3 2026:

Model Size	Minimum RAM	Recommended GPU	Cost per hour (cloud)
Llama 3.5 8B (LoRA)	24GB VRAM	RTX 4090, A4500	$0.30 - $0.80
Llama 3.5 8B (full)	48GB VRAM	A6000 48GB, 2x RTX 4090	$1.20 - $2.50
Llama 3.5 70B (LoRA, 4-bit)	48GB VRAM	A6000 48GB, H100	$3.00 - $8.00
Llama 3.5 70B (full)	160GB+ VRAM	8x A100 80GB	$30+ per hour

QLoRA (Quantized LoRA) is your friend. It loads the base model in 4-bit, then applies LoRA adapters in 16-bit. You lose maybe 2% accuracy but cut VRAM requirements in half. For most applications, it's the right choice.

We run our 8B fine-tunes on a single RTX 4090 with 24GB. Takes about 4-6 hours for 1,000 examples. The 70B we do on an A6000 using Unsloth's optimizations – 8 hours for 500 examples at rank 32.

Fine-Tune Any LLM 2026: 10 Tools Tested, Cheapest Wins did a cost comparison. Unsloth was the cheapest for local runs. For cloud, Modal and Together AI tied on price-per-epoch.

How to Fine Tune Llama 3.5 on Custom Dataset: Step-by-Step

Step 1: Prep your dataset

Format: each example as a JSON line. Use the chat template for Llama 3.5 – it expects messages with roles: system, user, assistant.

json
{
  "messages": [
    {"role": "system", "content": "You are a support agent for AcmeCorp. Respond helpfully and concisely."},
    {"role": "user", "content": "My order hasn't shipped and it's been 4 days."},
    {"role": "assistant", "content": "I apologize for the delay. Let me check your order status. Can you provide your order number? (e.g., AC-12345)"}
  ]
}

Save as data/train.jsonl. We also keep a data/val.jsonl (10% of training set) to monitor overfitting.

Step 2: Choose your fine-tuning method

You have three options:

1. Full fine-tuning – All parameters update. Best accuracy. Most hardware intensive. Use only if you have >100K examples and budget for 8xA100.
2. LoRA – Adds adapters. 99% of full fine-tuning quality at 1% of the cost. Our default.
3. QLoRA – LoRA on a 4-bit base model. Saves VRAM. Minimal quality loss. Use when you're GPU-poor.

For 99% of users, QLoRA wins. Here's why: you can fine-tune a 70B model on a single A6000. That was impossible a year ago. LLM Fine-Tuning Best Practices: Complete Guide for 2026 confirms QLoRA's quality is within 1-2% of full fine-tuning on most benchmarks.

Step 3: Write the training script

I'll give you the Unsloth version (our go-to). It handles gradient checkpointing, memory optimization, and mixed precision automatically.

python
# train_llama35.py
from unsloth import FastLanguageModel
from datasets import load_dataset
import torch
from transformers import TrainingArguments
from trl import SFTTrainer

# Load model with QLoRA
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Llama-3.5-8B-bnb-4bit",  # 4bit base
    max_seq_length=4096,  # trim context to save memory
    dtype=torch.bfloat16,
    load_in_4bit=True,
)

# Add LoRA adapters
model = FastLanguageModel.get_peft_model(
    model,
    r=16,  # rank
    lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
    lora_dropout=0.05,
    bias="none",
    use_gradient_checkpointing=True,
)

# Load dataset
dataset = load_dataset("json", data_files={"train": "data/train.jsonl", "val": "data/val.jsonl"})

# Training args
training_args = TrainingArguments(
    output_dir="./llama35-finetuned",
    per_device_train_batch_size=4,
    per_device_eval_batch_size=4,
    gradient_accumulation_steps=4,  # effective batch size 16
    num_train_epochs=3,
    learning_rate=1e-4,
    warmup_ratio=0.1,
    evaluation_strategy="steps",
    eval_steps=50,
    logging_steps=10,
    save_total_limit=3,
    bf16=True,
)

# Trainer
trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    args=training_args,
    train_dataset=dataset["train"],
    eval_dataset=dataset["val"],
    dataset_text_field="messages",  # works with chat format
    max_seq_length=4096,
)

trainer.train()

That's it. Run it. python train_llama35.py

On an RTX 4090 with 500 examples, expect the first epoch in about 90 minutes. Three epochs – 4.5 hours. Grab coffee. Or sleep.

Step 4: Monitor and stop

Watch the eval loss. It should drop in the first 100 steps. If it doesn't move, your data is wrong or your learning rate is too high.

If eval loss starts increasing while train loss keeps dropping – that's overfitting. Stop immediately. That's catastrophic forgetting knocking at the door.

Step 5: Merge and test

After training, save the LoRA adapters. To use the model in production, you can either load the base model plus adapters at inference time (slower, but flexible), or merge them once.

python
# Save LoRA adapters
model.save_pretrained("./llama35-lora-adapter")

# Merge and save full model (for deployment)
merged_model = model.merge_and_unload()
merged_model.save_pretrained("./llama35-merged")

Then test a few examples manually. Compare outputs before and after. If the fine-tuned model hallucinates on simple facts, you didn't include enough general data.

Picking the Right Approach: RAG vs Fine-Tuning

Most people ask: should I fine-tune or use RAG?

Let me make it simple.

Use RAG when: you need to answer questions based on a changing corpus of documents (company wiki, product docs, recent articles). The model doesn't need to learn anything new – it just needs to retrieve and summarize.

Use fine-tuning when: you need the model to adopt a specific behavior, tone, or reasoning pattern. When your support team has a 5-step resolution protocol. When your medical notes follow a specific format. When you want the model to never say "I don't know" but instead say "Let me connect you with a specialist."

The RAG vs Fine-Tuning in 2026: A Decision Framework article breaks this down with actual case studies. The key insight: fine-tuning teaches the model how to respond, RAG teaches it what to respond with.

We combine both at SIVARO. Fine-tune for tone and protocol. Then layer RAG on top for factual grounding. That's the sweet spot.

Common Mistakes (We Made All of Them)

Mistake 1: Fine-tuning on too few examples

We had a client who wanted to fine-tune on 50 support tickets. The model memorized them. Couldn't generalize.

Rule of thumb: Minimum 200 high-quality examples. Above 1,000, diminishing returns kick in. Above 5,000, you need a larger rank or full fine-tuning.

Mistake 2: Not cleaning your data

Bias, typos, contradictions – the model will amplify them all. We found a dataset where 30% of the "good" responses started with "Unfortunately...". The fine-tuned model became a pessimist. "Unfortunately, your order is ready for pickup."

Solution: Run a validation script that spot-checks for unwanted patterns. Use a tool like CleanLab or just hand-inspect 10% of your data.

Mistake 3: Using the wrong template

Llama 3.5 uses a specific chat template. If you pass raw text without the system/user/assistant structure, the model will behave weirdly. Double-check that your dataset_text_field and tokenizer's apply_chat_template match.

Mistake 4: Ignoring context length

Your fine-tuning examples should not max out at 128K. The model will learn to ignore long contexts because it never sees them. Keep your training sequences under 4K tokens unless you specifically need long-context reasoning.

Fine-Tuning Large Language Models for Specialized Use – that paper from 2024 is still the best academic reference for these trade-offs. I re-read it every six months.

How to Test Your Fine-Tuned Model

Don't just eyeball test set accuracy. Build a small evaluation suite.

1. Perturbation tests – Feed the model the same question with slightly different phrasing. Does it still answer correctly? If not, your model is brittle.
2. Invariant tests – "What's the capital of France?" should still work. If the model refuses, you forgot to avoid catastrophic forgetting.
3. Edge case tests – Empty prompts, all caps, offensive input. Does the model handle it gracefully?
4. Latency tests – LoRA inference is fast. Full fine-tuning can be slower due to larger model size. Benchmark with your hardware.

We use a small eval framework called lmeval (LMsys's fork) to run standard benchmarks alongside our custom tests. SuperAnnotate's guide on fine-tuning LLMs in 2026 covers evaluation methodology in more detail.

FAQ

Q: Can I fine-tune Llama 3.5 for free?

You can use Google Colab Pro ($10/month) for 8B QLoRA with a T4 GPU. It'll be slow – maybe 10 hours for 500 examples – but it works. Free tiers won't cut it; VRAM is too limited.

Q: How long does fine-tuning take?

For 8B with 1,000 examples: 4-6 hours on RTX 4090. For 70B with 500 examples: 8-12 hours on A6000. Cloud rentals (Lambda, Vast.ai) can cut time if you use H100s, but cost more.

Q: What if my dataset is small (under 100 examples)?

Don't fine-tune. Use few-shot prompting instead. Or generate synthetic data from a stronger model and then fine-tune. 50 real examples + 450 synthetic is better than 50 alone.

Q: Do I need to fine-tune on the same hardware I'll use for inference?

No. You can fine-tune on a datacenter GPU and run inference on consumer hardware. The LoRA adapter is tiny – a few MB.

Q: How to avoid catastrophic forgetting when fine tuning on very specific domains?

Include 15-20% general instruction data in your mix. Use LoRA with rank 8-16, not 64+. Stop training when validation loss flattens. And always test general knowledge after.

Q: Can I fine-tune Llama 3.5 on a Mac?

Only with MLX (Apple's framework). The 8B model with QLoRA works on M2 Ultra with 64GB unified memory. M1/M2 with 16GB won't work – too little memory.

Q: Do I need to fine-tune the 70B or is 8B enough?

Depends on task complexity. For single-turn response generation (support, QA), 8B is sufficient 80% of the time. For multi-turn reasoning or code generation, 70B gives noticeably better quality. We benchmark before committing.

Final Thoughts

Fine-tuning Llama 3.5 on a custom dataset is now accessible to anyone with $50 and a weekend.

But accessible doesn't mean easy. The tools have gotten better – Unsloth, Axolotl, Hugging Face TRL – but the core challenges remain: data quality, avoiding catastrophic forgetting, and evaluating properly.

I've seen startups ship fine-tuned models that delighted users. I've also seen enterprises burn $50K on training runs that produced models that were worse than the base. The difference wasn't budget. It was discipline.

Start small. Validate fast. Don't fall in love with the first training run.

And when your fine-tuned Llama 3.5 model actually solves a customer's problem in the right tone – that's the moment it's all worth it.

---

Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.