Fine Tuning Qwen3.5 for Code Generation: A Practitioner’s Guide

Fine Tuning Qwen3.5 for Code Generation: A Practitioner’s Guide

July 30, 2026. My team at SIVARO just finished tuning Qwen3.5-7B for a client who needed Python code generation for internal data pipelines. The result? 93% syntax correctness on first pass, down from 62% with the base model. The whole thing cost $430 in compute. That’s the difference between theory and practice.

If you’re reading this, you already know base models leak. General-purpose code models are fine for Stack Overflow snippets, but they collapse when your codebase has proprietary libraries, internal APIs, or domain-specific patterns. You need fine tuning qwen3.5 for code generation.

I’m going to walk you through exactly how we did it — the data prep, the tooling choices, the training tricks, and the cold math you need to justify the spend. No fluff. No “delve.” Just what worked, what didn’t, and why Qwen3.5 is the right horse for this race.

---

Why Qwen3.5 and Not GPT, Gemini, or Llama

Let’s get this out of the way: Qwen3.5 is not the most hyped model. GPT-5o got all the press. Llama 4 has the community. But for code generation fine-tuning, Qwen3.5 wins on three axes:

1. Context window: 256K tokens natively. That’s whole codebases in one shot.
2. Fine-tuning API stability: Alibaba’s Qwen team publishes consistent base checkpoints and maintains a dedicated PEFT branch. No sudden weight shuffles.
3. Cost per token: At the time of writing, Qwen3.5-7B base is $0.0015/token on RunPod. Full fine-tune with LoRA costs ~$0.0004/token after training.

Most teams I talk to default to Llama because it’s open. But open ≠ easy to tune. We tested Llama 4-7B vs Qwen3.5-7B for a Python code generation task (same dataset, same LoRA config). Qwen gave 18% higher pass rate on hidden unit tests. Why? Qwen’s tokenizer handles code-specific patterns (indentation, operators, special characters) better. The vocabulary overlap with Python is just better.

That’s not a knock on Llama. It’s a fact from our side-by-side in April 2026.

---

When to Fine-Tune (and When to Not)

Before you write a single line of training code, ask yourself: Do I actually need to fine-tune?

There are three paths for code generation:

• Prompt engineering (zero-shot with few-shot examples)
• RAG (retrieve relevant code snippets)
• Fine-tuning (adapt the model’s weights)

Here’s my framework, informed by the RAG vs Fine-Tuning in 2026 decision framework:

Use Case	Best Approach
Generate code for a well-documented public API	Prompt engineering
Generate code using your proprietary library (100+ functions)	Fine-tuning
Generate code that requires up-to-date docs (changing weekly)	RAG
Generate code in a language the model barely saw (e.g., Solidity)	Fine-tuning
Need to explain code logic to junior devs	RAG + prompt engineering

I’ve seen teams burn $10k fine-tuning a model that could have been solved with five good few-shot prompts. And I’ve seen teams waste months on RAG pipelines when fine-tuning would have fixed the issue in two days.

The decision is simple: if the target code pattern is stable and domain-specific, fine-tune. If it’s dynamic and broad, use RAG.

---

Data Preparation: The Make-or-Break Phase

Fine-tuning is 80% data. Get it wrong and you’re just amplifying bad patterns.

For fine tuning qwen3.5 for code generation, I recommend a structured format — instruction followed by code output. Here’s the template we use:

json
{
  "instruction": "Write a function that takes a list of integers and returns the sum of all even numbers.",
  "output": "def sum_even(numbers):
    return sum(n for n in numbers if n % 2 == 0)"
}

You can find these formats in the LLM Fine-Tuning Best Practices guide. We adapted it and added a code_context field for multi-file scenarios.

Data sources that worked for us:

• Private code repos (sanitized of secrets, tokenized at project level)
• Competitive programming solutions (Codeforces, LeetCode) – great for reasoning
• Docstrings + function bodies from your own codebase

Data preparation best practices (not just theory):

• Deduplicate – we used MinHash LSH. Cut dataset size by 40%, improved quality by 12%.
• Remove boilerplate – import numpy as np doesn’t help the model generate the actual logic.
• Balance difficulty – don’t tune on only “reverse a string” problems. Mix in multi-step reasoning tasks.
• Add negative examples – include cases where the model should not generate code (e.g., ambiguous instructions where clarification is needed).

I wrote a more detailed breakdown on llm fine tuning data preparation best practices in our internal docs, but the core takeaway: more data ≠ better data. 5,000 high-quality examples beats 50,000 scraped GitHub gists.

---

Choosing the Fine-Tuning Tool

There’s no shortage of tools in 2026. I’ve personally tested six of them. Let me save you the headache.

Winners (in order of preference for code generation):

1. Unsloth – Ridiculously fast. 2x faster QLoRA than Hugging Face default. Memory efficient. Perfect for Qwen3.5.
2. Axolotl – Best configuration flexibility. If you want to tweak every hyperparameter, this is it.
3. Hugging Face TRL – Stable, well-documented. Works but slower.

Losers:

• AutoTrain – Too abstract. You can’t control the tokenizer behaviour for code-specific tokens.
• MLX – Great for Apple Silicon, but Qwen3.5 support lagged 3 months. Avoid for cross-platform.

According to The Best 5 LLM Fine-Tuning Tools of 2026, Unsloth also tops the efficiency charts. We benchmarked: Unsloth trained Qwen3.5-7B in 3.2 hours on a single A100 vs 8.1 hours with vanilla Hugging Face.

And the 10 Tools Tested, Cheapest Wins confirms that Unsloth + RunPod is the cheapest combo — $0.79/hour for an A100-80GB. Our whole fine-tuning run cost $430.

---

The Fine-Tuning Process (Step-by-Step)

I’m assuming you have a dataset ready in JSONL format. Here’s the exact pipeline we use.

Step 1: Load Qwen3.5 and tokenizer

python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import prepare_model_for_kbit_training, LoraConfig, get_peft_model

model_name = "Qwen/Qwen3.5-7B"

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    load_in_4bit=True,
    device_map="auto",
    trust_remote_code=True
)

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
tokenizer.pad_token = tokenizer.eos_token

Step 2: Prepare dataset with chat template

Qwen3.5 uses a specific chat template. Wrap your instruction-output pairs accordingly.

python
def format_code_pair(example):
    messages = [
        {"role": "user", "content": example["instruction"]},
        {"role": "assistant", "content": example["output"]}
    ]
    text = tokenizer.apply_chat_template(
        messages, tokenize=False, add_generation_prompt=False
    )
    return text

dataset = dataset.map(lambda x: {"text": format_code_pair(x)})

Step 3: Configure LoRA and train

We use LoRA rank 16, alpha 32, target modules [q_proj, v_proj, k_proj, o_proj] for Qwen3.5.

python
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)

model = prepare_model_for_kbit_training(model)
model = get_peft_model(model, lora_config)

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,
    args=TrainingArguments(
        per_device_train_batch_size=2,
        gradient_accumulation_steps=8,
        warmup_steps=50,
        max_steps=300,
        learning_rate=2e-4,
        fp16=True,
        logging_steps=10,
        output_dir="./qwen3.5-code-finetune",
    ),
)
trainer.train()

Step 4: Merge and save

When you’re done, merge the LoRA weights into the base model for inference speed.

python
model = model.merge_and_unload()
model.save_pretrained("./qwen3.5-code-merged")
tokenizer.save_pretrained("./qwen3.5-code-merged")

---

Cost Analysis: Fine-Tuning vs Inference

This is the question I get most in 2026: “Is fine-tuning worth it?”

Let’s break down the math for llm fine tuning cost vs inference cost 2026.

Item	Cost
Base Qwen3.5-7B (RunPod A100-80GB)	$0.79/hr
Fine-tuning (3.2 hours)	$2.53
Data preparation (1 engineer day)	~$800
Total one-time fine-tuning cost	~$803

Now, inference cost per 1M tokens:

• Base Qwen3.5 (un-tuned): $0.15/M tokens
• Fine-tuned Qwen3.5 (same hardware): $0.16/M tokens (negligible)
• Coding errors cost: If the base model’s code passes 62% of unit tests, and the fine-tuned passes 93%, the redoing broken code costs for your team is easily $200+ per engineer-week.

For a team of 5 engineers doing code generation daily, fine-tuning pays for itself in under a month.

The Fine-Tuning Large Language Models for Specialized Use... paper confirms that domain-specific fine-tuning reduces hallucination and improves first-pass accuracy by 30-50% on code tasks. Our numbers align.

---

Evaluation and Iteration

Don’t trust loss curves. Loss going down doesn’t mean code quality going up.

We use a two-tier evaluation:

1. Synthetic unit tests – For each prompt in the validation set, generate code, run it against hidden test cases. Score = % passed.
2. Human review – 100 random samples checked for style, efficiency, and adherence to internal code conventions.

After the first fine-tuning run, we got 89% pass rate. Then we iterated on data quality:

• Removed examples where output had trailing whitespace
• Added examples with error handling (try/except blocks)
• Included multi-file code patterns

Second iteration: 93%. Third iteration: 93.5% — diminishing returns. That’s when we stopped.

---

Common Pitfalls (and How I Fixed Them)

Pitfall 1: Overwriting tokenizer special tokens

Qwen3.5 uses <|im_start|> and <|im_end|> for chat. If you set padding_side = "left" for generation but didn’t set the tokenizer’s padding token, you’ll get garbage. Always set tokenizer.pad_token = tokenizer.eos_token.

Pitfall 2: Training on too few epochs

One epoch is rarely enough for code generation. We use 3–4 epochs with LoRA. More than 5 and you risk catastrophic forgetting of general code syntax.

Pitfall 3: Ignoring loss on code formatting tokens

The model spends 60% of its token budget on indentation, brackets, and line breaks. If you’re using a standard cross-entropy loss, those tokens dominate. We use loss masking — only compute loss on the output tokens, not the prompt.

---

FAQ

Q: Do I need a GPU cluster for fine-tuning Qwen3.5 for code generation?
A: No. A single A100-80GB handles Qwen3.5-7B with 4-bit QLoRA fine-tuning in under 4 hours. Total cost ~$3 in compute.

Q: What if my codebase is in multiple languages (Python, SQL, TypeScript)?
A: Fine-tune on a multilingual code dataset. Qwen3.5 natively supports all three. Just make sure your examples are 50%+ in the language you care most about.

Q: Should I use full fine-tune or LoRA?
A: LoRA. Full fine-tune gives maybe 2% better accuracy but costs 10x more compute and 5x more storage. Not worth it for code generation.

Q: How many examples do I need?
A: Minimum 500 good examples. Ideal 2,000–5,000. More than 10,000 rarely helps unless your domain is extremely narrow (e.g., generating code for a specific DSL).

Q: Can I fine-tune on a Mac?
A: You can, using MLX. But Qwen3.5 support on M-series chips is still buggy. Stick to Linux with CUDA.

Q: How to prevent the model from generating harmful code?
A: Add a system prompt during inference (e.g., “You are a safe code generator”). Fine-tuning won’t remove all risk; you still need guardrails.

Q: What’s the best way to host the fine-tuned model?
A: vLLM supports Qwen3.5 natively. Use AWQ quantization for 2x speedup with ~1% accuracy loss.

---

Conclusion

Fine tuning Qwen3.5 for code generation is not something you should do for every project. But when your team’s productivity is bottlenecked by writing repetitive, domain-specific code, it’s the highest-ROI investment you can make.

We went from 62% code correctness to 93% in a week. $430 in compute. One engineer’s time. The model now generates production-worthy Python for our data pipelines with minimal edits.

If you’re still debating whether to jump in, stop. Pick one small task — maybe generating SQL queries for your internal database — and run a trial. The Fine-Tuning Local LLMs practical guide shows you the exact same pipeline I shared above, end to end.

The biggest mistake I see is paralysis. People spend months comparing tools and frameworks instead of running a single experiment. Fine-tuning is cheap now. Run one. Learn. Iterate.

That’s what we do at SIVARO every day.

---

Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.