We can't decide between fine-tuning and RAG.

Teams that want a model to know their domain, and are stuck choosing between training it on their data and feeding it their documents at answer time.

Usually seen inTeams planning their first domain AI product

First testWhat the model gets wrong today

PracticeAI

What you're probably seeing

If this sounds familiar, you're in the right place.

Every article and vendor gives a different answer.
Nobody has measured what the model gets wrong today.
The data for fine-tuning isn't clean, or isn't labelled.
The facts change often, and nobody knows how a trained model would keep up.
The budget is set before the approach is.

The usual ways out

What teams usually try. Each works somewhere. None works everywhere.

The way	Works when	The catch
Better prompts and examples	The model knows enough and needs direction	Hits a ceiling on domain knowledge
RAG over your documents	Facts change, and answers must cite sources	Quality depends on retrieval, which needs testing
Fine-tune a model	You need a style, format or task, at volume	Doesn't keep up with changing facts, and needs data
Both: fine-tune for form, RAG for facts	You need both	Two systems to run and evaluate
A bigger general model	The gap is reasoning, not knowledge	Costs more per answer
Don't use a model for this part	Rules or search already solve it	Less impressive, often better

What other companies did. Their published results, not ours.

We read each company's own write-up and asked one question: would our diagnostic's tests have led to the same call? We judge the approach, not the result.

Company	What they did	What they published	Source
CheckrBackground checks for 100,000+ customers, millions each month. The hardest 2% of records must be sorted into 230 categories, needing high accuracy and near real-time speed.	Ran GPT-4 with prompts, GPT-4 with retrieved examples (RAG) and fine-tuned small open models on the same records, measuring accuracy, speed and cost for each. They chose a fine-tuned Llama-3-8B served on Predibase.	On the hard set: GPT-4 81.8% at 15 s, GPT-4 with RAG 79.3% at 7 s, fine-tuned Llama-2-7b 85.0% at 0.5 s. Final fine-tuned Llama-3-8B: 90% on the hard 2%, about 0.15 s per answer, about 5x lower cost than GPT-4.They noted: RAG helped on easy cases but lowered accuracy on hard ones, because the retrieved examples pulled the model away from the right answer. Combining the fine-tuned model with GPT-4 added nothing. Confidence scores skewed high until temperature and top_k were lowered.	Fine-Tuned SLMs Help Checkr Optimize Background ChecksOct 2024 · vendor story
HarveyGenerative AI platform for law, tax and finance professionals. 100+ staff at the time.	Tried fine-tuning through public APIs and tried RAG. Retrieval handled simple questions but not open-ended case-law research. They then worked with OpenAI on a custom model trained on US case law (about 10 billion tokens).	83% increase in factual responses. Attorneys at 10 large law firms preferred its answers over GPT-4's 97% of the time.They noted: The work needed lawyers to explain their research process and joint research with OpenAI. No cost figure is given.	Customizing models for legal professionals (Harvey)Undated · vendor story
IndeedJob site writing 'why this job fits you' explanations for job recommendations, at millions of messages.	Started with few-shot prompts on a GPT model. These worked but used too many tokens at Indeed's volume. They then fine-tuned a smaller GPT model, using GPT-4 to add training data, human-labelled ground truths for automated evaluation, and A/B tests.	60% fewer tokens than the few-shot approach, with similar performance. Started applications +20% and downstream success +13% versus traditional matching. Scaled from 1 million to nearly 20 million messages per day over several months.They noted: The reason was high token use at Indeed's scale, not missing knowledge. Quality is described only as similar to few-shot.	Delivering contextual job matching for millions with OpenAIUndated · vendor story
Microsoft ResearchResearch study on giving farmers location-specific advice, using agricultural PDFs and Llama2-13B, GPT-3.5 and GPT-4.	Built a pipeline that turns PDFs into question-answer pairs. They compared fine-tuning, RAG and both together, with GPT-4 grading the answers.	Fine-tuning raised accuracy by over 6 percentage points. RAG added a further 5 points, and the gains stacked. In one experiment, the fine-tuned model drew on information from other regions and raised answer similarity from 47% to 72%.They noted: Fine-tuning has a high upfront cost. The GPT-4 fine-tune took about 1.5 days on seven 8xA100 nodes.	RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on AgricultureJan 2024

Our read

Same call. Teams that published their choice tested the options on their own task, and the right answer depended on the task.

Checkr tested prompting, RAG and fine-tuning on the same records, with cost and response time for each. It picked fine-tuning for a fixed 230-category task at high volume, which matches the 'task at volume' case in the table above. Indeed fine-tuned a smaller model to cut token use at scale, with similar performance. Harvey found RAG alone did not cover open-ended legal research and moved to a custom model trained on case law. Microsoft found the two gains add up. We would still have tested how often the facts change and how a trained model would keep up. None of these write-ups says how the fine-tuned models are refreshed when facts change, and Harvey's case law keeps changing.

What we would test first: Run a test set of real questions with known good answers, and sort what the model gets wrong today by cause.

What the diagnostic would test

The tests that rule the wrong ways out.

On your own systems and data, read-only first. The number of tests sets the diagnostic's price, quoted before it starts.

1
A test set of real questions with known good answers.
2
What the model gets wrong today, sorted by cause.
3
How often your facts change.
4
What data you have for training, and its quality.
5
Cost and response time for each approach, on the same test set.

This week, without us

Three things you can do yourself.

1
Collect a few dozen real questions with the answers you'd accept.
2
Run them through a good general model with your documents pasted in, and see how far that gets you.
3
Count how often your key facts changed in the last six months.

From the book

Fine-tuning LLMs

Read the whole book →

See it

AI: product audits and improvement, tested before building →

Sound like you? Bring it to a 30-minute call. We'll tell you which tests your case needs, and what they cost.

Back to every case

This page describes what teams in this situation commonly face. It is not a client story.