This decision comes up in nearly every AI project, and it is usually resolved by whoever is most confident in the room rather than by the actual requirement. The distinction is clean enough to settle properly.
What each one does
RAG gives the model information at the moment of answering. You search your documents, find the relevant passages, and include them in the prompt. The model reads them and answers. Nothing about the model changes.
Fine-tuning changes the model itself. You train on examples of input and desired output, adjusting the weights so it responds differently in future. It is not learning facts so much as learning a pattern of behaviour.
The analogy that holds up: RAG is handing someone the reference manual open at the right page. Fine-tuning is sending them on a training course.
The decision in one table
| RAG | Fine-tuning | |
|---|---|---|
| Teaches | Facts and knowledge | Behaviour, tone, format |
| Updating | Re-index, minutes | Retrain, expensive |
| Cites sources | Yes | No |
| Setup cost | Low | High |
| Needs | Documents you already have | Hundreds to thousands of examples |
| Reduces invention | Yes, substantially | No |
| Good for | Company knowledge, policies, products | Consistent output shape, house style |
Use RAG when
The information changes. Pricing, policies, inventory, documentation, personnel. Anything that will be different in six months. Re-indexing is minutes; retraining is a project.
You need to show your working. A RAG system can say "according to section 4 of the returns policy" and link it. A fine-tuned model cannot tell you where an answer came from, because it does not know — the information is distributed across weights.
You are worried about invention. Grounding answers in retrieved text is the main defence against a model making things up. Fine-tuning does nothing for this.
Access control matters. With RAG you can filter retrieval by permission, so a user never receives content from a document they cannot read. A fine-tuned model has absorbed everything it was trained on, with no way to partition it per user. For internal knowledge systems this alone usually settles the question.
You want to start this month. RAG on existing documentation is a matter of weeks. Fine-tuning starts with building a dataset.
Use fine-tuning when
Output format must be exact. If every response must be valid JSON matching a precise schema, every time, fine-tuning is more reliable than instructing and hoping.
You need a distinctive voice. A genuinely specific house style, consistently, across thousands of outputs. Prompting gets you most of the way; fine-tuning gets you the rest.
One narrow task, very high volume. A fine-tuned smaller model can match a large model on a specific classification task at a fraction of the cost per call. At serious volume that saving compounds.
Latency matters a lot. A small fine-tuned model is faster than a large general one. For real-time applications such as voice, this can be decisive.
The misconception worth naming
Fine-tuning does not stop hallucination. This is the single most common misunderstanding in AI projects.
Teams fine-tune on company documents expecting the model to then "know" the content reliably. What they get is a model that produces text in the style of those documents, with no improvement in factual accuracy and no ability to cite. Often it is worse, because the output now sounds more authoritative while being just as unreliable.
If the goal is accurate answers from your own material, that is RAG, every time.
Using both
They are complementary, and the combination is genuinely powerful for mature systems.
A support assistant might be fine-tuned to always reply in your brand voice, always use a specific structure, and never make commitments outside policy — and use RAG to pull the current pricing, the actual policy text and the customer's real order status.
Behaviour from training, facts from retrieval. That division is the right mental model.
What a fine-tuning project actually involves
If you decide you need it, the effort is mostly not the training:
- Build the dataset. Hundreds to thousands of input-output pairs showing exactly the behaviour you want. This is the bulk of the work and it needs human review.
- Split for evaluation. Hold back examples the model never sees, so you can measure honestly.
- Train. Usually the quickest and cheapest step, which surprises people.
- Evaluate against the holdout and against the base model with good prompting. Frequently the base model with a well-written prompt is close enough, and that result should be taken seriously rather than ignored because you have already committed.
- Iterate, which means improving the dataset rather than the training.
Step four is the one teams skip. A meaningful share of fine-tuning projects would have been better served by a better prompt, and the only way to know is to measure.
Start with prompting
Before either, try the simplest thing: a well-written prompt with a few examples included in it.
Modern models follow detailed instructions well, and few-shot prompting handles a surprising amount of what people reach for fine-tuning to solve. It costs nothing to test, it takes an afternoon, and if it works you have saved a project.
The sensible escalation is: prompt engineering → RAG → fine-tuning, stopping as soon as the output is good enough.
In practice
Across the client work we do, the split is lopsided. The overwhelming majority of business problems — "the AI should know our products", "it should answer from our policies", "staff should be able to ask questions of our documentation" — are RAG problems.
Fine-tuning comes up for high-volume classification where cost per call matters, and occasionally for strict output formats. It is the right answer considerably less often than the amount of discussion it attracts would suggest.
If you want help working out which category your problem falls into, describe it to us — it is usually a short conversation.
