A general-purpose AI model has read an enormous amount of the public internet and absolutely none of your business. It has never seen your pricing, your contracts, your service documentation, or the four years of support tickets that contain the answer to almost every question a customer will ask.

Ask it about your returns policy and it will produce something plausible and wrong, because it is doing the only thing it can: generating a likely-sounding answer.

RAG fixes this without retraining anything.

The mechanism

Retrieval-augmented generation does exactly what the name describes, in three steps:

  1. Retrieval. When a question arrives, search your document collection for the passages most relevant to it.
  2. Augmentation. Insert those passages into the prompt alongside the question.
  3. Generation. The model answers using the supplied material, instructed to rely on it rather than on memory.

The model is not learning anything. It is being handed the relevant pages open at the right place and asked to read them. That framing explains most of RAG's properties: it updates instantly, it can cite sources, and it is bounded by what you gave it.

A worked example

A customer asks: "Can I return a custom-engraved item?"

Without RAG: the model generates a generic answer about return policies. It might be right. It is equally likely to invent a thirty-day window you do not offer.

With RAG:

  • The question is converted into a vector and used to search your indexed content
  • The search returns the three most relevant passages, including the custom-items clause from your returns policy
  • Those passages go into the prompt
  • The model answers: "Custom-engraved items cannot be returned unless faulty, as they are made to order. Standard items can be returned within 14 days." — with a link to the policy

Grounded, correct, verifiable.

Where the engineering actually is

The common misconception is that RAG quality is about the model. It is not, in any meaningful sense. Switch from one good model to another and answers change little. Improve retrieval and they change enormously.

Chunking

Documents must be split before indexing, and how you split them matters a great deal.

Chunks that are too small lose context — a sentence about "the 14-day window" without knowing which product category it refers to is worse than useless. Chunks that are too large dilute relevance, so the search returns a wall of text containing the answer somewhere inside it.

Different content needs different strategies. Contracts split naturally by clause. Manuals split by section. Support tickets are usually one chunk each. A single chunking strategy applied to everything is the most common reason a RAG system underperforms.

Embeddings and vector search

Each chunk is converted into a vector — a list of numbers representing its meaning. The question is converted the same way, and the system finds chunks whose vectors are closest.

This is what lets it match "can I send this back" to a passage about returns, with no shared keywords.

Hybrid search

Pure semantic search has a specific weakness: exact terms. Product codes, clause numbers, names and version numbers are precisely what semantic similarity handles worst.

So production systems combine semantic search with traditional keyword search and merge the results. Someone searching for "SKU-4471" wants the chunk containing that exact string, not the one that is conceptually similar.

Re-ranking

Retrieve more candidates than needed, then use a model to re-rank them for actual relevance before passing the top few to the generator. This measurably improves answer quality and is frequently skipped.

Metadata filtering

Attach metadata to each chunk — document type, date, department, access level — and filter before searching. This is how you prevent a salesperson's question returning a passage from an HR file, and how you stop a superseded 2023 policy being quoted as current.

RAG versus fine-tuning

The question that comes up in every project.

RAG Fine-tuning
Teaches the model Facts, at answer time Style, format, behaviour
Updating content Re-index, minutes Retrain, expensive
Citing sources Yes No
Cost Low Considerably higher
Risk of invention Lower, grounded Unchanged

Use RAG for anything factual, anything that changes, and anything where you need to show where an answer came from — which is almost all business knowledge.

Use fine-tuning when you need consistent output format, a specific tone, or performance on a narrow specialised task.

They are complementary rather than competing. A system can be fine-tuned for tone and use RAG for facts.

Does it stop hallucination?

It reduces it a great deal and does not eliminate it. Worth being precise about why.

With grounding, most remaining errors are not the model inventing — they are retrieval failures. The search returned the wrong passage, or no relevant passage, and the model answered anyway from what it had. The fix is better retrieval and a stronger instruction to decline.

An honest "I could not find that in our documentation" is a feature, and getting a model to produce it reliably takes deliberate work, because models default to being helpful.

Build the evaluation set first

The single most valuable thing you can do, and the step most often skipped.

Before building, collect thirty to a hundred real questions with known correct answers. Support tickets are an excellent source. Then every change — a new chunking strategy, a different embedding model, a re-ranker — can be scored against it.

Without this you are guessing. Teams without an evaluation set routinely make changes that improve one kind of question while silently breaking another, and only discover it from a user complaint.

Access control

Any RAG system over internal documents needs permissions enforced at retrieval time, not after.

If a user cannot read a document, chunks from it must never enter their prompt. Filtering the answer afterwards is far too late — the content has already been used to generate it.

This is straightforward to build in from the start and painful to retrofit, which is why it belongs in the architecture conversation on day one.

What content should not be indexed

Finally, the unglamorous part that determines whether the system is trusted.

Contradictory documents produce contradictory answers. Superseded policies get quoted as current. Draft documents get treated as authoritative. A significant part of any RAG project is deciding what stays out of the index and establishing who keeps it current afterwards.

A RAG system is a mirror of your documentation. If the documentation is a mess, the system will be an articulate, confident mess.

If you want to know whether your content is in good enough shape to build on — and what it would take to get there — tell us what you are working with.