What is retrieval-augmented generation, and where does it go wrong?

Retrieval-augmented generation means fetching relevant material and giving it to a language model along with the question, so the answer is grounded in your content rather than in whatever the model absorbed during training.

The name puts generation last, which is roughly the inverse of where the difficulty is. Almost every disappointing result traces back to the retrieval step fetching the wrong passages, and no amount of model upgrading fixes an answer built from material that did not contain the answer.

What we'll cover

• The two halves, and which one fails
• Chunking, which decides what can be found
• Retrieval that actually retrieves
• How much context to send
• Grounding and citations
• Evaluating each half separately
• Keeping the index current
• Permissions, which the index quietly flattens
• Building it with CodeWords
• What to get right first
• Frequently asked questions

The two halves, and which one fails

Retrieval finds passages it believes are relevant.

Generation writes an answer using them.

When the answer is wrong, one of two things happened: the right passage was not retrieved, or it was retrieved and the model used it badly. These need different fixes and the first is far more common.

A fluent wrong answer is the characteristic failure, because the model will write confidently from whatever it was given, including material that does not contain the answer.

Check retrieval first, always. Look at what was actually fetched for the failing question before changing a prompt or a model. Teams routinely spend weeks on the generation half for a problem entirely in the other one.

Chunking, which decides what can be found

The most consequential and least discussed design decision.

Fixed-size chunks cut through meaning. A boundary in the middle of a procedure produces two chunks, neither of which answers the question.

Split on structure instead — headings, sections, paragraphs — so a chunk is a coherent unit that means something on its own.

Carry the context. A chunk saying "this is not permitted" is useless without the heading it sat under, so prepend the document title and section path to each chunk.

Overlap modestly so a sentence spanning a boundary appears whole in one of them.

Size for the content, not by rule. Reference material with short self-contained entries wants small chunks; narrative documentation wants larger ones.

Test by reading. Take fifty chunks at random and ask whether each makes sense alone. If they do not, retrieval cannot succeed no matter what else you do.

Retrieval that actually retrieves

Embedding search finds semantic similarity, which is good at paraphrase and poor at exact terms — product codes, error numbers, names.

Keyword search finds exact terms and misses paraphrase.

Hybrid search combines both, and it is the default worth starting from rather than an optimisation.

Filter by metadata first. Restricting to the right product, version, or language before ranking removes a whole class of plausible-but-wrong results.

Re-rank the candidates. Fetching a wider set and re-ranking it with a more expensive model is one of the larger quality improvements available and is frequently skipped.

Watch for near-duplicates. Three chunks saying the same thing crowd out the one that adds something, so deduplicate before sending.

How much context to send

Fewer, better passages beat more. Three relevant chunks usually produce a better answer than ten mixed ones, and cost a third as much.

Long context is not free, in money or in quality. Models attend unevenly across a long context, and material in the middle is used less reliably.

Do not fill the budget because it exists. The temptation is to send everything that might be relevant, which dilutes what is actually relevant.

Order matters. Put the most relevant material where the model attends best, which in practice means not buried in the middle.

Measure it. Try three, five, and ten passages against your evaluation set. The answer is usually smaller than the intuition.

Grounding and citations

Instruct the model to answer only from the provided material, and to say so when the material does not contain the answer. This is a substantial improvement and it is a prompt-level change.

Require citations per claim, which makes the answer checkable and makes retrieval failures visible rather than hidden behind fluent prose.

Show the sources to the user, so somebody can verify rather than trust.

"I do not know" is a correct answer and worth explicitly permitting. A system that always answers will always answer, including when it should not.

Check that the citation supports the claim. A cited answer where the citation does not contain the assertion is worse than an uncited one, because it looks verified.

Evaluating each half separately

The discipline that makes improvement possible rather than guesswork.

Build a set of real questions with known-good answers and, crucially, the passages that contain them.

Measure retrieval on its own: for each question, was the passage containing the answer among those retrieved? This is a number you can improve directly.

Measure generation given correct passages: when the right material is provided, is the answer right? A separate number with separate fixes.

Fifty real questions beat any benchmark, including the awkward ones — ambiguous phrasings, questions whose answer spans two documents, questions with no answer in the corpus.

Re-run on every change to chunking, embedding model, retrieval strategy, prompt, or model. All five move quality and none announces it.

Include questions with no answer, to check the system says so rather than inventing one.

Keeping the index current

Content changes and the index does not, unless something updates it.

Re-embed on change, keyed on the source document, and remove chunks for deleted documents — the second is the step most often missed, leaving the system confidently citing material that no longer exists.

Track the source version on each chunk, so an answer can be traced to what the document said at the time.

Re-embedding everything is necessary when the embedding model changes, which is worth planning for rather than discovering.

Stale answers are the quiet failure. The system keeps working and starts being wrong, which nothing reports — so a periodic check that indexed content matches the source is worth having.

Permissions, which the index quietly flattens

A problem that arrives the moment a RAG system covers more than one audience.

The index does not know who is asking. Embed everything into one collection and every user can retrieve from all of it, regardless of what they could open directly.

The answer reveals the content. Even without a link, a generated summary of a restricted document has disclosed it.

Filter at retrieval, not after generation. Removing a citation from a finished answer does not remove the material the answer was built from.

Store the source's access metadata on the chunk, and pass the asker's identity into the query so the filter is applied before ranking.

Permissions change after indexing. A document restricted last week is still indexed under its old visibility unless something updates it.

Test it deliberately. Ask a question as a low-privilege user about material they should not see, and check both the citations and the text of the answer — that test finds the problem, and almost nobody runs it.

Building it with CodeWords

CodeWords connects to more than 3,000 integrations, which covers the connective work of keeping content flowing into an index and answers flowing out to wherever people ask.

1. Open CodeWords and start a new automation.
2. Describe what should happen in plain language to Cody, the automation builder: which content source, what triggers a refresh, and where answers should appear.
3. Describe the exceptions: a document that has been deleted, an answer with no supporting material, a question outside the corpus.
4. Connect the content sources once and reuse the connections.

You describe the outcome; Cody builds it, connects it, and deploys it. The free plan covers light use, with Pro at $39 per month and Business at $100 per month as usage grows; details are on the pricing page.

What to get right first

The evaluation set: fifty real questions, with the answers you consider correct and the passages that contain them. Without it, every change is a guess and nobody can tell whether last week's adjustment helped. With it, the retrieval number tells you where the problem actually is.

Two habits make the difference. Measure retrieval separately from generation, since that single split tells you which half to work on and most teams work on the wrong one. And include questions the corpus cannot answer, because a system that always produces an answer will confidently produce one for those too.

Frequently asked questions

Our answers are wrong — is it the model or the retrieval?

Check retrieval first. Look at the passages actually fetched for a failing question; if the answer was not among them, no model change will help. Retrieval accounts for the large majority of RAG quality problems.

How should documents be chunked?

On structure — headings, sections, paragraphs — with the document title and section path prepended to each chunk so it makes sense alone. Fixed-size splitting cuts through the middle of the thing somebody is asking about.

Should I use vector search or keyword search?

Both. Embedding search handles paraphrase and misses exact terms like error codes and product names; keyword search is the reverse. Hybrid is the sensible default rather than an advanced optimisation.

How many passages should I send?

Fewer than the context budget allows — usually three to five. More passages dilute the relevant ones, cost more, and models attend unevenly across long contexts. Measure it against your evaluation set rather than guessing.

How do I reduce invented answers?

Instruct the model to answer only from the provided material, permit "I do not know" explicitly, require citations per claim, and check that citations actually support what they are attached to.

What happens when a document is deleted?

Nothing, unless you remove its chunks. This is the most commonly missed step, and it leaves the system confidently citing material that no longer exists — a failure that produces no error and looks entirely normal.

Do I need to rebuild the index if I change embedding models?

Yes, entirely. Embeddings from different models are not comparable, so a mixed index returns nonsense. It is worth planning for before choosing a model rather than discovering it later.

Does RAG respect document permissions?

Only if you build it to. A single index flattens everything into one searchable pool, so filter by the asker's access at retrieval time using metadata on each chunk — filtering after generation is too late, since the answer already contains the content.

How do I test that access control works?

Ask a question as a low-privilege user about material they should not see, and check the answer text as well as the citations. A summary of a restricted document has disclosed it even with no link attached.

Is RAG better than fine-tuning?

They solve different problems. RAG supplies facts the model did not have and can be updated the moment content changes; fine-tuning shapes behaviour and format. For "answer from our documentation", RAG is the right tool and fine-tuning is not a substitute.

Related reading

• Model Context Protocol
• Intelligent automation
• Semantic caching