Retrieval-augmented generation without the hype

The pitch for retrieval-augmented generation is simple: instead of hoping the model memorised your documentation, you look up relevant passages at question time and paste them into the prompt. The model answers from the evidence in front of it.

The pitch is correct. The trouble is that the first version most teams build — split documents into chunks, embed them, take the top five by cosine similarity — works well enough in a demo and then quietly fails on real questions. Here is where it breaks, in rough order of how often I have seen each problem.

Failure #1: the answer was never retrieved

If the passage containing the answer is not in the context, the model has two options: say it doesn't know, or improvise. Models are trained to be helpful, so they often improvise. That gets logged as "hallucination", and people start tuning the prompt. The prompt was never the problem.

Chunking matters more than the embedding model

People spend days comparing embedding models and five minutes on chunking. It should be the other way round. Typical problems:

• Chunks that cut through the answer. Fixed 500-character windows happily split a table from its header or a step from its command. Split on structure — headings, paragraphs, list items — and add a modest overlap.
• Chunks without context. "Set it to 30 seconds" is useless without knowing what "it" is. Prepend the document title and section path to every chunk before embedding: Billing › Invoices › Retry policy: Set it to 30 seconds…
• Chunks that are too big. A 2,000-token chunk about five topics embeds as a blurry average of all five and matches nothing strongly.

Use hybrid search

Dense embeddings are good at meaning and bad at exact strings. Ask about error E1047, a product SKU or a function name, and pure vector search will cheerfully return passages about similar errors. Classic keyword search (BM25) has the opposite strengths.

Running both and merging the results is cheap and robust. A simple and popular merge is reciprocal rank fusion:

def rrf(rankings, k=60):
    scores = {}
    for ranking in rankings:            # e.g. [bm25_ids, vector_ids]
        for rank, doc_id in enumerate(ranking):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
    return sorted(scores, key=scores.get, reverse=True)

No score normalisation, no tuning beyond k, and it rarely does worse than the better of its two inputs.

Retrieve wide, then rerank

Embedding search compares a query vector to document vectors that were computed independently. A cross-encoder reranker reads the query and a candidate passage together, which makes it much better at judging relevance — and much slower. The standard pattern is to retrieve 30–50 candidates cheaply, rerank them, and keep the top 5–8 for the prompt. In my experience this one step fixes more bad answers than any other single change.

Measure retrieval separately

You need a small set of real questions with the passages that answer them — 50 to 100 is enough to start. Then track a retrieval metric on its own, before generation is involved at all:

If recall@8 is 70%, then roughly three answers in ten are being written without the evidence. That number tells you where to work next far better than reading chat logs.

Then make the generator honest

Once retrieval is solid, a few prompt-level habits help:

• Number the passages and ask the model to cite the numbers it used. Uncited claims become easy to spot.
• Explicitly allow "the documents don't say" as an answer — and test that it actually happens on questions you know aren't covered.
• Put the question after the passages, close to where generation starts.

The short version

Structure-aware chunks with context, hybrid search, a reranker, and a recall number you check every time you change something. None of it is glamorous. All of it works better than swapping in a bigger model and hoping.