How RAG Retrieval Actually Works

Every "intro to RAG" post shows you the same diagram: user query → embed → vector search → stuff context into prompt → LLM responds. That diagram is correct in the same way "car = wheels + engine + fuel" is correct: technically true, but tells you nothing about how to drive one across a city.

If you've ever demoed a RAG system that worked great on 10 PDFs and fell apart on 10,000, this post is for you. We're going to cover what actually matters, with numbers.

The four hard parts of RAG

Nobody ships RAG and complains about the prompt template. The four things that break are:

1. Chunking — what slice of document counts as "one thing to retrieve"?
2. Embedding model choice — 768 dimensions or 1536? Is cosine similarity actually the right metric?
3. Retrieval — ANN vs exact KNN, hybrid search, re-ranking, top-K.
4. Evals — how do you know retrieval is good before users tell you it's bad?

Let's go.

1. Chunking: the most impactful, least discussed step

Chunking strategy accounts for ~40% of your end-to-end retrieval quality in my experience. Everything else is tuning around it.

Why you don't want "just split by 512 tokens with 10% overlap"

Because real documents have semantic structure. A heading followed by two paragraphs + a code block is one semantic unit; splitting it in the middle means neither chunk has the full context to answer questions about it.

What works better (ranked)

I default to semantic chunking for new projects. Here's roughly what that looks like:

# Pseudocode — not production-ready
def semantic_chunk(text: str, threshold: float = 0.75) -> list[str]:
    sentences = split_into_sentences(text)
    embeddings = embed_model.encode(sentences)
 
    chunks = []
    current = [sentences[0]]
 
    for i in range(1, len(sentences)):
        sim = cosine(embeddings[i-1], embeddings[i])
        if sim < threshold:
            chunks.append(" ".join(current))
            current = [sentences[i]]
        else:
            current.append(sentences[i])
    chunks.append(" ".join(current))
    return chunks

2. Embeddings: more dimensions ≠ always better

Quick decision tree for picking an embedding model today:

• Budget unlimited, best accuracy: text-embedding-3-large (OpenAI, 3072-d) or Vertex text-embedding-005 (Google, 768-d, surprisingly competitive).
• Self-hosted, English-only: BAAI/bge-large-en-v1.5 (1024-d). Hugging Face weights, runs on a T2 CPU instance fine.
• Self-hosted, multilingual: BAAI/bge-m3 (1024-d). Chinese, English, and 100+ others.

The surprise here is that dimension count isn't the bottleneck. A well-tuned 768-d model will outperform a poorly-tuned 3072-d model every time because you're usually hitting chunking or retrieval issues long before you're limited by embedding dimensionality.

Distance metric

If you're using a modern embedding model, use cosine similarity. Dot product is equivalent when vectors are normalized (which all modern embedding APIs return). L2 distance works, but gives worse recall on the same corpus. Don't overthink this one.

3. Retrieval: KNN is a baseline, not a finish line

Vector DBs default to top-K nearest neighbors. This gets you to demo-able. It does not get you to production-grade.

The retrieval stack I use for production

User query
   │
   ▼
Query rewriting (LLM: "rephrase this user query for vector search, output JSON only")
   │
   ▼
Hybrid search: dense (embedding ANN) + sparse (BM25 on keywords)
   │
   ▼
Reciprocal Rank Fusion (RRF) to combine two ranked lists
   │
   ▼
Cross-encoder re-ranker (top-50 results → top-5)
   │
   ▼
Context stuffing into prompt

Numbers on a real 200-document internal-knowledge eval:

That re-ranking step is 70% of your quality gain for ~10ms of latency. Worth it.

4. Evals: stop guessing

If you ship without an eval set, you will discover retrieval bugs through angry Slack messages. I've been there.

Minimum viable eval:

• 50–100 question–answer pairs where the answer is in your corpus, written by someone who didn't build the retrieval system.
• Measure Recall@K (is the correct chunk in the top-K retrieved?) and Faithfulness (did the LLM answer only using the provided context?).
• Run the eval on every change to chunking, embedding, or retrieval config. CI gate it if you can.

The trap everyone falls into

You demo on 10 docs. Everything works. You import 10,000 docs. Accuracy collapses.

The reason is almost always one of:

1. Chunk size too large — embeddings become "about everything, so about nothing."
2. No re-ranking — the chunk you need is #7 in the ANN results; you stuff top-4 into the prompt.
3. Mixed-document domain — your embedding model handles engineering docs well but is terrible at legal contracts; run separate evals per domain.

Fix those three and you've solved 90% of what breaks when you scale.

Recommended reading (not just "popular" posts)

• "Advanced RAG Techniques" — Anthropic team, surprisingly practical section on retrieval.
• BGE M3 paper — the current state of the art for open multilingual embeddings.
• RAGAS framework — automated eval metrics, saves you from writing a faithfulness judge from scratch.

Happy shipping. If you're working on a RAG system in production, I'd love to hear your war stories.