How RAG Actually Works: A Practical Guide

Every LLM has a knowledge cutoff and a context window. Retrieval-augmented generation (RAG) is the standard answer to both problems: instead of asking the model to know everything, you fetch the relevant documents at question time and hand them to the model as context. The idea is simple. The engineering is where projects succeed or fail.

The pipeline in four stages

1. Ingest: split source documents into chunks small enough to embed and retrieve precisely.
2. Embed: convert each chunk into a vector that captures its meaning, and store it in a vector database.
3. Retrieve: embed the user's question, find the most similar chunks, and optionally re-rank them.
4. Generate: pass the question plus retrieved chunks to the LLM with instructions to answer only from that context.

Each stage has one decision that dominates quality. For ingestion it is chunking. For retrieval it is whether you add re-ranking. For generation it is how strictly you force the model to ground its answer.

Chunking is the highest-leverage decision

A chunk is the unit of retrieval. Chunk too large and you retrieve pages of noise that crowd the context window. Chunk too small and you retrieve sentences with no surrounding meaning. A reliable default: split on document structure (headings, paragraphs) with a size target of 300 to 500 tokens and 10 to 15 percent overlap.

def chunk_markdown(text: str, target_tokens: int = 400, overlap: int = 50) -> list[str]:
    """Split on headings first, then merge sections up to the token target."""
    sections = re.split(r"(?=^#{1,3} )", text, flags=re.MULTILINE)
    chunks: list[str] = []
    current = ""
    for section in sections:
        if count_tokens(current) + count_tokens(section) > target_tokens and current:
            chunks.append(current.strip())
            # Carry a small tail forward so context is not cut mid-thought.
            current = current[-overlap * 4 :] + section
        else:
            current += section
    if current.strip():
        chunks.append(current.strip())
    return chunks

Retrieval quality: measure it, then fix it

Before touching the LLM, evaluate retrieval alone. Build a golden set of 30 to 50 real questions, note which document each answer lives in, and measure recall at k: how often the right chunk appears in the top k results. If recall@5 is below roughly 85 percent, no prompt engineering will save the final answer.

• Hybrid search (vector similarity plus keyword BM25) fixes most failures on exact terms like error codes and API names.
• A cross-encoder re-ranker over the top 20 candidates is the cheapest large quality win in most pipelines.
• Metadata filters (product version, document type) beat clever embeddings whenever the user's intent is categorical.

Grounding: make the model cite or decline

The generation prompt should do three things: restrict the model to the provided context, require citations back to chunk identifiers, and give it explicit permission to say the answer is not in the documentation. That last instruction is what separates a trustworthy assistant from a confident liar.

A RAG system that says 'I don't know' at the right moments earns more user trust than one that always answers.

If you want to build this end to end — ingestion, pgvector, re-ranking, evaluation, and a clean API — the Docs RAG Assistant project on this site walks through a complete production implementation.