Naive RAG end to end

Why it matters

A model answers from two places: its weights and its context. The weights are frozen at training time, know nothing about your documents, and fill gaps with fluent guesses. Retrieval-augmented generation moves the facts into the context at query time, so the model reads rather than remembers. It is the default architecture for assistants over private or changing text, and the system interviewers ask you to design first and improve second. Naive RAG is the baseline every later lesson in this module improves.

The idea

RAG is two pipelines that share one index: offline turns documents into searchable pieces; online turns one question into an answer grounded in them.

Offline: build the index

Load documents from wherever they live. Parse each into clean text with headings, tables and lists preserved as far as the format allows. Chunk the text into pieces of a few hundred tokens, small enough to embed and precise enough to retrieve. Embed each chunk with an embedding model, which maps text to a vector so that similar meanings sit close together. Store the vector beside the chunk text and its metadata (source, title, section, date, who may read it). Rerun it when documents change; the index is only as fresh as its last run.

Online: answer one query

Embed the query with the same embedding model. Retrieve the k chunks whose vectors sit nearest the query vector, usually by cosine similarity, with k around 5. Assemble a prompt: a system instruction that binds the answer to the supplied text, the chunks each labelled with an id, then the user's question. Generate the answer. Cite: the model names the chunk ids it used and the application turns them into links back to the source.

The assembled prompt has a fixed shape:

SYSTEM: Answer only from the sources below. Cite the source id after
each claim. If the sources do not contain the answer, say so.

[1] returns-policy > Window: Items may be returned within 30 days...
[2] returns-policy > Exceptions: Sale items are final unless faulty...
[3] shipping-faq > Delays: ...
[4] ...
[5] ...

USER: Can I return a sale item I bought three weeks ago?

Why it works

The model is not asked to know anything. It is asked to read five short passages and answer from them. Attention over provided text is far more reliable than recall from weights, and a passage in context can be cited, checked and replaced tomorrow by re-indexing. That is the whole claim of RAG: the knowledge problem moves out of training and into a data pipeline you control.

A token and cost budget

Take one query with five chunks of 400 tokens, a 600-token system prompt, a 50-token question and a 300-token answer: about 2,650 input tokens and 300 output. At a mid-range price of $1 per million input tokens and $5 per million output, that is $0.0027 in and $0.0015 out, about $0.004 per query; 100,000 queries a day cost about $420. The system prompt is nearly a quarter of the input and never changes, so prompt caching on that prefix cuts its share to a tenth. Note what dominates: the chunks. Every chunk added costs 400 tokens on every query; every chunk dropped risks losing the answer.

What "naive" means

Naive RAG has fixed-size chunks, a single query embedded as typed, top-k by cosine similarity, no reranking, no query rewriting, no evaluation. On real data a vague question retrieves vaguely related chunks, a table split across two chunks loses its header, k is a guess, and nobody measures whether the right passage was in the context. Each is a lever, and interviewers expect you to name them in order: better parsing, better chunking, query rewriting, hybrid search, reranking, and an evaluation set that shows which lever moved the number.

Trade-offs

Against no retrieval, naive RAG buys freshness and grounding at the cost of a retrieval system and roughly three times the input tokens per query. Against stuffing whole documents into a long context window, it is cheaper and faster per query, but the retrieval step can miss, and a miss produces a confident answer from the wrong passage. The naive version has specific weak points: fixed chunks cut tables and arguments in half; cosine similarity rewards passages that sound like the question rather than ones that answer it; one query cannot cover a two-part question; and without reranking, k must be large to be safe and is expensive once it is. Without evaluation, none of this is visible until a user reports a wrong answer.

In an interview

Draw the two pipelines first: "Offline I parse, chunk, embed and index with metadata; online I embed the query, retrieve the top five chunks, put them in a prompt that binds the answer to them, generate, and cite chunk ids." Then name the budget: "about 2,700 input tokens and under half a cent per query, dominated by the chunks." At junior depth, draw the pipelines and say why provided text beats memory. At mid depth, give the budget, define naive and list the levers. At senior depth, say which lever you would pull first for a given symptom and how context recall and faithfulness would show it. Expect: "Why not put all the documents in the context?" Cost, latency, and the window has a limit; retrieval keeps all three bounded.