Blueprint: document Q&A

Why it matters

Document Q&A is the prompt most loops open with, because everyone can picture the product and the depth is hidden in the details: messy sources, permissions, citations, a latency and cost ceiling, and a pipeline where most failures happen before the model sees anything. This blueprint is one complete answer in the 45-minute structure. Read it with a timer once, then cover it and produce your own board.

The idea

Scope (0 to 5)

The prompt: an internal assistant answering support staff over 2 million resolved tickets and 50,000 policy pages. The answers: 20,000 questions a day in working hours, a peak near 2 a second; citations required; a complete answer within 3 seconds; $0.01 per question. Tickets carry customer data and are visible only to the owning team and region; policy pages are visible by role. Policies change weekly; yesterday's tickets are fresh enough. Out of scope: write actions and answering customers directly.

The requirement that changes the design is permissions: access control must live inside retrieval, not on the answer.

Eval plan (5 to 10)

A golden set of 200 questions sampled from what staff already ask in chat: 120 policy lookups, 60 "has this happened before and what fixed it" questions over tickets, and 20 the system must decline, each with a reference answer and supporting passage ids. Release gates: context recall at least 0.85, faithfulness at least 0.9, correct citations on every claim. The guardrail: zero answers that cite or paraphrase a document the user cannot open, tested with 50 questions answerable only from restricted documents, asked as unprivileged users. In production, an LLM judge scores faithfulness on 5 percent of traffic, calibrated against 100 human labels, and judged failures become candidate golden questions.

High-level design (10 to 20)

Offline: pull sources, parse with a layout-aware parser so tables survive, strip quoted replies and signatures from tickets, chunk, attach access-control metadata to every chunk, embed, and write to a hybrid index holding both a keyword index and vectors. Online: check an exact-match cache, embed the question, search both indexes inside the user's permissions, fuse the lists, rerank the top 40 to the best 6, assemble a prompt that binds the answer to numbered sources, generate with a mid-tier model, stream the answer with citations.

Deep dives (20 to 35)

Chunking and parent-child. Policy pages split on headings into child chunks of about 400 tokens, each carrying its title and section path. Retrieval runs on children for precision; for the top two policy hits the prompt gets the parent section, capped at 800 tokens, so a procedure is not cut in half. A long ticket thread retrieves poorly, so a batch job at half price writes a problem-and-resolution card per ticket and indexes the card, with the thread as its parent. About 2.7 million vectors; quantised to 8 bits they fit in one index.

Hybrid plus reranker. Support questions are full of exact strings: error codes, plan names, ticket ids. Keyword search finds those; vectors find paraphrase. Run both, fuse by reciprocal rank, rerank the top 40 with a cross-encoder, keep 6. Watch recall at 40 against recall at 6: the reranker closes that gap, for about 150 ms.

Permission-aware retrieval. Each chunk stores the groups that may read it, synced from source systems within minutes, revocations first. The filter runs inside both searches, before ranking. Filtering after generation fails twice: the answer may already paraphrase forbidden text, and late filtering can leave fewer than 6 chunks. The answer cache is keyed by access group as well as question.

Cost. System prompt 800 tokens from the prompt cache at a tenth of the input price; chunks and two parent sections, about 3,000 tokens; question and history 200; answer 350. At a mid-tier $1 per million input and $5 per million output: $0.0032 in, under $0.0001 for the cached prefix, $0.0018 out, about $0.005 a question or $100 a day. The reranker is a fixed GPU cost, a fraction of a cent per question at this volume; the judge adds about $10 a day. Latency: about 300 ms to retrieve and rerank, first token by 800 ms, the full answer streamed by 3 seconds.

	Naive vector RAG	Hybrid plus reranker	Agentic retrieval
Recall on codes and ids	Weak: embeddings blur exact strings	Strong: keyword search catches them	Strong, after several searches
3-second budget	Comfortable	Fits: rerank adds about 150 ms	Fails: each search round adds a model call
One-cent budget	About half a cent	About half a cent plus a fixed rerank cost	Several cents for multi-step questions
Permissions	Filter in one index	Filter in both indexes before fusion	Filter in every tool call the agent makes
Verdict	The baseline to beat	Ship this	Keep for a later multi-hop slice if the eval demands it

What ships first (42 to 45)

Policy pages only, for one support team, behind a flag: hybrid search, reranker, citations, the permission filter and the golden set as the release gate. Tickets follow in the second slice, once PII scrubbing and the card job are proven, because they carry the most risk. The first number to watch is faithfulness on sampled traffic; the second is how often staff click a citation.