Prompt assembly and citations

Why it matters

Retrieval found the passages. The prompt decides whether the model uses them, invents around them or quotes the wrong one. Assembly is where grounding is enforced or lost: the instruction that binds the answer to the sources, the order of the chunks, the ids that make a citation checkable, the history that keeps a follow-up coherent. A citation is the user's only evidence that the answer came from somewhere, and an interviewer will ask how you know a citation is real.

The idea

Binding instructions

The system instruction states the contract: answer only from the sources; cite the source id after every factual claim; if the sources do not contain the answer, say so. Each clause does work: "only from the sources" suppresses recall from weights, "cite every claim" makes the output checkable, "say so" gives the model a way out other than invention. Keep it short and put it first.

Labelled sources

Give each chunk a stable id, a source title with section path, and the text. The id is what the model cites; the title and path are what the user sees. Short numbers in brackets work.

SYSTEM: Answer only from the sources. After each claim write the
source id in brackets. If the sources do not contain the answer,
reply: The sources do not say.

[1] returns-policy > Window
Items may be returned within 30 days of delivery for a full refund.

[2] returns-policy > Sale items
Sale items may be returned within 14 days; refunds are store credit.

HISTORY: user asked about the standard window; assistant: 30 days [1].
USER: And for sale items?

Ordering and lost in the middle

Models attend more reliably to the start and end of a long context than to the middle. With five chunks this barely matters; with twenty it does. Put the highest-ranked chunk first and the second-highest last, or order by relevance and keep k small enough that nothing is buried.

Asking for citations in a structured form

Free-text citations ("according to the returns policy") cannot be checked. Ask for the id inline after each claim, or for a JSON answer with a claims array where each claim carries source_ids. Either way, validate in code that every cited id exists in the prompt: a citation to a source that was not provided is a hallucination caught by a set lookup.

Rendering

Turn ids into links to the cited section, or into footnotes with the title and a short quoted span, so the user can land on the passage. For high-stakes domains, show the quoted span beside the claim so verification needs no click.

The post-check

Existence is not support. A second cheap model call, or an entailment model, takes each claim and its cited chunk and answers one question: does this passage support this claim? This is the faithfulness metric run online, per answer, and it catches the most dangerous failure: a real citation under a wrong claim.

"The sources do not say"

Make abstention a first-class output: instruct it, test it with questions whose answers are absent from the corpus, and render it well, with the nearest sources offered so the user can judge. Monitor the abstention rate: too high and retrieval is failing or the instruction is too strict; too low and the model is answering from memory.

History and budget

A follow-up needs the recent turns, but history competes with sources for tokens. A workable split for a 6,000-token input budget: 600 for instructions, 3,000 for six or seven chunks, 1,500 for recent turns with older ones summarised, and the question. Count with the model's tokeniser, and drop the lowest-ranked chunk before the instruction or the question.