Query rewriting, multi-query and HyDE

Why it matters

Users type "did it break again?", "the thing from last week" and "refund policy" when they mean "can I return a sale item after 30 days?". The retriever embeds exactly what was typed and finds chunks that look like the words, not chunks that answer the question. Pronouns have no referent outside the conversation, dates are implicit, and one question often hides three. That gap closes with one cheap model call. Interviewers raise it as "users ask follow-ups and retrieval returns nothing relevant".

The idea

Put a model between the user and the retriever to turn a question into the query, or queries, most likely to retrieve the passages that answer it.

Rewriting, with history

The simplest step is a small model that restates the question as a self-contained search query: abbreviations expanded, chat filler dropped, the entity named. Alone it can do little with "any update on the thing?", which is the point: the rewriter needs the conversation. Give it the last few turns and ask for a standalone question, so "Does it apply to sale items too?" becomes "Does the 30-day return window apply to sale items?". This is the fix for follow-ups, because a bare follow-up says nothing by itself and its embedding matches noise.

Decomposition

"Compare our refund and exchange policies for international orders" is two or three retrievals. Have the model split it into sub-questions, retrieve for each, and union the results; a single embedding sits between the topics and matches none of them well.

Multi-query

Generate three to five paraphrases, retrieve for each, union and deduplicate. Different phrasings land in different parts of embedding space, so the union covers more than any one query. Rank the union by how many queries retrieved each chunk and how highly (reciprocal rank fusion is the usual recipe) and keep the top k.

HyDE

Hypothetical document embeddings: ask the model to write the answer it would expect, even if wrong, and embed that instead of the question. A question and its answer look different in embedding space; a hypothetical answer looks like the real passage, and the hallucination is harmless because it is only used to search, never shown. It works best on short factual questions over prose.

Step-back prompting

Ask for the more general question behind the specific one ("what are the return rules for discounted items?" behind "can I return this jacket I bought on sale?"), retrieve for both, and provide both sets. Useful when the corpus is organised by principle and the user asks about an instance.

Query routing

With several corpora (product docs, tickets, code, the HR handbook), classify the question and search the right one, or search several and merge. Routing by a small model or a rule beats searching everything, because an off-topic corpus adds noise a reranker then has to remove.

The cost of each call

Every technique is at least one extra model call before retrieval, and multi-query and HyDE add generation latency plus several searches. A small, fast model handles rewriting and decomposition well at 200 to 500 milliseconds and a fraction of a cent per query.

When a rewrite is enough

Most of the win comes from history resolution and a clean restatement. Add decomposition when questions are compound, multi-query when context recall is low and the corpus is large and varied, HyDE when questions are short and factual, and routing when there is more than one corpus. Measure each with context recall against the plain query; if the number does not move, remove the step.

	No rewrite	Rewrite with history	Multi-query	HyDE
Recall gain	Baseline	Large on follow-ups and vague questions	Moderate on large, varied corpora	Moderate on short factual questions
Latency added	None	One small-model call, 200 to 500 ms	One call plus 3 to 5 searches	One generation of a short passage
Cost added	None	A fraction of a cent	A fraction of a cent plus search load	Output tokens for the hypothetical answer
Use when	Single-turn, precise users, tight latency	Any conversational product	Recall is low and the corpus is varied	Questions and passages differ in form