Reranking with cross-encoders

Why it matters

Retrieval returns the nearest chunks, and nearest is not the same as most relevant. The embedding was computed before anyone asked a question, so it cannot know which part of a chunk the question cares about. The model reading the prompt then sorts through ten candidates of which three matter, worse than a dedicated scorer would and at a higher price per token. Reranking adds a second, more careful look at a small set of candidates. It is the cheapest quality win in most retrieval systems; interviewers ask to see whether you understand why two stages beat one.

The idea

Bi-encoder versus cross-encoder

A bi-encoder is the embedding model you already have. It encodes the query and each document separately, and similarity is a function of the two vectors. That independence lets documents be embedded once and indexed, which is what makes nearest-neighbour search over millions of chunks possible; it is also the limitation: the document vector was fixed without seeing the query, so a chunk that answers the question in its third sentence looks like any other chunk on the general topic.

A cross-encoder reads the query and one document together, as a single input, and outputs one relevance score. Every attention layer can compare query words to document words directly, so it notices that "cancel within 14 days" answers "how long is the refund window" while a chunk about delivery times does not. The score is far better. Nothing can be precomputed, though: each pair is a full forward pass, so scoring a million chunks per query is out of the question.

The two-stage pattern

Stage one, retrieval, pulls a wide candidate set cheaply: hybrid search, as in the Hybrid search lesson, returning the top 50 to 100. Stage two, reranking, runs the cross-encoder over the query paired with each candidate and keeps the top 5 to 10 for the prompt. Retrieval optimises recall, getting the answer somewhere in the list; reranking optimises precision at the top, getting it into the few slots the context budget allows.

Where the reranker comes from

Hosted reranking APIs take a query and a list of passages and return them scored: the easiest route, billed per document. Open cross-encoder models, from a few hundred million parameters to several billion, run on your own GPU or, for small ones, a CPU. An LLM as reranker is the third option: a prompt that lists the candidates and asks a general model to score or order them. It follows instructions ("prefer official documentation over forum posts") and judges nuanced relevance, and it costs the most, because every candidate's tokens are billed on every query. Listwise prompting, all candidates in one prompt, is the cheaper way to run it.

Latency budgets

A cross-encoder over 100 passages of 300 tokens is 30,000 tokens of attention per query. On a GPU with batching that is tens to a few hundred milliseconds depending on model size; on a CPU it is seconds. Set a retrieval budget, say 300 milliseconds end to end, and spend it deliberately: query embedding and hybrid search in under 50, reranking in about 150, payload fetch and prompt assembly in the rest. If the reranker does not fit, cut candidates before you cut the reranker: 30 candidates reranked well beat 100 unreranked.

When it is not worth it

Skip reranking when retrieval is already precise, as with small, clean corpora; when the prompt takes every candidate anyway; when the whole budget is tens of milliseconds; and when an eval shows no gain. Reranking cannot rescue a candidate list that lacks the answer, so low recall at 100 is a retrieval problem, not a reranking problem.

Measuring the gain

Context precision, in the Ragas vocabulary, asks what fraction of the chunks handed to the model are relevant, with relevant ones ranked high; compute it on the top ten before and after reranking. Expect a large jump at the top, with recall at ten rising too as relevant chunks from rank 30 move up.

	No reranker	Cross-encoder	LLM reranker
Accuracy	Bi-encoder order: nearest, not most relevant	Large gain in precision at the top	Highest on nuanced relevance; follows instructions
Latency	Nothing added	Tens to a few hundred milliseconds for 100 pairs on a GPU	Seconds unless candidates are batched into one listwise prompt
Cost per query	Nothing added	Small: one small-model pass per pair	Highest: every candidate token billed on every query
Fits	Tight budgets, easy corpora	The production default	Hard domains, small candidate sets, offline evaluation