Hybrid search: BM25 plus dense

Why it matters

Dense retrieval understands paraphrase and misses the obvious. A user types a part number, an error code, a surname or a function name, and the embedding smooths it into a vague neighbourhood where nothing matches exactly. Keyword search has the opposite failing: it finds the string and misses the meaning. Every production retrieval system ends up running both. Interviewers ask about hybrid search because it separates people who have shipped retrieval from people who have read about embeddings.

The idea

What sparse catches

BM25 scores a document by the query terms it contains, weighting each term by how rare it is across the corpus and how often it appears in the document, with a correction so long documents do not win by bulk. It runs on an inverted index, a map from each term to the documents containing it. The representation is sparse, one weight per vocabulary term, nearly all zero, and that makes it exact on identifiers. SKU-88213, ERR_CONN_RESET, Okonkwo and parseConfig are rare terms, so they carry high weight and a document containing one ranks first. A dense model sees a rare token as a few subword pieces with no learned meaning.

What dense catches

A dense model maps "my laptop will not switch on" and "notebook fails to power up" to nearby vectors although they share no useful word. It handles synonyms, paraphrase, misspellings that tokenise similarly and, with a multilingual model, a query in one language against documents in another. BM25 scores all of those at zero, because no term matches. This is the vocabulary mismatch problem.

Running both and fusing

Hybrid search runs the query through both retrievers and merges the two ranked lists into one. The scores cannot be added directly: BM25 scores are unbounded and corpus-dependent, and cosine scores sit on a model-specific scale. Reciprocal rank fusion ignores scores and uses only positions. Each document gets, from each list it appears in, one divided by a constant plus its rank in that list, and the contributions are summed. The constant, usually 60, flattens the difference between rank one and rank two so that no single retriever dominates: a document near the top of both lists beats one that tops a single list and is absent from the other.

A worked example with the constant at 60. BM25 returns A, C, B in that order; dense returns B, A, C.

A: 1/(60+1) + 1/(60+2) = 0.0164 + 0.0161 = 0.0325
B: 1/(60+3) + 1/(60+1) = 0.0159 + 0.0164 = 0.0323
C: 1/(60+2) + 1/(60+3) = 0.0161 + 0.0159 = 0.0320
fused order: A, B, C

A wins because it is high in both lists. The gaps are tiny: fusion is a tie-breaker between two orderings, not a precise score.

Weighting

Plain fusion treats both lists as equal. Multiplying one list's contributions by a weight shifts the balance: lean lexical for a catalogue of part numbers, lean dense for a help centre of prose. Tune the weight on judged queries and re-tune when the corpus or the model changes; a weight that is right for one corpus is a guess on another.

Learned sparse models

Between BM25 and dense sit learned sparse models: a transformer outputs weights over the vocabulary rather than a dense vector, expanding "notebook" toward "laptop" while keeping exact terms exact. The output still fits an inverted index, so one engine serves it and the vocabulary mismatch gap closes without a second index. They cost a model pass at index and query time, and suit a team whose constraint is a single search engine.

Why hybrid plus a reranker is the baseline

Each retriever is a cheap, broad net with a known blind spot. Fusion covers the blind spots and produces a candidate list wide enough to contain the right answer most of the time. What it cannot do is order the top of that list precisely, because neither retriever reads the query and the document together. That is the reranker's job, covered in the Reranking with cross-encoders lesson. The stack teams converge on after their first retrieval incident is hybrid retrieval of fifty to a hundred candidates, fusion, and a cross-encoder over the fused list.