Choosing an embedding model

Why it matters

Every retrieval system starts with one decision that is cheap to make and expensive to undo: which model turns text into vectors. It sets the quality ceiling, because no index, fusion trick or reranker recovers meaning the embedding never captured, and it sets a lock-in: change the model and every vector in the corpus must be regenerated. Interviewers ask because candidates tend to name a model and stop, when the real answer is a list of constraints and an evaluation plan.

The idea

An embedding model maps text to a fixed-length vector so that texts with similar meaning land close together. Queries and documents go through the same model into the same space, and retrieval is nearest-neighbour search in that space.

What differs between models

Dimensions. Output vectors run from a few hundred to a few thousand floats; each costs storage, index memory and compare time on every query.

Maximum input length. A model reads at most a fixed number of tokens per input; anything past that is cut off silently. That limit decides your chunk size, not the other way round.

Languages and domain. A model trained mostly on English prose handles other scripts worse, and a general model handles legal clauses, chemical names or source code worse than one trained on them.

Cost. Hosted models bill per million input tokens; open-weight models bill in GPU hours and engineering time. Both bills return on every re-embed.

Licence and hosting. A hosted API means no infrastructure and a dependency on a vendor's pricing and deprecation schedule. Open weights mean you run it and own the latency; some open licences restrict commercial use.

Reading a leaderboard

Public retrieval benchmarks rank embedding models across many datasets, and leaderboards in the MTEB style aggregate them into one score per model. Read one as a shortlist generator, not a verdict: look at the retrieval column rather than the overall average, at the datasets closest to your domain and language, and at the dimension count and input limit beside the score. A model that leads the average can trail on your data, because your documents, queries and notion of relevant are not in the benchmark. Build an eval set of a few hundred real queries with judged relevant chunks, run the shortlist against it, and let recall at ten decide.

Asymmetric embeddings

Many models are trained with a separate prefix for each side, so that "how do I reset my password" lands near a paragraph about resetting passwords rather than near another question. Use the query prefix for queries and the document prefix for documents; swapping or omitting them degrades recall quietly.

Truncatable dimensions

Models trained with a Matryoshka objective put the most informative values first, so the first 256 of 1,024 dimensions are a usable embedding on their own. Keep the prefix that fits the memory budget and accept a small recall loss, or store the prefix for a cheap first pass and the full vector for rescoring. Without that training, truncation destroys the vector.

Lock-in and fine-tuning

Vectors from two models, or two versions of one model, live in different spaces and cannot be compared. Changing the model means re-embedding the whole corpus, rebuilding the index and cutting over, with two spaces running side by side until it finishes. Store the raw text and the model version next to each vector so the re-embed is a batch job, not a rescue.

If a general model falls short on your eval set, fine-tuning an open-weight model on your own query-document pairs, with hard negatives, usually buys more than a larger general model; a few thousand pairs from search logs are enough to start. The fine-tuned model is then yours to host, version and re-train.

When a small model is enough

Short, well-written chunks in one language, a corpus under a few million chunks, queries that resemble the documents: a small open-weight model embeds all of that well, runs on a CPU and keeps the index tiny. Reach for a large model when the eval set says the small one fails, not before.

	Hosted API	Open-weight large	Open-weight small
Quality	Strong and general; vendor decides what changes	Best you can own; fine-tunable on your pairs	Good on clean single-language text; caps on hard queries
Cost	Per million tokens, recurring	GPU hours plus someone to run them	CPU-class; near zero per query
Latency	Network round trip, tens of milliseconds	GPU-bound, milliseconds per batch	Low; runs beside the application
Control	None over deprecation or price	Full: version, host, tune	Full, and cheap to replace
Re-embed cost	Paid again at vendor price on every change	Your GPU time, on your schedule	Small: hours, not days

Trade-offs

A hosted model trades control for convenience: a price change or a deprecation notice forces a re-embed on the vendor's schedule. A large open-weight model gives the best quality you can own, for GPUs, serving latency and someone to run them. A small model is cheap and fast and may quietly cap recall on hard queries. Fine-tuning raises quality on your domain and lowers it elsewhere. Every one of those choices is reversible only by re-embedding, so make the first with an eval set in hand.

In an interview

Lead with constraints, then the test: "I shortlist by language, domain, input length and licence, then pick by recall at ten on a few hundred judged queries from our own logs, because leaderboard averages do not transfer." At junior depth, know that queries and documents share a model and a space, and that switching models means re-embedding. At mid depth, explain asymmetric prefixes, truncatable dimensions and how to build the eval set. At senior depth, talk about the migration behind a model change and about fine-tuning on logged pairs. Expect the follow-up: "How would you switch models without downtime?" Embed into a second index while the first serves, compare on the eval set, cut queries over, delete the old space.