Embeddings as meaning

Why it matters

A language model's working currency is not the token id but a vector: a list of a few hundred to a few thousand numbers that stands for a piece of text. Every retrieval system, semantic cache, deduplicator, recommender and "find similar tickets" feature you will be asked to design rests on one property of those vectors: two inputs that mean roughly the same thing land close together. Engineers who treat embeddings as a black box pick the wrong model, compare vectors from two different models, or ship search that matches wording instead of meaning. Interviewers ask "what is an embedding?" to see whether you can explain the geometry before you reach for a vector database.

The idea

An embedding is a fixed-length vector that represents an input so that distance tracks similarity. Fixed length is the point: a three-word query and a three-page document both become, say, 1,024 numbers, which is what makes them comparable at all. No single number means anything on its own; only the direction of the whole vector does.

Where the vectors come from

Inside a transformer every token carries a vector that each layer reads and updates; the Attention and the transformer lesson calls this the residual stream. A text embedding is made by running the text through such a model and collapsing the per-token vectors into one, usually by averaging them (mean pooling) or by taking the vector at a special final token. A generative model can do this, but its stream was shaped for prediction, not comparison, so production systems use a dedicated embedding model: a smaller encoder trained with a contrastive objective that pulls matched pairs (a question and its answer, a title and its body, a sentence and its paraphrase) together and pushes random pairs apart. The same recipe gives code and image embeddings, user embeddings built from clicks and purchases, and multimodal spaces where a caption can find a photo.

Dimensions and distance

Typical output sizes run from 384 to 3,072 dimensions. More dimensions carry finer distinctions and cost proportionally more memory and comparison time. Some models let you truncate to the first N dimensions and trade accuracy for storage after the fact. Similarity is almost always cosine similarity: the cosine of the angle between two vectors, 1 for the same direction, near 0 for unrelated, computed as the dot product of the two unit-normalised vectors. Most embedding models emit unit vectors already, so cosine and dot product coincide and the cheaper dot product is what the index computes.

Similar depends on the task

The embedding model decides what counts as similar, and it decided during training. A model trained on question-answer pairs puts "how do I reset my password" next to the article that answers it; a model trained on paraphrases puts it next to other ways of asking and may rank the answer below them. The same space can place "the server is up" beside "the server is down", because they share a topic and nearly every token. That is why retrieval quality is measured on your own queries, why re-ranking with a cross-encoder that reads query and document together often beats raw similarity, and why vectors from two different models, or two versions of one model, must never be compared: the axes mean different things.

The bridge to retrieval

The index that finds nearest neighbours among millions of vectors without scanning them all is the System Design track's Vector databases lesson. This lesson's claim is narrower: an embedding turns "which documents are about this?" into "which points are near this one?", a question a database can answer in milliseconds.

	Sparse (BM25)	Dense (embeddings)	Hybrid
Catches	Exact terms, ids, codes, rare names	Paraphrase, synonyms, intent	Both, merged by rank fusion
Misses	Synonyms and rewordings	Exact strings and unseen jargon	Little, at the cost of running both
Index	Inverted index, small, cheap to update	Vector index, memory-heavy, rebuilt when the model changes	Two indexes kept in step
Out-of-domain text	Degrades gracefully	Degrades silently	The sparse side holds it up
Natural home	Logs, code, product catalogues	FAQ, support, semantic cache	General retrieval

Trade-offs

Dense retrieval misses what its training never showed it: exact part numbers, error codes, rare names and fresh jargon, where the lexical match is the whole signal and BM25 wins. Sparse retrieval misses paraphrase. Hybrid search runs both and merges the two ranked lists, which is the default in 2026 and doubles the index you maintain. Embeddings also age: change the model and every stored vector must be recomputed, so pin the model version and plan the re-embedding job before you need it. And a vector is not anonymous: inversion attacks recover much of a sentence from its embedding, so stored vectors need the same access control as the documents they came from.

In an interview

Open with the geometry: "An embedding is a fixed-length vector where distance tracks similarity, so search becomes a nearest-neighbour query." At junior depth, define it, name cosine similarity, and know that query and documents must be embedded by the same model. At mid depth, explain pooling, contrastive training, the dimension trade-off, and why hybrid beats either side alone. At senior depth, talk about task-specific similarity, re-ranking, model versioning and re-embedding cost, and the privacy of stored vectors. Expect the follow-ups: "Why not embed with the chat model?" (a contrastive encoder is cheaper and better at comparison). "What happens when you upgrade the embedding model?" (re-embed everything; the two spaces are incompatible).