Vector stores for retrieval

Why it matters

Once the embeddings exist, something has to hold them and answer "which are nearest to this" fast, under a filter, for one tenant, with the text the prompt will need. Engineers argue about which vector database to buy as if the index were the whole decision. The index internals, HNSW and IVF, are covered in the System Design lesson Vector databases and matter less in practice than the plumbing around them: filters, tenancy, updates, hybrid queries and what comes back. Interviewers ask to see whether you treat the store as a component with requirements or as a product name.

The idea

Inside a retrieval system, the vector store is the component that takes a query vector plus constraints and returns candidate ids with scores and payloads. Chunking, embedding, fusion, reranking and prompt assembly live outside it. That framing turns "which database" into "which of these needs does it meet".

What retrieval needs from the store

Metadata filtering. Real queries carry constraints: this language, documents the user may read, published after a date, not archived. They must apply inside the search.

Namespaces or tenants. A multi-customer product must never return one tenant's chunks to another. A namespace per tenant is the boundary; a hard partition is safer than a filter you hope every caller remembers.

Upserts and deletes. Documents change. The store must replace a chunk's vector by id and drop deleted documents promptly without a full rebuild; graph indexes handle deletes least gracefully.

Hybrid support. Most production retrieval fuses lexical and vector results, as the Hybrid search lesson explains. Running BM25 and vectors in one query saves a second system.

Payload storage. The prompt needs the chunk text, its source and its position. Either the store carries that payload beside the vector or you fetch it from the document store by id, which adds a hop.

Consistency after writes. When a document is indexed, is it searchable at once, after a refresh interval, or eventually? An agent that writes a note and reads it back in the next step notices.

Filtering before, during and after

With post-filtering, the index finds the k nearest overall and then drops the ones that fail the filter. If the filter keeps one chunk in a hundred, a top ten returns, on average, nothing; over-fetching ten times k helps and then fails on a narrower filter. With pre-filtering, the candidate set is narrowed first and the search runs inside it; a graph index starved of candidates may fall back to a scan, slow but correct. Good stores integrate the filter into the traversal, checking metadata as they walk, so recall holds and latency stays bounded.

The options

Postgres with pgvector. Vectors as a column beside the row that owns them, with SQL filters, joins and transactions. It carries a few million to low tens of millions of vectors per table well.

A dedicated vector store. Built around the index: filtered search inside the traversal, quantisation, sharding to hundreds of millions of vectors, namespaces as a first-class concept, payload storage. It is a new service with its own failure modes and bill.

A search engine with vector support. Lucene-based engines hold BM25, structured filters and vectors in one index, so hybrid is a single query. They are heavy to run, but a team that already runs one gains hybrid for free.

An in-process index. For a corpus under a million chunks, a library index in the application's memory answers in microseconds, rebuilds from the document store on deploy and needs no operations. Filtering is a scan you write yourself.

Deciding

Decide by corpus size and operational shape, in that order. Under a million chunks, in-process or pgvector. Up to low tens of millions with relational filters and one database team, pgvector. Beyond that, or for sub-10-millisecond filtered queries at high throughput, a dedicated store or the search engine you already run. Thousands of small tenants push toward namespaces in a dedicated store.

	pgvector	Dedicated store	Search engine with vectors	In-process index
Scale	Up to low tens of millions per table	Hundreds of millions to billions, sharded	Tens to hundreds of millions, sharded	Under a million; must fit in RAM
Filtering	Full SQL, joins, row policies	Payload filters inside the index walk	Rich filters plus BM25 in one query	A scan you write yourself
Ops	A system you already run	A new service: backups, upgrades, capacity	Heavy but mature; a cluster to run	None; rebuild on deploy
Cost	The Postgres you already pay for	Per node or per vector, plus a second payload copy	Cluster cost, often already present	Memory only