Full-text search engines

Why it matters

Users type "red runing shoes" and expect red running shoes, ranked by how well they match, with counts per brand on the side, in under 100 milliseconds across ten million products. No relational query does that. Search engines are a separate kind of store built around one data structure, and nearly every product with a search box eventually adds one. The design question is how it relates to the database that owns the data.

The idea

A search engine indexes documents so that queries by words, phrases and fuzzy matches return ranked results fast. The heart is the inverted index: instead of document to words, it maps each word (term) to the list of documents containing it, with positions and frequencies.

Analysis: from text to terms

Before indexing, text passes through an analyzer: tokenisation splits it into words, lowercasing normalises case, stemming reduces "running" to "run", stop words like "the" may be dropped, and synonyms can be expanded. Queries go through the same analyzer, which is why "Running Shoes" finds "running shoe". Choosing analyzers per field and per language is most of the quality work in search.

Relevance: BM25 and beyond

Matching is not enough; ordering is the product. BM25 scores a document by how often the query terms appear in it (term frequency, with diminishing returns), how rare each term is across the corpus (inverse document frequency, so "shoes" matters less than "Pegasus"), and document length normalisation. On top of that, systems add field boosts (title matches outrank description matches), recency and popularity signals, and increasingly a re-ranking stage using machine-learned or vector models over the top results. Tuning relevance is iterative and needs real query logs and click data.

Beyond matching

Facets and aggregations count results per brand, size or price range for the filter sidebar, computed over the matched set in the same request. Autocomplete uses prefix-indexed fields or edge n-grams. Fuzzy matching tolerates typos through edit distance. Highlighting returns the matched snippets. Each is an index-time or query-time feature with a cost in index size or latency.

A derived view, not the source of truth

Search indexes are optimised for reads and rebuilt often; they are not where data should live. The standard architecture keeps the relational or document database as the system of record and feeds the search engine asynchronously: change data capture or an outbox publishes updates, an indexer transforms rows into search documents (denormalised, with the fields the search needs), and bulk-indexes them. Freshness lags by seconds. Mappings change frequently as relevance is tuned, so reindexing into a new index and switching an alias is a routine, zero-downtime operation, which is only possible because the data can be regenerated from the source.

Scaling and operating

Indexes are split into shards distributed across nodes, each shard an independent inverted index; a query fans out to every shard and merges results, so shard count affects latency and more shards is not free. Replicas serve reads and provide failover. Indexing is CPU-heavy; separating indexing from query nodes keeps search latency steady during bulk updates. Monitor query latency percentiles, indexing lag and refresh intervals.