Freshness, incremental indexing and re-embedding

Why it matters

Every index is a snapshot, and the world keeps moving after it. A policy is amended, a price changes, a page is deleted, and the assistant keeps answering from the old chunk with a confident citation to a document that no longer says that. Users notice stale answers faster than wrong ones, because they can see the current page themselves. Interviewers ask because freshness is where a demo that indexed a folder once turns into a pipeline with lag, cost and failure modes, and "how do you keep it up to date" is the follow-up to every RAG design.

The idea

Detecting change

Three sources, in rising order of quality. Content hashes: store a hash of each document, re-crawl, compare; cheap, works with any source, finds changes only as often as you crawl. Modified timestamps: ask the source for everything changed since the last run; cheaper than hashing, trusts the source's clock. Change feeds: the source emits an event per create, update and delete; lowest lag, needs a source that offers one, and deletes arrive as events rather than absences you must infer. Most real pipelines use a feed where one exists and a periodic hash sweep to catch what the feed missed.

Updating only what changed

A changed document does not mean every chunk changed. Re-chunk the new version, hash each chunk, and compare against the stored hashes: unchanged chunks keep their embeddings, changed and new chunks are embedded, chunks that no longer exist are deleted. Deletion is the step teams forget. An upsert that writes new chunks without removing the old ones leaves both versions in the index, and the stale one can outrank the fresh one. Give each chunk a document id and a version, so "delete every chunk of document D below version 12" is one operation.

Versioning and the promised lag

Store the document version with every chunk and the pipeline's high-water mark with the index: "all changes up to event 48,201 applied". The lag between a change at the source and its appearance in answers is then measurable, and it should be promised. A product that says "answers reflect the knowledge base within fifteen minutes" has a service level to monitor; one that says nothing has users filing bugs. Different corpora deserve different promises: pricing in minutes, policy in an hour, archives in a day.

Embedding-model upgrades

Embeddings from two models are not comparable, so a new embedding model means every chunk is re-embedded: a full re-index whatever the pipeline does day to day. Do it as a dual-write cutover. Build a second index with the new model while the old one keeps serving, and run the incremental pipeline against both so the new index is not stale on the day it is finished. Compare retrieval quality on an evaluation set, flip traffic, keep the old index for a rollback window, then drop it. The cost is one embedding call per chunk plus double writes for the overlap; for ten million chunks that is a budget item, not a Friday job.

Consistency with the metadata store

Most systems keep chunk text and metadata in a database and only the vectors in the index. The two drift: a chunk deleted from one and not the other is a vector with no text, or text no one can find. Write the metadata store first and the index second, make every step idempotent so a retry is safe, and run a reconciliation job that compares the two by id and repairs the difference.

Invalidating cached answers

If answers or retrieved contexts are cached, a fresh index is not enough; the cache serves the old answer. Record the document ids each cached answer depended on and invalidate those entries when a document changes. Where the dependency is unknown, bound the damage with a time-to-live no longer than the promised lag.

Monitoring index age

Three numbers on a dashboard: the age of the newest applied change, the backlog of unprocessed events, and chunks per document version. The first measures lag, the second predicts it, the third catches forgotten deletes.

	Full rebuild	Incremental update	Dual-index cutover
Lag	As long as the rebuild, repeated	Minutes from the feed	None at flip; the new index is kept current during the build
Cost	Every chunk re-embedded, every time	Only changed chunks	Every chunk once, plus double writes for the overlap
Risk	Stale during the build; simple otherwise	Forgotten deletes, drift between stores	Two indexes to keep consistent; needs a rollback plan
Use when	Small corpus, rare changes	Day-to-day operation of any corpus	Embedding model or chunking strategy changes