Module 06 · Caching · Lesson 7 of 8
Cache stampede
When a hot key expires, every concurrent request misses at once and hits the origin together. Single-flight locks, early expiry, stale-while-revalidate and TTL jitter keep the herd from trampling the database.
The video walkthrough is part of Pro.
See what Pro includesWhy it matters
A cache turns a 10,000-request-per-second endpoint into a 100-request-per-second database load. The moment the cached key expires, the arithmetic reverses: for the few hundred milliseconds it takes one request to rebuild the value, every request misses and goes to the origin. That is a cache stampede (also called a thundering herd or dog-piling). The database that was sized for 100 requests per second gets 10,000, slows down, which makes the rebuild take longer, which lets more requests through. Outages start this way.
The idea
The problem is concurrency on a miss. One rebuild is fine; ten thousand identical rebuilds are the failure. Every mitigation does one of three things: make sure only one request rebuilds, make sure the key never fully expires while it is hot, or spread expiries out so they do not line up.
Only one rebuilds
Locking / single-flight. On a miss, try to acquire a short lock on the key (for example SET lock:key NX PX 2000 in Redis). The winner rebuilds and writes the value; the others wait briefly and re-read the cache, or serve a stale copy if one exists. Go's singleflight package and most mature cache libraries implement this in-process; a distributed lock extends it across servers. Give the lock a TTL so a crashed rebuilder cannot wedge the key.
Request coalescing at the edge. CDNs and reverse proxies do the same thing for HTTP: collapse concurrent identical requests into one origin fetch. Nginx's proxy_cache_lock and Varnish's request coalescing are this feature.
Never fully expire a hot key
Stale-while-revalidate. Keep serving the expired value for a grace period while one background request refreshes it. Users never wait on a miss, and the origin sees exactly one refresh. HTTP has a header for it; application caches implement it by storing a soft TTL alongside the hard TTL.
Probabilistic early expiration. Each request recomputes the value early with a probability that rises as the key approaches expiry, scaled by how long the recompute takes. Formally, refresh when now - delta * beta * ln(rand()) >= expiry, where delta is the recompute time (this is the XFetch algorithm). One request refreshes slightly early; the key never actually hits zero while traffic flows.
Refresh-ahead / warming. For a known hot set, a background job refreshes keys before expiry on a schedule. Simple and effective when you know the keys; the next lesson covers warming.
Spread the expiries
TTL jitter. If a deploy fills ten thousand keys at the same second with the same TTL, they expire together an hour later. Add a random few percent to every TTL so expiries smear out over time. Cheap, universal, and it prevents the synchronised version of the problem even when each individual miss is harmless.
| Single-flight lock | Stale-while-revalidate | Early expiration | TTL jitter | |
|---|---|---|---|---|
| Protects against | Concurrent misses on one key | Any miss on a key that has a stale copy | Expiry of hot keys | Synchronised expiry across keys |
| User-visible latency on refresh | Waiters pause for the rebuild | None; stale served | None; refreshed early | Unchanged |
| Serves stale data | No (unless combined) | Yes, for the grace period | No | No |
| Cost | A lock per miss | Extra memory for the stale copy | A random draw per read | Nothing |
| Use when | Expensive rebuilds, strict freshness | Latency matters more than seconds of staleness | Very hot keys, steady traffic | Always |