Caching

Why it matters

Every large system has the same shape of problem: the data lives somewhere durable and slow, and the people asking for it want it fast and often. A database can answer a few thousand queries a second from disk; a web page may need fifty lookups to render. Multiply by traffic and the storage tier becomes the bottleneck long before the application tier does.

Caching is the standard answer. Keep a copy of the hot data in memory, close to the request, and let the storage tier see only the requests the cache cannot answer. Done well, it cuts latencyThe time between sending a request and receiving its response, usually measured at a percentile such as p50 or p99. from tens of milliseconds to under one, and it shrinks the database bill with it.

The idea

A cache sits in front of a slower store. A request checks the cache first. If the data is there, that is a hit and the response comes straight back. If not, that is a miss: the request goes to the origin, the answer is written into the cache, and the next identical request hits.

The number that describes a cache is its hit ratio: hits divided by total requests. A 90 percent hit ratio means the database sees one request in ten. A 99 percent ratio means one in a hundred. Small changes in hit ratio are large changes in origin load, which is why the second number matters more than the first.

Caches work because real traffic is skewed. A small fraction of keys gets most of the reads (popular products, today's posts, the logged-in user's profile). That skew is called locality, and without it a cache is just a slower database with more moving parts.

Where caches live

Caching is not one component. It is a layer that appears at every hop:

• Browser: HTTP cache headers keep images and scripts on the user's device.
• CDN: edge servers cache responses near the user. Covered in its own lesson.
• Application memory: an in-process map or LRU inside each server. Fastest, but every server has its own copy.
• Distributed cache: a separate fleet such as Redis or Memcached shared by all application servers.
• Database buffer pool: the database itself caches pages in RAM. Often the first cache you already have.

	In-process cache	Distributed cache
Latency	Nanoseconds to microseconds, no network	Sub-millisecond, one network hop
Consistency across servers	Each server has its own copy; they drift	One copy everyone reads
Capacity	Bounded by a single process's RAM	Scales horizontally by adding nodes
Failure	Lost on restart or deploy	Survives app restarts; needs its own operations
Good for	Config, feature flags, small hot sets	Sessions, rendered fragments, query results

Trade-offs

A cache trades freshness for speed. The moment data is copied, the copy can go stale. Everything in the rest of this module (strategies, eviction, invalidation) is about managing that one fact.

It also adds a failure mode. If the cache fleet dies, every request becomes a miss at once and the database receives the full load it was never sized for. A cache that lets you run a database at 10 percent of real traffic is also a cache whose loss is a 10x surge.

Finally, caches cost memory and operational attention. Caching something that is read once is pure waste. Caching something that changes every second is worse than waste: it serves wrong answers.

In an interview

Introduce a cache when the read path is hot and the data tolerates a little staleness. Say what you will cache (rendered responses, query results, objects), where it lives, how long it lives, and what the hit ratio target is. "A Redis layer in front of the product database, caching product pages for five minutes, aiming for a 95 percent hit ratio" is a complete sentence; "add caching" is not.

Expect follow-ups on invalidation and on what happens when the cache is cold. If you can answer both, you understand caching.