Distributed cache architecture

Why it matters

A single cache node tops out at its memory and its network card. Past that you have a cluster, and a cluster raises every question a single node hid: which node holds this key, what happens when a node dies, how a key read a million times a second avoids melting one machine. Interviewers ask "how does your cache scale" expecting these answers, not "add more Redis".

The idea

A distributed cache spreads keys across many nodes so that memory and throughput scale with the node count. Three decisions define the design: how keys map to nodes, how clients find the right node, and what happens when the node set changes.

Mapping keys to nodes

Naive modulo (hash(key) mod N) breaks the moment N changes: nearly every key moves, the cache empties, and the database takes the full load. Consistent hashing places nodes and keys on a ring; a key belongs to the next node clockwise. Adding or removing a node moves only the keys in its arc, roughly 1/N of them. Virtual nodes (each physical node appears many times on the ring) even out the distribution. Redis Cluster uses a fixed 16,384 hash slots assigned to nodes, which is the same idea with explicit slot ownership that can be migrated live.

Who does the routing

	Client-side	Proxy	Server-side (cluster-aware)
How	The client library hashes the key and connects to the right node	Clients talk to a proxy (Twemproxy, Envoy, mcrouter) that routes	Any node accepts the request and redirects or forwards to the owner
Hop count	One	Two	One or two
Topology knowledge	Every client must learn and track it	Centralised in the proxy	Clients learn from redirects and cache the slot map
Operational cost	Library per language; coordinated config changes	One more tier to run and scale	Built into the store
Examples	Memcached with ketama hashing	mcrouter at Meta, Twemproxy	Redis Cluster MOVED/ASK redirects

Node failure and replication

A cache is supposed to be disposable, but losing a node still means a burst of misses against the database for that node's arc. Two mitigations: replication, where each shard has one or more replicas that take over on failure (Redis Cluster replica failover), and graceful degradation, where the application's miss path is rate-limited or uses request coalescing so a cold arc does not become a stampede. Rebalancing after a change should move slots gradually, not all at once.

Hot keys

One viral item can send a million reads per second at a single key, which lives on a single node no matter how many you have. Remedies: a small local in-process cache with a short TTL in front of the distributed cache for the hottest keys; key splitting, where item:42 becomes item:42:0 through item:42:9 and readers pick one at random, spreading load across ten nodes; and read replicas for the hot shard. Detect hot keys with sampling in the client or the proxy rather than guessing.

Large values and the network

A cache node's network interface is often the real ceiling before memory. Values over a few hundred kilobytes should be compressed or split, and fan-out patterns such as fetching a hundred keys per request should use multi-get so one round trip retrieves many keys. Measure bytes per second per node, not just hit rate.

Trade-offs

Sharding scales memory and throughput and introduces routing, topology management and partial failures. Consistent hashing bounds the damage of topology changes but still causes a miss burst for the moved arc. Replication costs memory for availability. Client-side routing is the fastest and the hardest to operate; a proxy centralises control at the price of a hop; cluster-aware servers are the usual modern answer.

In an interview

Say it in one pass: "The cache is a Redis Cluster of N shards with one replica each, keys distributed by hash slot so adding a node moves roughly 1/N of keys. Clients are cluster-aware and cache the slot map. For hot keys we add a short-TTL local cache and split the hottest keys into ten sub-keys. We size shards by network throughput as well as memory, and the miss path uses request coalescing so a failed shard cannot stampede the database."