Load balancing algorithms

Why it matters

"Spread the load evenly" sounds like one problem. It is several, because requests are not equal (a search costs more than a health check), servers are not equal (last year's instances are slower), and some workloads need the same client to reach the same server. Each algorithm encodes an assumption about those facts. Choose one whose assumption is wrong for your traffic and you get hot servers, slow tails and a balancer that looks fine on its own dashboard.

The idea

A balancing algorithm picks a backend for each new connection or request. The common ones, in rough order of sophistication:

Round robin. Rotate through the pool. Assumes requests are similar in cost and servers similar in capacity. Simple, predictable, and the default almost everywhere. Weighted round robin gives bigger servers proportionally more turns, fixing the unequal-servers case but not unequal requests.

Least connections. Send the request to the backend with the fewest active connections. Assumes connection count tracks load, which is true when requests vary in duration: a slow backend accumulates connections and automatically gets less new work. Weighted least connections combines it with capacity weights.

Least response time. Pick the backend with the lowest recent latency (often combined with fewest connections). Reacts to backends that are slow for any reason, not just busy ones. Needs the balancer to measure, and can oscillate if measurements are noisy.

Hashing. Hash something in the request (client IP, a session id, a URL) and map it to a backend. Gives affinity: the same key always reaches the same server, which helps server-side caches hit. Plain modulo hashing remaps most keys when the pool changes size; consistent hashing remaps only about 1/N of them, which is why caches and sharded stores use it (it gets its own lesson in Database scaling).

Random and power of two choices. Pick two backends at random, send to the less loaded. It sounds like a trick and it is one of the best results in the field: nearly as even as least-connections with far less coordination, so it scales across many independent balancer instances.

Algorithm	Assumes	Breaks when	Use for
Round robin	Equal requests, equal servers	Request cost varies a lot	Stateless pools of identical servers; the default
Weighted round robin	Known capacity ratios	Request cost varies	Mixed instance sizes
Least connections	Connection count tracks load	Many cheap keep-alive connections	Variable-duration requests, slow backends
Least response time	Latency reflects health	Noisy measurements, few requests	Latency-sensitive pools
IP / key hash	Affinity is worth uneven load	Many clients behind one NAT (one key, huge load)	Server-side caches, stateful protocols
Consistent hash	Pool membership changes	Hot keys	Cache tiers, sharded stores
Power of two choices	Many balancers, little coordination	Rarely; it degrades gracefully	Large fleets, service meshes

The hot-spot problem

Hashing has a failure mode worth naming: a single key (one tenant, one viral item, a corporate office behind one NAT address) can send a disproportionate share of traffic to one backend. Mitigations are to hash on a finer key, to add a bounded-load variant of consistent hashing that spills over when a backend is full, or to accept that affinity-based balancing is for caches, not for the stateless tier.

Trade-offs

Smarter algorithms need state (connection counts, latency samples) and make the balancer more complex and harder to run as many independent instances. Affinity improves cache hit rates and worsens evenness. Round robin is dumb and almost always good enough for a stateless tier; the time to change it is when measurements show uneven load, not before.

In an interview

Say the default and the reason you would deviate: "Round robin across the stateless API pool; if request cost varies a lot I would move to least connections. The cache tier uses consistent hashing so the same key hits the same node and the hit rate stays high when nodes join or leave." Mentioning power of two choices when the fleet is large is a small, credible flourish.