Load balancers

Why it matters

One server has a ceiling and a failure mode. The moment you have two, something must decide which one gets each request, notice when one dies, and present a single stable address to the world. That something is a load balancer, and it is the component that turns "a server" into "a service". Almost every other scaling technique in this track assumes it is there.

The idea

A load balancer receives connections or requests on one address and distributes them across a pool of backend servers. Besides distribution it does three jobs that matter as much:

1. 1

   Health checking

   Probe each backend on a schedule; stop sending traffic to one that fails; bring it back when it recovers. Without this, balancing spreads failures as evenly as it spreads load.
2. 2

   Hiding the pool

   Clients see one address; servers can be added, removed, replaced and deployed behind it without clients noticing.
3. 3

   Absorbing the edge

   Terminate TLS, buffer slow clients, limit connection counts, so backends deal only with clean, fast, internal traffic.

Layer 4 and layer 7

A layer 4 balancer works on TCP or UDP: it forwards connections based on IP and port, never looks inside, and so handles millions of connections at line rate and any protocol. A layer 7 balancer terminates HTTP: it reads the request, so it can route by host and path, rewrite headers, cache, compress, and apply per-route policies, at the cost of CPU and a little latency. Cloud providers sell both (network load balancers and application load balancers); many architectures chain them, L4 in front for raw distribution, L7 behind for routing.

Where the balancer runs

Stickiness

Some applications keep per-user state on the server (an in-memory session, an open upload) and need the same client to hit the same server. Sticky sessions pin a client to a backend via a cookie or a hash of its address. It works, and it quietly undoes the point of a stateless tier: uneven load, lost sessions when a server dies, painful deploys. Prefer moving the state to a shared store and keeping the pool interchangeable; use stickiness as a transition, not a design.

Trade-offs

The balancer is in the path of every request, so it must be redundant and it adds a hop. Layer 7 features cost CPU and require the balancer to hold certificates and understand your protocol. Managed balancers remove operations work and limit customisation; software balancers reverse that. Stickiness trades scalability for the convenience of server-side state.

In an interview

Draw it as the first box after the clients and state its type and its redundancy: "A managed layer 7 load balancer, multi-zone, terminating TLS, health-checking the API pool every five seconds and routing by path." If the interviewer asks about sessions, say where they live and why the pool stays stateless. If scale is extreme, mention a layer 4 tier in front.