Health checks and failover

Why it matters

Redundancy without detection is decoration. Two servers behind a balancer that cannot tell one has died will send half of all traffic into a hole. Health checks are the detection, failover is the response, and the details (what the check tests, how often, how many failures it takes) decide whether an incident lasts five seconds or until a human notices. The same mechanics, scaled up, are how a database replica becomes primary and how traffic leaves a failed region.

The idea

A health check is a probe the balancer (or orchestrator) sends to each backend. Pass and the backend receives traffic; fail enough times in a row and it is drained: removed from rotation until it passes again. Two distinctions do most of the work.

Liveness versus readiness. Liveness asks "is this process alive at all?" and a failure means restart it. Readiness asks "can this instance serve traffic right now?" and a failure means stop routing to it but leave it running: it may be warming a cache, waiting on a dependency or shutting down gracefully. Conflating them causes restart loops during a dependency outage, which turns a slow service into a dead one.

Shallow versus deep. A shallow check returns 200 if the process responds. A deep check also verifies dependencies: the database connection, the cache, the downstream service. Deep checks catch more, and they create a trap: if every instance's deep check fails because one shared dependency is down, the balancer drains the entire pool and a partial outage becomes a total one. The rule is that a readiness check should test what this instance needs to serve degraded-but-useful responses, not everything it ever calls.

Tuning the numbers

Interval, timeout, unhealthy threshold and healthy threshold together set detection time and stability. A 5-second interval with a 3-failure threshold detects a dead instance in about 15 seconds; a 2-second interval with 2 failures detects in 4 but is more sensitive to a single slow response. Make the healthy threshold at least as strict as the unhealthy one so an instance does not flap in and out of rotation under marginal conditions.

Deploys and graceful shutdown

Most "random 502s during deploy" come from killing an instance while requests are in flight. The sequence that avoids it: the instance fails its readiness check first (or the orchestrator removes it from the pool), the balancer stops sending new requests, in-flight requests finish (connection draining, typically 10 to 30 seconds), then the process exits. Readiness failing before shutdown is what makes zero-downtime deploys work.

The same pattern at larger scale

Level	What is checked	What fails over	Typical time
Instance	HTTP readiness endpoint	Balancer drains the instance	Seconds
Dependency	Client-side timeouts and error rates	Circuit breaker opens; callers degrade	Seconds
Database	Replication heartbeat, primary reachability	A replica is promoted; writers reconnect	Tens of seconds; risk of split brain
Region	Synthetic probes from outside	DNS or anycast routes traffic elsewhere	Minutes, bounded by TTL

Trade-offs

Fast detection means low thresholds, which means more false positives and flapping; slow detection means longer partial outages. Deep checks find more problems and can turn one dependency failure into a full drain. Automatic database failover removes humans from the critical path and introduces split-brain risk if the old primary was only partitioned, not dead. Every one of these is a dial, and the right setting depends on how expensive a false failover is compared with a late one.

In an interview

Make the check concrete: "Readiness at /healthz every five seconds, three failures to drain, two passes to restore; it checks the database connection pool but not downstream services, so a payments outage degrades checkout rather than draining the whole API." Then mention deploys: "Instances fail readiness and drain for twenty seconds before exit." And for the database: "Automatic promotion of a replica, with fencing so the old primary cannot accept writes." That covers detection, response and the classic trap.