Heartbeats and failure detection

Why it matters

Since a remote node's death cannot be observed directly, every distributed system invents a way to guess it, and every failover, rebalance and leader election is triggered by that guess. Guess too eagerly and healthy nodes are evicted during a hiccup, triggering needless rebalancing and sometimes cascading failures; guess too slowly and a dead leader holds the system hostage. Failure detection is the knob that decides how a system behaves under stress, and most incident post-mortems involving "flapping" are about it.

The idea

A heartbeat is a periodic message a node sends to prove it is alive; a failure detector declares a node dead when heartbeats stop arriving for longer than a timeout. Variants push (the node sends to a monitor), pull (the monitor probes the node), or both. Because the network is unreliable, no detector is perfect: it can be complete (eventually detects every real failure) and more or less accurate (rarely declares a live node dead), and tuning trades one for the other.

Tuning the timeout

	Short timeout (a few seconds)	Long timeout (tens of seconds)
Detection time	Fast failover; less unavailability after a real crash	Slow; a dead leader blocks longer
False positives	Garbage-collection pauses, load spikes and brief network blips cause evictions and leader churn	Rare; tolerates transient trouble
Consequence of a false positive	Unnecessary rebalancing, failover storms, split-brain risk if fencing is weak	Same, but seldom
Fits	Stateless services behind a balancer where eviction is cheap	Stateful leaders, databases, anything where failover is expensive

Rules of thumb: set the timeout to several heartbeat intervals so one lost packet does not trigger it; make the cost of a false positive proportional to the caution of the timeout (cheap to evict a web server, expensive to fail over a database); and never trigger irreversible actions from a single detector reading.

Adaptive detection: phi accrual

Fixed timeouts ignore that network conditions vary. The phi accrual failure detector (used by Cassandra and Akka) records the distribution of recent heartbeat inter-arrival times and outputs a continuously growing suspicion level phi instead of a boolean. Phi expresses how unlikely it is that a live node would be this late given recent history: on a quiet network, a two-second gap is highly suspicious; on a jittery one it is normal. Callers choose their own threshold, so a cheap action (stop routing reads) fires at low phi and an expensive one (fail over) waits for high phi. The detector adapts automatically as conditions change.

Membership at scale: gossip and SWIM

Having every node heartbeat every other node is quadratic traffic. Gossip-based membership (the Gossip lesson later) spreads liveness information through random peer exchanges, and SWIM-style protocols add indirect probing: if A cannot reach B, A asks C and D to probe B before declaring it dead, which distinguishes "B is down" from "the A to B link is down". Suspicion propagates as a state (alive, suspect, dead) with incarnation numbers so a wrongly suspected node can refute it.

After the verdict

Detection is only useful if the reaction is safe. Evict from load balancing, stop sending it work, trigger a leader election with quorum and fencing, start re-replication with a delay (a node that returns in 30 seconds should not have triggered a terabyte of data movement), and alert. Separate liveness (the process responds) from readiness (it can do useful work) as Kubernetes does, so a node warming up is not treated as failed, and a node that is up but cannot reach its database stops receiving traffic.

Trade-offs

Fast detection buys availability after real failures and pays with false positives whose cost depends on what they trigger. Adaptive detectors buy robustness to varying networks at the cost of complexity and a less predictable timeline. Indirect probing buys accuracy under partial partitions for a little extra traffic. The right design pairs each action with a detection threshold matched to how expensive and reversible that action is.

In an interview

Tie detection to consequence: "Service instances send heartbeats every second; the load balancer drops an instance after three misses, which is cheap and reversible. The database leader uses a ten-second lease with phi-accrual suspicion, and failover requires a quorum plus fencing, because a false failover is expensive. Re-replication after a node loss waits five minutes so a reboot does not move terabytes." Different thresholds for different costs is the senior signal.