Module 13 · Distributed fundamentals · Lesson 3 of 18
Split brain
Two nodes both believe they are the leader and both accept writes. How it happens (partitions, pauses, slow failure detection), what it destroys, and the three defences: majority quorums, fencing tokens that make stale leaders harmless.
The video walkthrough is part of Pro.
See what Pro includesWhy it matters
Automatic failover sounds simple: if the primary stops responding, promote a replica. The nightmare case is that the primary did not stop; it was partitioned or paused, and it comes back to find a second primary. Now two nodes accept writes for the same data, clients are split between them, and when they reconnect there are two incompatible histories. This is split brain, and it has destroyed databases at companies that thought their failover was solid. Every high-availability design must answer how it is prevented.
The idea
Split brain occurs when a system that should have exactly one authority for some resource (a leader, a primary, a lock holder) ends up with two or more that each believe they are the only one. The cause is always the same: the mechanism that decided the old leader was dead was wrong, and the old leader kept acting.
How it happens
- 1
Network partition
The leader is isolated from the nodes that monitor it; they elect a new leader; the old one still serves its local clients. - 2
Process pause
A long garbage-collection pause, a VM migration or a swapped-out process makes the leader miss heartbeats. It is declared dead and replaced, then wakes up with no idea time has passed and continues as leader. - 3
Slow or asymmetric detection
The monitor cannot reach the leader but clients can, or detection takes long enough that a second promotion races with the first recovering. - 4
Human error
Two operators fail over manually at the same time, or a runbook promotes without demoting.
Defence one: majority quorums
Require a leader to hold the support of a majority of nodes, re-confirmed through a lease that expires. A leader that cannot renew its lease with a majority must stop serving before the lease expires; a new leader can only be elected after it does. Because two majorities cannot coexist, two leaders cannot both hold valid leases. This is how Raft and ZooKeeper-based systems work, and it depends on nodes honouring the lease timeout, which a paused process cannot do, hence defence two.
Defence two: fencing tokens
Every leadership term gets a monotonically increasing epoch (also called a generation, term or fencing token). The leader includes its epoch in every write to the shared resource (the storage system, the database, the lock-protected service), and the resource rejects any write with an epoch lower than the highest it has seen. A paused old leader wakes up, writes with epoch 7, and the storage, having seen epoch 8, refuses. The old leader cannot do harm even though it still believes it is in charge. This is the single most important technique in the lesson: do not trust the leader to know it was demoted; make the resource enforce it.
Defence three: STONITH
"Shoot the other node in the head": before promoting a replica, forcibly power off or network-isolate the old primary through an out-of-band channel (a management API, a power controller, a cloud instance stop). Blunt and effective where fencing at the resource is impossible, with the risk that the shooter is itself the partitioned minority. Pacemaker-style clusters and some database HA tools use it.
| Defence | Prevents | Depends on | Weakness |
|---|---|---|---|
| Majority quorum with leases | Two simultaneous valid leaders | Odd membership, synchronised-enough clocks for lease timing | A paused leader may overrun its lease |
| Fencing tokens (epochs) | Stale leaders doing damage | Every shared resource checking the token | Resources that cannot be made to check (legacy systems) |
| STONITH | Old leader acting at all | An out-of-band kill channel | Shooter may be on the wrong side of the partition |
Robust systems combine the first two: quorum leases to make dual leadership unlikely, fencing to make it harmless when it happens anyway.
Where it shows up
Database failover (Postgres with Patroni, MySQL orchestrators), distributed locks (a lock holder paused past its lease), message consumers that must be single-active, job schedulers, and multi-region active-passive setups where DNS or a load balancer flips to the standby while the primary still serves local traffic.