Leader election

Why it matters

Many jobs must be done by exactly one node at a time: running the cron scheduler, consuming a partition, holding the primary role, coordinating a migration. Running them on zero nodes is an outage; running them on two is data corruption. Leader election is how a group picks the one node, and the split-brain lesson showed how badly it goes when the group picks two. This lesson gives the practical recipe that production systems use and the two rules that make it safe.

The idea

Leader election selects one node from a group as the leader for a role, makes every node agree on who it is, and replaces it when it fails or steps down. The hard parts are agreement (two nodes must never both believe they lead at the same time) and detection (the group must notice a dead leader without a live one being wrongly replaced).

Election by consensus

Consensus systems elect leaders as part of their protocol (Raft's terms and votes). Each term has at most one leader by construction, because winning requires a majority of votes and each node votes once per term. This is the gold standard but applies to the consensus group itself. Application services typically do not run their own consensus; they borrow it from a coordination service.

Election by lease in a coordination service

The common production pattern uses etcd, ZooKeeper, Consul or a database with strong consistency as an arbiter.

1. 1

   Try to create a key

   Each candidate attempts to create the same key (e.g. /leader/scheduler) with its own identity, using a compare-and-set that succeeds only if the key does not exist. Exactly one succeeds; it is the leader.
2. 2

   Attach a lease

   The key is tied to a lease (etcd) or is an ephemeral node bound to the session (ZooKeeper) that expires unless the holder keeps renewing it with heartbeats. A dead or partitioned leader stops renewing and the key disappears after the lease interval.
3. 3

   Others watch

   Non-leaders watch the key. When it is deleted they race to create it again; one wins. ZooKeeper's recipe has each candidate create a sequential node and watch only the one before it, avoiding a thundering herd.
4. 4

   Carry the fencing token

   The successful create returns a monotonically increasing revision or version. The leader includes it in every action it takes on shared resources, and those resources reject tokens lower than the highest seen.

The two safety rules

Rule one: the old lease must expire before a new leader acts. If the lease is 10 seconds, a partitioned leader must stop acting as leader within 10 seconds of its last successful renewal, measured on its monotonic clock, with margin for clock rate differences. If it cannot (a process pause), rule two saves you.

Rule two: every action carries a fencing token checked by the resource. The coordination service's revision number is the token. The database, storage system or downstream service rejects any write whose token is lower than the highest it has seen. A paused former leader that wakes and writes with revision 42 is rejected once revision 43 exists. Without this rule, leader election is only probabilistically safe.

Textbook algorithms and why they are rarely used

The textbook algorithms assume you can know a node is dead, which the earlier lessons showed you cannot. Delegating to a consensus-backed service imports its majority guarantee.

Alternatives to electing a leader

Sometimes you can avoid the problem: partition the work so every node owns a disjoint share (consumer groups in Kafka assign partitions, no single leader needed), make operations idempotent so duplicate execution is harmless and run them on every node, or use a database row with a conditional update as a lightweight lock for low-frequency jobs. Elect a leader only when exactly-one semantics truly matter.