Network partitions

Why it matters

A partition is the failure that turns theory into incidents: two groups of nodes are both alive and both believe the other has failed. Whatever each group does next, serve writes or refuse them, is a decision with consequences for correctness or uptime, and the decision must be made in advance by the design because there is no one to ask in the moment. Understanding partitions concretely, not just as the P in CAP, separates designs that survive cloud outages from designs that corrupt data during them.

The idea

A network partition is a communication failure that splits a set of nodes into groups that can talk within themselves but not across. Nodes on both sides are healthy and running. From inside either group, the other group looks exactly like a set of crashed machines, and nothing in the system can tell the difference until connectivity returns.

Where partitions come from

Switch and router failures, misconfigured firewalls and security groups, cloud availability-zone outages, cross-region link degradation, garbage-collection pauses long enough to miss heartbeats (a one-node partition in effect), overloaded nodes that drop packets, and bad deploys of network policy. They are not rare; large systems see partial connectivity failures weekly. Partitions are often asymmetric (A can reach B but B cannot reach A) or partial (A reaches B and C, but B cannot reach C), which produce stranger behaviour than a clean split.

The forced choice

Suppose clients on both sides send writes. If both sides accept them, the two groups diverge and must be reconciled later, possibly with conflicts that have no clean answer (availability chosen, consistency lost). If only one side accepts writes, clients on the other side see errors or read stale data until the partition heals (consistency chosen, availability lost on that side). CAP says you cannot have both during the partition; the design must say which side gives way and how the system knows which side it is on.

	Choose consistency (CP)	Choose availability (AP)
During partition	Only the side with a quorum (majority) serves writes; the minority refuses or serves stale reads	Both sides serve reads and writes
After healing	Minority catches up from the majority; no conflicts	Divergent writes must be merged: last-write-wins, vector clocks, CRDTs or application rules
User experience	Some users see errors for the duration	Everyone keeps working; some see inconsistent data and later surprises
Fits	Money, inventory, configuration, anything where two truths are unacceptable	Shopping carts, likes, presence, collaborative edits, anything mergeable
Examples	ZooKeeper, etcd, Spanner, single-leader databases with fencing	Dynamo-style stores, Cassandra at low consistency levels, CRDT-based apps

Most real systems are CP for some data and AP for other data, which is a per-feature decision, not a per-company one.

Quorums: knowing which side you are on

A node cannot see the whole network, but it can count who answers. If a group can reach a majority of the configured membership, it knows no other group can also have a majority, so it may safely act as the authority. The minority side knows it is the minority and steps down. This is why clusters have odd sizes (3, 5, 7): with 4 nodes, a 2 versus 2 split has no majority and both sides must stop. Quorums are the mechanism behind leader election, consensus and replicated databases, each later in this module.

Detecting and healing

Detection is heartbeats with timeouts (next lesson) and is inherently late and uncertain. During the partition, the consistent side keeps a log of what happened; on healing, the other side replays it. The availability side instead reconciles divergent state. In both cases fencing matters: a former leader on the minority side must be prevented from acting on stale authority when connectivity returns, typically with epoch numbers that make its old writes rejectable, covered in Split brain and Leader election.