Replication

Why it matters

One database server is a single point of failure and a ceiling on reads. Replication fixes both: copies on other machines serve reads and stand ready to take over. It also introduces the first truly distributed-systems problem most engineers meet: the copies lag, and a user can write something and then not see it. Every interview that mentions "read replicas" is a test of whether you know that lag exists and what you do about it.

The idea

Replication keeps the same data on several nodes by shipping changes from where they are written to the others. The designs differ in how many nodes accept writes and whether a write waits for the copies.

Single-leader (primary and replicas)

One node, the primary, accepts all writes and streams its change log (write-ahead log or binlog) to replicas, which apply it and serve reads. Simple, no write conflicts, and the dominant pattern in Postgres, MySQL and most managed services.

Synchronous versus asynchronous

Most production setups mix them: one synchronous standby in the same region for zero-loss failover, and asynchronous replicas elsewhere for reads and disaster recovery.

Replication lag and what to do about it

Replicas apply changes after the primary commits them, so a read from a replica can return a state from 50 milliseconds, or under load several seconds, ago. Anomalies this causes and their fixes:

1. 1

   Read-your-writes

   A user updates their profile and the next page shows the old value. Fix: route reads for recently written data to the primary (for a few seconds after a write, or for the user's own data), or pass the primary's log position with the request and wait for the replica to reach it.
2. 2

   Monotonic reads

   Two successive reads hit different replicas and time appears to go backwards. Fix: pin a session to one replica (consistent hashing on user id).
3. 3

   Lag spikes

   Big transactions, index builds or a slow replica make lag jump. Fix: monitor lag per replica, remove lagging replicas from the read pool automatically, and keep transactions small.

Failover

When the primary dies, a replica must be promoted. That requires detection (heartbeats with a timeout long enough to avoid false positives), choosing the most up-to-date replica, repointing the application (DNS, a proxy, or a cluster-aware driver), and fencing the old primary so it cannot come back and accept writes (split brain). Automated orchestrators (Patroni, Orchestrator, managed-cloud failover) do this in tens of seconds. With asynchronous replication, the promoted replica may be missing the last few writes; the application must tolerate that or use synchronous replication for the data that cannot be lost.

Multi-leader and leaderless

Multi-leader replication lets several nodes accept writes, typically one per region, and replicates between them asynchronously. It gives local write latency and regional independence, and it creates write conflicts that must be resolved (last-write-wins, merge functions, or application rules), which is hard enough that most teams avoid it unless they truly need active-active regions. Leaderless systems (Dynamo-style, Cassandra) let any node accept writes with quorum reads and writes for consistency; the wide-column lesson covered the model.

Trade-offs

Replication buys read scaling and availability for the cost of lag and failover complexity. Synchronous replication buys durability for latency; asynchronous buys latency for a small window of possible loss. Multi-leader buys regional write locality for conflict resolution. Choose per data class: the ledger can wait for a synchronous replica; the activity feed cannot afford to.

In an interview

State the topology and the lag handling: "Postgres primary with one synchronous standby in-region for zero-loss failover and two asynchronous read replicas. Reads go to replicas except for a user's own recent writes, which go to the primary for ten seconds after a write. Lag is monitored and a replica over two seconds behind drops out of the pool. Failover is automated with fencing of the old primary." That covers scaling, durability, consistency and operations in four sentences.