Module 13 · Distributed fundamentals · Lesson 7 of 18
Clock synchronization
Why two machines never agree on the time, how NTP and PTP narrow the gap, monotonic versus wall clocks, leap seconds and clock jumps, and what Spanner's TrueTime does differently: expose the uncertainty and wait it out.
Why it matters
Timestamps feel like the natural way to order events across machines, and they are the most common source of subtle distributed bugs: a replica with a clock three seconds ahead wins every last-write-wins conflict, a lease computed on one machine expires early on another, a log appears to show an effect before its cause. Understanding how far clocks can disagree, and which clock to use for which purpose, is prerequisite to the logical-clock lessons that follow.
The idea
Every machine has a quartz oscillator that drifts, typically 10 to 100 parts per million, which is a second a day or more. Clock synchronization protocols periodically compare the local clock to a reference and adjust it. The result is never exact: the adjustment itself has error, and between adjustments the clock drifts again. The achievable agreement between machines is called skew, and your design must tolerate whatever skew your environment delivers.
NTP and PTP
| NTP (Network Time Protocol) | PTP (Precision Time Protocol) | |
|---|---|---|
| Accuracy | Milliseconds over the internet; sub-millisecond on a good LAN | Microseconds, with hardware timestamping at the network card |
| How | Client exchanges timestamps with servers, estimates round-trip delay and offset, slews the clock gradually | Master clock multicasts; switches and NICs timestamp packets in hardware to remove software jitter |
| Topology | Hierarchy of strata from atomic or GPS references | Grandmaster plus boundary and transparent clocks |
| Fits | General servers, cloud VMs (cloud providers run NTP fleets) | Trading, telecom, databases that depend on tight bounds (Spanner-style) |
| Failure modes | Bad servers, asymmetric network paths, VM pauses causing jumps | Needs supporting hardware end to end |
Two clocks, two jobs
The wall clock (time-of-day) tells you the date and time and can jump forwards or backwards when synchronised, when an operator changes it, or at a leap second. The monotonic clock only ever moves forward at a steady rate and has no meaningful absolute value; it exists to measure durations. The rule: use the wall clock to timestamp events for humans and for coarse ordering across machines; use the monotonic clock for every timeout, lease, rate limit and elapsed-time measurement within a machine. A timeout computed from wall-clock differences fires instantly or never when the clock jumps.
Leap seconds and jumps
A leap second inserts 23:59:60, which many systems cannot represent; the historical result has been crashes and spinning CPUs. Large operators smear the leap second over a day so every second is slightly longer and no jump occurs. Virtual machines paused for migration, hosts resumed from sleep and misconfigured NTP all produce jumps of seconds or more, forwards or backwards. Designs that depend on wall-clock monotonicity break on those days.
Living with skew
Treat any cross-machine timestamp comparison as uncertain by at least the expected skew (tens of milliseconds in a well-run cloud, worse elsewhere). Do not use wall-clock timestamps for conflict resolution where correctness matters; last-write-wins with skewed clocks silently discards writes. Do not expire sessions or leases by comparing timestamps from different machines without margin. Where ordering matters, use the logical clocks of the next lessons, which order by causality rather than by physical time.
TrueTime: embracing uncertainty
Google's Spanner gets externally consistent transactions across continents by not pretending clocks are exact. TrueTime returns an interval, earliest and latest, within which the true time certainly lies, kept to a few milliseconds by GPS and atomic clocks in every data centre. A transaction assigns its commit timestamp at the latest bound and then waits out the uncertainty before releasing locks, so by the time anyone can see the commit, every clock in the system agrees it is in the past. The cost is a few milliseconds of commit latency; the benefit is a global order without a global coordinator. CockroachDB uses a similar approach with looser bounds and hybrid logical clocks.