Why distribution is hard

Why it matters

On one machine, a function call either returns or the whole process dies; there is no in-between. Across a network, a request can succeed and its response be lost, and you cannot tell that apart from the request never arriving or the server being slow. Every hard problem in this stage, consensus, transactions, clocks, replication, exists because of that ambiguity. Understanding exactly what you lose when you add a second machine is the foundation for everything that follows.

The idea

A distributed system is several processes on separate machines communicating by messages over a network. Three assumptions that hold inside one process fail across machines.

1. 1

   There is no shared clock

   Each machine has its own clock, drifting at tens of parts per million and jumping when corrected. Two events on two machines have no reliable 'before' and 'after' from timestamps alone, so ordering must come from messages, not time. The Time and Ordering lessons cover the fixes.
2. 2

   The network is unreliable

   Messages are delayed arbitrarily, reordered, duplicated or dropped. A link, a switch, a cloud availability zone or a misconfigured firewall can isolate healthy machines from each other for seconds or hours: a network partition.
3. 3

   You cannot know if a remote process is alive

   The only evidence is messages. Silence could mean the process crashed, is paused by garbage collection, is overloaded, or is fine but the network dropped your packets. All look identical from outside: no reply.

Partial failure and the timeout

In a single process a failure is total: everything stops. In a distributed system failure is partial: some nodes work, some do not, and the working ones cannot be sure which is which. The client must decide, after some timeout, to give up or retry, knowing that the right action differs for each hidden reality. Too short a timeout treats slow nodes as dead and causes needless retries and failovers; too long leaves users waiting and resources held. There is no correct value, only a trade-off, and this is why idempotency (API module) and failure detection (later in this module) exist.

The fallacies of distributed computing

The impossibility results, briefly

The Two Generals problem shows that two parties on an unreliable channel can never be certain they agree, because the last acknowledgement might always be lost. FLP proves that in a purely asynchronous system with even one crash failure, no deterministic consensus algorithm can guarantee termination. Real systems get around both by assuming partial synchrony (messages usually arrive within some bound) and using timeouts and randomness, which is why practical consensus works most of the time but not under arbitrary network behaviour. The CAP trade-off from Core concepts is the application-level face of the same constraint: under partition, choose consistency or availability.

What this means for design

Every remote call needs a timeout, a retry policy and an idempotent target. Every "is it alive" check is probabilistic. Every ordering guarantee must be built from message causality, not wall clocks. Every write to two places is a dual-write problem. The patterns in this stage are not cleverness for its own sake; each is the disciplined response to one of the three lost assumptions.

Trade-offs

Distribution buys scale, availability and geographic reach that one machine cannot offer, and pays with partial failure, uncertainty and the coordination machinery needed to live with it. The honest design question is always: what is the smallest amount of distribution that meets the requirement, because every added boundary adds the whole catalogue of problems above.

In an interview

Show the mindset early: "Every call between services has a timeout and retries against an idempotent endpoint, because a lost response is indistinguishable from a failed request. Ordering between services comes from sequence numbers and causality, not timestamps. Where two systems must change together we use an outbox, not a dual write." That sentence signals you understand why the rest of the design looks the way it does.