Handling failures

Why it matters

Most large outages are not caused by the component that failed first; they are caused by how everything else reacted. A slow database makes callers time out and retry, which triples its load, which makes it slower, which makes more callers retry. Handling failure well is mostly about making those reactions calm: bounded, idempotent, and willing to give up. This lesson is the operational core of distributed systems and the one interviewers use to find out whether you have been on call.

The idea

Failures will happen; the design decides whether they stay local or spread. The tools below each limit one way that failure propagates.

Timeouts and deadlines

Every remote call has a timeout, chosen from the callee's latency distribution (around its p99, not its average) and the caller's own budget. Better than per-hop timeouts is a deadline carried with the request: the total time the end user will wait, decremented at each hop, so a service with 200 ms left never makes a downstream call with a 2-second timeout. gRPC propagates deadlines natively; HTTP systems pass a header. Work that has already exceeded its deadline should be cancelled, not finished for nobody.

Retries, done carefully

1. 1

   Retry only what is retryable

   Timeouts, connection errors, 503s and explicit retryable codes. Never retry 400s or business failures, and never retry a non-idempotent operation without an idempotency key.
2. 2

   Exponential backoff with jitter

   Wait 100 ms, 200, 400, 800, capped, each with a random factor, so a thousand clients that failed together do not retry together. Full jitter (random between 0 and the cap) spreads load best.
3. 3

   Bound the attempts

   Two or three retries; more rarely helps and multiplies load. Total retry time must fit within the deadline.
4. 4

   Retry budgets, not just counts

   Allow retries to be at most, say, 10 percent of normal request volume across the service. When the budget is exhausted, failures return immediately. This is what stops a retry storm: retries cannot amplify load beyond a fixed ratio.
5. 5

   Retry at one layer

   If the client, the gateway and the service each retry three times, one failure becomes 27 requests. Decide where retries live and make the other layers pass failures through.

Circuit breakers

A circuit breaker wraps calls to a dependency and tracks failures. Closed: calls flow. When failures exceed a threshold in a window, it opens: calls fail immediately without touching the dependency, giving it time to recover and giving callers a fast, predictable failure to degrade from. After a cooldown it goes half-open, letting a few trial calls through; success closes it, failure reopens it. Breakers are per dependency (and often per endpoint), and their open state is a signal worth alerting on.

Bulkheads and isolation

Ships survive hull breaches because compartments are sealed. A bulkhead gives each dependency or tenant its own bounded resources, a thread pool, a connection pool, a queue, so one slow dependency exhausts only its compartment, not the whole service. The Microservices patterns module details bulkheads and breakers.

Load shedding and backpressure

When demand exceeds capacity, serving everyone badly is worse than serving most people well. Load shedding rejects excess requests early (at the edge, with 503 and Retry-After) based on queue length, concurrency or CPU, preferring to drop low-value traffic first. Backpressure pushes the signal upstream: bounded queues, flow control in streaming protocols, and producers that slow down when consumers lag. A system without backpressure buffers until it falls over.

Graceful degradation

Decide in advance what the product does when each dependency is gone. Recommendations down: show popular items. Search down: show category browsing. Payment provider down: queue the order and confirm later. Each is a code path that must be built and tested before the outage, with feature flags to force it. The reliability lesson in Core concepts framed this; here it is the fallback a circuit breaker switches to.

Testing failure on purpose

Fault injection and chaos experiments (kill an instance, add latency to a dependency, partition a zone) in staging and, carefully, in production are how you learn whether timeouts, breakers and fallbacks actually work. Game days rehearse the runbooks. A failure path that has never run will not run correctly when it matters.