Reliability

Why it matters

A system can be up and still be wrong. It can answer every request and silently drop one write in a thousand, serve yesterday's balance, or return a 200 with an empty body. Availability measures whether it answers; reliability measures whether the answers are correct and the system keeps behaving the way it promised, through hardware faults, bad deploys and overload. Reliable systems are the ones people stop thinking about, which is the goal.

The idea

Reliability is the probability that the system performs its function correctly for a given period under stated conditions. The useful vocabulary comes from hardware engineering and still fits software:

• Fault: a component misbehaves (a disk dies, a process leaks memory, a dependency times out).
• Failure: the system as a whole stops meeting its specification because of a fault.
• Fault tolerance: the property that faults do not become failures.
• MTBF, mean time between failures, and MTTR, mean time to recovery. Availability is MTBF divided by MTBF plus MTTR, so you can raise it by failing less often or by recovering faster, and recovering faster is usually the cheaper lever.

The design moves

1. 1

   Redundancy

   Two of anything that matters, with automatic failover. Covered in Availability; it is the foundation of reliability too, because it turns a crash into a capacity dip.
2. 2

   Retries, with care

   Transient faults (a timeout, a dropped connection) are the most common kind, and a retry fixes most of them. But retries multiply load on a struggling dependency. Use exponential backoff with jitter, cap the attempts, and only retry operations that are safe to repeat.
3. 3

   Idempotency

   An operation is idempotent if doing it twice has the same effect as doing it once. It is what makes retries safe. Design write APIs so a repeated request is detected and ignored, usually with a client-supplied key.
4. 4

   Timeouts and fail-fast

   A call with no timeout can hang forever and take its caller's threads with it. Every outbound call gets a deadline; when a dependency is clearly down, stop calling it (a circuit breaker) rather than queueing up failures.
5. 5

   Graceful degradation

   When a non-critical dependency fails, serve without it. No recommendations is better than no page. Decide in advance which features are optional and make the code path real, not theoretical.

Trade-offs

Retries trade latency and load for success rate; done without backoff they cause the retry storms that turn a blip into an outage. Idempotency costs a key store and some design discipline. Graceful degradation means building and testing paths that most users never see. Timeouts set too short cause false failures; too long and they are useless. Each move has a dial, and the dial is set by what the product can tolerate.

In an interview

Interviewers probe reliability with "what happens when X fails". Have the pattern ready: "The call to the payment provider has a two-second timeout and retries twice with backoff; the request carries an idempotency key so a retry cannot double-charge; if the provider is down, the circuit opens and we queue the order for later rather than failing checkout." Four moves in one sentence is a senior answer.