Common network failures

Why it matters

Network failures rarely announce themselves. They arrive as a latency bump, a rise in one error code, a single region or client type misbehaving. The same dozen failure modes account for most of them, and each has a recognisable signature. Knowing those signatures, and the order to check them in, is what turns a three-hour war room into a ten-minute diagnosis, and the drill at the end of this module is built around exactly these cases.

The idea

Read the symptom precisely before guessing. "Connection refused", "timed out", "reset by peer" and "no route to host" are different facts about different layers. Combine that with scope (one client or all, one region or all, new connections or existing ones, small requests or large ones) and most failures identify themselves.

The catalogue

Partial partitions and gray failures

The hardest failures are partial. A partial partition lets A reach B and B reach C, but not A reach C, so health checks from one place look fine while real traffic from another fails. A gray failure is a component that is degraded rather than dead: a switch port dropping 2 percent of packets, a load balancer node with a slow disk, a host with a flaky NIC. Health checks pass, averages barely move, and the p99 explodes for the subset of traffic that touches the bad element. Detect them by measuring from many vantage points and by slicing metrics by zone, host, client type and network path, then comparing slices rather than looking at the global average.

How networks turn small faults into outages

Timeouts shorter than the dependency's normal p99 cause needless failures; timeouts much longer tie up threads and connections. Retries without backoff and jitter synchronise into waves. Retries at every layer of a call chain multiply: three retries at each of three layers is up to 64 attempts per user request. The defences, covered in the System Design reliability lessons, are timeouts set from measured latency, retries with exponential backoff and jitter, retry budgets, and circuit breakers.

A triage order

1. 1

   Scope it

   Which clients, regions, endpoints and time? New connections or established ones? Small or large payloads? Scope eliminates most candidates.
2. 2

   Name resolution

   Does the name resolve to the expected addresses from the affected location?
3. 3

   Reachability by layer

   Ping or connect to the IP and port: refused, timeout or success separates listener, filtering and application problems.
4. 4

   Phase timing

   A curl breakdown localises time to DNS, connect, TLS or server.
5. 5

   Path quality

   mtr for loss and latency per hop from the affected side; compare with an unaffected side.
6. 6

   Limits and state

   Ports, descriptors, NAT allocations, conntrack tables, load balancer surge queues.
7. 7

   Prove it

   A packet capture on one or both ends settles what actually happened.

Trade-offs

Exhaustive monitoring catches gray failures early and costs money and noise; minimal monitoring is cheap and finds out from users. Aggressive retries hide transient blips and amplify real outages; conservative retries surface errors sooner. Fast timeouts protect resources and risk false failures. Every one of these is a dial to set from measurements, not a default to accept.

In an interview

Answer failure questions by signature, not by guess: "If clients get connection refused, the host is reachable and nothing is listening, so check the deploy. If they time out, something is dropping SYNs: security groups, ACLs or routes. If only the first request after idle fails, it is a middlebox idle timeout; add keepalives. If only large responses hang, it is MTU." Then mention retry storms and how you bound retries.