Measuring networks

Why it matters

"The network is slow" is not actionable. "p99 connect time from eu-west to the payments API went from 3 ms to 300 ms at 14:05" is. Measuring the network well turns vague complaints into specific claims about latency, throughput or loss on a specific path, which tells you whether to look at routing, capacity, a middlebox or the application. Measuring it badly, with averages, the wrong tool or the wrong path, sends teams chasing the wrong problem for days.

The idea

A network path has four properties worth measuring, and each needs its own tool and its own interpretation.

Breaking a request into phases

The most useful single measurement for an HTTP service is a phase breakdown of one request. curl reports it directly with -w, and browsers show the same phases in their network panel.

TCP connect time is one round trip and therefore a clean latency measurement of the path. TLS adds about one more round trip plus server CPU. Time to first byte after the request is sent is mostly server work. Transfer time depends on size, bandwidth and congestion window growth. Measured from several locations and repeated, this breakdown localises most performance problems in minutes.

Distributions, not averages

Network latency has a long tail. An average hides it: a path with 99 requests at 10 ms and one at 2 seconds averages 30 ms and looks healthy. Report percentiles (p50, p95, p99, p99.9) and look at histograms. Tail latency matters disproportionately because a page or request that fans out to many calls is as slow as its slowest one: with 50 parallel calls, the request's median is near each call's p99. When comparing before and after, use enough samples and compare the same percentiles over the same time windows.

Throughput testing honestly

iperf3 between two hosts measures what the path and stacks can do. Run it in both directions, with one stream and with several (-P 8), and for long enough to get past slow start (30 seconds or more). A single stream far below link speed on a long path usually means window or buffer limits, not a bad link; several streams reaching link speed confirm it. Test inside the same environment your service uses (same instance types, same network path, through the same NAT or proxy), because each hop can be the bottleneck.

Benchmarking traps

1. 1

   Measuring the wrong path

   Testing from your laptop over VPN says little about traffic between two cloud services in the same region.
2. 2

   Cold versus warm

   The first request pays DNS, TCP, TLS and slow start; later ones on a reused connection do not. Measure both and say which you report.
3. 3

   Coordinated omission

   A load generator that waits for each response before sending the next stops sending exactly when the system is slow, hiding the worst latency. Use open-loop tools that send at a fixed rate.
4. 4

   Averages and short runs

   Short tests miss periodic events (garbage collection, batch jobs); averages hide tails.
5. 5

   Clock assumptions

   One-way latency needs synchronised clocks; round trips do not, which is why most tools measure round trips.

Continuous measurement

Spot checks diagnose; continuous measurement detects. Synthetic probes from several regions that perform real DNS, TCP, TLS and HTTP requests every minute, plus client-side timing from real users, catch regressions and regional problems before support tickets do, and give you the baseline that makes "slower than usual" a measurable statement.

Trade-offs

Simple tools are fast and approximate; packet captures and distributed tracing are precise and expensive. Synthetic probes are consistent but do not see real user paths; real-user monitoring sees reality but is noisy. A practical stack uses metrics and synthetic checks to detect, phase breakdowns to localise, and captures to prove.

In an interview

Show the localisation habit: "First I take a curl phase breakdown from the client's region: if connect and TLS are each about one round trip, the path is fine and the time is in the server. Then I check p99, not the average, and whether it is cold-connection or warm. Only if connect time is high or variable do I go to mtr and captures for loss on the path."