Latency vs throughput vs bandwidth

Why it matters

"Make it faster" can mean three different things, and optimising the wrong one wastes a quarter. A page that takes 200 milliseconds for one user may take two seconds for the thousandth concurrent user, not because the code got slower but because requests are queueing. Separating latency from throughput from bandwidth, and measuring latency at the tail rather than the average, is how you diagnose that correctly.

The idea

• Latency is the time for one operation to complete, from request to response. Measured in milliseconds, per request.
• Throughput is how many operations complete per unit of time. Measured in requests per second (or writes per second, messages per second).
• Bandwidth is the maximum rate at which data can move through a link. Measured in bits or bytes per second. It bounds throughput for large payloads; a 1 Gbps link cannot deliver more than about 125 MB/s no matter how fast the servers are.

Averages lie; percentiles tell the truth

Latency is a distribution, not a number. A service with a 50 ms average can have one request in a hundred taking three seconds, and that one request is the one a user remembers. Report percentiles: p50 is the median experience, p99 is what one in a hundred requests sees, p99.9 is the tail. Set targets on p99 (for example, "p99 under 300 ms") because a page that makes twenty backend calls will hit somebody's p99 on nearly every load.

Why throughput and latency fight

Up to a point, a server handles more requests per second at the same latency. Past that point, requests wait in queues (for a CPU, a connection, a lock) and latency climbs steeply while throughput flattens. The relationship is captured by Little's lawLittle's law: the average number of items in a system equals arrival rate multiplied by average time in the system (L = λW).: concurrency equals throughput times latency. Hold concurrency fixed and the only way to raise throughput is to cut latency; raise arrival rate beyond capacity and latency has to grow.

	Latency	Throughput	Bandwidth
Unit	ms per request	requests per second	bytes per second
Improve with	Caching, fewer hops, faster storage, less work per request	More servers, more cores, async processing, batching	Bigger links, compression, CDNs, smaller payloads
Measured as	Percentiles (p50, p99)	Rate at a given latency target	Peak sustained rate
Fails as	Slow responses, timeouts	Queues grow, errors under load	Saturated links, dropped packets
Typical target	p99 under 300 ms	Peak rps with 2x headroom	Under 60 percent utilisation

Batching: the classic trade

Batching ten writes into one network call raises throughput and lowers per-item cost, but the first item in the batch waits for the other nine, so latency rises. Message queues, database bulk inserts and HTTP/2 multiplexing are all this trade at different layers. Know which side the product needs.

Trade-offs

Lowering latency often means more hardware per request (more cache, more replicas), which costs. Raising throughput often means batching or asynchrony, which raises latency or weakens freshness. Protecting the tail (p99) usually means capping concurrency and shedding load before queues form, which means deliberately returning errors to a few so that the many stay fast. Every one of those is a product decision disguised as an engineering one.

In an interview

Quote targets as percentiles and pair them with throughput: "Reads at p99 under 200 ms at 5,000 requests per second peak." When asked to make something faster, ask which of the three they mean. And when a design involves fan-out (one page, many backend calls), say the tail-latency sentence: "With twenty parallel calls, the page's latency is the slowest call's, so I would set per-call timeouts and hedge or degrade the slow ones."