Reliability and flow control

Why it matters

The network beneath TCP loses, duplicates and reorders packets routinely, yet applications see a clean stream of bytes. The machinery that achieves that also determines throughput on long links, explains why one lost packet can stall a whole HTTP/1.1 or HTTP/2 connection, and is why a slow consumer eventually blocks a fast producer through the socket. When a transfer is slow despite a fast link, the answer is usually in this lesson or the next.

The idea

TCP numbers every byte it sends. The receiver acknowledges the next byte it expects (a cumulative ACK), so the sender knows what arrived. Anything not acknowledged in time is retransmitted. Out-of-order arrivals are buffered and reassembled before the application sees them, so the application always reads bytes in the order they were written.

Detecting and repairing loss

1. 1

   Retransmission timeout (RTO)

   The sender keeps a smoothed estimate of the round-trip time and its variance. If a segment is not acknowledged within the RTO (at least 200 ms to 1 s depending on the stack), it is resent and the timeout doubles on each retry. Timeouts are slow and also signal congestion.
2. 2

   Fast retransmit

   When a segment is lost but later ones arrive, the receiver keeps acknowledging the missing byte. Three duplicate ACKs tell the sender exactly what is missing, and it retransmits immediately without waiting for the timer.
3. 3

   Selective acknowledgement (SACK)

   The receiver also reports which later blocks it already has, so the sender resends only the holes rather than everything after the first loss.

The sliding window

Waiting for an acknowledgement after every segment would cap throughput at one segment per round trip. Instead the sender may have a window of unacknowledged bytes in flight; as ACKs arrive, the window slides forward. Throughput is roughly window size divided by round-trip time, so a long path needs a large window to stay busy. The amount needed is the bandwidth-delay product: a 1 Gbit/s link with 100 ms round trip needs about 12.5 MB in flight. TCP's original 64 KB window limit is lifted by the window scaling option negotiated in the handshake.

Flow control: protecting the receiver

The receiver advertises a receive window in every ACK: how much free buffer space it has. If the application reads slowly, the buffer fills, the advertised window shrinks toward zero, and the sender stops. This is back-pressure across the network: a slow consumer automatically slows a fast producer, and the producer's writes eventually block or report "would block". Congestion control, next lesson, protects the network in the same way; the sender is limited by whichever window is smaller.

Head-of-line blocking

Because TCP delivers bytes strictly in order, one lost segment holds up everything behind it until the retransmission arrives, even data that has already been received. For a single stream that is fine. For protocols that multiplex many independent requests over one TCP connection (HTTP/2, many RPC frameworks), a single loss stalls every request on the connection: head-of-line blocking at the transport layer. That is the main reason QUIC and HTTP/3 implement their own streams over UDP, where a loss affects only the stream it belongs to.

Trade-offs

TCP's reliability is general and automatic, and it pays with latency on loss (a retransmission costs at least one round trip, a timeout far more), with ordering that creates head-of-line blocking, and with kernel buffering that must be sized for long fast paths. Applications that would rather drop late data than wait for it (live audio, games) prefer UDP-based protocols; applications that need every byte in order take TCP's trade.

In an interview

Use it to explain throughput and stalls: "Cross-region replication was capped at about 40 Mbit/s on a 1 Gbit/s link because the socket buffers limited the window to 512 KB at an 100 ms round trip; raising buffers to cover the bandwidth-delay product fixed it." Or: "On lossy mobile networks HTTP/2's single connection suffers head-of-line blocking; HTTP/3 over QUIC avoids it."