Reading packet captures

Why it matters

"The client says the server is slow, the server says it answered in 5 ms." Both logs are honest and both are partial. A packet capture shows what actually crossed the wire and when, which settles arguments about who is slow, who closed the connection, whether a request ever arrived and whether loss is involved. It is the most decisive network debugging tool there is, and most engineers never learn to read one.

The idea

A packet capture records copies of packets as they pass a network interface, with precise timestamps. tcpdump captures on servers from the command line; Wireshark (or its terminal sibling tshark) decodes captures into protocols and conversations. You capture where the problem is visible, filter to the traffic you care about, and read the timeline.

Capturing without hurting anything

Capture with a filter so you record only relevant traffic, write to a file rather than printing every packet, and cap the size. A typical server capture: tcpdump -i any -nn -s 0 -w /tmp/api.pcap 'host 10.0.4.17 and tcp port 443' -c 200000. The -nn skips name lookups, -s 0 captures whole packets, and -c stops after a count. Capturing on both ends of a connection, at the same time, is the gold standard: differences between the two views show exactly where packets were lost or delayed. Treat captures as sensitive data, since unencrypted payloads and headers can contain credentials and personal information.

Filter	Captures
host 10.0.4.17	Traffic to or from one address
tcp port 5432	PostgreSQL traffic
tcp[tcpflags] & (tcp-syn|tcp-rst) != 0	Only connection setups and resets: cheap way to watch failures
udp port 53	DNS queries and answers
net 10.20.0.0/16 and not port 22	A subnet, excluding your own SSH session

Reading a TCP conversation

Wireshark's "Follow TCP stream" isolates one connection; its time column set to "seconds since previous displayed packet" makes gaps jump out. Ask, for each gap: which side sent the last packet before it? If the client sent a request and the server took 800 ms to answer while the handshake took 2 ms, the network is fine and the server (or something behind it) is slow. If the server answered quickly but the client saw the response late, look for retransmissions or a congested path in between.

Patterns worth recognising

Pattern	What it means	Usually caused by
SYN, SYN, SYN with growing gaps, no reply	Connection attempts are not reaching anything that answers	Firewall or security group dropping, wrong IP, host down
SYN answered by RST	Host reachable, nothing listening on that port	Service down or wrong port
Retransmissions and duplicate ACKs	Packet loss on the path	Congestion, bad link, overloaded middlebox, MTU black hole for full-size packets
Zero window from the receiver	Receiver's buffer is full; the reading application is not keeping up	Slow consumer, blocked thread, GC pause
RST in the middle of a session from a middlebox's address	Something in between dropped the connection's state	Load balancer or NAT idle timeout shorter than the connection's idle period
ClientHello then immediate alert or RST	TLS negotiation failed	Protocol or cipher mismatch, SNI not recognised, certificate rejected
Large gap before the first response byte	Server think time	Slow query, lock, dependency call

Encrypted traffic

With TLS you cannot read payloads, but the capture still shows connection setup, the TLS handshake messages (and alerts), timing, sizes and TCP behaviour, which answers most questions. When you must see content, capture at the point where TLS terminates, or have the client export session keys (the SSLKEYLOGFILE environment variable in many TLS libraries) so Wireshark can decrypt in a test environment.

Trade-offs

Captures are the most detailed evidence available, at the cost of being heavy (size, CPU at high rates), sensitive, and narrow (one host's view at a time). Flow logs and metrics are cheap and broad but cannot show individual retransmissions or exact timings. Start broad with metrics to find where to look, then capture narrowly to prove what happened.

In an interview

Describe it as the tie-breaker: "Client logs showed 900 ms, server logs 50 ms. A capture on the server's host showed the request arriving 850 ms after the client's SYN, with the client retransmitting the SYN twice, so connection setup was the problem: the load balancer's backlog was full. Without the capture we would have tuned the wrong service."