Firewalls and NAT traversal

Why it matters

A large share of "it can't connect" problems are a rule somewhere saying no: a security group without the right port, a network ACL missing the return traffic, a corporate firewall blocking UDP, an egress policy nobody remembered. Another large share are stateful devices silently forgetting idle connections. And any feature that connects two users directly (video calls, file sharing, games) has to get through two NATs that were designed to block exactly that. Knowing how these devices think shortens these investigations dramatically.

The idea

A firewall inspects packets and allows or drops them according to rules. The crucial distinction is whether it remembers connections.

Cloud security groups versus network ACLs

Security groups are stateful and attached to instances or interfaces: allow inbound 443 and the replies flow out automatically. Network ACLs are stateless and attached to subnets: allowing inbound 443 is not enough, because the response leaves toward the client's ephemeral port (1024 to 65535), which needs its own outbound rule, and they are evaluated in numbered order. The classic misconfiguration is a tight ACL that permits the request but not the response, so connections hang after the SYN. Default security groups allow all outbound traffic; tightening egress is good practice, but every new dependency then needs a rule, which is a common source of mysterious timeouts after a deploy.

State has a lifetime

Stateful firewalls, NATs and load balancers keep an entry per connection and delete it after an idle timeout: often 350 seconds for cloud NAT gateways and load balancers, sometimes minutes or less on corporate firewalls, and as low as 30 seconds for UDP. After that, the next packet on the "open" connection is dropped or answered with a reset. Both ends still believe the connection is up. Symptoms: pooled database connections or long-lived streams that fail on first use after a quiet period. Fixes: TCP keepalives or application pings at an interval shorter than the shortest timeout on the path, and pools that validate or retire idle connections.

NAT traversal for peer-to-peer

Two peers each behind NAT have no public address to connect to, and each NAT drops unsolicited inbound packets. The standard toolkit, bundled as ICE in WebRTC:

1. 1

   Gather candidates

   Each peer lists addresses it might be reachable at: its local address, its public address and port as seen by a STUN server, and a relay address on a TURN server.
2. 2

   STUN

   A STUN server simply tells a client the public IP and port its NAT assigned, so the peer can share it through the signalling channel.
3. 3

   Hole punching

   Both peers send packets to each other's public mappings at about the same time. Each outbound packet creates a mapping in its own NAT, so the other side's packets now match it and get through. Works with most home NATs, especially for UDP.
4. 4

   TURN relay

   Symmetric NATs (which assign a different mapping per destination) and strict firewalls defeat hole punching; both peers then connect out to a TURN server that relays all traffic. Reliable, but it costs server bandwidth for every relayed session.

A debugging order for "cannot connect"

Check, from the source: DNS resolves to the expected address; a TCP connect to the port succeeds (nc -vz), times out (something drops it: security group, ACL, firewall, route) or is refused (reachable, nothing listening). Then check the destination's inbound rules, the source's outbound rules, any network ACLs in both directions, route tables, and finally host firewalls on the server itself. Cloud flow logs record accepted and rejected flows and show exactly which hop said no.

Trade-offs

Stateful firewalls are easy to configure and safe by default, at the cost of state that expires and can be exhausted. Stateless filters scale and are predictable but need return-traffic rules that are easy to get wrong. Tight egress rules contain breaches and exfiltration at the cost of friction for every new dependency. For peer-to-peer, direct connections save money and latency, and TURN relays guarantee connectivity at bandwidth cost; production systems need both.

In an interview

Show layered checks and the timeout trap: "Inbound is allowed by the load balancer's security group only; instances accept traffic only from that group. Egress is restricted to known dependencies. Database pools send keepalives every 60 seconds because the NAT and firewall drop idle flows after a few minutes. For video calls, ICE tries direct paths first and falls back to TURN, which we size for about 15 percent of sessions."