Routing and BGP

Why it matters

Several of the largest internet outages in recent years were not server failures but routing failures: a company withdrew its own routes and vanished from the internet, a small provider announced someone else's addresses, a misconfigured route sent a continent's traffic the long way round. Routing also explains latency you cannot fix in code (packets take the path the networks agree on, not the shortest one), how anycast and CDNs steer users to the nearest site, and what a cloud route table actually does.

The idea

A router forwards each packet independently, hop by hop, using a routing table: a list of destination prefixes, each with a next hop (the neighbouring router to hand the packet to) and an outgoing interface. No router knows the whole path; each only knows the next step toward each destination.

Longest prefix match

When several entries cover a destination, the router chooses the most specific one, the longest prefix. A default route (0.0.0.0/0) matches everything and catches whatever nothing else does, which is how a home router sends all internet traffic to the provider.

Inside a network versus between networks

The internet is about 75,000 autonomous systems (ASes): networks run by one organisation (an ISP, a cloud provider, a large company), each identified by an AS number. Routing works at two levels.

How BGP works

Each AS announces the prefixes it owns to its neighbours ("I can reach 203.0.113.0/24, path: AS64500"). Neighbours prepend their own AS number and pass the announcement on, so every route carries the list of ASes it traverses. Each router chooses among the routes it hears using policy first (prefer routes from paying customers over peers over transit providers) and path length second, then installs the winner. Withdrawing an announcement tells everyone the prefix is no longer reachable that way.

When routing goes wrong

1. 1

   Self-inflicted withdrawal

   A configuration change withdraws all routes to an organisation's own prefixes; within minutes the internet forgets how to reach it, including its DNS servers, and engineers may lose remote access to fix it.
2. 2

   Route leak

   A network re-announces routes it learned from one provider to another, attracting traffic it cannot carry; the traffic detours through a small network and much of it is dropped.
3. 3

   Hijack

   A network announces someone else's prefix, or a more specific slice of it (longest prefix wins), and draws that traffic to itself, by mistake or deliberately.
4. 4

   Mitigations

   RPKI cryptographically signs which AS may originate which prefix, and route filters on announcements from customers stop most leaks; adoption is growing but incomplete.

Routing you touch as an application engineer

Cloud route tables are static routes with longest prefix match: 0.0.0.0/0 to an internet or NAT gateway, VPC peering and VPN prefixes to their attachments. Anycast announces the same prefix from many locations so BGP delivers each user to the topologically nearest one, which is how public DNS resolvers and CDNs work. Asymmetric routing, where the reply takes a different path from the request, is normal on the internet and confuses stateful firewalls that only see one direction.

Trade-offs

Hop-by-hop routing with local decisions scales to the whole internet and survives failures by rerouting, at the cost of no single place to see or fix a path and convergence that can take minutes. BGP's policy-first design lets networks express business relationships, and its default trust lets one misconfiguration spread globally. Static routes are simple and predictable; dynamic routing adapts to failures at the cost of complexity.

In an interview

Use it to explain reachability and latency: "Our public prefix is anycast from several regions, so BGP brings users to the nearest one; if a region fails we withdraw its announcement and traffic shifts within a minute or so. In the VPC, a more specific route sends 10.50.0.0/16 to the peering connection while 0.0.0.0/0 goes to the NAT gateway." Mentioning that you keep out-of-band access in case your own routes vanish shows hard-won wisdom.