The TLB and page faults

Why it matters

Two services with identical code and data can differ in latency by a large factor because of memory events nobody logs: TLB misses and page faults. A process that touches memory spread across a huge heap, a service whose first requests after start are slow, a latency spike that coincides with a large file being read: these trace back to the mechanisms here. They are invisible in application metrics and easy to measure once you know they exist.

The idea

The TLB

Walking four levels of page tables on every memory access would multiply memory latency. The translation lookaside buffer is a small, fast cache inside each core that holds recent virtual-to-physical translations: typically a few dozen entries in the first level and a thousand or two in the second. A TLB hit costs essentially nothing. A TLB miss triggers a page-table walk by the hardware, several memory reads that may themselves hit or miss the data caches, costing from a few nanoseconds to over a hundred.

TLB reach is small: 1,500 entries times 4 KB covers only 6 MB of memory. A workload that randomly touches a 50 GB heap misses the TLB constantly, which is the main reason huge pages help large in-memory systems: 1,500 entries times 2 MB covers 3 GB. When a process changes its mappings (unmapping memory, changing permissions), other cores holding stale entries must be told to flush them, a TLB shootdown by inter-processor interrupt, which is why frequent large unmaps in a heavily threaded process can be surprisingly expensive.

Page faults

A page fault is raised when the translation exists but the page is not present or the access is not permitted. The kernel's handler decides what to do.

Demand paging is the design that makes faults normal: memory is not given to a process until it touches it, and file contents are not read until they are accessed. That is why allocations and process starts are fast, and also why the first pass over freshly allocated or freshly mapped memory is slow.

Measuring

Per process, Linux reports minor and major fault counts (/proc/PID/stat, ps -o min_flt,maj_flt, /usr/bin/time -v). perf stat reports dTLB and iTLB misses alongside instructions per cycle. A rising major-fault rate means the process is waiting on disk for memory, an early sign of swapping or of a working set that no longer fits in the page cache.

Reducing their cost

1. 1

   Improve locality

   Touch fewer distinct pages: compact data structures, arenas instead of scattered allocations, sequential access patterns.
2. 2

   Use huge pages for big heaps

   Fewer TLB entries for the same memory; explicit huge pages for databases and JVMs with large heaps.
3. 3

   Pre-fault where latency matters

   Touch or lock memory at startup (mlock, MAP_POPULATE, JVM AlwaysPreTouch) so request paths never take first-touch faults; warm memory-mapped files by reading them before serving.
4. 4

   Avoid needless remapping

   Allocators that return memory to the OS and take it back constantly cause fault and shootdown churn; tune them to keep freed memory.

Trade-offs

Demand paging makes allocation and startup fast and moves the cost to first access. Pre-faulting and locking memory buy predictable latency at the cost of slower startup and memory committed whether used or not. Huge pages raise TLB reach and can waste memory and stall during compaction.

In an interview

These come up as performance follow-ups: "The first minute after start has high latency because the 30 GB index is memory-mapped and every first access is a major fault; we read it sequentially at startup before marking the instance ready." Or: "With a 100 GB heap, TLB misses dominated; enabling explicit 2 MB pages cut p99 noticeably."