Module 06 · Memory management · Lesson 6 of 8
Garbage collection basics
How managed runtimes reclaim memory: reachability from roots, reference counting and its cycles, mark-sweep and compaction, the generational hypothesis, and concurrent collectors.
Why it matters
Most backend code runs on a garbage-collected runtime: Java, Go, C#, JavaScript, Python. The collector decides a large share of tail latency (pauses), CPU use (collection work) and memory footprint (headroom it needs), and the three trade against each other. When a JVM service shows p99 spikes every few seconds, or a Go service's CPU is a quarter garbage collection, or a Node process is killed at its memory limit, the explanation and the fix are in how the collector works.
The idea
A garbage collector frees objects the program can no longer reach. An object is live if it can be reached by following references from a root: local variables on thread stacks, global and static variables, CPU registers. Everything not reachable is garbage, however recently it was used.
Two families
| Reference counting | Tracing (mark-sweep and relatives) | |
|---|---|---|
| How | Each object counts references to it; when the count hits zero it is freed immediately | Periodically start at the roots, mark everything reachable, reclaim the rest |
| Pauses | No collection pauses; work happens on every reference change | Collection cycles that may pause the program (less so with concurrent collectors) |
| Cycles | Cannot free cycles (A references B, B references A) without a backup cycle collector | Handles cycles naturally |
| Overhead | Count updates on every assignment, atomic if shared across threads | Work proportional to live data (marking) and to heap size (sweeping) |
| Used by | CPython (plus a cycle collector), Swift and Objective-C (ARC), Rust's Rc and Arc, C++ shared_ptr | Java, Go, C#, JavaScript engines, Ruby |
Mark, sweep, compact
Mark-sweep marks reachable objects, then sweeps the heap, adding unmarked objects to free lists. It leaves the heap fragmented. Compacting collectors also move live objects together, eliminating fragmentation and making allocation a simple pointer bump, at the cost of updating every reference to moved objects. Copying collectors split the heap and copy live objects from one half to the other, which is fast when few objects survive.
The generational hypothesis
Most objects die young: request buffers, temporary strings, iterators. Generational collectors exploit this by allocating new objects in a small young generation, collecting it often and cheaply (cost proportional to the few survivors, not to the garbage), and promoting objects that survive several collections to an old generation collected rarely. The JVM's collectors, .NET and V8 are generational. The consequence for code: allocating many short-lived objects is cheap, while objects that live just long enough to be promoted and then die (medium-lived caches, large request batches held across collections) are expensive because they fill the old generation.
Pauses and concurrent collection
A stop-the-world collection pauses every application thread. Early collectors paused for seconds on large heaps. Modern ones do most work concurrently with the program, using barriers in compiled code to track references changed during marking: Java's G1 targets pauses of tens of milliseconds, ZGC and Shenandoah keep pauses under a millisecond regardless of heap size, and Go's collector is concurrent with sub-millisecond pauses. Concurrency is not free: it uses extra CPU and needs headroom, because the program keeps allocating while the collector works.
The three-way trade-off
- 1
Pause time
How long the application stops. Matters for tail latency. - 2
Throughput
The fraction of CPU spent on the application rather than collection. - 3
Footprint
How much memory beyond the live set the collector needs to work efficiently. Less headroom means more frequent collections.
You can usually improve two at the expense of the third. Go exposes it as a single knob (GOGC: how much the heap may grow before a cycle, plus a soft memory limit); the JVM through collector choice and heap sizing. The universal lever is allocation rate: allocating less garbage reduces all three costs at once.
Trade-offs
Reference counting gives prompt, pause-free reclamation and pays per-assignment overhead and leaks cycles. Tracing handles cycles and batches the work, paying in pauses or concurrent CPU and needing heap headroom. Generational collection makes short-lived objects nearly free and punishes medium-lived ones. Manual memory management (C, C++, Rust ownership) avoids collector costs entirely and moves the burden to the programmer and the type system.
In an interview
Tie GC to latency and sizing: "The service is latency-sensitive with a 12 GB heap, so it runs a low-pause collector with about 30 percent headroom over live data; we keep the allocation rate down by reusing buffers on the hot path, and we alert on GC CPU share and pause percentiles. Container memory limits are set above max heap plus off-heap and thread stacks, or the OOM killer gets there before the collector does."
Check yourself
A Java service caches results for 30 seconds in a large map. Its p99 latency shows long pauses, and GC logs show the old generation filling quickly and frequent full collections. Most cached entries are dropped after 30 seconds. What is going on?