Context switching

Why it matters

A machine with 8 cores routinely runs thousands of threads. The illusion that they all run at once is produced by context switches, hundreds of thousands per second on a busy server. Each switch is cheap, but not free, and the hidden costs dominate at high rates. When a thread-per-request server falls over under load with CPUs pegged and little useful work done, context switching is usually part of the story.

The idea

A context switch stops one thread on a core and starts another. The kernel saves the outgoing thread's context (registers, program counter, stack pointer, and some CPU state) into its kernel record, picks the next thread to run, restores that thread's saved context, and returns to user mode where the new thread continues exactly where it left off.

Why switches happen

High voluntary switch counts mean threads are blocking a lot (I/O-heavy work). High involuntary counts mean more runnable threads than cores, fighting over CPU time. Linux exposes both per process (/proc/PID/status) and system-wide (vmstat's cs column).

What a switch really costs

The direct cost, saving and restoring state and running the scheduler, is around one to a few microseconds. Switching between threads of different processes also changes the address space: the CPU loads new page tables, and on many processors that invalidates TLB entries, so the new thread's first memory accesses are slower. The bigger indirect cost is that the new thread finds the CPU caches full of the previous thread's data, so it runs from main memory until its working set is reloaded. For work with good cache locality, that warm-up can cost far more than the switch itself.

Too many threads

1. 1

   Memory

   Every thread has a stack, often reserving 1 to 8 MB and committing what it touches; 10,000 threads can mean gigabytes.
2. 2

   Scheduling overhead

   More runnable threads than cores means more involuntary switches and more time spent in the scheduler.
3. 3

   Cache thrashing

   Threads evict each other's cached data, so each runs slower than it would alone.
4. 4

   Lock contention

   More threads chasing the same locks means more blocking, which means more voluntary switches.

This is why thread pools are sized deliberately. For CPU-bound work, about one thread per core is optimal; more only adds switching. For work that blocks, more threads keep the cores busy while others wait, roughly cores multiplied by one plus the ratio of wait time to compute time. And for very high concurrency with mostly idle connections, event loops and lightweight threads avoid kernel switches altogether by switching tasks in user space, which costs nanoseconds rather than microseconds.

Trade-offs

Preemptive switching gives fairness and responsiveness at the cost of switch overhead and cache disruption. Longer time slices reduce switching for batch work and hurt interactive latency; shorter slices do the reverse. User-space task switching is far cheaper but relies on tasks yielding at await points, so one CPU-hungry task can stall its neighbours.

In an interview

Use it to size pools and justify async: "The CPU-bound encoding pool has one thread per core; the I/O-bound request pool is larger because threads spend most of their time waiting on downstream calls. Beyond a few thousand concurrent connections we switch to an event loop, since context switching and stack memory start to dominate."