Threading models

Why it matters

The threading model decides what a blocking call costs, how many concurrent tasks one process can hold, and how many cores it can use. It is why a Java servlet container of 2015 topped out at a few thousand connections while nginx held a hundred thousand, why GOMAXPROCS matters under a container CPU limit, and why every thread pool needs a size.

The idea

A kernel-level thread is one the kernel knows about: it has its own task_struct (The process control block lesson) and the scheduler places it on a core. A user-level thread is managed by a library inside the process: its own stack and saved registers, switched in a few dozen instructions, invisible to the kernel. The threading model is the mapping between the threads a program creates and the kernel threads the scheduler runs.

Why a blocking system call is the crux

The kernel does not know user-level threads exist. When one calls read on a socket with no data, the kernel sleeps the only thing it can see, the kernel thread, and every user-level thread stacked on it stops too. Every M:N runtime is a set of tricks for getting round that. Go keeps GOMAXPROCS logical processors (P), each bound to one kernel thread (M) at a time, and runs goroutines (G) on them. Network I/O never blocks an M: the runtime registers the socket with epoll and parks the goroutine until the netpoller says it is ready. For calls it cannot convert, such as file I/O and cgo, the goroutine blocks its M, and after about 20 µs the monitor thread hands the P to another M. Java virtual threads do the same with carrier threads: a socket read or Object.wait unmounts the virtual thread, while file I/O and, before JDK 24, synchronized blocks pin the carrier, and the pool compensates by adding carriers. Erlang's BEAM runs one scheduler per core and makes all I/O asynchronous inside the virtual machine.

Scheduler activations

The 1990s answer was to make the kernel tell the runtime. With scheduler activations the kernel gives the process a set of virtual processors and sends an upcall whenever one of its threads blocks or wakes, so the runtime can put another user thread on the freed processor. Solaris and NetBSD shipped it and both removed it, in 2002 and 2009: the upcalls were hard to get right alongside signals, debuggers and fork, and kernel threads had become cheap. Linux never adopted it.

Why Linux is one-to-one

The NPTL library of 2003 replaced LinuxThreads with a strict one thread, one kernel task design, made cheap by clone sharing the address space and by futex keeping uncontended locks out of the kernel. One scheduler sees everything, so signals, blocking, nice values and affinity behave uniformly, and gdb and perf see every thread. The cost is a kernel stack and a kernel switch per thread: ten thousand platform threads are workable, a million are not.

Thread pools

1. 1

   Pool, do not spawn

   A fixed set of kernel threads pulls tasks from a queue, amortising creation and bounding concurrency; a thread per request lets load set the thread count.
2. 2

   Size with Little's law

   Tasks in flight equal arrival rate times time per task: at 400 requests per second and 80 ms each, about 32 are in progress, so a pool of 8 queues and 200 idles. CPU-bound work wants the core count; waiting work wants cores times one plus the wait-to-compute ratio, from the Context switching lesson.
3. 3

   Steal work

   Each worker owns a deque and idle workers steal from busy ones, as Go, Java's ForkJoinPool and Tokio do; it removes contention on one shared queue and keeps related tasks on a warm cache.

Trade-offs

One-to-one is uniform and observable, and every thread costs kernel memory and a kernel switch, so it holds thousands, not millions. M:N holds millions with nanosecond switches, at the price of two schedulers that cannot see each other: a loop with no yield point can hog a carrier, and an opaque blocking call such as a native library or a pinned synchronized block takes a carrier out of service. Many-to-one is simple and, on a multi-core machine doing I/O, wrong.

In an interview

Tie the model to the blocking profile: "The gateway holds 50,000 mostly idle WebSocket connections. A platform thread each would mean 50,000 kernel stacks and a scheduler busy switching; on virtual threads a parked connection costs a few kilobytes and no kernel thread." For pools: "The pool is 32 threads on 8 cores because each task spends three-quarters of its time waiting on the database; Little's law gives the same 32."