The CPU and the instruction cycle

Why it matters

A modern CPU core can retire several instructions every nanosecond, yet real programs often run at a small fraction of that because the core spends most of its time waiting: for memory, for a mispredicted branch, for another core. Understanding what the core is trying to do lets you read profiler output, explain why one loop is ten times faster than an equivalent one, and know when adding cores helps and when it does not.

The idea

A CPU core runs a loop forever. It reads the next instruction from memory at the address held in the program counter, works out what the instruction means, carries it out, stores the result, and moves the program counter on. Everything a computer does is built from that loop.

The parts that matter

Registers are a handful of tiny storage slots inside the core; arithmetic happens between registers, and everything else must be loaded into them first. The clock ticks a few billion times a second (3 GHz means one tick every third of a nanosecond), and each stage of work takes one or more ticks. The arithmetic logic unit does the actual adding, comparing and bit-twiddling. Load and store units move data between registers and memory, which, as the next lesson shows, is where most of the waiting happens.

How cores go faster than one instruction per tick

1. 1

   Pipelining

   Like an assembly line, each stage works on a different instruction at once. A 15-stage pipeline has up to 15 instructions in flight, so throughput approaches one instruction per stage per tick.
2. 2

   Superscalar execution

   Several execution units run in parallel, so a core can complete four or more independent instructions per tick when the code allows it.
3. 3

   Out-of-order execution

   If an instruction is waiting on memory, the core runs later, independent instructions in the meantime and puts results back in order. It hides some memory latency automatically.
4. 4

   Branch prediction

   At every if and loop the core guesses which way it will go and keeps the pipeline full along the guess. Right guesses are free; a wrong guess throws away the speculative work, costing around 15 to 20 ticks.

Multiple cores and threads

Clock speeds stopped rising around 2005 because of power and heat, so chips added cores instead. Each core runs its own instruction stream; a program only gets faster on more cores if it has independent work to spread across them, which is why concurrency and parallelism became everyday concerns. Simultaneous multithreading (hyperthreading) lets one physical core interleave two instruction streams, filling idle execution units; it typically adds tens of percent, not double.

What this means for your code

A profiler's "CPU time" includes time the core spends stalled on memory, so high CPU does not mean the core is computing. Code with predictable branches and sequential memory access runs at many instructions per tick; code that branches randomly or jumps around memory runs at a fraction of one. Most speedups in hot loops come from helping the core keep its pipeline full, not from clever arithmetic.

Trade-offs

Speculation and out-of-order execution make ordinary code fast without effort, at the cost of unpredictable performance cliffs (mispredictions, stalls) and, famously, security issues where speculative work leaks information (Spectre and Meltdown). More cores raise total throughput but not single-thread speed, which pushes complexity into software as concurrency.

In an interview

Use it to explain CPU-bound behaviour: "This service is single-threaded in the hot path, so adding cores will not reduce latency; we either parallelise the work or reduce it." Or to explain a surprising benchmark: "Sorting the input made the loop faster because the branch became predictable, so the pipeline stopped flushing."