I/O scheduling and buffering

Why it matters

Disk-heavy services rarely fail because the device is slow on paper; they fail because of how I/O is shaped on the way to it. A service writing logs one line at a time with a sync after each, a backup job flooding the device so request latency triples, a database fighting the page cache for memory, a write burst stalling every thread when dirty pages hit a limit: all of these are buffering and scheduling problems, and the fixes are configuration and code shape rather than faster hardware.

The idea

Between your program and the storage device sit several layers that buffer, merge, reorder and prefetch I/O.

Buffering at each layer

User-space buffers collect many small writes into one system call; writing a log line per write() call costs a system call each, while a buffered writer issues one per few kilobytes. The page cache holds dirty pages and writes them back in the background, coalescing many writes to the same page and batching neighbouring pages. Linux starts background writeback when dirty pages exceed a fraction of memory (dirty_background_ratio) and makes writers block when they exceed a higher one (dirty_ratio), which is the source of sudden multi-second write stalls on machines with huge RAM and slow disks.

Read-ahead

When the kernel sees sequential reads, it reads ahead, fetching the next pages before they are requested so the program rarely waits. Read-ahead makes sequential scans fast and wastes bandwidth on random access patterns, which is why databases and mmap users can hint their pattern (posix_fadvise, madvise with sequential or random).

Sequential versus random

Device	Sequential throughput	Random 4 KB reads	Ratio
Hard disk	~150–250 MB/s	~100–200 per second (~1 MB/s)	Random is about 100–200× slower
SATA SSD	~500 MB/s	Tens of thousands per second	Several times slower
NVMe SSD	3–7 GB/s or more	Hundreds of thousands to a million per second at high queue depth	Small gap, if enough I/O is in flight
Network block volume	Provisioned MB/s	Provisioned IOPS	Bounded by the purchased limits

Much storage-system design (LSM trees, log-structured file systems, Kafka's append-only logs) is about turning random writes into sequential ones. On NVMe the remaining lesson is parallelism: the device needs many requests in flight (queue depth) to reach its rated IOPS, which single-threaded synchronous I/O never provides.

The block layer and I/O schedulers

The block layer merges adjacent requests and decides their order. Modern Linux uses multi-queue block I/O with per-core queues and simple schedulers: none (pass through, typical for NVMe, which schedules internally), mq-deadline (bounds how long reads and writes wait, good for SATA SSDs and disks), BFQ (fair sharing between processes, good on desktops) and Kyber (latency targets). cgroup I/O controls (io.max, io.weight) limit and share bandwidth between containers, the I/O counterpart of CPU limits.

Direct I/O

Databases that manage their own cache (InnoDB's buffer pool, Oracle, ScyllaDB) often bypass the page cache with direct I/O (O_DIRECT), avoiding double caching and giving them control over when data hits the device. In exchange they must align buffers and sizes to the device's block size and do their own read-ahead and write batching. Postgres, by contrast, deliberately relies on the page cache. io_uring adds asynchronous, batched submission for both buffered and direct I/O.

Measuring

iostat -x shows per-device throughput, IOPS, average request size, queue size, await (time per request including queueing) and utilisation. High await with low utilisation on NVMe often means not enough parallelism; high await with full queues means the device or its provisioned limit is saturated. Per-process I/O and pressure stall information for I/O complete the picture.

Trade-offs

Buffering and write-back batch small writes into efficient large ones, at the cost of durability (until fsync) and of stalls when dirty limits are hit. Read-ahead speeds sequential access and wastes bandwidth on random access. Direct I/O gives databases control and removes double caching, at the cost of doing the kernel's job yourself.

In an interview

Shape I/O deliberately: "The log shipper buffers lines and writes 64 KB blocks, compresses before sending, and fsyncs on a timer rather than per line. The backup job runs in a cgroup with an I/O limit so it cannot push the database's write latency up. On NVMe the import uses io_uring with 64 requests in flight, because one synchronous reader leaves most of the device idle."