Journaling and crash consistency

Why it matters

Durability bugs are invisible until the power fails. A system that "saved" a file can come back with the old version, an empty file, or garbage, depending on how the write was done. Databases spend enormous care on this, and so must any application that writes its own state to disk: configuration agents, queues, embedded stores, anything with a local file it trusts. The mechanisms here are also the origin of the write-ahead logs you met in the Databases module.

The idea

Writing a file touches several on-disk structures: the data blocks, the inode (size, pointers), the free-space bitmap, possibly the directory. The disk writes them independently and in any order, and a crash can stop it after any subset. Without care, the file system ends up inconsistent: an inode pointing to blocks still marked free, or blocks marked used that no inode owns.

Journaling

A journaling file system writes a description of the changes to a journal (a log area) first, marks the transaction committed, and only then applies the changes to their real locations (checkpointing). After a crash, recovery replays committed transactions from the journal and ignores incomplete ones, in seconds, instead of scanning the whole disk with fsck.

Copy-on-write file systems

ZFS and btrfs never overwrite live blocks. Changes are written to new blocks, then the tree of pointers above them is updated, ending in an atomic switch of the root pointer. A crash leaves either the old tree or the new one, which gives consistency without a journal and makes snapshots nearly free. The cost is fragmentation of files that are rewritten in place, such as databases, which is why such workloads often disable copy-on-write for their files.

What fsync promises

A successful write puts data into the page cache. It reaches the disk later, and a crash before then loses it. fsync(fd) blocks until the file's data and metadata are on stable storage, including flushing the device's own volatile write cache. fdatasync skips metadata that does not affect reading the data back (such as modification time), so it is a little cheaper. Subtleties that have bitten real systems:

1. 1

   New files need their directory synced

   Creating or renaming a file changes the directory; fsync the directory too, or the file may vanish after a crash.
2. 2

   fsync errors are sticky and lossy

   If writeback fails, the kernel may report the error once and then mark the pages clean. Retrying fsync can then report success while the data is lost. Postgres changed its behaviour in 2018 to treat an fsync failure as fatal and recover from its log.
3. 3

   Devices can lie

   Consumer drives with volatile caches and no power-loss protection may acknowledge flushes early; servers use drives or controllers with capacitors, or disable the write cache.
4. 4

   It is slow

   An fsync waits for the device: tens of microseconds on enterprise NVMe, milliseconds on a disk or network volume. Databases batch many commits into one fsync (group commit).

Crash-safe patterns for applications

To replace a file: write a temp file, fsync it, rename it over the target, fsync the directory. To append records: write each record with a checksum and length, fsync at commit points, and on recovery truncate any torn tail whose checksum fails. To update structured state: use a write-ahead log of intended changes, fsync it, then apply the changes, and replay the log on start. Or let a database do all of this, which is usually the right answer.

Trade-offs

Journaling makes recovery fast and consistent at the cost of extra writes; full data journaling doubles data writes. Copy-on-write avoids the journal and enables snapshots at the cost of fragmentation. fsync turns a fast cached write into a slow durable one, and batching restores throughput at the cost of slightly later acknowledgement.

In an interview

Say exactly when data is durable: "The queue appends each message with a checksum and acknowledges the producer only after fsync, batching acknowledgements every 2 ms so one fsync covers hundreds of messages. On restart it scans to the last valid checksum and truncates any torn tail." That shows you know a write is not a commit.