Module 07 · File systems and I/O · Lesson 7 of 8
RAID
Why RAID exists and why it is not a backup: levels 0, 1, 5, 6 and 10, parity and the small-write penalty, the write hole, rebuilds at 20 TB, mdadm and hot spares, and how replication and erasure coding took over.
Why it matters
A server loses a disk at 3 a.m.; whether that pages someone depends on how the disks were arranged. An array takes four days to rebuild a 20 TB drive and a second fails on day two. A team mirrors two disks, calls it a backup, and rm -rf is faithfully written to both. Cloud volumes and object stores replicate underneath, so few engineers run RAID now, yet the same arithmetic of stripes, parity and rebuild windows governs those services.
The idea
RAID (redundant array of independent disks) combines several disks into one logical device for some mix of performance (I/O across disks), capacity (one large volume) and availability (surviving a disk failure). It is not a backup: every write, including corruption, deletion and ransomware, reaches every disk.
Striping, mirroring, parity
- 1
Striping
Split data into 64 KB to 1 MB chunks across consecutive disks, so a large transfer uses every disk. - 2
Mirroring
Write every block to two disks; reads come from either, and a failed disk is rebuilt from its twin. - 3
Parity
Per stripe, store one extra chunk, the XOR of the others; any one missing chunk is recomputed from the rest. - 4
Double parity
A second chunk from Reed-Solomon arithmetic recovers any two missing chunks.
The levels
| Level | Min disks | Usable capacity | Reads / writes | Failures tolerated | Rebuild cost |
|---|---|---|---|---|---|
| 0 (stripe) | 2 | All | Fastest both ways | None | None possible |
| 1 (mirror) | 2 | Half | Reads from either; writes twice | 1 per pair | Copy one disk |
| 5 (parity) | 3 | n − 1 disks | Small writes cost four I/Os | 1 | Read every disk; heavy |
| 6 (double parity) | 4 | n − 2 disks | Small writes cost six I/Os | 2 | Heavier still |
| 10 (mirror + stripe) | 4 | Half | Fast both ways | 1 per pair, up to n/2 | Copy one disk |
Parity and the small-write penalty
A full-stripe write computes parity from data in hand and costs one extra chunk. A small write that changes one chunk cannot: the controller reads the old data and old parity, computes new parity as old parity XOR old data XOR new data, then writes both. Four I/Os per logical write, six on RAID 6, two on a mirror. This read-modify-write penalty is why parity RAID was avoided under random-write databases and why controllers cache writes to assemble full stripes.
The write hole
Data and parity are written separately. If power fails between them, parity no longer matches data, and nothing notices until a disk dies and the rebuild quietly reconstructs wrong bytes: the RAID 5 write hole. Hardware controllers close it with a battery-backed write cache that replays unfinished stripes after power returns; mdadm offers a write journal on an SSD (--write-journal). ZFS RAID-Z avoids it by design: copy-on-write with variable-width stripes puts new data and parity in fresh space and flips the block pointers only once both are durable.
Rebuild at 20 TB
A rebuild reads every survivor in full. At 200 MB/s a 20 TB disk takes 100,000 seconds, about 28 hours, at best; production traffic stretches that to days, with the array degraded throughout and, on RAID 5, one failure from loss. The second worry is the unrecoverable read error rate: consumer disks are rated at one per 10^14 bits, about 12.5 TB, and a RAID 5 rebuild on 20 TB disks must read 60 TB or more without one, since a bad sector aborts most rebuilds. Enterprise disks (10^15) and RAID 6 make the odds tolerable; RAID 5 on large disks is now considered unsafe. A hot spare, an idle disk in the chassis, starts the rebuild the moment a disk fails.
Hardware and software
Hardware RAID is a controller card with its own CPU and cache; it presents one device, hides SMART data and ties the array to that controller family. Software RAID on Linux is mdadm (mdadm --create /dev/md0 --level=10 --raid-devices=4, status in /proc/mdstat): host CPU, portable, usual on commodity servers.
What replaced it
Local RAID protects one machine; modern infrastructure moved redundancy up a level. Cloud block volumes are replicated across servers in a zone, so the instance never sees a disk. Distributed stores keep three copies across racks (HDFS, Cassandra) or use erasure coding across a dozen nodes (S3, Ceph), RAID 6 stretched across a data centre at lower overhead. The System Design track's Replication, Erasure coding and Block storage lessons cover those designs.
Trade-offs
Mirrors are simple and quick to rebuild, at half the capacity. Parity levels recover that capacity at the price of slow small writes, a long fragile rebuild and the write hole. RAID 0 is pure speed with every disk's failure rate added together. None protect against deletion, application corruption, a controller bug or a fire; backups and off-site replication do.
In an interview
Separate availability from durability: "RAID 10 keeps the database serving when a disk fails and rebuilds in hours; backups and cross-region replication protect the data." On levels: "With 20 TB disks I would not run RAID 5: the rebuild reads 60 TB and one unrecoverable read error loses the array, so RAID 6 or RAID 10 with a hot spare."
Check yourself
A team proposes RAID 5 across six 20 TB disks to maximise capacity. What is the strongest objection?