Module 09 · Storage systems · Lesson 4 of 4
Erasure coding
How object stores reach eleven nines of durability at half the cost of three copies. Reed-Solomon in plain terms, the k+m trade between overhead and failures tolerated, repair traffic and why it matters, and where replication still wins.
The video walkthrough is part of Pro.
See what Pro includesWhy it matters
Keeping three copies of everything survives two simultaneous failures and costs three times the storage. At exabyte scale that multiplier is the bill. Erasure coding gets equal or better durability for 1.2 to 1.5 times the storage, which is how object stores can charge cents per gigabyte while promising eleven nines. It is the single most important cost lever in large storage systems, and it has a real downside that makes replication still the right choice for hot data.
The idea
Split an object into k data fragments and compute m parity fragments from them with an erasure code such as Reed-Solomon. Store the k+m fragments on different disks, racks or zones. The object can be rebuilt from any k of the k+m fragments, so the system tolerates the loss of any m fragments at once. Storage overhead is (k+m)/k instead of the 3× of triple replication.
Reed-Solomon in plain terms
Treat the k data fragments as k numbers. Choose m extra equations over those numbers such that any k of the k+m equations are independent and can be solved for the original k values. Parity fragments are the results of those equations; recovery is solving the system. The arithmetic runs in a finite field so that bytes stay bytes. The simplest case, k = 2 and m = 1, is RAID 5's XOR parity: lose any one of three fragments and XOR the other two to get it back. Reed-Solomon generalises that to any m.
Choosing k and m
| Scheme | Overhead | Failures tolerated | Typical use |
|---|---|---|---|
| 3× replication | 3.0× | 2 | Hot data, small objects, low-latency reads |
| RS(6,3) | 1.5× | 3 | Hadoop default; general object storage |
| RS(10,4) | 1.4× | 4 | Large cold storage; Facebook-style warm tiers |
| RS(12,4) or wider | 1.33× and below | 4 | Archive; wide stripes need many failure domains |
| Local reconstruction codes | ~1.3× | 3 to 4, with cheaper single-failure repair | Azure Storage; large clouds |
More parity (larger m) tolerates more simultaneous failures; more data fragments (larger k) lowers overhead but widens the stripe, meaning more fragments to fetch per read and more failure domains required. Durability depends on both and on how fast repairs happen relative to how often disks fail.
The real cost: repair and reads
With replication, repairing a lost copy reads one copy and writes one. With RS(6,3), repairing one lost fragment requires reading six fragments from six other nodes and recomputing: six times the network traffic per repair, and a single-disk failure in a large cluster triggers repairs of every stripe it held. Local reconstruction codes exist precisely to make the common single-failure repair cheaper. Reads of a full object also touch k nodes instead of one, adding latency and tail risk, which is why erasure coding suits large, sequentially read or cold objects, and small hot objects stay replicated or cached.
Where it lives in a design
Object stores apply erasure coding across availability zones by default, which is the source of their durability claim. Hadoop and Ceph offer it per directory or pool for cold data. Within a single machine, RAID 5 and 6 are erasure codes across disks. In a design interview it appears when you justify how backups, media or data-lake files are both durable and cheap, and when you explain why the hot tier in front of them uses replication or caching instead.