Checksums and data integrity

Why it matters

Networks drop and corrupt packets, disks rot, memory flips bits, and a bug in one service can truncate a file that six others then copy faithfully. At the scale of petabytes and billions of requests, "rare" events happen daily. A checksum is a few bytes that let the receiver know the data it got is the data that was sent. Every serious storage system, transfer protocol and backup scheme verifies integrity this way, and designs that skip it discover corruption only when a customer does.

The idea

A checksum is a small value computed from a block of data such that a change to the data almost certainly changes the value. The sender computes and attaches it; the receiver recomputes and compares. Equal means intact with high probability; different means corrupted, so retransmit, re-read from a replica or raise an alarm.

Three strengths, three costs

	Simple checksum	CRC	Cryptographic hash
Examples	Internet checksum (IP, TCP, UDP headers)	CRC-32, CRC-32C	SHA-256, BLAKE3, MD5 (legacy)
Size	16 bits	32 bits	128–256 bits
Detects	Random single errors; misses some swaps	Bursts of bit errors reliably	Any change, including deliberate tampering
Speed	Trivial	Very fast, often hardware-accelerated	Fast, but noticeably more CPU
Use	Packet headers	Ethernet frames, ZIP, storage blocks, Kafka messages	Content addressing, downloads, deduplication, signatures

A CRC catches the random errors networks and disks produce but can be forged by anyone who wants to; a cryptographic hash cannot be matched by a different input in practice, which is what makes it suitable for verifying downloads from an untrusted mirror or for addressing content by its hash.

Where you meet them in a design

• Transport: TCP and UDP carry a checksum per segment; Ethernet carries a CRC per frame. Corrupt packets are dropped and TCP retransmits, which is one reason TCP's reliability works.
• Storage: object stores return an ETag or explicit checksum on upload and verify it; file systems such as ZFS checksum every block and repair from a replica on mismatch, which is how they survive silent disk corruption.
• Replication and backup: comparing hashes of blocks or files is how replicas detect divergence cheaply (Merkle trees, covered in Data structures for scale).
• Content addressing: name a blob by its hash and you get deduplication, immutable caching and tamper evidence for free. Git, container registries and CDN asset versioning all rely on it.
• APIs: an upload endpoint that accepts a client-computed hash and rejects a mismatch turns a silent corruption into a 400.

Trade-offs

Checksums cost CPU and a few bytes per block, both negligible against the data they protect, which is why the only real trade-off is strength versus speed. CRCs are cheap enough to run everywhere; cryptographic hashes are reserved for where tampering or deduplication matters. The practical failure is not cost but omission: pipelines that copy data without verifying it end to end.

In an interview

It is a detail, so spend it where it counts. In a file-upload or storage design: "The client sends a SHA-256 with the upload; the object store verifies it and we store the hash as the object's key, which gives us deduplication and lets the CDN cache immutably." In a replication design, mention comparing block hashes to detect divergence. One sentence, placed well, signals you have run data at scale.