Distributed file systems

Why it matters

Before object storage became the default, the problem of storing petabytes across commodity machines was solved by distributed file systems, and the design that Google published and Hadoop reimplemented still shapes how every large storage system is built: separate metadata from data, chunk files into large blocks, replicate blocks across failure domains, and optimise for the access pattern you actually have. Understanding it explains the trade-offs in object stores, data lakes and streaming logs.

The idea

A distributed file system presents a single file namespace across a cluster. Files are split into large chunks (64 to 256 MB), each stored on several chunk servers (DataNodes), while a metadata server (the NameNode or master) tracks which chunks make up each file and where each replica lives. Clients ask the metadata server where a chunk is, then read or write the data directly from the chunk servers, so the master is never in the data path.

Design decisions and why

1. 1

   Large chunks

   A 128 MB chunk means a petabyte is only eight million chunks, small enough for one master to hold the map in memory, and means sequential reads stream for a long time between metadata lookups. It also means tiny files waste space and metadata, the small-files problem.
2. 2

   Metadata out of the data path

   The master answers "where", clients and chunk servers move the bytes. Throughput scales with the number of chunk servers; the master handles only small messages.
3. 3

   Replication across failure domains

   Three replicas by default, placed on different machines and at least two racks, so a rack switch failure loses no data. Rack awareness also keeps most read traffic within a rack.
4. 4

   Append-only, write-once files

   Random writes are not supported; files are written sequentially and then read many times. This matches batch analytics and logs and avoids distributed consistency problems for concurrent writers.
5. 5

   Self-healing

   Chunk servers heartbeat and report their blocks; when one dies, the master schedules re-replication of its chunks from surviving replicas until the replication factor is restored.

Reads, writes and consistency

A write pipelines data through the replicas in a chain and commits once all acknowledge, so a successful write is on three nodes. Concurrent appends from multiple clients are supported in GFS with relaxed, at-least-once semantics (duplicates and padding are possible and readers must tolerate them); HDFS allows a single writer per file. Reads choose the nearest replica. The model is strongly consistent for metadata (one master) and simple for data because files do not change after close.

The weaknesses that shaped successors

The single master is a bottleneck for metadata operations and a single point of failure, mitigated by standby masters and, later, federation across namespaces. Small files overwhelm the master's memory, which pushed teams to pack them into larger containers. Fixed three-way replication costs three times the storage, which erasure coding later reduced. And a POSIX-like namespace turned out to be unnecessary for most workloads, which is why object storage, with a flat key space, a simpler consistency model and HTTP access, replaced distributed file systems as the default for everything except the tightly coupled compute clusters they were built for.