Key Components of a DFS

The Moving Parts Behind a DFS

A distributed file system is not one program — it is a small set of cooperating roles that each solve one piece of the "many machines, one filesystem" problem. This lesson goes under the hood of those roles using HDFS as the reference implementation, because its design (and its public numbers) generalize cleanly to GFS, Colossus, and most block-based DFS designs you'll meet in an interview or in production.

• NameNode (metadata server) — the single source of truth for the namespace: which files exist, which blocks make up each file, and which DataNodes hold each block. It never stores file bytes, only metadata, and keeps almost all of it in memory for speed.
• DataNode (storage server) — stores the actual block contents on local disk, serves read/write requests from clients, and reports block state back to the NameNode.
• Block — the unit of storage and replication. Files are split into fixed-size blocks (128MB is the common HDFS default) so that a single huge file can be spread — and replicated — across many machines instead of living on one disk.
• Client library — talks to the NameNode to resolve a file into its block locations, then talks directly to the relevant DataNodes to read or write bytes. The NameNode is never on the data path, which is what lets it stay a scalable single point of coordination instead of becoming a throughput bottleneck.

The rest of this lesson traces how these four pieces cooperate to place replicas, write a block, and detect failure — and then closes with the one judgment call every DFS design forces on you: replication or erasure coding.

Placement: Why 1 Local Rack + 1 Remote Rack

With the default replication factor of 3, HDFS's block placement policy does not scatter replicas randomly — it follows a fixed rule that balances two competing risks: losing a whole rack (top-of-rack switch or power failure) and paying too much cross-rack network cost on every write.

1. Replica 1 goes on the writer's own node (or a random node if the client is off-cluster) — this is the "local" copy.
2. Replica 2 goes on a node in a different rack — this guarantees survival of a full rack failure.
3. Replica 3 goes on a different node in that same remote rack — this keeps only one rack-to-rack hop on the write path instead of two, saving cross-datacenter-switch bandwidth, while still tolerating any single node failure.

The net effect: a full rack outage can take out at most one of the three replicas, and only one of the three write hops crosses a rack boundary — read bandwidth and fault tolerance are bought without paying for two full cross-rack copies on every block write.

Metadata: Why the NameNode Lives (Almost Entirely) in Memory

Every file, directory, and block in the namespace is an object the NameNode must track: its name, permissions, block list, and replica locations. Because the client's every read and write starts with a metadata lookup, this has to be fast — so the NameNode keeps the entire namespace in memory and only uses disk for durability (the edit log and periodic checkpoints via the fsimage).

A commonly cited operational rule of thumb in the Hadoop community is that each namespace object — a file, a block, or a directory — costs roughly on the order of 150 bytes of NameNode heap, which is why cluster operators often plan for roughly 1GB of heap per ~1 million objects tracked. Treat both figures as a back-of-envelope sizing heuristic, not a documented guarantee: actual per-object overhead varies by Hadoop version, JVM and object-header layout, and — most importantly — by the mix of files, blocks, and directories in your namespace, since a directory entry, a file inode, and a block record don't cost the same number of bytes. Always validate against `jmap`/heap-dump numbers on your own NameNode before sizing a production cluster from this rule alone.

This in-memory design is also why "lots of small files" is the classic HDFS anti-pattern: a million 10KB files consumes the same namespace-object budget as a handful of 128MB-block files holding far more actual data, so small-file workloads exhaust NameNode heap long before they exhaust DataNode disk.

The Write Path: Lease + Pipeline

Writing a block is coordinated through two mechanisms working together: a lease that gives one client exclusive write access to a file, and a replication pipeline that streams the data to all replicas in one pass instead of the client uploading to each replica separately.

Lease acquisition

Before writing, the client asks the NameNode for a lease on the file — this is what prevents two clients from writing the same file concurrently and corrupting it. The NameNode tracks two timeouts on that lease:

• Soft limit — 60 seconds. If the client goes silent for this long, another client is allowed to preempt the lease and trigger recovery, on the assumption the original writer may just be slow.
• Hard limit — 60 minutes. If no one has recovered the lease by this point, the NameNode itself force-closes the file and reclaims it, so a dead writer can never block the file indefinitely.

Pipeline write and packet chunking

Once the client has the lease and the NameNode has assigned it a set of DataNodes for the next block (placed using the rack-aware rule above), the client does not send the block to each replica independently. Instead it opens a pipeline: client → DataNode 1 → DataNode 2 → DataNode 3. Data streams down the pipeline in 64KB packets, and each DataNode forwards a packet to the next hop as soon as it has buffered it, rather than waiting for the whole block. Acknowledgements flow back up the same pipeline once all three DataNodes have persisted a packet.

Concretely, for the default 128MB block: 134,217,728 bytes ÷ 65,536 bytes per packet = 2,048 packets streamed through the pipeline per block. This pipelined, packet-level design is what lets HDFS overlap network transfer across all three replicas instead of serializing three full-block transfers, and it's why a slow or dead node mid-pipeline can be detected and spliced out packet-by-packet rather than only after an entire block transfer times out.

Failure Detection: Heartbeats and the 10.5-Minute Rule

Every DataNode sends a heartbeat to the NameNode on a fixed interval (3 seconds by default) to prove it's alive and to report block state. The NameNode doesn't declare a node dead the moment one heartbeat is missed — that would make the cluster wildly over-sensitive to a single slow GC pause or network blip. Instead it uses a two-part formula:

dead-node timeout = 2 × heartbeat-recheck-interval + 10 × heartbeat-interval

With the defaults — a 5-minute recheck interval and a 3-second heartbeat interval — that's 2 × 300s + 10 × 3s = 600s + 30s = 630 seconds, i.e. 10.5 minutes, before a silent DataNode is marked dead and the NameNode starts scheduling re-replication of the blocks it was holding. This is a deliberately conservative window: false-positively declaring a healthy node dead triggers unnecessary network-wide re-replication traffic, so the system trades a slower failure-detection reaction for avoiding that self-inflicted load spike.

When to Use It, and When Not: Replication vs. Erasure Coding

3x replication is the default because it's the simplest way to buy durability, but it isn't free: storing 3 copies of every block means 200% storage overhead — 3 units of disk for every 1 unit of data. Erasure coding (EC) is the named alternative that trades some of that storage cost for CPU and read-path complexity, and knowing when to reach for which one is the actual judgment call this topic is testing.

HDFS's EC implementation (e.g. the default Reed-Solomon RS-6-3 scheme) splits data into 6 data blocks and computes 3 parity blocks, so 9 physical blocks store 6 blocks worth of data — a 1.5x storage multiplier, i.e. only 50% overhead, versus 200% for triplication. That's a real, large storage win. The cost shows up on the read and recovery paths:

Prefer 3x replication when: the data is hot (frequently read), files are small, workloads depend on data locality for compute scheduling, or reads need consistently low latency without risking a decode penalty — i.e. most "live" operational data in the cluster.

Prefer erasure coding when: the data is cold or archival (write-once, rarely read), files are large, and the 150-percentage-point storage saving (200% → 50% overhead) matters more than occasional decode latency or CPU cost on the rare reconstruction read — i.e. long-term retention tiers, compliance archives, backups.

Avoid erasure coding when: you have many small files (parity overhead per stripe stops paying for itself, and it multiplies the small-file NameNode-object problem described earlier), the workload is latency-sensitive random access, the cluster's compute layer relies on block-local scheduling, or the cluster doesn't have CPU/network headroom to absorb decode-on-read and background re-encoding traffic. In practice, production HDFS clusters run both side by side: replication for the hot working set, EC policies applied to directories once data cools off — the choice is a per-directory storage policy, not an all-or-nothing cluster setting.

NameNode High Availability and namespace scale

A single NameNode is the metadata authority, but production HDFS does not leave it as an unrecoverable single machine. NameNode High Availability runs two NameNodes in Active/Standby states. The Active serves clients and writes namespace edits; the Standby continuously reads those edits and applies block reports so it can take over with the same namespace image. ZooKeeper-based failover coordination ensures only one NameNode is Active at a time, avoiding split brain.

The shared edit log is commonly implemented with Quorum Journal Manager (QJM). The Active writes each namespace edit to a quorum of JournalNodes; the Standby tails those JournalNodes. Because a majority has each committed edit, failover can promote the Standby without losing acknowledged metadata updates. This protects metadata availability; it does not put the NameNode on the data path, so DataNodes still serve block bytes directly to clients.

HDFS Federation solves a different problem: metadata scale, not failover. Instead of one namespace, multiple independent NameNodes each own a namespace volume, while the same DataNodes store blocks for all of them. A path or mount table routes clients to the right namespace. Federation partitions metadata across NameNodes, so one huge cluster can scale past the heap and inode limits of a single namespace while sharing the storage fleet.

Sources

Facts, defaults, and formulas above are drawn from: Apache Hadoop HDFS Architecture Guide (hadoop.apache.org/docs — HDFS Architecture); Apache Hadoop HDFS Erasure Coding documentation (HDFS-EC design doc and admin guide, RS-6-3 default policy); Ghemawat, Gobioff & Leung, The Google File System (SOSP 2003), for the general replica-placement and chunk-pipeline design HDFS derives from; and Tom White, Hadoop: The Definitive Guide, for the NameNode heap sizing rule of thumb and lease/heartbeat operational defaults. NameNode per-object memory figures are community operational heuristics rather than values guaranteed by the Apache documentation and should be validated against actual heap usage before capacity planning.

When NOT to custom-build DFS components

• Object storage API fits — use S3-compatible rather than namenode+datanode yourself.

Interviewer follow-ups & drills

1. Why separate metadata service? Consistency and small ops vs large sequential data plane.
2. Ops: master failover, under-replicated blocks, rebalance bandwidth.
3. Drill: client write path — pipeline to datanodes, ack when RF satisfied.