Distributed File System — Durability Arithmetic, Erasure Coding, Hot Blocks & NameNode Split-Brain (Deep Dive)

Durability Arithmetic — Why Replication Factor 3

A DFS is durable because losing a block requires every replica of it to die before the system repairs the damage — so the real question is never "is replication factor 3 enough," it is "how many independent replicas do I need so that a single failure never leaves me at zero spare copies while I repair." That framing is why 3, specifically, is the number almost every large-scale DFS (HDFS, GFS, Colossus) converged on, and it is worth deriving instead of memorizing.

Worked example: the naive independence model

Start from a per-disk annualized failure rate (AFR). Industry drive-reliability studies (Backblaze's annual drive-stats reports are the most widely cited public source) put typical AFR in the 1–2% per year range depending on drive age and model, so take p = 0.02 (2%/yr) as a round, illustrative number — plug in your fleet's real AFR and the arithmetic below is unchanged.

Treat each replica's annual failure as an independent Bernoulli event with probability p, and ask: what is the chance a specific block, replicated RF times, loses all its copies sometime in the same year (i.e., no repair happens in between — a deliberately pessimistic, "worst case" simplification)?

Going from RF=2 to RF=3 divides the annual loss probability by 50× for one extra full copy (a 50% storage increase). Going from RF=3 to RF=5 divides it by a further 2,500× — a much bigger jump in the raw math, for a 67% storage increase on top of RF=3. So the naive math alone does not fully explain why the industry stops at 3 rather than pushing to 5: at RF=3 the per-block loss probability (~5 nines) is already far below the loss rates contributed by correlated failures, software bugs, and operator error — the dominant real-world causes of data loss — so paying for RF=5 buys durability the rest of the stack cannot actually deliver on. The deciding argument for exactly 3 is operational, not purely probabilistic — see below.

The real reason it's 3: never zero redundancy during repair

The naive year-long model above is pessimistic in the wrong direction — it assumes failures accumulate with no repair in between, which is not how a healthy cluster behaves. A more realistic model accounts for the repair window: the time between a failure being detected and the block being re-replicated back up to full redundancy. Redo the RF=3 calculation assuming a T = 6 hour repair window (illustrative — detection plus network copy time at scale):

P(loss) ≈ (3p) × (2p·T⁄8760h) × (1p·T⁄8760h) — first, any of the 3 replicas fails (rate 3p); then, within the repair window, a second of the remaining 2 fails; then, within a further repair window, the last one fails.

With p = 0.02 and T = 6h: 3p = 0.06, 2p·(6/8760) = 2.74 × 10⁻⁵, 1p·(6/8760) = 1.37 × 10⁻⁵. Multiplying: P(loss) ≈ 2.3 × 10⁻¹¹ — roughly 10.6 nines of annual durability, not 5. The gap between the two models (5 nines vs. 10.6 nines) is entirely explained by repair speed: the faster a cluster detects a dead replica and re-replicates, the shorter the window in which a second, independent failure can turn into data loss. This is exactly why the block-report heartbeat timeout (the ~10.5-minute dead-node rule covered on the Key Components page) is not just an availability knob — it is a durability knob.

This gives the real, operational reason RF=3 (not 2, not 5) is standard: RF=2 means that the instant one replica fails, you are down to a single copy — zero spare redundancy — for the entire repair window. Any second failure in that window, however small the probability, is unrecoverable, and repair windows are exactly when correlated risk is highest (the same maintenance operation, software rollout, or power event that caused the first failure is often still in progress). RF=3 guarantees that even during a repair (down to 2 copies), you still have one full spare — you are never one bad disk away from data loss. RF=5 buys the same property with a bigger cushion, but the cushion beyond "1 spare during repair" mostly protects against correlated-failure scenarios that redundancy count alone cannot fix anyway (see next section) — so the marginal storage cost stops being worth it past 3.

The independence assumption breaks: correlated failure is the real risk

Every calculation above assumes replica failures are statistically independent. In practice they often are not: disks from the same manufacturing batch share defect rates and fail in clusters as they age; machines in the same rack share a single power feed and top-of-rack switch, so one power-supply or switch failure takes out every node in that rack at once, not one disk. An "independent 2% AFR" model says nothing about the risk of 20 disks dying in the same 30 seconds because someone tripped a breaker.

This is exactly why replicas are placed rack-aware rather than randomly (the placement rule traced on the "What is a DFS" and "Key Components" pages: one replica on the writer's rack, the other two on a different, shared remote rack). It is worth being precise about what that rule does and does not buy you: a single rack failure takes out at most the replicas that live in that one rack. Under the standard HDFS placement (1 local rack, 2 on one remote rack), a failure of the remote rack takes out two of the three replicas simultaneously — the block survives (one copy remains on the local rack), but redundancy silently drops to RF=1 for every block that had its "2" and "3" replica in that rack, until re-replication catches up. A rack event is not "no big deal" — it burns through most of a block's safety margin in one shot. Rack-awareness guarantees you never lose all copies to one rack event; it does not make rack failures cheap. The only real fixes for correlated risk are: spreading replicas across independent failure domains (rack, power feed, and — at larger scale — data-center or availability zone), diversifying hardware batches/vintages across those domains, and repairing fast so a correlated event's damage doesn't have time to compound with an unrelated, independent failure.

Erasure Coding vs. Replication — More Durability, Half the Storage

Erasure coding (EC) protects the same data with math instead of full copies: split the data into k equal blocks, compute m additional parity blocks from them (Reed-Solomon is the standard algorithm), and store all k+m blocks — any k of the k+m blocks, in any combination, is enough to reconstruct the original data. HDFS's default EC policy is RS(6,3): 6 data blocks + 3 parity blocks = 9 physical blocks holding 6 blocks' worth of data.

Worked example: comparing durability fairly (same data, same failure model)

To compare EC against replication honestly, hold the underlying data constant: 6 data units. Under 3x replication, that is 6 separate triplication groups — 18 physical blocks, 3× storage (200% overhead) — and the group is lost if any single group loses all 3 of its copies. Under RS(6,3), the same 6 units live in one 9-block stripe — 1.5× storage (50% overhead) — and the stripe is lost only if 4 or more of its 9 blocks are unavailable at once (losing any 3 is still fully recoverable).

Using the same independent per-disk AFR model as above (p = 0.02, no repair-window credit, for a fair comparison against the RF numbers already derived):

Under this model, RS(6,3) is roughly 2.6× less likely to lose data than replication for the identical 6 units of data — while using half the storage (1.5× vs. 3×). This is the genuinely surprising part of EC: it is not simply "replication, but cheaper and a bit less safe" — done right, it can beat replication on both axes at once, because spreading redundancy across 9 independently-failing blocks (need 4+ simultaneous losses) is harder to trigger by chance than concentrating it in groups of 3 (need only 3 simultaneous losses per group, and there are 6 chances for some group to hit that).

Where EC pays it back: reconstruction cost

The durability and storage numbers look like a free lunch, so the catch has to be somewhere — and it is on the read/repair path, not the failure math. To serve a read of a failed block, or to rebuild a lost block after a DataNode dies, EC must fetch k = 6 surviving blocks across the network and run a Reed-Solomon (Galois-field) decode to recompute the missing one. Replication's equivalent operation is: copy 1 intact block from 1 surviving replica. So, per lost block, EC reconstruction costs roughly 6× the network I/O of a replication repair, plus real CPU time for the decode — that CPU and network cost was negligible for replication (zero encode/decode, one straight copy). EC also erases data locality: a replicated block sits whole on one node, so compute (Spark/MapReduce) can schedule a task right next to its data; an EC-striped block is scattered across 9 nodes, so no single node holds a complete copy and every read of a degraded block is a fan-out network operation.

This is why EC is the right tool for cold data (write-once, rarely read, rarely if ever failing-and-being-read-mid-failure) and the wrong tool for hot data: on hot, frequently-read, frequently-updated data, you pay EC's reconstruction tax constantly (every degraded read, and any in-place mutation requires recomputing parity across the whole stripe), while on cold archival data you almost never trigger reconstruction at all, so the tax is rarely charged and the 50%-vs-200% storage saving dominates.

The Hot-Block Read Bottleneck

Replication factor sets a hard ceiling on how many machines can serve one block: with RF=3, at most 3 DataNodes hold a copy of that block, so at most 3 DataNodes' worth of read throughput can ever serve it — no matter how large the surrounding cluster is. A "hot block" — a viral file, a heavily-joined small dimension table, a popular video segment — runs into this ceiling directly, and adding more unrelated cluster capacity does nothing for it, because the other 9,997 DataNodes in a 10,000-node cluster simply don't have a copy of that block to serve.

Worked example (illustrative numbers)

Say each DataNode can sustain about 200 concurrent read requests for a given block before it saturates (an illustrative round number — the exact figure depends on disk/network hardware). At RF=3, that one block is capped at 3 × 200 = 600 concurrent reads, cluster-wide, regardless of cluster size. Now say the file goes viral and demand spikes to 5,000 concurrent requests for that block: only 600 are served at any instant; the other 4,400 queue, time out, or get retried — a classic hotspot, even while the rest of a 10,000-node cluster sits nearly idle.

Three levers fix this, and they attack different parts of the ceiling:

• Raise replication factor for that specific file/block. HDFS supports a per-file/per-directory replication override. Bumping this one hot file to RF=15 raises the cap to 15 × 200 = 3,000 — better, but on its own still short of 5,000 concurrent requests.
• Client-side or edge caching. If a CDN or application-tier cache absorbs, say, 90% of requests before they ever reach a DataNode, only ~500 of the 5,000 requests actually hit the DataNode layer — comfortably under even the original RF=3 cap of 600, without touching replication at all.
• Fan the file across more blocks. Splitting a large hot file into more, smaller blocks spreads different byte-ranges across disjoint sets of DataNodes, multiplying aggregate capacity roughly by the number of distinct block-groups in play. This only helps to the extent readers actually want different byte-ranges — if every single reader wants the exact same bytes (one small, uniformly-hot block), sharding doesn't spread that specific hotspot; raising RF or caching is what actually helps there.

In practice, production systems combine the levers: raise RF for the hot tier (3,000-request cap) and add caching (cuts real demand to ~500) rather than relying on either alone.

NameNode Split-Brain: When "Dead" Isn't Actually Dead

HDFS High Availability (covered on the "What is a DFS" page) assumes that when the Active NameNode stops responding, it is safe to promote the Standby. The dangerous edge case is the one that assumption ignores: what if the old Active is not dead, only paused — frozen by a long stop-the-world garbage-collection pause, a hung disk I/O, or a transient network partition — and it resumes exactly where it left off, still fully convinced it is the one and only Active? If nothing stops it, you now have two processes simultaneously believing they own the namespace: a split-brain, and if both are allowed to write, the shared edit log — and therefore the whole filesystem's metadata — can become inconsistent.

The fix is not "detect the old Active is dead more reliably" (you fundamentally cannot distinguish "dead" from "paused for a while" from the outside with 100% certainty). The fix is fencing: make it structurally impossible for the old Active's writes to ever be accepted again, whether or not it later wakes up.

Traced: how QJM epoch numbers fence a GC-paused Active

HDFS's Quorum Journal Manager (QJM) solves this with monotonically increasing epoch numbers rather than requiring anyone to physically kill the old process (that harder guarantee — STONITH, "Shoot The Other Node In The Head," an administrator-configured fencing script that force-kills or power-cycles the old Active — is still used as a belt-and-braces backstop, but the epoch mechanism alone is enough to stop corruption). Trace it step by step:

The property doing all the work here is the quorum itself: with an odd number of JournalNodes (3 or 5, so a majority is well-defined), it is mathematically impossible for two disjoint majorities to exist at the same time — so epoch 5's "majority" and epoch 6's "majority" must overlap in at least one JournalNode, and that shared node is what guarantees the stale epoch can never again be acknowledged by enough nodes to commit. Fencing does not require certainty that Active-A is dead; it only requires certainty that Active-A's writes can no longer be durably recorded.

Pitfalls

• Treating "5 nines of block durability" as the whole story. The naive p³ math ignores repair speed entirely and is needlessly pessimistic; but real production durability still depends on fast failure detection and re-replication, and on defenses against correlated failure that no amount of independent-AFR math will show you.
• Assuming rack-awareness eliminates correlated risk. It bounds the damage from one rack event to "at most the replicas in that rack" — for the standard 2-in-one-rack placement, that can be 2 of 3 replicas at once, silently dropping a block to RF=1 until repair finishes.
• Applying erasure coding to hot or small-file data. The storage win is real, but every degraded read costs 6× the network I/O of a replication repair plus CPU decode time, and EC destroys data locality for compute — fine for cold archives, expensive for a live working set.
• Assuming more replicas fixes a hotspot. Raising RF helps a hot block up to a new, still-finite ceiling; it does not eliminate the ceiling, and combining it with caching is usually cheaper than chasing the ceiling with replicas alone.
• Trusting liveness detection to prevent split-brain. A GC pause, not a crash, is the classic way an "Active" NameNode looks dead without being dead. Failover must assume the old Active might come back, and fence it structurally (epoch numbers, and STONITH as backup) rather than merely declaring it dead and hoping.

Selection & Trade-offs

Four judgment calls this topic actually tests, and how a senior engineer decides each one:

• Choosing a replication factor: RF=2 is the wrong default because a single failure leaves zero spare redundancy for the entire repair window — any second failure during that window, correlated or not, is unrecoverable. RF=3 is the minimum factor that keeps one spare copy alive even mid-repair, and its raw durability (~5 nines naive, ~10+ nines once realistic repair speed is credited) already exceeds what correlated failure and operator error will let you achieve end-to-end — so RF=5+ buys mostly unusable headroom for real storage cost, and belongs only where you specifically need to survive correlated multi-domain events (e.g., an entire availability zone) that RF=3 across a single set of racks cannot.
• Replication vs. erasure coding: choose EC when data is cold/archival, files are large enough to amortize a stripe, and reads rarely if ever race a failure — you get better durability AND lower storage cost than replication, for a reconstruction-cost and locality tax that a cold workload rarely pays. Choose replication when data is hot, small, or latency-sensitive, or when compute needs data locality to schedule efficiently — you pay 2× the storage of EC for a system that never has to decode anything to serve a normal read.
• Fixing a hotspot: reach for RF-per-file first when the hot data is small and the read pattern is uniform across the whole block; reach for caching first when a large fraction of reads are repeat reads of the same bytes (the common case); reach for re-sharding only when reads are spread across different byte-ranges of a large logical object, since sharding does nothing for a single byte-identical hotspot.
• Preventing split-brain: never rely on "the old Active looks dead" as your only signal — build fencing into the write path itself (epoch/generation numbers checked by a quorum) so a falsely-declared-dead process is harmless even if it wakes back up; treat STONITH as a defense-in-depth backstop, not the primary mechanism.

Takeaways

• RF=3 is not chosen for a specific "nines" target — it is the minimum factor that guarantees at least one spare replica survives even during the repair window after a failure; correlated failures (rack, power, batch), not independent AFR, are the real durability risk that redundancy count alone cannot fully fix.
• Erasure coding is not merely "cheaper, slightly less safe replication" — compared fairly on the same data, RS(6,3) can be both more durable and half the storage cost of triple replication; what it actually costs is 6× the network/CPU on any reconstruction and the loss of data locality, which is why it belongs on cold data.
• A hot block's read throughput is capped at RF replicas' worth of capacity no matter how large the cluster is — fix it by raising RF for that data, caching, or sharding, not by scaling the cluster.
• NameNode split-brain is prevented by making the old Active's writes structurally unable to commit — epoch numbers enforced by a JournalNode majority — because a paused process can always wake up believing it is still in charge, and no liveness check can rule that out with certainty.

Sources: Apache Hadoop documentation, "HDFS Erasure Coding" (HDFS-EC design and administration guide, RS-6-3 default policy and storage-overhead figures); Apache Hadoop documentation, "HDFS High Availability Using the Quorum Journal Manager" (epoch numbers, JournalNode quorum, fencing behavior); Backblaze, annual "Hard Drive Stats" reports (illustrative annualized failure rate figures used for the durability arithmetic); Ghemawat, Gobioff & Leung, "The Google File System," SOSP 2003 (rack/fault-domain placement rationale). Re-authored/Deepened for this guide.