Distributed File System

Chain: a single-node file caps you on both capacity and throughput → split the 10 GB file into ~80 chunks of 128 MB each → scatter chunks across many machines → a client reading the file now pulls from many disks in parallel.

• Throughput: 1 disk at 125 MB/s reads 10 GB in ~82s; 80 disks reading their chunk in parallel finish in ~1s. Throughput scales with the number of disks you can spread the file across.
• Capacity: a 50 TB file no longer needs to fit on one 4 TB disk — it spreads across the cluster.
• Recovery: if a disk dies, you re-replicate the lost 128 MB chunks, not a whole monolithic 10 GB file, and the re-copies fan out across many source/target nodes in parallel.

Cost/trade-off: why 128 MB and not 4 KB like a normal OS block? Each chunk needs a metadata entry in the master's RAM. At 4 KB blocks, 10 GB = ~2.6M entries; at 128 MB, ~80 entries. Large chunks keep metadata tiny and cut the master's lookup load — but a 1 KB file still consumes a full metadata entry (though only ~1 KB of disk, since chunks aren't pre-padded), so the master's RAM, not disk, is what makes DFS terrible for billions of tiny files.

Chain: 1 copy per chunk → any disk death = permanent data loss + a read that returns an error. Instead, write each chunk to 3 machines (the GFS/HDFS default), placed on at least two different racks.

• Disk dies → master notices the chunk is now under-replicated (2 copies) → it schedules a copy of the chunk onto a fresh node, restoring 3 → zero data loss, reads keep working from the surviving 2.
• The danger window is the minutes between a disk dying and re-replication finishing. For data loss you'd need all 3 copies to fail inside that same window. Treating failures as independent, that's ~(0.02)³ ≈ 8 in a million — and rack-aware placement makes a correlated event (a rack/switch/power failure taking out copies together) far less likely than the independent estimate.

Cost/trade-off: 3x replication means 3x raw storage — store 1 PB of data, buy 3 PB of disk — and every write must land on 3 disks. This is why cold/archival data often switches to erasure coding (e.g. Reed–Solomon 6+3: 6 data + 3 parity shards), which also tolerates 3 simultaneous shard losses but at only ~1.5x overhead instead of 3x. The catch: reconstructing a lost shard requires reading 6 other shards and doing CPU-heavy math, so recovery and degraded reads are much more expensive than replication's plain copy.

Chain: split responsibilities → one master/NameNode holds only the metadata (the file→chunk→location map) → many chunkservers/DataNodes hold only the bytes.

• Client asks master: "where is /logs/april.txt?" → master replies with the chunk IDs and the 3 locations of each → client then streams bytes directly from the data nodes.
• Crucially, the master is out of the data path — the multi-gigabyte transfer never flows through it. The master only serves the tiny lookup, then steps aside.
• All hot metadata (file tree, chunk map) lives in master RAM → lookups are memory-speed, not disk seeks, so one master can serve a large cluster.

Cost/trade-off: the master becomes a single point of failure and a scaling ceiling — the namespace must fit in one machine's RAM. As a concrete rule of thumb, an HDFS NameNode uses on the order of ~150 bytes of RAM per object (file or block), so tens of GB of heap holds a few hundred million objects, no more. It's defended with a hot standby plus an operation log that the standby replays, but namespace size is hard-capped by RAM. (HDFS Federation later partitioned the namespace across multiple NameNodes to push that ceiling higher.)

Chain: concurrent writers + independent replicas → replicas could apply mutations in different orders → the 3 copies diverge and are no longer byte-identical. Fix: the master grants one replica a time-limited lease, making it the primary that defines the order.

• Data flow (push): the client pushes the bytes to the nearest replica, which pipelines them to the next, then the next — bytes stream along the chain so every link's bandwidth is used and the data is buffered on all 3 before any commit.
• Control flow (order): once all 3 have the bytes buffered, the client sends the write request to the primary. The primary assigns a serial order, applies it locally, and forwards that same order to the 2 secondaries to apply.
• The write succeeds only after all 3 secondaries+primary ack → all copies converge on identical state.

Cost/trade-off: the write is only as fast as the slowest of the 3 replicas (tail latency drags the whole write), and if a replica fails mid-commit the client retries, which can leave duplicate or padded records in the chunk. GFS deliberately exposes only at-least-once record-append semantics, pushing dedup/idempotency up to the application rather than paying for stronger guarantees in the storage layer.

Chain: master polling 1,000 nodes on a tight loop → 1,000 outbound RPCs every cycle → the master burns its scarce CPU/network on liveness checks instead of serving lookups, and the cost grows O(N) on the one bottleneck node. Invert it: each node pushes a heartbeat (every ~3s in HDFS) carrying its status, and periodically a full block report of the chunks it holds.

• Master just receives heartbeats and resets a per-node timer → tracking liveness is near-free for the bottleneck node.
• No heartbeat for ~10 missed intervals (~30s in HDFS's default) → master marks the node dead → it schedules re-replication of every chunk that node held.
• This is the push/event-driven vs pull/polling trade applied to failure detection: the work scales onto the 1,000 nodes (each does a trivial amount), while the bottleneck (master) only reacts to events.

Cost/trade-off: detection is not instant — a node can be dead for ~30s before the master reacts, and reads aimed at it during that window must fail over to its other replicas. Shorten the timeout and you risk declaring a node dead during a long GC pause or a transient network blip, triggering a needless re-replication storm that floods the network copying chunks that were never actually lost.

Chain: someone must remember the file→chunk→location mapping — that's irreducible state. Concentrate the authoritative copy in one stateful master and keep everyone else effectively stateless, so failures of the many are cheap.

• Near-stateless clients hold no authoritative state — they cache a chunk's location for a short TTL, and if it's stale (the chunk moved or its node died) the read fails, the client simply re-asks the master and retries. A crashed client loses nothing the system needs.
• Stateful master must durably persist its metadata: it appends every namespace change to an operation log on disk and replicates that log, and periodically writes a checkpoint of the in-RAM state. On restart it loads the latest checkpoint and replays the tail of the log to rebuild RAM state.
• Chunk locations are deliberately not persisted in the log — the master rebuilds the location map from the block reports nodes send when they (re)join. Less to persist, and the rebuilt map can't go stale relative to reality.

Cost/trade-off: the stateful master needs heavyweight machinery the stateless nodes don't — durable log writes (each metadata op pays a disk sync/replication round-trip before it's acknowledged), a hot standby kept in sync, and a leader-election/failover protocol. The lesson: statelessness is cheap to scale and recover, statefulness is expensive — so you minimize it and isolate it to one well-defended place.

Chain: a node holding 50,000 chunks sends 50,000 IDs; 1,000 nodes do this repeatedly → this is hot internal machine-to-machine traffic on the critical path to the bottleneck master, where bytes-on-the-wire and parse CPU dominate.

• Text (JSON/XML): a 64-bit chunk ID becomes ~19–20 ASCII digits instead of 8 binary bytes, and the parser must scan delimiters and convert ASCII→int for every field. JSON runs roughly 2–3x larger than packed binary here (XML is worse — closing tags and attribute names inflate it further). At 50,000 IDs per node that's the difference between a report of a few hundred KB and ~1 MB, multiplied by every node, every cycle.
• Binary (Protobuf/Thrift/Avro): fixed-width or varint-packed IDs, no field-name repetition, near-zero parse cost. The master spends its CPU on real work instead of string parsing. This is why GFS/HDFS use compact RPC encodings internally.

Cost/trade-off: binary buys size and speed but costs human-debuggability and loose coupling — you can't curl it and eyeball it, and both sides must share a schema and stay version-compatible. That's exactly why the answer flips for a public API: there, the consumers are external and unknown, developer ergonomics and easy debugging beat raw efficiency, so human-readable JSON usually wins. The rule of thumb: binary for hot internal paths you control both ends of; text for public, low-volume, human-facing interfaces.