hard Partition × Replication: Consistency Under Failover

Mechanism: partitioning and replication are two independent axes that compose

Partitioning splits data across the key axis so no single node has to hold everything; replication copies each partition across multiple nodes so no single node's failure loses anything. Real systems do BOTH at once: each partition gets its own leader, which accepts writes for that partition's key range and replicates them to that partition's own follower replicas. Because leadership is assigned PER PARTITION, not per cluster, a client's consistency and availability for a given key depend entirely on the state of THAT partition's leader and replicas — not on the health of the cluster as a whole. That is the detail that trips people up: "is the system available, is it consistent" is not one yes/no answer, it is a per-key-range answer that can differ partition by partition at the same instant.

Traced: partition 2's leader fails over while partitions 1 and 3 keep serving

3 partitions, RF=3 (leader + 2 followers) each, asynchronous replication. Client traffic is spread across all three key ranges.

1. t0 — healthy. All three partitions have a leader and two followers, in sync or within a small lag budget. Reads and writes proceed normally everywhere.
2. t1 — P2's leader dies. A crash, a network partition, a failed heartbeat during a long GC pause — the mechanism doesn't matter here, only that P2's leader stops responding. P1's and P3's leaders are entirely unaffected; they are different processes serving different key ranges.
3. t2 — the failover window. The controller (a ZooKeeper/etcd/Raft-based metadata service) has not yet confirmed a new leader for P2. Any read or write that hashes into P2's key range now either (a) reaches a follower still serving stale data — async replication means a follower can lag the leader by some replication delay — which can break read-your-writes for a client who just wrote through the old leader, or (b) times out entirely, because no node will currently accept writes for that range. Reads and writes to P1 and P3's ranges are untouched — different partitions, different leaders, a different failure domain.
4. t3 — a new leader for P2 is promoted. The most-caught-up follower is chosen and promoted, but only after a fencing token is issued and a quorum of the metadata service agrees — exactly the mechanism traced in Split-Brain, Fencing & Safe Failover. Skip that step and the old leader could keep accepting writes after "coming back," corrupting the range with two writers. If the promoted follower was lagging, any writes the old leader had already acked but never shipped are gone — not rolled back, simply never present on the new leader.
5. t4 — P2 is healthy again. The new leader accepts writes, its followers catch up. However many seconds the failure detector plus the controller took to act, the episode was scoped entirely to P2's key range.

Partial availability: the rest of the cluster doesn't care

This is the single most important consequence of combining partitioning with replication: availability is no longer a cluster-wide property, it is a PER-PARTITION property. If partition 2's leader is down and no follower is currently promotable — none has caught up enough, or the quorum needed to promote hasn't been reached — THAT partition's key range is unavailable, full stop, while every other partition's key range keeps serving normally. A dashboard showing "99.7% of requests succeeded" during this window is not lying, but it is hiding that 100% of requests to one specific slice of the keyspace failed. Whether this matters operationally depends on whether traffic is spread evenly across partitions, or whether the down partition happens to hold a disproportionately hot or important range of keys (see the shard-skew discussion in Partitioning Methods) — "the down partition is also the hot one" turns a small-sounding partial outage into a major one.

Judgment: read-from-leader vs. read-from-any-replica vs. quorum reads

Read-from-leader

Every read for a partition goes to that partition's leader, same as every write. This gives linearizable reads relative to that partition's own writes — there is only ever one place a "latest value" can live. The cost: the leader alone absorbs all read+write load for its partition (followers never shed any of it), and reads are unavailable during exactly the failover window traced above.

Read-from-any-replica

Reads are load-balanced across a partition's leader and its followers. This buys read scalability — followers absorb read traffic the leader would otherwise carry alone — at the cost of staleness: a follower that hasn't caught up can return an old value, and a client that writes through the leader and immediately reads from a lagging follower can fail to see its own write.

Quorum reads

Read from R of the N replicas and reconcile by version, the same mechanism as a Dynamo-style store. As covered in full in Quorum ≠ Linearizability, picking R + W > N guarantees a read overlaps a previously-COMPLETED write's ack-set — it does not make reads and writes linearizable, and it does not by itself solve the leader-failover problem above. Quorum reads are a replication-level consistency knob; partition leadership and failover are a separate, orthogonal mechanism that quorum reads don't touch.

Sync vs. async replication, per partition

Independently of how reads are served, each partition's leader can replicate to its followers either synchronously — wait for at least one follower to ack before acking the client, giving zero data loss on failover at the cost of higher write latency, and the leader can't ack at all if that follower is down — or asynchronously — ack the client immediately and ship to followers in the background, giving low write latency and keeping the leader available even when followers lag, at the price of exactly the acked-but-lost-write scenario traced at t3 above. Different partitions in the same cluster can even run different replication modes: money-holding key ranges synchronous, analytics-event ranges asynchronous. This is a per-partition dial, not a cluster-wide setting.

The composite picture: your real consistency and availability story is the product of all three dials at once — which partition owns the key you're touching, which replica of it you land on, and whether that partition's replication is sync or async — not a single number you can quote for "the system."

Pitfalls

• Treating "the cluster is up" as "every key is available." 99 of 100 healthy partitions tells you nothing about the 1 that is mid-failover; alert on per-partition leader health, not just aggregate cluster health.
• Read-from-any-replica silently breaking read-your-writes. If a client's reads aren't pinned to the leader (or a replica known to be caught up) right after that client's own write, "I just saved this and it's gone" bug reports are the direct consequence.
• Assuming quorum reads make partition failover safe. R + W > N is about replica overlap for a completed write; it says nothing about which replica gets promoted to leader or whether that replica had the latest data — that is the fencing/quorum-promotion mechanism covered in Split-Brain, Fencing & Safe Failover, a different mechanism entirely.
• Sizing capacity off the average partition, not the hot one. The partition that fails over isn't chosen by traffic volume — if it happens to be the hottest one, "partial availability" can still mean most users are affected.

Takeaways

• Partitioning and replication compose: each partition has its own leader and followers, so consistency and availability are per-partition properties, not cluster-wide ones.
• A partition's leader failover can make that partition's key range stale (lagging follower) or briefly unavailable (no confirmed leader) while every other partition keeps serving normally — partial availability.
• Read-from-leader (consistent, no read scaling), read-from-any-replica (scales reads, risks staleness), and quorum reads (a replica-overlap guarantee, not linearizability) are three different knobs; sync/async replication is a fourth, applied per partition.
• Quorum reads and leader-failover safety (fencing, quorum promotion) are separate mechanisms that must both be reasoned about — one does not substitute for the other.

---

Synthesized from Martin Kleppmann, Designing Data-Intensive Applications (O'Reilly, 2017), Ch. 5 (Replication) & Ch. 6 (Partitioning), and Apache Kafka's per-partition leader/ISR model (see Kafka Internals). Re-authored/Deepened for this guide.

I just read a lesson on **Partition × Replication: Consistency Under Failover** (System Design) and want to truly understand it. Explain Partition × Replication: Consistency Under Failover from first principles using ONE vivid real-world analogy and a visual mental model — draw it as ASCII art or a clear step-by-step diagram — with a concrete example using real numbers. Then ask me one question to check I got the mental picture, and wait for my reply. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Teach me **Partition × Replication: Consistency Under Failover** interactively. Ask me ONE guiding question at a time, wait for my answer, and adapt to my confusion — build the idea with me step by step instead of explaining it all at once. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Quiz me on **Partition × Replication: Consistency Under Failover** with 5 questions, easy to tricky, ONE at a time. Tell me if each answer is right; at the end, explain clearly what I got wrong and why. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Help me remember **Partition × Replication: Consistency Under Failover** for the long term: give the one-sentence intuition, a memorable hook/mnemonic, a tiny worked example, and 3 active-recall flashcards (Q -> A). If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.