Redundancy — Active-Passive Failover, RPO/RTO, Semi-Sync Replication & Correlated Failure (Deep Dive)

The "What is Redundancy" page already derives the availability-multiplication math (0.01 × 0.01 = 0.0001) and traces a ~20-second active-passive failover timeline; the "Redundancy and Replication" page already walks synchronous, asynchronous, semi-synchronous and leaderless mechanics with a worked replication-lag example. This page does not repeat either derivation. It adds the layer interviewers actually probe on top of them: why the multiplication assumption breaks under correlated failure, and the two numbers — RPO and RTO — that turn "we have a standby" into a quantified, defensible recovery guarantee.

1. Split-brain under partition: the old active isn't dead, only unreachable

Active-passive failover has one structural trap: a network partition looks, from the monitor's side, identical to a dead primary — but the primary itself is still up, still holding client connections, still willing to accept writes. If the passive gets promoted the instant it stops hearing heartbeats, you now have two nodes that both believe they own the write path. This is exactly the failure mode traced in depth on the Heartbeat deep dive (symmetric partitions, phi-accrual detection, the blast radius of a false positive) and the Leader/Follower deep dive (externally-orchestrated failover with no term/epoch, fencing tokens, STONITH) — read those for the full mechanism. The short version this page needs: fencing (a monotonically increasing token that a downstream write path rejects if it's lower than the highest it has seen) is what makes a promotion safe even if the old active wakes back up, and it is a precondition for everything below — none of the RPO/RTO numbers in this page mean anything if a resurrected old active can still silently accept writes after "failover" completes.

2. Availability multiplication needs independence — correlated failure breaks it

The parallel-availability rule — unavailability multiplies across redundant components — is only valid if the two components fail for unrelated reasons. Two nodes in the same rack are not independent: they share a power feed, a top-of-rack switch, and often a software rollout or a config push. If either of those goes, both nodes go together, and the independence assumption the multiplication relied on is gone.

Worked contrast. Take two nodes each independently unavailable 1% of the year (p = 0.01). The naive parallel calculation says the pair is down only when both fail independently at once: 0.01 × 0.01 = 0.0001 (99.99% available, ~53 min/year downtime) — the number already derived on the "What is Redundancy" page. Now add one shared failure mode: the rack's power distribution unit fails 0.3% of the year, taking out both nodes simultaneously regardless of anything else. The real unavailability is no longer the product of the two independent terms — it is (at minimum) the sum of the independent-overlap term and the shared-cause term: 0.0001 + 0.003 ≈ 0.0031, or roughly 99.69% available — ~31× worse than the naive figure, and the correlated term is the one doing almost all of the damage. Redundancy multiplies away independent failure; it does nothing to a cause that takes out every replica in the group at once. This is the same mechanism traced in full, with the RF=3 durability arithmetic, on the DFS durability deep dive (same-rack power/switch failure, why rack-awareness bounds but doesn't eliminate the risk, why an entire remote rack going down can silently drop a block to its last copy).

Design implication: a passive standby (or a semi-sync acking replica) that lives in the same rack, same power domain, or same availability zone as the active is not buying you the independence the math assumes. Spread the standby across a genuinely separate failure domain — different rack at minimum, different AZ for anything that must survive a zone-level event — or the "two nines of protection" you think you bought is actually however unreliable the shared power/network path is, which is usually worse.

3. What actually makes sync replication block

Mechanism, in one sentence: under synchronous replication the primary withholds the client's acknowledgment until at least one replica has durably persisted the write, so "the client was told success" and "the write is unrecoverable" become the same event — at the cost of every write now depending on that replica's liveness, not just the primary's. Async replication decouples the two: the primary acks off its own local durability and ships the write afterward, so client-perceived latency stops depending on replica health, but any write acked-and-not-yet-shipped is gone if the primary dies before it ships. This mechanism, the replication-lag arithmetic, and the leaderless/quorum variant are traced with a full worked LSN-gap example on the "Redundancy and Replication" page — this page only needs the one-line mechanism above as the hook for RPO below.

4. Semi-synchronous / quorum replication: the real-world default

Pure synchronous replication (wait for every replica) and pure asynchronous (wait for none) are both extremes nobody runs at scale for a primary write path: the first couples write availability to the slowest or least healthy follower, the second accepts an unbounded loss window. Semi-synchronous / quorum replication is the standard middle ground: the primary waits for an acknowledgment from one replica (or a quorum of k out of n), not all of them, before acking the client — bounding the loss window without coupling write availability to every follower's health.

Concretely, this is what production systems actually ship:

• MySQL semi-sync replication — the source waits for at least rpl_semi_sync_source_wait_for_replica_count replicas (default 1) to acknowledge receipt of the transaction before returning success to the client; if no replica acks within a timeout, it falls back to async rather than blocking forever. (That variable was named rpl_semi_sync_master_wait_for_slave_count before MySQL 8.0.26, which introduced the rpl_semi_sync_source/rpl_semi_sync_replica plugins and deprecated the master/slave-named variables — you will still meet the old name on older deployments and in older runbooks.)
• PostgreSQL synchronous standbys — synchronous_standby_names can name a specific standby, or use ANY k (...) to commit once any k of a named set have confirmed — a direct quorum-commit knob.
• Kafka acks=all + min.insync.replicas — the producer's write is acknowledged only once at least min.insync.replicas members of the in-sync replica set (ISR) have the record; if the ISR shrinks below that count, the broker rejects further writes with NotEnoughReplicas rather than silently accepting an under-replicated write.

The mechanism in all three: the primary/leader's ack is gated on a count, not a set membership guarantee — "one of my followers, whichever answers first" (MySQL semi-sync default) is weaker than "a specific named quorum that must include the one that will be promoted" (Postgres ANY k, or Kafka's ISR tracking). That distinction is exactly the pitfall in section 6.

5. RPO and RTO, computed — two clocks, not one number

RPO (Recovery Point Objective) is a data-loss measurement: how far back in time you might lose data, fixed at the instant of failure and looking backward. It is set entirely by the replication mode: async replication's RPO equals however much was acked-but-unreplicated at the moment the primary died (the replication lag at that instant); sync or quorum-commit replication's RPO is effectively 0 for any write the client was actually told succeeded, because that ack didn't happen until the surviving replica already had it.

RTO (Recovery Time Objective) is a service-restoration measurement: how long until the system accepts traffic again, starting from the same failure instant and looking forward. It decomposes into three phases that stack: detection (how long until the failure is confirmed, not just suspected — the phi-accrual / health-check-threshold tuning from the Heartbeat deep dive lives here), failover/promotion (electing or designating the new primary, replaying any queued-but-unapplied writes, opening it for traffic), and DNS/connection re-establishment (clients and load balancers noticing the new endpoint — bounded by DNS TTL, connection-pool retry/backoff, or VIP re-binding time).

Traced example

t	Event	Which clock it sets
t = −45ms	Under async replication, the primary has acked three writes to its clients that are still only durable locally; the replica's applied position is 45ms of writes behind.	Fixes RPO — this is the loss window, decided the instant the crash happens, not something that grows afterward.
t = 0	Primary crashes. The three acked-but-unreplicated writes are gone the moment this happens — RPO ≈ 45ms, already final.	Crash instant: RPO is now fixed; RTO's clock starts.
t = 0 to 10s	Health checker polls every 5s, requires 2 consecutive failures before declaring the primary dead (standard "interval 5s, unhealthy-threshold 2" config).	Detection phase of RTO: 10s.
t = 10s to 17s	Orchestrator confirms the primary is actually down (not just slow), promotes the standby, and the standby replays whatever was queued in its apply buffer, then binds the virtual IP.	Promotion phase of RTO: 7s.
t = 17s to 20s	DNS/service-discovery propagation and client-side connection-pool reconnect complete; new connections land on the promoted node.	Reconnect phase of RTO: 3s.
t = 20s	Traffic is flowing again.	RTO ≈ 20s, total.

The point the two clocks make together: RPO would have been 0, not 45ms, if those three writes had gone through semi-sync (waited for the replica that ends up promoted) instead of async — a replication-mode choice, made long before the crash. RTO would still have been roughly the same 20 seconds regardless of which replication mode was in play, because RTO is a function of detector tuning and orchestration speed, not of how durable the data was. Tightening one does not tighten the other; they are separate dials answering separate questions ("how much data can we lose" vs. "how long can we be down"), and a design review that only asks for "an SLA number" without splitting the two is asking an underspecified question.

Pitfalls

• Treating RPO and RTO as one "recovery SLA" number. They are set by different design choices — replication mode sets RPO, detector tuning and orchestration speed set RTO — and optimizing one does nothing for the other.
• Assuming "semi-sync, wait for one" gives zero loss no matter which replica dies. If the acking replica is whichever answers first (MySQL semi-sync's default) rather than a specific set that includes the one that will actually be promoted, the write can still be lost if the other replica is the one promoted. A true RPO=0 guarantee needs the acking set to provably overlap the promotion candidate — the same quorum-overlap argument as Raft's leader-completeness property on the Leader/Follower deep dive, not just "some replica acked."
• Chasing RTO by shortening the detection window alone. A shorter health-check interval/threshold detects real failures faster but raises false-positive risk — the flapping and re-replication storms covered on the Heartbeat deep dive. RTO has three phases; the cheapest wins are often in promotion (pre-warming the standby, faster log replay) or reconnect (shorter DNS TTL, faster client retry), not squeezing detection to zero.
• Believing the 0.01 × 0.01 parallel-availability math without checking failure-domain placement. If the "redundant" pair shares a rack, power feed, or deploy pipeline, the real unavailability is dominated by the correlated-failure term, not the independent product.
• Forgetting DNS/connection re-establishment as part of RTO. A perfectly fast promotion is invisible to users still holding a stale DNS answer or a connection pool that hasn't noticed the endpoint moved — this phase is easy to omit from an RTO estimate and is often the least-tuned of the three.

Judgment layer: choosing by RPO need

• RPO must be 0 (financial ledgers, inventory decrements, anything where "we told the client it succeeded" must remain true) → synchronous replication or Raft-style quorum commit. Accept the coupling of write availability to reaching a quorum/replica, and budget for the added write latency.
• RPO can be small and bounded, but not zero (most transactional application data — orders, comments, user records) → semi-synchronous / quorum replication is the real-world default: MySQL semi-sync, Postgres synchronous_standby_names = ANY k (...), or Kafka acks=all with min.insync.replicas tuned to the durability you need. Bounds the loss window to "whatever the acking quorum hadn't gotten yet" without paying full-sync's availability coupling to every follower.
• RPO can be seconds to minutes (analytics replicas, cross-region DR copies, read-scaling followers) → plain asynchronous replication is correct: lowest write latency, and the loss window is an accepted, budgeted risk rather than an oversight.
• RTO is tuned independently of all of the above: pick detection thresholds against the false-positive cost (Heartbeat deep dive), keep the standby warm and its log-replay path fast, and keep DNS TTLs/connection-pool retry settings short enough that the reconnect phase doesn't dominate the total. A design that nails RPO=0 with synchronous replication but leaves a 5-minute DNS TTL still has a multi-minute RTO.

Takeaways

• Availability multiplication (parallel unavailability = product of the parts) only holds for statistically independent failures; two replicas sharing a rack, power feed, or config push are correlated, and the shared-cause term — not the independent product — usually dominates real-world unavailability.
• RPO (data-loss window, backward-looking) is set by the replication mode; RTO (service-restoration time, forward-looking) is set by detection + promotion + reconnect. They are two separate clocks answering two separate questions, not one SLA number.
• Semi-synchronous/quorum replication (MySQL semi-sync, Postgres ANY k, Kafka acks=all + min.insync.replicas) is the standard middle ground between full-sync's availability coupling and async's unbounded loss window — but "wait for one" only gives a real zero-loss guarantee if the acking set is guaranteed to overlap whichever replica gets promoted.
• None of this matters without fencing: an old active that isn't actually dead, only unreachable, can silently undo every RPO/RTO guarantee by continuing to accept writes after a "successful" failover — see the Heartbeat and Leader/Follower deep dives for the full mechanism.

Related pages

• What Are RPO and RTO, and How Do They Differ in Disaster Recovery Planning — the basic definitions this page builds on
• What Is the Difference Between Active-Active and Active-Passive Architectures — contrasts the failover model this page assumes
• Consensus with Raft — Leader Election & Log Replication — the leader-completeness property referenced in the pitfalls
• Data Backup vs Disaster Recovery — where RPO/RTO fit into a broader DR strategy
• Split-Brain, Fencing & Safe Failover — deeper treatment of the fencing precondition in section 1

I just read a lesson on **Redundancy — Active-Passive Failover, RPO/RTO, Semi-Sync Replication & Correlated Failure (Deep Dive)** (System Design) and want to truly understand it. Explain Redundancy — Active-Passive Failover, RPO/RTO, Semi-Sync Replication & Correlated Failure (Deep Dive) from first principles using ONE vivid real-world analogy and a visual mental model — draw it as ASCII art or a clear step-by-step diagram — with a concrete example using real numbers. Then ask me one question to check I got the mental picture, and wait for my reply. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Teach me **Redundancy — Active-Passive Failover, RPO/RTO, Semi-Sync Replication & Correlated Failure (Deep Dive)** interactively. Ask me ONE guiding question at a time, wait for my answer, and adapt to my confusion — build the idea with me step by step instead of explaining it all at once. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Quiz me on **Redundancy — Active-Passive Failover, RPO/RTO, Semi-Sync Replication & Correlated Failure (Deep Dive)** with 5 questions, easy to tricky, ONE at a time. Tell me if each answer is right; at the end, explain clearly what I got wrong and why. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Help me remember **Redundancy — Active-Passive Failover, RPO/RTO, Semi-Sync Replication & Correlated Failure (Deep Dive)** for the long term: give the one-sentence intuition, a memorable hook/mnemonic, a tiny worked example, and 3 active-recall flashcards (Q -> A). If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.