Replication Lag & Failover — Read-Your-Writes, Split-Brain & Fencing

Replicas make reads fast — and consistency hard

You add read replicas to scale reads and survive a node loss. The catch: replication is usually asynchronous, so a follower is always slightly behind the leader. That replication lag (tens to hundreds of ms, or seconds under load) produces real, user-visible bugs.

The three anomalies lag causes

• Read-your-writes: you update your profile, the next read hits a lagging replica, and your change is "gone." Fix: route a user's reads to the leader (or a pinned replica) for a short window after they write.
• Monotonic reads: two reads hit replicas at different lag → the user sees data go backwards. Fix: pin a user to one replica.
• Consistent prefix: across partitions, effects can appear before causes. Fix: causal ordering.

Sync vs async: synchronous replication (wait for a replica to ack) removes lag but adds write latency and stalls if the replica is down. Most systems use semi-sync — one synchronous replica for durability, the rest async.

Failover: when the leader dies

A follower is promoted to leader. The dangers:

• Data loss: async writes not yet replicated are lost on promotion (bounded by your durability/RPO).
• Split-brain: the old leader comes back and also thinks it's leader → two leaders accept conflicting writes. Prevent with a quorum/consensus election (only a majority can elect) and fencing (a monotonically increasing epoch token; storage rejects the stale old leader).
• Detection: too-aggressive failover on a transient blip causes needless churn; tune heartbeat timeouts.

Takeaways

• Async replication → lag → read-your-writes / monotonic-read anomalies; fix by routing/pinning reads.
• Sync = no lag but slower writes; semi-sync is the common middle ground.
• Failover risks data loss (async) and split-brain — prevent with quorum election + fencing tokens.

---

Re-authored for this guide; replication-lag timeline hand-authored as SVG. Follows DDIA ch. 5. See also: Quorum, Leader and Follower, CAP Theorem, and (Concurrency) the same race/visibility intuition.

I just read a lesson on **Replication Lag & Failover — Read-Your-Writes, Split-Brain & Fencing** (System Design) and want to truly understand it. Explain Replication Lag & Failover — Read-Your-Writes, Split-Brain & Fencing from first principles using ONE vivid real-world analogy and a visual mental model — draw it as ASCII art or a clear step-by-step diagram — with a concrete example using real numbers. Then ask me one question to check I got the mental picture, and wait for my reply. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Teach me **Replication Lag & Failover — Read-Your-Writes, Split-Brain & Fencing** interactively. Ask me ONE guiding question at a time, wait for my answer, and adapt to my confusion — build the idea with me step by step instead of explaining it all at once. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Quiz me on **Replication Lag & Failover — Read-Your-Writes, Split-Brain & Fencing** with 5 questions, easy to tricky, ONE at a time. Tell me if each answer is right; at the end, explain clearly what I got wrong and why. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Help me remember **Replication Lag & Failover — Read-Your-Writes, Split-Brain & Fencing** for the long term: give the one-sentence intuition, a memorable hook/mnemonic, a tiny worked example, and 3 active-recall flashcards (Q -> A). If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.