hard Split-Brain, Fencing & Safe Failover

Why promoting a second primary can silently corrupt your data

A failover controller promotes a replica when it decides the primary is dead. But over a network it cannot distinguish “primary crashed” from “primary is alive but I can’t reach it” — that is the partition. If the old primary is actually alive and still taking writes while a replica is promoted, you now have two nodes that both believe they are primary. Both accept writes, their histories diverge, and there is no correct automatic way to merge them. That is split-brain.

A second, quieter failure rides along with it. With asynchronous replication the primary acks a write to the client before the replica has it. Promote that lagging replica and every acked-but-unshipped write vanishes — the client was told “committed,” yet the new primary never saw it. Durability is violated even without two primaries.

A traced failover timeline

Three nodes: primary P, replicas R1, R2, asynchronous replication, one key x.

The three fixes (they solve different halves of the problem)

1. Quorum-based promotion — prevents two winners

Require a majority of nodes to agree before anyone is promoted. With 3 nodes the majority is 2. When the partition splits the cluster into {P} and {R1, R2}, only the {R1, R2} side has 2 votes ≥ majority, so only that side can elect a leader. The lone P cannot assemble a majority, so a quorum-respecting P steps down and stops accepting writes. Because two disjoint groups can never both hold a majority of the same cluster, you can never elect two leaders. This is exactly how Raft and Paxos leader election avoid split-brain.

2. Fencing — stops a deposed primary that didn’t get the memo

Quorum stops a well-behaved old primary. But a node paused in a long GC or a network stall may wake up believing it is still primary and try to write. Fencing blocks that write:

• Fencing token: the lock/lease service hands out a monotonically increasing token with each leadership grant. Say P holds token 33, pauses, its lease expires, and R1 is granted token 34. When P wakes and writes to shared storage with token 33, the storage layer — which remembers it has already seen 34 — rejects the stale 33. The old primary is fenced out. (This is Martin Kleppmann’s canonical fencing-token example.)
• STONITH (“Shoot The Other Node In The Head”): physically power-off or isolate the old node via a power controller / management API before promoting. Works even when the shared resource cannot check a token.

3. Synchronous replication — prevents the lost acked write

Have the primary wait for a replica to durably ack before it acks the client. Now a promoted replica already holds every acked write, so W1 can never be lost. The cost: every write pays the round-trip to the replica, and if the replica is down the primary cannot ack at all — you have traded availability for durability. Semi-synchronous (wait for any one of several replicas) is the common middle ground.

Pitfalls

• Distributed locks without fencing are unsafe. A lease can expire during a GC pause; the holder wakes and writes anyway. Only a fencing token checked at the resource makes the lock safe.
• Auto-failover on a flaky link causes flapping — primary demoted and re-promoted repeatedly. Require sustained failure detection + quorum.
• Dueling STONITH: in a partition each side may try to shoot the other; without a tiebreaker (quorum disk / witness) both can power-cycle each other.

The judgment layer

Synchronous vs asynchronous replication. Sync gives zero data loss on failover (RPO=0) but raises write latency and reduces availability when a replica is slow or down — the right choice for money and source-of-truth state. Async keeps writes fast and the primary available even when replicas lag, at the price of a bounded window of writes that can be lost on promotion — the right choice when a few lost seconds of the newest data is survivable (analytics, feeds). Semi-sync (ack after ≥1 replica) buys most of the durability for a fraction of the latency hit.

Fencing token vs STONITH. A fencing token is clean and hardware-free but requires the shared resource (storage, DB) to be token-aware and reject stale tokens — cooperative fencing. STONITH needs no cooperation from the resource (it removes the node entirely) but requires a reliable power/management channel and a tiebreaker to avoid mutual kills. Use a token when the resource can enforce it; fall back to STONITH when it cannot.

Takeaways

• Split-brain = two primaries under a partition; async replication additionally loses acked-but-unshipped writes on promotion.
• Quorum guarantees a single winner (two groups can’t both hold a majority); fencing stops a stale ex-primary from writing; sync replication stops data loss.
• They are complementary, not substitutes — quorum + fencing + a replication choice that matches your RPO.
• A lease-based lock without a fencing token is not a safe lock.

Synthesized from Kleppmann, Designing Data-Intensive Applications (Ch. 5, 8, 9) and “How to do distributed locking” (fencing tokens), the Raft paper (leader election / quorum), and Linux-HA / Pacemaker STONITH practice. Re-authored/Deepened for this guide.