High Availability and Fault Tolerance

A load balancer becomes highly available by putting two or more LB instances behind one address that clients keep using — a virtual IP (VIP) or a DNS/anycast entry — and running a failure detector that continuously decides which instance currently owns that address, so a dead instance is silently replaced by a survivor before most clients notice.

That single sentence hides two independent control loops, and conflating them is the most common source of confusion:

• LB → backend health checks. Each LB probes its pool (e.g. GET /healthz every 2s, eject after 3 consecutive failures = 6s) and stops routing to sick backends. This protects users from broken application servers.
• LB → LB failure detection. The LBs watch each other (VRRP/keepalived heartbeats, or a lease in etcd/Consul/ZooKeeper) and move VIP ownership when a peer dies. This protects users from a broken load balancer.

High availability is only real when both loops exist. A pool of healthy backends behind a single LB is still a single point of failure; a redundant LB pair in front of dead backends just fails over to more failure.

A traced failover: VRRP active-passive with keepalived

Two LBs share VIP 203.0.113.10. LB-A has priority 150 (elected MASTER), LB-B has priority 100 (BACKUP). VRRP advertisements go out every 1s. The BACKUP arms a master-down timer = 3 × advert_interval + skew, where skew = (256 − priority)/256 = 156/256 ≈ 0.609s, so the timer is 3.609s.

Outcome: a ~3.2s traffic black-hole (from failure at 2.4s to VIP takeover at 5.61s). TCP/TLS flows that LB-A was terminating are reset unless connection state was mirrored to LB-B (Linux conntrackd for L4, or session state parked in Redis for L7); clients then transparently reconnect. Tightening advert_interval to 200ms cuts the gap to under a second but multiplies heartbeat traffic and raises the risk of a false failover under CPU/network stress.

Active-active and the split-brain trap

Active-active runs both LBs live (traffic split by DNS, an L4 layer, or router ECMP), so no capacity sits idle. The danger appears when the failure detector — not a real node — is what breaks. If the heartbeat link between two peers is cut but both nodes are otherwise healthy, each concludes "my peer is dead, I must take over." This is split-brain.

Traced split-brain (2-node VRRP, partitioned link):

1. The cross-connect carrying VRRP adverts fails; both nodes stop hearing each other.
2. LB-B's master-down timer expires; it promotes to MASTER and sends a gratuitous ARP for the VIP.
3. LB-A never saw itself fail — it is still MASTER and still owns the VIP.
4. Both nodes now answer for 203.0.113.10. The switch's CAM table flaps between two MACs on every GARP; packets for a single TCP flow get sprayed across both LBs, breaking sequence numbers → RST storms.
5. Worse, if both LBs write session/counter state to different local stores, the two halves diverge and can't be cleanly merged after the partition heals.

The fix is fencing via a quorum, not a bigger timeout. Instead of a peer-to-peer heartbeat, ownership becomes a lease in etcd/Consul/ZooKeeper: a node may hold the VIP only while it holds a lease that the majority of the coordination cluster has granted, renewed on a TTL. With a 3-node etcd cluster, a partitioned LB that can't reach the majority cannot renew its lease, so it voluntarily drops the VIP — only the side with quorum keeps it. This is exactly why you need an odd number of coordination members: two LBs alone can never break a tie about which one is really isolated; a third witness can.

Pitfalls

• The passive node was never actually working. A standby that only proves itself during a real failover often fails to take over — expired TLS certs, drifted config, a firewall rule that was never applied. Force periodic failovers (game days) so the standby path is exercised, not assumed.
• Sizing both active nodes at 90%. In active-active, when one of two nodes dies the survivor inherits 100% of traffic. If each was at 90%, the survivor is instantly at 180% and collapses — turning a one-node failure into a total outage. Run active-active nodes below the N−1 line (each ≤50% for a pair).
• Failover gap swallows in-flight requests. The seconds between failure and VIP takeover reset every non-replicated connection. If clients don't retry idempotently, users see errors even though "failover worked." Replicate connection state (conntrackd / Redis) or make clients retry.
• Health-check flapping. Thresholds that are too tight eject a backend on one slow response, then re-add it, then eject it — thrashing the pool. Use asymmetric rise/fall counts (e.g. fall after 3, rise after 5) and check a cheap dedicated endpoint, not "/".
• Health check that lies. A /healthz returning 200 while the DB pool is exhausted keeps a useless backend in rotation. Health checks must exercise the real dependency path, but not so deeply that a slow dependency ejects every backend at once (a self-inflicted outage).
• Two-node quorum. Building split-brain protection on an even number of coordinators gives you no majority to arbitrate a partition. Always deploy 3 (or 5) etcd/Consul/ZK members.

When Health Checks Cascade — Panic / Fail-Open

The health-check ejection loop from the top of this page (probe, eject after N failures) has a failure mode that is worse than the problem it solves. Ejection is built on an independence assumption: a few backends go bad while the rest stay healthy, so removing the sick ones concentrates traffic on the good ones. That assumption holds for uncorrelated hardware faults. It shatters when the failure is correlated — a bad deploy rolled to every host, a shared dependency (the auth service, a config server, a DNS resolver) that just blipped, or a health check whose probe path itself started timing out.

When the cause is shared, backends don't fail one at a time — they fail together, within one or two probe intervals. Naive ejection now does exactly the wrong thing: it dutifully removes host after host as each crosses its failure threshold, and the pool of eligible targets shrinks toward zero. The moment it hits zero, the load balancer has no one to route to, and every request returns a connection error. A partial degradation — say 60% of backends returning slow 500s while 40% still serve — has been converted by the health-check logic into a 100% outage. The safety mechanism became the outage.

The fix: a panic threshold (fail-open)

The countermeasure is to cap how much of the pool ejection is allowed to remove. Below a configured floor — the panic threshold — the load balancer stops trusting the health signal and reverts to routing across all hosts, healthy or not. Envoy calls this panic mode; its healthy_panic_threshold defaults to 50%: if fewer than half the hosts in a cluster are healthy, Envoy ignores health status entirely and load-balances over the whole set.

The reasoning is a comparison of two bad options, not a claim that the sick hosts are fine. If 90% of your backends are failing health checks, one of two things is true: either 90% of your fleet is genuinely down (in which case the check is telling the truth, but ejecting them changes nothing — there is no healthy capacity to shift onto), or — far more likely — the check itself is lying because of a correlated blip, and most of those hosts can still serve some traffic. In both cases, a struggling backend that answers 40% of requests beats a guaranteed 0% from an empty pool. Fail-open bets that spreading load thin across everything degrades more gracefully than routing to nothing.

The drill it answers

"100% of your backends just failed the health check. What does the load balancer do?"

The junior answer is "it ejects them all and returns 503s" — which is the cascade. The staff answer names the panic threshold: below the floor, the LB stops ejecting and routes to every host, because when almost nothing is healthy the health signal has lost its discriminating power and an empty pool is strictly worse than a degraded one. Then it goes one level deeper — this is also why the health check must not probe so deep that a single shared-dependency stall trips every host at once (the "health check that lies" pitfall above): panic mode is the backstop for when that discipline fails, not a substitute for it.

Traced, with a 20-host pool and a 50% panic threshold:

Trade-off: strict ejection vs panic/fail-open — why you need both thresholds

These are not competing philosophies; they defend against opposite failure shapes, and a correct configuration carries both.

The panic threshold is the hand-off point between the two regimes. Above it, you trust the health signal and let ejection protect users from the sick minority. Below it, you distrust the signal and fail open to protect users from an empty pool. Set the floor too high and you fail open on ordinary churn, sending traffic to hosts you should have ejected; set it too low (or disable it) and you re-expose the cascade cliff. Envoy's 50% is a reasonable default precisely because it splits "most hosts fine, a few sick" from "the failure is correlated" — the exact boundary where ejection flips from protective to catastrophic.

When to use active-passive vs active-active

Signals that point to active-passive: traffic comfortably fits one node; the LB holds hard-to-replicate state (deep L7 session affinity, sticky TLS); you're running a vendor HA pair (F5, Citrix) where active-passive is the supported default; simplicity and a provably-correct failover matter more than utilization.

Signals that point to active-active: peak traffic exceeds a single node's capacity; a multi-second failover gap is unacceptable; you can push the LBs toward stateless (state in Redis/DB, or consistent-hash routing) so any node can serve any request.

Choose active-passive when one node is enough and the LB is stateful — the simplest thing that survives a failure wins. Prefer active-active when you've outgrown a single node or can't tolerate a failover gap, and you've done the work to make the LBs near-stateless. Reach for stateless ECMP/anycast when you operate at hyperscale (many LBs, own the routers) and want failure handled by routing rather than by a heartbeat racing a timer.

How availability composes: nines, series, and redundancy

"Highly available" is only meaningful as a number, and the number you promise is a budget. Translate nines into downtime before anything else:

Availability composes along the request path, and the rules are not intuitive. Components a request must pass through in series multiply, because the request survives only if every hop is up. Take a request that traverses an LB at 99.99%, an app tier at 99.9%, and a database at 99.9%:

A_system = 0.9999 × 0.999 × 0.999 ≈ 0.9979  →  99.79%  (~1.5 h/month down)

Two things fall out immediately. First, a serial chain is always less available than its weakest link — 99.79% is below even the 99.9% tiers. Second, the weakest tier dominates: the two 99.9% tiers, not the 99.99% LB, set the result, so spending to push the LB to five nines is wasted while the DB sits at three. Fix the weakest serial link first.

You raise a tier by adding redundancy — putting components in parallel, where the group is up unless all members are down. For independent components the group's unavailability is the product of the individual unavailabilities, so:

A_parallel = 1 − ∏(1 − a_i)
Two independent 99% replicas:  1 − (1 − 0.99)² = 1 − 0.01² = 1 − 0.0001 = 0.9999  →  99.99%

Two ordinary 99% nodes in parallel yield 99.99% — redundancy is how you buy nines. This is the whole game of HA work: find the weakest tier on the serial path and add parallelism there, rather than gold-plating a tier that was never the bottleneck. (The math assumes independent failures; correlated failure domains — same rack, same AZ, same bad deploy — break the product rule, which is why redundancy must span failure domains to actually deliver the computed nines.)

Takeaways

• HA needs two loops: backend health checks (eject sick servers) and LB-to-LB failure detection (move the VIP). One without the other still has a single point of failure.
• Failover is not instant — a VRRP master-down timer (3 × advert + skew ≈ 3.6s at defaults) plus ARP re-convergence is a real, measurable outage window; replicate connection state or retry to hide it.
• The signature failure of active-active is split-brain; solve it with a quorum-backed lease (etcd/Consul/ZK, odd member count) that fences the isolated node, never with a longer timeout.
• Size active-active nodes for N−1: the survivor must absorb the dead node's full load, so a pair should each run at half capacity.
• Ejection assumes failures are independent; below a panic threshold (~50%) stop trusting the health signal and fail open — an empty pool is strictly worse than a degraded one.

Sources: RFC 5798 (VRRP v3) for the advertisement/master-down timing; the keepalived and Linux conntrackd documentation for VIP failover and connection-state sync; HashiCorp Consul and etcd docs (Raft, leases/TTLs, odd-sized clusters) for quorum-based fencing; Google's "Maglev: A Fast and Reliable Software Network Load Balancer" (NSDI 2016) and the HAProxy/NGINX operations guides for health-check and active-active practice. Re-authored and deepened for this guide.