Fault Tolerance vs High Availability

Both strategies fight the same enemy — a component failure becoming a user-visible outage — but with opposite mechanisms: fault tolerance (FT) runs redundant components in lockstep so a surviving copy carries the in-flight request the instant a peer dies (the failure is masked, not recovered from), while high availability (HA) accepts a brief interruption and shrinks it by detecting the failure fast and failing over to a standby. FT spends money to make the gap invisible; HA spends money to make the gap short.

This distinction is where two absolutes commonly stated about FT break down and must be corrected. Real fault-tolerant systems do not guarantee "zero downtime, ever" and do not guarantee "no data loss ever" — they bound both quantities for a class of faults. A single independent hardware fault can be masked in well under a second; a correlated fault (the same software bug crashing both replicas, a bad deploy, an entire AZ losing power) defeats lockstep. And "no data loss" is a property of synchronous replication (RPO = 0), not of FT in general — an asynchronously replicated "fault tolerant" setup loses whatever was in the replication lag window.

How each mechanism actually works

Fault tolerance — masking. Two (or more) replicas execute the same instruction stream and stay bit-for-bit identical. VMware vSphere FT and Stratus ftServer do this in hardware lockstep; a synchronous state-machine-replicated database (Raft/Paxos with synchronous commit) does it in software. When one replica faults, the other is already at the exact same state, so it keeps answering with no re-computation. The cost is structural: you run N copies doing identical work, and every write must be confirmed by the peer before it commits, which adds latency to the hot path. FT masks independent faults; it cannot mask a fault whose cause is shared by all replicas.

High availability — fast recovery. One node serves; one or more standbys wait, kept warm by replication. A failure detector (heartbeats) notices the primary is gone, a promotion step makes a standby the new primary, and traffic is redirected (VIP move, DNS, or a connection pool re-pointing). During that window the service is unavailable and any in-flight requests fail — clients must retry. HA is dramatically cheaper because standbys don't do redundant work in lockstep and writes don't pay a synchronous cross-node round trip (if replication is async), but you accept a measurable RTO (recovery time) and, with async replication, a non-zero RPO (data loss).

Worked example 1 — the nines, in real time

Availability targets are quoted as "nines." Convert the percentage into an annual downtime budget (one year = 525,600 minutes):

Now spend the budget. Suppose your HA cluster fails over 4 times a year and each failover takes the 12 s traced below: 4 × 12 s = 48 s of outage → comfortably inside the 5.26 min five-nines budget. But if a subtle bug makes detection take 60 s and you failover 10 times a year: 10 × 60 s = 600 s = 10 min → you blow five nines and even four nines (52.6 min) is now at risk if a couple of failovers stall. The lesson: with HA, your annual downtime is (failover count) × (failover duration) — and availability is 1 minus that over the year. Both factors are engineering variables, and detection time usually dominates the duration.

Worked example 2 — trace one HA failover (the 12-second gap)

A primary Postgres (or an API-gateway upstream) crashes at t = 0. A typical managed-failover timeline:

RTO ≈ 12 s. Now the data question. If replication was synchronous, every committed write was already on the standby → RPO = 0, no data loss, but every commit on the primary had paid a cross-node round trip (e.g. same-region cross-AZ ≈ 1–2 ms added per commit). If replication was asynchronous with, say, 5 s of lag at crash time → RPO ≈ 5 s of writes are gone. This is exactly why "FT guarantees no data loss" is wrong as an absolute: the guarantee comes from synchronous replication, and you pay for it in write latency.

Side-by-side

Pitfalls

• Treating "fault tolerant" as "invincible." Lockstep only masks independent faults. A bad deploy, a poison-pill message, or a logic bug executes identically on every replica and takes them all down at once. FT buys you nothing against correlated failure — that needs canary deploys, blast-radius limits, and rollbacks.
• Assuming failover is free of data loss. Most cloud "HA" defaults to async replication for write performance. Under load, replication lag grows exactly when you're most likely to crash — so your RPO is worst precisely at the worst moment. Measure lag; alert on it.
• Split-brain. If the old primary isn't truly dead (just partitioned) and a standby is promoted, you now have two primaries accepting writes. Without fencing / a quorum, they diverge and you get silent corruption. This is why detection needs a majority quorum, not a single observer.
• False-positive failovers. A too-aggressive heartbeat (e.g. 1 miss = dead) turns a 200 ms GC pause into a full failover, adding outage instead of preventing it. The 3-miss rule in the trace is deliberate.
• Buying five nines and losing it at the gateway. A five-nines database behind a single-instance API gateway inherits the gateway's availability. Redundancy must be end-to-end; the chain's availability is the product of each independent tier.
• No bulkheads — one slow dependency melts the whole fleet. Multi-AZ redundancy (HA) keeps green health checks even while users get errors, because a shared worker pool has no fault isolation. Give each downstream its own bounded pool — e.g. 100 gateway connections reserved for orders, 100 for search — so a search meltdown exhausts only its 100 and orders keep succeeding. Without the bulkhead, a stalled search backs up until it holds every worker and the "healthy" fleet serves a total outage. Bulkheading is fault isolation within a redundant fleet — it complements HA rather than replacing it.
• Retries without idempotency. HA leans on clients retrying the requests that failed during the gap. If those writes aren't idempotent (idempotency keys, dedup), retries double-charge or double-book.

When to use which — and the trade-off

Reach for true fault tolerance when a single failed request is itself catastrophic and cannot simply be retried: a flight-control loop, an infusion pump, a telecom voice switch mid-call, the innermost ledger commit of a payment. Signals: human safety or irreversible money is on the line per operation; the operation is not naturally retryable; and you can afford 2x+ hardware plus the synchronous write-latency tax on every commit.

Reach for high availability — the right default for almost all web-scale systems — when a short, rare blip is tolerable because clients (or a queue) can retry: e-commerce checkout, SaaS APIs, an API gateway fleet. Signals: requests are idempotent or can be made so; users tolerate a 10–30 s reconnect a few times a year; budget matters.

The trade-off vs the named alternative. Choosing FT over HA buys you a sub-second, invisible gap — but it costs you: double the compute doing redundant work, added commit latency from the synchronous write path, and far more coordination complexity — and it still doesn't cover correlated faults, which are the failures that actually cause most real outages. Choosing HA over FT saves that money and latency and keeps the architecture simpler — but it costs you a measurable RTO (seconds of downtime) and, unless you pay for synchronous replication, a non-zero RPO. Crisply: choose FT when a single operation failing is unacceptable and unrepeatable; prefer HA when a brief, retryable interruption is acceptable — which is why the pragmatic pattern is HA everywhere, with synchronous replication (RPO = 0) reserved for the narrow slice of state where losing even one committed write is not allowed. Most systems marketed as "fault tolerant" in the cloud are, precisely, HA + redundancy.

Takeaways

• FT masks a fault (sub-second, invisible) for a class of independent faults; HA recovers from it fast (a measurable RTO). Both are defeated by correlated failure.
• "Zero downtime" and "no data loss" are not free properties of FT — they are bounded quantities: sub-second gaps for covered faults, and RPO = 0 only with synchronous replication (paid for in write latency).
• With HA, downtime = (failover count) × (failover duration), and availability is 1 minus that over the year; detection time usually dominates, so the nines you actually hit are an engineering choice, not a purchase.
• Default to HA + idempotent retries; spend on true FT / synchronous replication only for the state or operations where a single lost or interrupted request is genuinely unacceptable.

Re-authored and deepened for this guide. Nines-to-downtime figures and failover reasoning follow Google's Site Reliability Engineering (availability tables, error budgets). Replication, RPO/RTO, leader election, synchronous vs asynchronous replication and split-brain draw on Martin Kleppmann, Designing Data-Intensive Applications (ch. 5–9). Lockstep fault tolerance references VMware vSphere Fault Tolerance and Stratus ftServer documentation; the HA-first, redundancy-end-to-end guidance aligns with the AWS Well-Architected Reliability Pillar. The original page's absolutes ("no downtime even during failure," "no data loss") were corrected to reflect real failover windows and RPO > 0.