Redundancy and Replication

Redundancy and replication

Redundancy is the deliberate duplication of critical components or functions so that a failure in one place does not take down the whole system. If a file lives on exactly one server, losing that server loses the file; keeping a second copy on a second server removes that single point of failure.

Database replication is the mechanism that keeps those duplicate copies (replicas, or followers) in sync with the original. Two independent questions define any replication design, and keeping them separate is half the battle:

Axis 1 — topology: who is allowed to accept writes?

1. Single-leader — one node (the leader, or primary) accepts every write and streams changes to its followers. The default in MySQL, PostgreSQL, and most managed relational services.
2. Multi-leader — several nodes accept writes (typically one leader per region or datacenter) and replicate to each other; each leader keeps an ordered log, and conflicting writes are resolved when the leaders' streams merge.
3. Leaderless — there is no fixed leader at all; any replica can accept a write, and quorums of replicas coordinate to stay convergent (the Dynamo model).

Axis 2 — acknowledgment mode, within leader-based replication: when does the leader tell the client "done"? Synchronous — after a follower confirms it has the write; asynchronous — immediately, replicating in the background; semi-synchronous — the tunable middle ground, waiting for just one fast follower instead of all of them. These are latency/durability settings on a leader-based topology, not topologies themselves: the same single-leader pair can run in any of the three modes.

The rest of this page walks through the ack modes of single-leader replication and the leaderless alternative — how each works, when to reach for it, when not to — plus a worked example of the lag that asynchronous replication introduces, including a precise definition of what "lag" actually measures.

Synchronous replication

In synchronous replication, the leader does not consider a write complete until at least one follower has confirmed it received and applied the change. This gives the pair strong consistency: whatever the client was told succeeded is guaranteed to exist on both the leader and that follower, even if the leader crashes the instant after acknowledging.

When to use it

• The write must survive a leader crash with zero loss (RPO = 0) — financial ledgers, payment postings, inventory decrements, anything where "we told the client it succeeded" must still be true after failover.
• The synchronous follower is close by (same AZ or region), so the extra network round trip adds only a few milliseconds to write latency.
• Regulatory or audit requirements demand a durability guarantee stronger than "probably replicated."

When not to use it

• The follower is in another region: every write now pays that cross-region round trip, and write latency is bounded by the slowest synchronous follower's network — a single degraded link stalls every write.
• The workload is write-heavy and latency-sensitive (bidding, telemetry ingestion, chat fan-out), where a few extra milliseconds per write compound into a real throughput ceiling.
• You only have one synchronous follower and no fallback: that follower's health becomes a de facto dependency for write availability, since the leader blocks on it. (Semi-synchronous replication, below, exists specifically to soften this.)

Asynchronous (and semi-synchronous) replication

In asynchronous replication, the leader commits and acknowledges the client immediately; the change reaches followers afterward, over a queue. This decouples client-perceived latency from follower distance or health, but it means a follower can be meaningfully behind the leader at any given moment. Semi-synchronous replication sits between the two extremes: the leader waits for an acknowledgment from at least one follower — whichever responds first — before committing, while the rest replicate asynchronously. That gives a bounded-loss guarantee at a much smaller latency cost than waiting for every follower.

When to use it

• Read-heavy workloads that spread reads across followers and can tolerate eventually-consistent results — product catalogs, dashboards, most feed reads.
• Cross-region disaster-recovery copies, where a synchronous round trip across regions would be too slow but you still want a warm standby.
• Any system where write latency is the primary tuning lever and losing the last unreplicated writes on a sudden leader crash is an acceptable, bounded risk (semi-sync narrows that window to "whichever follower was fastest").

When not to use it

• A client-visible successful write must never silently disappear on failover — order placement, financial postings. Pure asynchronous replication can lose exactly the writes that were acknowledged but not yet shipped when the leader dies.
• The product needs read-your-writes behavior immediately after a write (e.g., "did my comment post?") without extra plumbing — a stale follower can serve a read that doesn't yet reflect the user's own write.
• You need a durability guarantee tied to a specific follower (e.g., "always durable in region B"). Semi-sync's guarantee is "durable on whichever follower acked first," not a targeted one — if that matters, use synchronous replication to the specific follower you care about instead.

Leaderless replication

Leaderless (quorum-based) replication removes the fixed leader entirely. With N replicas, a write is sent to as many as are reachable and counts as successful once W replicas acknowledge it; a read queries R replicas and returns the most recent version it sees, using per-value versioning (vector clocks or timestamps) to detect which copy is newest. As long as W + R > N, every read quorum is guaranteed to overlap every prior write quorum by at least one replica, so a read can never miss the latest acknowledged write entirely — though it may have to reconcile competing versions. Replicas that fall behind (a network blip, a restart) are brought back in line later through read repair or background anti-entropy, and a temporarily unreachable replica's write can be handed to a stand-in node and reconciled once it returns (hinted handoff).

When to use it

• Multi-region, active-active writes, where every region needs to accept local writes without a round trip to a single leader elsewhere — shopping carts, session/presence state, IoT telemetry ingestion, collaborative-editing metadata.
• Know the named alternative for this exact use case: multi-leader replication also gives each region local writes — but each leader keeps an ordered per-leader log, so conflicts surface at replication time, when the leaders' streams merge, whereas leaderless surfaces them at read/quorum time as sibling versions to reconcile. Reach for multi-leader when per-region write ordering matters (each region's writes replay in order everywhere else); reach for leaderless when per-node write availability matters more than any ordering.
• Workloads that value write availability over strict ordering: the system should keep accepting writes even during a network partition or when a minority of nodes are down, and occasional conflicting writes are tolerable.
• Data types that merge cleanly — counters, sets, last-writer-wins fields, CRDTs — so conflict resolution is cheap and mostly invisible to the user.

When not to use it

• Anywhere a single global order or an exact count is required and can't be fixed up after the fact — unique-constraint enforcement, decrementing finite inventory, account balances, sequence-number allocation. A leaderless quorum can accept two conflicting writes concurrently on different replicas; reconciling that later either drops one write (last-writer-wins) or requires application-level merge logic that many of these domains cannot safely support.
• Teams without the appetite for the operational complexity that comes with it: vector clocks, sibling values, and read repair are real, ongoing complexity that a leader-based design sidesteps by construction.
• Small, single-region deployments (e.g., three replicas in one availability zone) — you take on quorum complexity without the multi-region write-availability payoff that motivates leaderless replication in the first place; plain synchronous or semi-synchronous replication gives comparable durability with a simpler mental model.

What "replication lag" actually measures

Replication lag is, at its core, a time quantity: the delay between the moment a write becomes durable on the leader and the moment that same write becomes durable (or visible to reads) on a follower. In practice it's often tracked via a position delta instead — the gap between the leader's current log sequence number (LSN, or binlog offset in MySQL terms) and a follower's applied LSN — because comparing two log positions is cheap and needs no clock synchronization between hosts.

That position delta is a count of log entries, not a duration, and the two are only related through the write throughput at the time the gap was measured: time-lag ≈ LSN-gap ÷ writes-per-second. Because throughput changes constantly, the same 18-entry gap can represent very different amounts of wall-clock delay depending on how fast the leader was writing when it opened up. Tools such as MySQL's Seconds_Behind_Master or PostgreSQL's replay_lag report a genuine time value because they compare timestamps — often via a small heartbeat row replicated like any other write — rather than reporting the raw LSN difference.

Worked example: watching an asynchronous follower's lag

A leader has one synchronous follower (A) and one asynchronous follower (B). At t = 0 ms the leader appends write W at LSN 1050. The table tracks LSNs and the resulting gap at each point in time; the chart further below plots the same five points, with the x-axis marking each row's timestamp rather than a linearly scaled clock, since the intervals between rows range from 8 ms to 100 ms.

Two things to notice. First, A's lag is never applied-before-acknowledged: the t = 0 ms row shows A one entry behind precisely because the leader is still waiting on its ack, and A only reaches lag 0 once that ack has actually happened, at t = 8 ms. Second, be careful whose rate you divide by when converting an entry gap into time — the table gives you two rates, and they answer different questions. At follower B's apply rate (roughly 15 ms per entry, visible as B drains 13 entries — 1038 → 1051 — over the 200 ms from t = 100 ms to t = 300 ms), an 18-entry backlog is about 270–360 ms of drain time: how long B needs to catch up once the leader goes quiet. Dividing by the leader's write rate instead answers a different question — how much wall-clock write history the gap spans. This leader appends only 5 entries over the whole 300 ms window (1050 → 1055, one every 40–60 ms), so by that measure the same 18-entry gap represents roughly 720–1080 ms of the leader's writing. The formula time-lag ≈ LSN-gap ÷ writes-per-second from the previous section only means something once you say which actor's rate sits in the denominator; here the two answers differ by a factor of two to four.

Choosing a replication design

None of these is a strictly better default — the right choice tracks how expensive a lost acknowledged write is versus how expensive extra write latency (or, for leaderless, conflict resolution) is for the specific data being replicated.

Whichever topology you pick, replicas only buy fault tolerance if they fail independently. A replication factor of 3 with all three copies in the same rack (shared power/switch) or same availability zone converts one correlated event — a rack PDU trip, an AZ network partition — into the loss of every copy at once, defeating the redundancy on paper. Spread replicas across independent failure domains (racks, then AZs, then regions as durability demands rise); the further apart they sit, the more independent the failures but the higher the synchronous-write latency, which is the same latency-vs-durability trade-off the topologies above make, now expressed as physical placement.

What an interviewer probes here

Everything above compresses into a handful of follow-up questions that come up almost verbatim. Check that you can answer each one from this page alone:

• "What's your data-loss window if the leader dies?" Answer per ack mode: synchronous — zero for acknowledged writes (RPO = 0); asynchronous — every write acknowledged but not yet shipped is gone, so the window is unbounded; semi-synchronous — bounded, because each acknowledged write exists on at least one follower.
• "So with semi-sync my write is safe on the standby I care about?" No — the guarantee is durable on some follower, whichever acked first, not on a follower you chose. If durability must live in a specific place (say, always in region B), that calls for synchronous replication to that specific follower.
• "Why does W + R > N guarantee a read sees the latest write — and what does it not guarantee?" Because the write quorum and the read quorum together contact more than N replicas, they must share at least one, so a read can never entirely miss the last acknowledged write. It does not give a single global order — concurrent writes can still land divergently and need reconciling — which is why quorum is not linearizability.
• "Your follower is 18 log entries behind — how many milliseconds is that?" A trick question until you name a rate: dividing by the follower's apply rate gives drain time (how long until it catches up); dividing by the leader's write rate gives how much wall-clock write history the gap spans. In the worked example above those two answers differ by a factor of two to four.
• "Replication factor 3 — so you're fault-tolerant?" Only if the three copies fail independently. Three replicas in one rack die together on one PDU trip; the defense is placement across failure domains (racks, then AZs, then regions), accepting more synchronous-write latency for each step up in independence.

Sources

Concepts and terminology in this lesson draw on: Martin Kleppmann, Designing Data-Intensive Applications (O'Reilly, 2017), Chapters 5 ("Replication") and 9 ("Consistency and Consensus"), for the synchronous/asynchronous/leaderless replication models and the W + R > N quorum condition; DeCandia et al., "Dynamo: Amazon's Highly Available Key-value Store" (SOSP 2007), for leaderless quorum mechanics, hinted handoff, and read repair; and the MySQL and PostgreSQL replication documentation for synchronous/semi-synchronous configuration and lag-monitoring conventions (Seconds_Behind_Master, replay_lag). LSNs and timings in the worked example are illustrative, not measurements from a specific running system.