PrimaryReplica vs PeertoPeer Replication

Both schemes hold multiple copies of the same data and keep them in sync by streaming a log of writes between nodes; they differ on exactly one decision — how many nodes are allowed to accept a write. Primary-replica names one node the sole writer and ships its change log to read-only followers, so copies converge by construction. Peer-to-peer replication (in databases this means multi-leader, or at the extreme leaderless) lets every node accept writes locally and reconciles the divergence afterward.

One correction up front, because it is a common trap: BitTorrent is peer-to-peer file distribution, not peer-to-peer database replication. BitTorrent chops one immutable file into chunks and lets peers fetch chunks from each other — there are never two conflicting versions of the same byte, so there is nothing to reconcile. Real peer-to-peer database replication is the opposite problem: the same mutable key can be written on two nodes at once, and the whole design is about resolving that clash. Think MySQL Group Replication, PostgreSQL BDR, CouchDB, Cassandra/DynamoDB (leaderless), or a calendar app syncing edits made offline on your phone and laptop.

Primary-replica: one writer, and the sync/async knob

All writes go to the primary. The primary appends each change to a replication log (MySQL binlog, PostgreSQL WAL) and streams it to followers, which replay it to stay identical. Reads can be served by any follower, which is why this pattern scales read-heavy workloads cheaply. The load-bearing choice is when the primary considers a write durable:

• Synchronous — the primary blocks the client's commit until at least one follower acknowledges the write. Zero data loss on primary failure, but every write pays the round-trip to that follower, and if the follower stalls, writes stall with it.
• Asynchronous — the primary commits and acks the client immediately, then replicates in the background. Fast and available, but if the primary crashes before a change ships, that tail of committed writes is gone.

Most production systems run semi-synchronous: one synchronous follower for durability, the rest asynchronous for throughput.

Worked example: replication lag breaks "read your own writes"

A user edits their profile bio from Dev to SRE. The primary has one asynchronous follower R2 that is running ~200 ms behind. The app load-balances reads across followers, so the user's next page load happens to hit R2. Trace it in milliseconds:

At t=50 the user just saved a change and the app tells them it never happened. Nothing is corrupt — R2 is simply behind. The standard fixes: route a user's reads to the primary for a few seconds after they write, pin them to one replica (monotonic reads), or track the write's log position and read only from a follower that has caught up to it.

Peer-to-peer / multi-leader: every node writes, then you reconcile

Give up the single writer and every node accepts writes locally — great for multi-region latency (each region writes to its nearby leader) and for offline clients. The price is that two nodes can update the same key concurrently and produce conflicting versions of the same row. You now need a conflict-resolution policy, and the choice determines whether you silently lose data:

• Last-Write-Wins (LWW) — attach a wall-clock timestamp to each write; on conflict keep the higher one, drop the rest. Dead simple, always converges, and is what Cassandra does by default. Its two flaws: it silently discards the losing write (a lost update), and it trusts wall clocks — clock skew between machines can make an older write win.
• Vector clocks (version vectors) — each node keeps a per-node counter. Comparing two versions tells you whether one causally happened-after the other (safe to overwrite) or whether they are concurrent (a true conflict). Concurrent writes are kept as siblings and handed to the application to merge — this is the Amazon Dynamo shopping-cart design. No data is dropped; the cost is app-level merge logic. CRDTs go further and make the merge automatic and deterministic (e.g. a cart is a set; merge = union).

Worked example: vector clocks vs LWW on the same conflict

Key cart:42 lives on two leaders, A (US) and B (EU). A shopper adds items concurrently from two devices, each hitting the nearer leader before replication catches up. Each object carries a version vector {A:_, B:_}.

1. Start: value [], version {A:0, B:0}.
2. Client adds milk at A → value [milk], version {A:1, B:0}.
3. Concurrently (A's write not yet seen) client adds eggs at B → value [eggs], version {A:0, B:1}.
4. Replication: A receives B's version {A:0, B:1} and compares it to its local {A:1, B:0}. On the A-axis 0 < 1; on the B-axis 1 > 0. Neither vector dominates the other → the writes are concurrent → conflict.

This is the concrete reason Amazon's Dynamo chose version vectors over LWW for carts: dropping an item from a shopping cart is a visible, revenue-losing bug, so keeping siblings and merging (never losing an add) was worth the extra complexity.

Failover and leader election

Primary-replica has a hidden cost peer-to-peer avoids: the write path has a single point of failure, so it needs an automatic failover procedure. When the primary dies, the cluster must (1) detect the death (missed heartbeats), (2) elect a new primary — usually the follower with the most up-to-date log, via a consensus protocol like Raft or Paxos — and (3) redirect clients and remaining followers to it. Three things bite here:

• Data loss on async failover. If the old primary had committed writes that never reached the new one, promoting the new primary discards them. GitHub's 2018 outage came from exactly this class of failover hazard.
• Split-brain. A network partition can leave two nodes both believing they are primary, each accepting writes. The guard is a fencing token (a monotonically increasing epoch number) so storage rejects writes from a stale, deposed primary.
• Failover gap. During election (seconds), the system usually accepts no writes at all — a small availability outage baked into the design.

Multi-leader / leaderless designs sidestep leader election for writes entirely (every node already writes), trading the failover problem for the conflict problem you saw above.

Pitfalls

• Treating async replicas as durable. An async follower promoted after a crash silently loses the primary's last few committed writes. If you promised the user their order was saved, that promise can evaporate.
• Reading your own writes off a lagging follower. The t=50 anomaly above. It looks like a data-loss bug to users; it is really routing a fresh reader to a stale replica.
• LWW on data you can't afford to lose. LWW is fine for "last cursor position" or cache-like state; it is quietly catastrophic for counters, carts, or balances, where a dropped write is a real lost update.
• Trusting wall clocks for ordering. NTP skew of tens of milliseconds is normal; under LWW that means a genuinely newer write can lose. This is why systems use logical clocks / version vectors for causality rather than NOW().
• Non-commutative multi-leader operations. "Set balance = 100" replicated both ways is fine; "balance += 10" applied in different orders on different leaders diverges. Multi-leader wants commutative operations (or CRDTs), not arbitrary read-modify-write.
• Split-brain without fencing. Auto-failover plus a flapping network, and no fencing token, gives you two primaries and corrupted data. Detection alone is not enough.

When to use which — and the trade-offs

Choose primary-replica when writes fit comfortably on one node, the workload is read-heavy, and you want strong-ish consistency with simple operations. Signals: a single primary region, a classic web app + relational DB, "scale reads by adding replicas." You gain conflict-free writes and simple reasoning; you pay a write bottleneck at one node, replication-lag read anomalies, and a failover gap when the primary dies. This is the default — reach for it first.

Prefer multi-leader (peer-to-peer) when you need low-latency local writes across regions or offline-capable clients that write while disconnected (mobile, calendars, collaborative docs). Signals: "users in three continents must write fast," "the app works on a plane and syncs later." You gain local write latency and no write-side failover; you pay conflict resolution — LWW loses data, vector clocks/CRDTs add real application complexity — and only eventual consistency across leaders.

Prefer leaderless (Dynamo-style: Cassandra, DynamoDB) when you want no failover step at all and tunable consistency at massive scale. Every replica takes reads and writes; you get consistency by quorum — with N replicas, requiring R + W > N read/write acks guarantees a read overlaps the latest write. You gain maximum availability and no leader to fail over; you pay with read-repair machinery, no free read-your-writes, and "sloppy quorum" edge cases under partition.

Crisp rule: one writable region → primary-replica; many writable regions or offline writes → multi-leader; extreme availability at scale with no failover → leaderless quorum.

Takeaways

• The one real distinction is how many nodes accept writes: one (primary-replica, converges by construction) vs many (peer-to-peer, must reconcile conflicts).
• Sync vs async replication is a durability-vs-latency dial; async replicas that get promoted can silently lose the primary's last writes.
• Multi-leader's whole cost is conflict resolution — LWW is simple but drops data and trusts clocks; vector clocks/CRDTs keep every write but push merge logic into the app.
• Primary-replica pays for its single writer with failover complexity (election, split-brain, fencing); peer-to-peer pays for its many writers with conflicts. You pick which problem you'd rather own.

Re-authored/Deepened for this guide. Sources: Martin Kleppmann, Designing Data-Intensive Applications, ch. 5 (single-leader, multi-leader, and leaderless replication; replication lag; version vectors); DeCandia et al., "Dynamo: Amazon's Highly Available Key-value Store" (SOSP 2007) for vector clocks, sibling reconciliation, and quorum R + W > N; Ongaro & Ousterhout, "In Search of an Understandable Consensus Algorithm (Raft)" for leader election and fencing; and the MySQL (binlog, semi-sync replication, Group Replication) and PostgreSQL (WAL streaming, synchronous_commit) replication documentation.