hard CAP/PACELC for Replication & Reads

Replication is a consistency decision, not just a scaling knob

CAP says that during a network Partition you must give up either Consistency or Availability — you cannot keep both. PACELC completes the sentence for the common case: if Partitioned, choose A or C; Else (normal operation) choose Latency or Consistency. The moment you add a replica, you have taken a PACELC stance whether you meant to or not, because when the write reaches the replica is the whole question.

• Synchronous replication — primary waits for the replica to durably ack before acking the client. Reads can be strongly consistent, but under a partition the primary must either block (unavailable) or drop the replica, and even with no partition every write pays the replica round-trip. This leans CP / EC (consistency, at a latency and availability cost).
• Asynchronous replication + read replicas — primary acks immediately; replicas catch up in the background. Writes are fast and the primary stays available even when replicas lag, but reads off a replica can be stale. This leans AP / EL (available and low-latency, at a staleness cost).

Traced anomaly: reading your own write and missing it

Async replication, replica lag ≈ 200ms, reads load-balanced across primary and replica.

Nothing is broken and no data is lost — the write is safely committed on the primary. The user simply read from a replica that hadn’t caught up. This is a read-your-writes violation, and it is the most common “the app feels buggy” symptom of async read replicas.

Fixes (buy back just enough consistency)

• Read-from-primary-after-write: for a short window after a user’s own write (e.g. read from the primary for the next N seconds, or whenever the thing being read is one the user can edit), route that user’s reads to the primary. Cheap and targeted — it gives read-your-writes without making all reads consistent.
• Monotonic reads / sticky routing: pin a given user to one replica so they never see time move backward (a newer value on one read, then an older value on the next from a further-behind replica).
• LSN / version token: capture the write’s log position (LSN) and only serve the read from a replica that has applied at least that LSN — otherwise wait or fall back to the primary. This is the precise, general form.

Pitfalls

• Round-robin reads break monotonicity: two consecutive reads land on replicas with different lag and the value appears to go backward.
• “Eventually consistent” hides an unbounded tail: replication lag spikes under load or during a large write, so the staleness window is not the average you measured.
• Read-from-primary defeats the point at scale: if too many reads fall back to the primary you lose the read-scaling you added replicas for — scope the fallback tightly.
• “We’re CA” is a red flag for any replicated store. Partitions are not optional in a distributed system, so you cannot forfeit P — the honest choice is C-vs-A during a partition. A single-node database can look “CA” only because it has no inter-replica network to partition; the moment you replicate, you have taken a CP or AP stance.

How much does the fallback actually shift? Back-of-envelope: say writes are ~5% of operations and you route a user’s reads to the primary for a window T = 1 s after each of their own writes. Only reads that land in a 1 s window trailing a write hit the primary — on the order of ~5–10% of reads, not all of them, so the replicas still absorb the bulk of read traffic. The lever is T: a longer window catches more read-your-writes cases but pushes more read load back onto the primary and erodes the scaling you paid replicas for; a shorter T (or scoping the fallback to just the entity the user edited) keeps it. T is the correctness-vs-scaling dial — set it just long enough to cover typical replication lag, no longer.

The judgment layer

Synchronous (CP/EC) vs asynchronous (AP/EL). Choose synchronous when a stale read is a correctness bug — account balances, inventory counts, permission checks, anything a user edits and immediately re-reads — and you can afford the extra write latency and the risk of blocked writes when a replica is down. Choose asynchronous when the workload is read-heavy and latency-sensitive and bounded staleness is acceptable — social feeds, product catalogs, analytics — and patch the sharp edges (read-your-writes, monotonic reads) surgically rather than making every read consistent. In PACELC terms: Cassandra/Dynamo are PA/EL (available and fast, tunable toward consistency per query); Google Spanner and HBase are PC/EC (they hold consistency and pay the latency). The lens forces the question out into the open: what does this specific read do if it’s 200ms stale?

What about quorums?

With leaderless replication, N=3/W=2/R=2 makes every read set overlap the latest write set — but R+W>N alone is not linearizability: overlap guarantees the new value is present somewhere in your read set, yet concurrent writes, partially-failed writes, and read-repair timing can still let a later read observe an older value than an earlier read did. If you need linearizable reads from a quorum system you must add machinery — synchronous read repair on the read path, or routing through a leader/consensus. See quorum arithmetic: why R+W>N.

PACELC is per-configuration, not per-product

Cassandra at consistency ONE/ONE is PA/EL; the same cluster at QUORUM/QUORUM pays a round-trip to a majority on every operation and behaves close to PC/EC in normal operation. The stance lives in the settings you deploy, not in the product name.

Worked follow-up: route two reads

Catalog page — tolerates 200ms staleness, so route it to any replica. Cart after “add to cart” — needs read-your-writes, so serve it via an LSN token (a replica caught up past the user’s write) or from the primary. One routing rule per read path, chosen by what a stale value costs there.

Takeaways

• CAP is only about partition time; PACELC adds the everyday latency-vs-consistency choice you make on every replicated read.
• Sync replication → CP/EC (consistent, slower, less available); async + read replicas → AP/EL (fast, available, stale).
• Read-your-writes breaks when read latency < replication lag; fix with read-from-primary, sticky/monotonic reads, or an LSN token.
• Decide per read path by what a stale value actually costs — not one global setting.

Synthesized from Gilbert & Lynch (CAP proof), Daniel Abadi’s PACELC formulation, and Kleppmann, Designing Data-Intensive Applications (Ch. 5, replication lag and read-your-writes / monotonic reads). Re-authored/Deepened for this guide.