hard CAP/PACELC for Replication & Reads

Replication is a consistency decision, not just a scaling knob

CAP says that during a network Partition you must give up either Consistency or Availability — you cannot keep both. PACELC completes the sentence for the common case: if Partitioned, choose A or C; Else (normal operation) choose Latency or Consistency. The moment you add a replica, you have taken a PACELC stance whether you meant to or not, because when the write reaches the replica is the whole question.

• Synchronous replication — primary waits for the replica to durably ack before acking the client. Reads can be strongly consistent, but under a partition the primary must either block (unavailable) or drop the replica, and even with no partition every write pays the replica round-trip. This leans CP / EC (consistency, at a latency and availability cost).
• Asynchronous replication + read replicas — primary acks immediately; replicas catch up in the background. Writes are fast and the primary stays available even when replicas lag, but reads off a replica can be stale. This leans AP / EL (available and low-latency, at a staleness cost).

Traced anomaly: reading your own write and missing it

Async replication, replica lag ≈ 200ms, reads load-balanced across primary and replica.

Nothing is broken and no data is lost — the write is safely committed on the primary. The user simply read from a replica that hadn’t caught up. This is a read-your-writes violation, and it is the most common “the app feels buggy” symptom of async read replicas.

Fixes (buy back just enough consistency)

• Read-from-primary-after-write: for a short window after a user’s own write (e.g. read from the primary for the next N seconds, or whenever the thing being read is one the user can edit), route that user’s reads to the primary. Cheap and targeted — it gives read-your-writes without making all reads consistent.
• Monotonic reads / sticky routing: pin a given user to one replica so they never see time move backward (a newer value on one read, then an older value on the next from a further-behind replica).
• LSN / version token: capture the write’s log position (LSN) and only serve the read from a replica that has applied at least that LSN — otherwise wait or fall back to the primary. This is the precise, general form.

Pitfalls

• Round-robin reads break monotonicity: two consecutive reads land on replicas with different lag and the value appears to go backward.
• “Eventually consistent” hides an unbounded tail: replication lag spikes under load or during a large write, so the staleness window is not the average you measured.
• Read-from-primary defeats the point at scale: if too many reads fall back to the primary you lose the read-scaling you added replicas for — scope the fallback tightly.

The judgment layer

Synchronous (CP/EC) vs asynchronous (AP/EL). Choose synchronous when a stale read is a correctness bug — account balances, inventory counts, permission checks, anything a user edits and immediately re-reads — and you can afford the extra write latency and the risk of blocked writes when a replica is down. Choose asynchronous when the workload is read-heavy and latency-sensitive and bounded staleness is acceptable — social feeds, product catalogs, analytics — and patch the sharp edges (read-your-writes, monotonic reads) surgically rather than making every read consistent. In PACELC terms: Cassandra/Dynamo are PA/EL (available and fast, tunable toward consistency per query); Google Spanner and HBase are PC/EC (they hold consistency and pay the latency). The lens forces the question out into the open: what does this specific read do if it’s 200ms stale?

Takeaways

• CAP is only about partition time; PACELC adds the everyday latency-vs-consistency choice you make on every replicated read.
• Sync replication → CP/EC (consistent, slower, less available); async + read replicas → AP/EL (fast, available, stale).
• Read-your-writes breaks when read latency < replication lag; fix with read-from-primary, sticky/monotonic reads, or an LSN token.
• Decide per read path by what a stale value actually costs — not one global setting.

Synthesized from Gilbert & Lynch (CAP proof), Daniel Abadi’s PACELC formulation, and Kleppmann, Designing Data-Intensive Applications (Ch. 5, replication lag and read-your-writes / monotonic reads). Re-authored/Deepened for this guide.