Examples of CAP Theorem in Practice

Every one of these systems runs the same reflex at its core: when a node cannot reach a majority (a quorum) of its peers, does it keep answering, or does it go silent? That single reflex — wired into the replication protocol, not chosen per request — is what drops a database into the CP or the AP bucket. The previous page named the buckets; this page opens the machine and shows the exact mechanism, with real defaults and a timed trace, so you can predict a system's behaviour under partition instead of memorising a label.

CP mechanism: quorum voting and the failover pause

CP systems make the minority side go dark on purpose. The mechanism is a quorum vote — a majority of ⌊N/2⌋ + 1 nodes must agree before any state advances.

• ZooKeeper uses Zab (ZooKeeper Atomic Broadcast). The leader turns each write into a numbered proposal and commits it only once a quorum acks. With a 5-node ensemble, quorum is 3. A 3∣2 partition leaves the 3-node side able to commit; the 2-node side cannot form a quorum, so its leader steps down and its nodes enter a LOOKING state — and a LOOKING server serves neither writes nor reads: it drops its client sessions, so the minority side is fully dark and clients must fail over to the majority. That total minority unavailability is exactly the CP price — the ensemble would rather answer nothing than answer wrong. (The famous "ZooKeeper reads can be stale" property belongs to healthy followers connected to a leader: they answer reads from local state, which can lag the leader's latest commit, and a client that needs an up-to-date read calls sync() before reading. It is a property of the healthy ensemble, not a partition-survival mode — you cannot read from the quorum-less minority.) This is exactly why ensembles are odd-sized: N=6 tolerates the same 2 failures as N=5 but needs a bigger quorum (4), so it only adds latency.
• MongoDB runs a Raft-derived election (protocol version 1). Only the primary takes writes. When it disappears, secondaries that miss heartbeats past electionTimeoutMillis (default 10 000 ms) call an election, and the winner needs a majority of votes. During that window there is no primary a majority can reach, so writes are refused — the availability sacrifice. Critically, a former primary stranded on the minority side cannot reach a majority for its own heartbeats, so it self-demotes to secondary — that is what prevents two primaries (split-brain).

Worked trace: a MongoDB failover with real timings

Three-node replica set — primary P, secondaries S1, S2; quorum = 2; defaults electionTimeoutMillis=10000, heartbeats every 2 s; the application writes with w:"majority".

Net effect: a ~10–12 s write outage per failover, but zero divergence. Because the client used w:"majority", every acknowledged write already lived on at least two nodes, so it survives on S1/S2 — nothing acknowledged is lost. That durability guarantee is the whole point of paying the pause.

AP mechanism: leaderless writes, reconcile later

AP systems keep every side writing and repair the mess afterward. There is no leader whose loss stalls the cluster.

• Cassandra is leaderless with a replication factor (say RF = 3) and tunable consistency per query. A write of consistency level QUORUM waits for 2 of 3 replicas; a read at QUORUM reads 2. You get strong consistency only when R + W > RF (here 2 + 2 = 4 > 3, so the read and write replica sets are guaranteed to overlap). Under partition you can drop to CL=ONE: the write succeeds on any one reachable replica, and the coordinator stores a hinted handoff for each unreachable replica (default replay window 3 h) to deliver when it returns. Conflicts are resolved by last-write-wins on a microsecond cell timestamp — highest timestamp wins — backed up by read-repair and nodetool repair (Merkle-tree anti-entropy).
• DynamoDB replicates each item across 3 availability zones and acks a write once it is durable on a quorum of them. Reads are eventually consistent by default (served from any replica, possibly <1 s stale); setting ConsistentRead=true routes the read to the leader replica for a strongly consistent answer within that region, at 2× the read-capacity cost. Global Tables make it active-active across regions — in one of two modes. Classic Global Tables (multi-Region eventual consistency, MREC) replicate asynchronously with last-write-wins reconciliation, so that cross-region view is eventual even if you request a consistent read — ConsistentRead=true is region-local. Since mid-2025 there is also an opt-in multi-Region strong consistency mode (MRSC): a strongly consistent read then reflects the latest write from any of the table's Regions (zero RPO), paid for with cross-Region write coordination on every write — the PACELC E-side latency-for-consistency trade made explicit.

When to use which — and what it costs

These four are not interchangeable; each exposes a different knob and charges a different price.

Choose ZooKeeper/etcd for coordination — locks, leader election, small linearizable config. It must be CP because two clients disagreeing on "who holds the lock" is a correctness bug. But it is not a database: it holds its dataset in memory and tops out around tens of thousands of writes/s, so never point application data at it.

Choose MongoDB with w:majority when you want a general document store with read-your-writes and can tolerate a brief write pause on failover — order records, user profiles, content. Prefer Cassandra/DynamoDB instead when a 10-second write outage per failover is unacceptable and the data is append-heavy or naturally last-writer-wins (time series, feeds, telemetry, carts). You gain zero-downtime writes and easy multi-region; you pay with lost-update risk and no cross-row transactions.

Choose DynamoDB over self-run Cassandra when you want the AP model without operating repair, compaction, and clock hygiene yourself, and you value the per-region strong-read escape hatch. Prefer Cassandra when you need control over the storage engine, multi-cloud placement, or want to avoid per-request pricing at very high write volume.

Pitfalls

• Trusting MongoDB's old default write concern. With w:1, a write is acked after only the primary has it. If that primary is the one that gets partitioned away, the write is rolled back on rejoin — an acknowledged write silently vanishes. Always use w:"majority" for data you cannot lose.
• Reading from MongoDB secondaries. Setting readPreference=secondary for scale quietly gives back the consistency you paid the failover pause for — secondaries lag the primary, so you serve stale reads.
• Cassandra quorum math you didn't do. Writing at ONE and reading at ONE gives R + W = 2 ≤ RF = 3: no overlap guarantee, so a read can miss the latest write. Consistency needs R + W > RF.
• Last-write-wins + clock skew. Cassandra and Global Tables pick the highest timestamp. If a node's clock runs fast, its stale write can beat and permanently overwrite a newer one. Run tight NTP, and never do read-modify-write on a hot key without lightweight transactions (Paxos-backed, and much slower).
• Assuming DynamoDB reads are strong. The default is eventually consistent; you must opt in with ConsistentRead=true — and on classic (MREC) Global Tables even that is region-local; only the opt-in multi-Region strong consistency mode makes a consistent read span Regions.
• Chasing "CA". No distributed system is CA. A single-node database looks CA only until the network between the client and that node partitions — then it is simply unavailable.

Takeaways

• The CP/AP label is not a property you choose per request — it is the emergent behaviour of one mechanism: what a node does when it loses quorum.
• CP buys a single consistent history by making the minority stop (ZooKeeper halts; MongoDB pauses ~10–12 s for an election and self-demotes the stranded primary).
• AP buys always-on writes by letting both sides diverge, then reconciling — Cassandra/DynamoDB via hinted handoff and last-write-wins timestamps; correctness now depends on R + W > RF and on well-synced clocks.
• Match the knob to the failure you fear: linearizable coordination → ZooKeeper/etcd; durable strong-ish store → MongoDB w:majority; zero-downtime write scale → Cassandra/DynamoDB.

Sources: Gilbert & Lynch, "Brewer's Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services" (2002); DeCandia et al., "Dynamo: Amazon's Highly Available Key-value Store" (SOSP 2007); Junqueira, Reed & Serafini, "Zab: High-performance broadcast for primary-backup systems" (2011); MongoDB Manual — Replica Set Elections and Write Concern; Apache Cassandra docs — Tunable Consistency, Hinted Handoff, and Repair; AWS DynamoDB Developer Guide — Read Consistency and Global Tables. Re-authored/Deepened for this guide.