Mental Models II — Coordination Avoidance, Dual-Write/Outbox, Latency Physics, Circuit Breakers & Framing Rules (Deep Dive)

The mental model behind the mental models

Every hard distributed-systems decision reduces to one question: is this constraint a law of physics/logic, or a dial you're free to turn? Coordination, latency, and failure each have a "floor" you cannot engineer past — and a lever you can pull instead. This page is the second layer of intuition on top of the guide's existing mental models (fan-out tail latency, idempotency, the consistency spectrum, back-pressure, mechanical sympathy, the USE method, time/clocks, designing for failure) — it adds the ones those pages don't cover: the dual-write/outbox trap, coordination-avoidance (the CALM theorem), the physics of cross-region latency, the circuit-breaker timeout-budget rule, and the framing rules for reaching for the right model under time pressure.

1. The dual-write problem and the outbox mental model

Writing to your database and publishing to a queue are two separate systems with no shared transaction. Do them one after another and a crash in between either loses the event (DB committed, publish never happened) or double-emits (publish succeeded, then the DB transaction rolled back) — this is the dual-write problem, covered mechanically with its own diagram in Transactional Outbox. The fix there is the same fix restated as a mental model: never let "write my state" and "tell the world" be two atomicity domains. Fold the second into the first by writing an outbox row in the same local DB transaction, then let a separate relay (poller or CDC log-tailer) publish it later. Atomicity moves from "DB + broker" (impossible without 2PC) to "DB + DB" (a single ACID transaction) — the broker gets involved only after the fact is already durable.

The part that mental-model pages skip: the outbox row itself needs a lifecycle, not just an insert. Give every row a status (PENDING → SENT) and a lease/timeout — when the relay picks up a PENDING row it stamps a leased_until timestamp instead of deleting it immediately. If the relay crashes mid-publish, the row is still there; once leased_until expires, a recovery sweep re-claims it and republishes. Trace a crash through it:

The recovery rule is simple — on ambiguity, re-send — because the outbox can only guarantee at-least-once delivery, never exactly-once. That is a deliberate trade, not a gap: the hard part (collapsing duplicates into one effect) is explicitly pushed onto the consumer via an idempotency key. Outbox solves "did the event get out at all"; idempotency solves "what happens when it gets out twice." Neither one alone is the fix — the pairing is.

2. Coordination-avoidance: the CALM theorem

The deeper reason the outbox pattern reaches for "avoid the join" instead of "add a distributed transaction" is a general principle: coordination is the expensive thing in a distributed system — expensive in latency (a round trip per decision), in availability (a coordinator that must be reachable), and in complexity. The end-to-end argument says the same thing from a different angle: don't pay for a guarantee in the middle of the system (say, a broker doing exactly-once) if the endpoint has to re-verify it anyway (the consumer's idempotency check) — build the guarantee once, at the edge that actually needs it, not at every hop.

The CALM theorem (Consistency As Logical Monotonicity, Hellerstein & Alvaro) makes this precise: a program has a coordination-free, eventually-consistent implementation if and only if it is monotonic — every step only adds facts or refines a conclusion, and never has to retract one already announced. Concretely:

• Monotonic (no coordination needed): grow-only sets, counters that only go up, "has event X ever happened" flags, unions/joins over ever-growing data. Once true, always true — any replica can answer immediately and can never be contradicted by a fact that arrives later.
• Non-monotonic (needs coordination): deletes, "is this username unique" (true only if you've seen every other claim), a balance that must never go negative, "the current top-1 leader" if leadership can be revoked. A fact that was true can be un-true once one more input arrives — so you cannot answer safely without knowing you've seen all the inputs, which is exactly what coordination buys you.

The design lever falls straight out of the theorem: reformulate a non-monotonic operation as a monotonic one and the coordination requirement disappears. A delete becomes a monotonic add of a tombstone fact (the record isn't retracted, a "deleted" marker is appended — this is exactly what CRDTs and log-structured stores do). A running total with corrections becomes a monotonic append of adjustment events summed at read time, instead of an in-place decrement. What's left after every reformulation you can manage — uniqueness constraints, "never go negative" invariants, single-leader elections — is the genuine, irreducible coordination in your system; that's where you spend a consensus protocol (Raft/Paxos) or a serializing lock, deliberately and sparingly, rather than by default.

3. Latency physics: the speed-of-light floor

Cross-region latency is not a performance bug waiting for a faster network card — it is arithmetic. Light in vacuum travels at c ≈ 300,000 km/s; in optical fiber it travels at roughly ⅔ c ≈ 200,000 km/s (the glass's refractive index slows it down). New York to London is about 5,600 km of undersea cable. Trace the calculation:

That 56ms is a hard floor — no amount of bandwidth, caching within the request, or code optimization changes it, because it is bound by the length of the wire and the refractive index of glass, not by any software you control. The only lever is to not make the trip: serve from an edge/CDN node near the user, keep a regional read replica so most reads never cross an ocean, replicate asynchronously so the write path doesn't wait on the far region, or bundle several round trips into one (batching, HTTP/2 multiplexing). This is the same "law vs. lever" split as tail-latency thinking and the same reason fan-out amplification matters — you cannot out-engineer physics, only route around it.

4. Circuit breakers and the timeout-budget rule

The mechanism (closed → open → half-open, with a worked failure/recovery trace and a step-through debugger) already has its own deep page: Circuit Breaker — The State Machine — reach for that page for the state-transition detail. What belongs here is the mental model interviewers actually probe once you know the three states: a circuit breaker only protects you from a dependency that is already down; it does nothing about a timeout that is set wrong in the first place, and most production incidents are the second problem.

The rule: per-hop timeouts must sum to less than the caller's overall budget. If a client gives itself a 500ms budget for a request, and every service in a 3-hop chain (Gateway → Service A → Service B → DB) sets its own downstream timeout to that same 500ms, the chain doesn't fail in 500ms — the client gives up at 500ms while A, B, and the DB call are all still separately waiting on their own 500ms clocks, each started slightly later than the one before it. Every hop is holding a thread/connection for work whose answer is already useless. Trace the fix — a shrinking waterfall, each hop's timeout carved out of what its parent has left, sized to the downstream's measured p99, not to "whatever budget remains":

Each row's timeout is strictly smaller than what its caller has left, and is anchored to what the downstream actually does (its p99), not copy-pasted from the top. This is also exactly where the breaker's own threshold should be set: trip on a failure/latency signal measured against that same p99, not an arbitrary round number — a breaker tuned looser than the real p99 never trips before users feel it; tuned tighter than natural variance, it trips on noise.

5. Framing rules — reaching for the right model under time pressure

A decision rule, problem-shape → reach for:

• "Write to a DB and notify other systems, must survive a crash" → outbox/CDC, not two-phase commit.
• "A counter/flag/set that many nodes update independently and rarely conflict" → ask if it can be made monotonic (CALM) — grow-only counters, CRDTs — before reaching for a lock or consensus.
• "Must never violate an invariant (unique username, non-negative balance, one leader)" → this is genuinely non-monotonic; accept the coordination cost (consensus, a serializing transaction) rather than fighting it.
• "A user on another continent is complaining about latency" → check the physics floor first (distance ÷ ⅔c) before assuming it's a code problem; if you're already near the floor, the fix is edge/cache, not "optimize the query."
• "One dependency's slowness is dragging the whole system down" → circuit breaker + bulkhead, with timeouts sized to that dependency's own p99, summed hop-by-hop under the total budget.

Under real interview time pressure, a 6-step framework (Requirements → Estimation → API → Data model → High-level design → Deep dive) rarely fits in full. Cut in this order when the clock is short: (1) trim estimation to stated assumptions instead of derived arithmetic — say "assume 10M DAU, ~1K QPS peak" rather than deriving it live; (2) pick one component to go deep on rather than spreading detail evenly — depth on one thing reads stronger than a shallow pass over six; (3) never cut requirements or the high-level design — those are what the interviewer is actually grading.

Two independence caveats worth stating out loud in an interview, because both are commonly assumed away and both are wrong in practice:

• The tail-math caveat: "probability at least one of N replicas fails = 1 − (1−p)^N" assumes independent failures. If the replicas share a power feed, a rack, an availability zone, or a config push, failures correlate — a single event can take out many of them at once, and the real probability of at least one failure is far higher than the independence formula suggests. This is the same trap as composing "five nines" across components that aren't actually independent (see Designing for Failure's blast-radius framing).
• The commit-wait caveat: even genuinely coordinated, linearizable writes across regions (Google Spanner) still pay a consensus round trip plus a deliberate extra wait. Spanner's TrueTime API returns not a single timestamp but an uncertainty interval [earliest, latest]; before acknowledging a commit, Spanner performs commit-wait — it blocks until the clock is certainly past the commit timestamp's uncertainty bound (on the order of a few milliseconds, ~7ms in the published design) — trading a small fixed wait for external consistency across the globe. It's the same "coordination is expensive, but sometimes unavoidable" lesson as §2, made concrete with a real number.

Pitfalls

• Treating outbox as exactly-once — it isn't; it's at-least-once, and skipping the consumer-side idempotency key just moves the duplicate-event bug from "sometimes" to "always eventually."
• Applying CALM as an excuse to skip an invariant that's genuinely non-monotonic (e.g. eventually-consistent unique usernames) — the theorem tells you when coordination is avoidable, not that it's always avoidable.
• Quoting a latency "budget" without checking whether it's below the physics floor for the regions involved — a 30ms cross-continent SLA is not a stretch target, it's mathematically impossible.
• Setting every hop's timeout to the parent's full remaining budget "to be safe" — this is the single most common cause of threads/connections pinned on calls whose answer nobody is still waiting for.
• Composing tail-failure probabilities 1−(1−p)^N across replicas that share a failure domain (rack, AZ, deploy) — correlated failure makes the real number far worse than the formula.

Judgment layer — when to coordinate vs. avoid

Default to avoidance: reformulate for monotonicity, push idempotency to the edge, cache to dodge the round trip, fail fast with a breaker instead of queuing behind a dead dependency. Reach for real coordination (consensus, distributed locks, synchronous cross-region commit) only for the small, named set of invariants that are truly non-monotonic — uniqueness, "never negative," single-leader — and even then, prefer to shrink the blast radius of that coordination (partition it, scope it to a region, batch it) rather than doing it on every request. Size breaker thresholds and per-hop timeouts to the downstream's measured p99, never to a round number pulled from habit — a badly-set timeout defeats the point of a well-designed breaker.

What to put on a dashboard for this. Each lever has a signal that tells you it is misbehaving: outbox age — the age of the oldest PENDING row and the relay's publish lag (a rising oldest-age means the relay is wedged or the broker is down, and events are silently backing up); orphaned work — count of requests where the client already timed out but a downstream call is still in flight (the copy-the-budget bug in §4); and cross-region p50 sitting near the physics floor (e.g. ~70–80ms NY–London) — when it does, stop filing "the app is slow" tickets, because you are at the wire's limit and only routing around it (edge/replica) will move the number.

Takeaways

• The dual-write problem is solved by moving atomicity from "DB + broker" to "DB + DB" — the outbox row is durable in the same transaction as the business change; a lease/timeout plus a re-send-on-ambiguity rule handles a crashed relay, and idempotency on the consumer absorbs the resulting duplicates.
• CALM: coordination is avoidable exactly when an operation is monotonic (only adds facts). Reformulating deletes/counters as append-only facts is the concrete lever; uniqueness/leader-election invariants are the genuine, irreducible cases that still need it.
• Cross-region latency has a calculable floor (distance ÷ ⅔c) — treat it as a law to route around (cache, edge, async), never as a target to "optimize" past.
• A circuit breaker only helps if the timeout it guards is sized correctly — per-hop timeouts must sum to less than the caller's budget and should track the downstream's measured p99, not an arbitrary constant.

Related pages

• Transactional Outbox — Solving the Dual-Write Problem — the full mechanism (with diagram) behind the outbox lifecycle traced in §1.
• Idempotency & “Exactly-Once Is a Myth” — the consumer-side dedup half of the outbox pairing in §1.
• Circuit Breaker — The State Machine (Closed → Open → Half-Open) — the state-transition mechanics behind the timeout-budget rule in §4.
• Designing for Failure — Blast Radius, Timeouts, Breakers & Bulkheads — the blast-radius framing behind the correlated-failure caveat in §5.
• Tail Latency & Fan-out Amplification — Why p99 Is the Number — the same “law vs. lever” framing applied to tail latency instead of physical distance.

Re-authored/Deepened for this guide.