CQRS & Event Sourcing in Depth — Optimistic Concurrency, Projection Ordering & Multi-Projection Offsets (Deep Dive)

What this page owns that the other CQRS pages don't

Several pages in this guide already introduce CQRS — splitting a normalized write model from a denormalized read model, and why (see the circuit-breaker/bulkhead/CQRS-intro deep dive and the CQRS architecture/example pages elsewhere in this guide for that motivation). None of them answer the three questions that actually decide whether a CQRS/event-sourced system is correct under concurrency and failure: how do two concurrent writers not corrupt one aggregate, precisely what order does the event log guarantee, and how does a read model stay correct — or silently rot — while it's rebuilt from that log? This page is the write-side and projection mechanics deep dive. It assumes you already know what CQRS is and why you'd split read/write; it does not re-derive that motivation.

1. Write-side optimistic concurrency control: compare-and-append

In event sourcing, an aggregate's state is never overwritten in place — it's the fold of every event ever appended to its stream, in order. A command handler that wants to change an aggregate must therefore: load the stream, replay it to get current state and its current version (the count of events so far), validate the command against that state, and append the resulting new event(s) — but only if the stream is still at the version it was loaded at. That last clause is optimistic concurrency control (OCC), and the mechanism is called compare-and-append: the append call carries an expectedVersion, and the event store atomically checks "is the stream still at expectedVersion?" before writing. If yes, the event lands and the version increments. If no — someone else appended in between — the store rejects the write with a concurrency conflict instead of silently accepting a decision made against stale state.

This is the direct analogue of a database's compare-and-swap: instead of "UPDATE balance SET balance = 470 WHERE id = 42" (which blindly overwrites whatever is there), OCC on an event stream says "APPEND Withdrawn(30) WHERE stream Account-42 is at version 5" — the write is conditioned on the state it was computed from, not just aimed at a row.

Traced example: two commands race the same aggregate

Account-42's stream sits at version 5 (balance $500, from 5 prior events). Two withdrawal commands arrive almost simultaneously — Command A withdraws $50, Command B withdraws $30 — each loading the stream independently at v5:

Step	Command A	Command B	Event store: Account-42
1	loads stream @ v5, balance $500	loads stream @ v5, balance $500	version = 5
2	appends Withdrawn(50), expected=5	—	5==5 → accepted, version → 6
3	—	appends Withdrawn(30), expected=5	5≠6 → rejected, 409 conflict, nothing written
4	—	reloads stream @ v6, balance now $450, recomputes withdrawal against the fresh balance	version = 6
5	—	appends Withdrawn(30), expected=6	6==6 → accepted, version → 7, balance $420

Without OCC, step 3 would have blindly appended Withdrawn(30) as if Command A never happened — both commands read $500, both decided $500 was enough to withdraw from, and the aggregate would end up debited twice from a stale premise (the split-brain: two commands, each individually valid against the state they saw, together violating the invariant that governs the real, current state). Compare-and-append turns that race into a detected conflict plus a retry, not a silent double-debit.

2. Ordering guarantees, stated precisely

"Message buses can reorder events, so design for it" is true but useless without the precise guarantee. Kafka (the most common backbone for this) gives exactly one ordering guarantee: events within a single partition are delivered in the order they were appended. There is no ordering guarantee across partitions — two events in different partitions can be consumed in either order, or concurrently, regardless of when they were produced.

That guarantee only helps you if every event for a given aggregate lands on the same partition. The remedy is to set the Kafka message key to the aggregate ID (e.g. Account-42) rather than leaving it null or using something unrelated (a request ID, a random UUID). Kafka's default partitioner hashes the key to pick a partition, so every event keyed Account-42 — Opened, Deposited, Withdrawn, ... — deterministically lands on the same partition and is therefore consumed in the order it was appended. Events for Account-43 may land on a different partition and interleave with Account-42's events in wall-clock time — that's fine, because nothing about Account-42's invariants depends on their relative order to Account-43's events. What must never happen is Account-42's own Withdrawn being processed before its own Opened because they were split across partitions with no ordering relationship between them.

• Partition by aggregate key → per-aggregate order preserved, cross-aggregate order undefined (and unneeded).
• Partition by something else (or unkeyed/round-robin) → an aggregate's own events can scatter across partitions and be consumed out of order — a projection can see Withdrawn before Opened, or a later version applied before an earlier one.
• Ordering is also bounded by partition count: more partitions raise consumer throughput/parallelism but each aggregate is still pinned to exactly one of them, so partition count doesn't threaten per-aggregate order — only a key choice that isn't the aggregate ID does.

3. Projection building: version-guarded drop is correct only for full-state events

A projection (read model) is built by consuming the event stream in order and applying each event to update materialized state. Because consumers can be redelivered a message (at-least-once delivery, consumer restarts and re-reads uncommitted offsets) or, per §2, occasionally see events slightly out of order, projections commonly guard against reprocessing with a simple rule: track the last-applied version per aggregate, and drop any incoming event whose version is ≤ that stored version.

That guard is correct only when each event carries the full new state (a snapshot-style event: "Order-9 is now: status=SHIPPED, items=[...], total=84.50"). If a duplicate or late arrival at version 4 shows up after version 6 was already applied, dropping it changes nothing — the full state at version 6 already supersedes whatever version 4 contained. Applying the same full-state event twice, or skipping an old one, is idempotent by construction.

It is wrong for delta/incremental events (an event that carries a relative change: "+15.00 to running total", "append item to cart"). Each delta is only meaningful applied exactly once, in order, on top of the state the previous deltas produced — there is no "already superseded" for a delta, because unlike a full-state snapshot it doesn't contain the information the skipped event contributed. Dropping a delta event doesn't just risk a duplicate application — it permanently loses whatever that delta changed.

	Full-state event	Delta event
Payload	entire new state: {balance: 420}	relative change: {delta: -30}
Version-guard drop (v ≤ last-applied)	safe — a later full-state event already subsumes it	unsafe — the delta's contribution is not recoverable from any other event
Out-of-order arrival (v4 after v6 applied)	drop v4, no harm; final state already correct	dropping v4's delta permanently understates/overstates the aggregate — e.g. balance ends up $30 too high forever
Safe strategy	version-guard drop is a legitimate optimization	must guarantee strict in-order, exactly-once delivery per aggregate (partition-by-aggregate-key from §2, plus a per-aggregate applied-version check that halts and reprocesses rather than dropping on any gap)

Concretely: an OrdersByCustomer projection built from full-state OrderSnapshot events can use the cheap version-guard and move on. A RunningTotalsByCustomer projection built from AmountAdded(+15.00) delta events cannot — it must detect a version gap (expected v5, saw v7) and treat that as a fault to resolve (reread from the log at v5) rather than a duplicate to silently skip.

4. Multi-projection offset ownership

A single event log commonly feeds several projections at once — OrdersByCustomer, OrdersByProduct, an Analytics rollup — each shaped for a different query pattern. The mechanism that makes this safe is that each projection owns its own consumer offset, independently of every other projection reading the same log. In Kafka terms, each projection runs as its own consumer group; the broker retains events regardless of which groups have consumed them, and each group's committed offset only describes that group's progress.

Two consequences follow directly, and this is the detail that separates someone who's memorized "CQRS has read models" from someone who understands the mechanism:

• Projections can legitimately sit at different points in the log at the same time. OrdersByCustomer caught up to the newest event is not "correct" while OrdersByProduct five events behind is "broken" — both are behaving exactly as designed; each is eventually consistent with the write log at its own pace.
• Rebuilding one projection means resetting only its own offset to 0 and replaying — the log itself, and every other projection's offset, is untouched. This is also how you fix a projection with a bug in its apply logic, or add a brand-new projection after the fact: point a new consumer group at offset 0 on the existing log and let it fold the entire history into a fresh read model, with zero risk to the projections already running.

The corollary failure mode this prevents: if all projections shared one offset (or one consumer), a single slow or crashing projection (say Analytics throws on a malformed event at position 2) would stall consumption for every projection behind that shared checkpoint. Per-projection offset ownership means Analytics can be stuck, paged on, and fixed independently while OrdersByCustomer and OrdersByProduct keep serving fresh reads without interruption — isolation of failure is a direct consequence of offset ownership being per-projection, not a separate feature bolted on.

5. CAP/PACELC stance and crash recovery

Label it explicitly rather than leaving it implicit: the CQRS read model is eventually consistent with the write model. A write completes as soon as its event is durably appended to the log — the projection update happens afterward, asynchronously, on its own consumer lag. In PACELC terms this is PA/EL: if the network partitions, the write side stays available and keeps accepting commands rather than blocking for the read side to catch up (choosing A over C); and even with no partition at all, there is still a real latency-vs-consistency trade-off (EL) — the system chooses low write latency and pays for it with a nonzero, variable window where a read can return state that is behind the most recent committed write. A client that writes and immediately reads its own write through the projection can observe stale data purely because of that lag, not because anything is broken.

Crash between events. Consider a handler (or, in the transfer scenario below, a saga step) that emits event 1, then crashes before emitting event 2 of the same logical operation. On restart it must not blindly re-run from the top — that would emit event 1 a second time. The fix is the same idempotency discipline projections already need: every event carries a stable identity (aggregate ID + expected version, or an explicit idempotency/correlation key), so a resumed handler can check "did event 1 for this operation already land?" before re-emitting it, and a projection replaying from an offset reset can re-apply an event it's already seen without double-counting (which is exactly why full-state projections are naturally safe to replay, and why delta projections must track applied-version per aggregate rather than assume replay is a no-op). Idempotent, replayable projections are what make "crash anywhere, resume from the last durable checkpoint" a correctness property instead of a hope.

6. The hard case: an atomic-looking change across two aggregates

Everything above assumes one command changes one aggregate's stream. A funds transfer from Account-42 to Account-77 needs two aggregates to change together — and OCC's compare-and-append is scoped to a single stream; there is no built-in "append to two streams atomically" primitive. Naively appending Debited(30) to Account-42 and then Credited(30) to Account-77 as two separate calls reintroduces the exact crash-between-events gap from §5: if the process dies after the debit lands and before the credit does, $30 has vanished from the system's own books.

The standard resolution is a process manager (saga): a transfer is its own short-lived stateful coordinator, not a single command. It emits TransferInitiated(transferId=T1), then drives the steps with the same idempotency discipline as §5 — each step keyed by transferId so it can be safely re-driven:

1. Append Debited(30) to Account-42, expected version N. On success, record "debit step of T1 done."
2. Append Credited(30) to Account-77, expected version M. On success, record "credit step of T1 done" → transfer complete.
3. If the process crashes between steps 1 and 2, the saga resumes on restart, sees "debit done, credit not done" for T1, and re-attempts only step 2 — it does not re-debit Account-42, because the idempotency check on T1's debit step short-circuits it.
4. If step 2 can never succeed (Account-77 closed, business rule violated), the saga issues a compensating event — DebitReversed(30) on Account-42 — rather than leaving the system in a partially-applied state.

This gives eventual atomicity across the two aggregates, not the true single-transaction atomicity a monolithic database transaction would give you for a two-row UPDATE. That gap — and the compensating-event bookkeeping it requires — is a real, named cost of aggregate boundaries drawn per entity rather than per business operation, and it's the concrete reason CQRS/ES system designs treat "what is one aggregate's consistency boundary" as a first-class modeling decision, not an afterthought.

7. When CQRS/ES is overkill

None of this machinery is free: two data models to keep in sync, an event schema to version and migrate, per-projection offset/replay infrastructure, and — as §6 shows — saga bookkeeping the moment an operation spans more than one aggregate. A modular monolith with one well-indexed relational schema (plus read replicas for scaling reads) gets you most of the read/write separation benefit with none of that cost, and is the right default when:

• Read and write load are roughly balanced and neither needs to scale or evolve independently of the other.
• You don't need a full audit trail or the ability to reconstruct "what was true as of time T" — a plain updated-in-place row is fine.
• The team is small enough that running an event store, multiple consumer groups, and projection-rebuild tooling is a genuine operational tax, not a rounding error.

Reach for CQRS/ES specifically when the read shape (not just read volume) is genuinely different from the write shape, when regulatory/audit requirements need the full event history rather than current-state-only, or when the write-side business process is itself event-driven across services and the event log is the natural integration point — not merely a scaling knob bolted onto CRUD.

Pitfalls

• Skipping OCC "because conflicts are rare." Rare concurrent writes to the same aggregate are exactly the case that goes undetected in testing and corrupts data in production under real traffic — compare-and-append costs one version check per write and removes an entire class of split-brain bugs.
• Keying events by anything other than the aggregate ID. Keying by request ID, timestamp, or leaving the key null scatters one aggregate's own events across partitions and silently breaks the only ordering guarantee Kafka gives you.
• Applying the full-state version-guard to a delta projection. This is the single most common projection bug in event-sourced systems: it looks correct in every test that delivers events in order, and only loses data on the redelivery/reorder path that tests rarely exercise.
• One shared offset/consumer for many projections. Couples every read model's availability to the slowest or buggiest one; per-projection offsets are what let you rebuild or debug one without freezing the rest.
• Treating projection lag as a bug to "fix" rather than a labeled trade-off. Read-your-writes complaints against an eventually consistent projection are a UX/routing problem (read from the write side for that one query, or show an optimistic local update) — not evidence the projection pipeline is broken.
• Multi-aggregate operations without a saga. Two sequential compare-and-append calls with no coordinator and no idempotency key is a silent way to reintroduce exactly the "half-applied transfer" bug event sourcing was supposed to prevent.

Judgment: the decisions this page actually tests

Decision	Choose this when	Choose the alternative when	vs. a named alternative
Optimistic concurrency (compare-and-append)	Concurrent writers to the same aggregate are possible and correctness matters more than avoiding a rare retry	A single-writer-per-aggregate guarantee already exists upstream (e.g. one partition-pinned actor) — the check becomes pure overhead	vs. pessimistic locking: a lock blocks the second writer until the first commits (no retries, but throughput drops and you own deadlock risk); OCC lets both proceed to the point of conflict and only one redoes work — better throughput when conflicts are actually rare
Full-state vs delta projection events	The read model just needs "current state" and can tolerate a slightly bigger event payload	The event volume/size makes shipping full state wasteful, or the domain event is naturally a delta (a ledger entry, a counter increment)	vs. each other: full-state buys you cheap version-guard idempotency and simple replay; delta buys smaller events and a natural audit log of individual changes, at the cost of needing strict in-order, gap-free delivery to stay correct
CQRS + event sourcing	Read shape/volume is genuinely asymmetric to write shape, or full history/audit reconstruction is a real requirement	Balanced CRUD, no audit requirement, small team — a modular monolith with read replicas is cheaper and simpler	vs. modular monolith + read replicas: replicas solve read scaling with one schema and far less operational machinery; CQRS/ES earns its cost only when the read shape differs, not just the read volume
Saga/process manager for cross-aggregate operations	Any business operation spans more than one aggregate's consistency boundary (transfers, multi-item orders touching separate inventory aggregates)	The operation truly fits inside one aggregate's boundary — no coordination needed	vs. a distributed transaction (2PC): 2PC gives real atomicity but blocks all participants on a coordinator and doesn't survive partitions well; a saga trades true atomicity for availability plus an explicit compensating-action step — the right trade for most cross-service business processes

Takeaways

• Compare-and-append (OCC on expected version) is what stops two concurrent commands from both computing a decision against the same stale state and silently corrupting one aggregate — a rejected conflict plus a retry, not a lock, is the mechanism.
• Kafka's ordering guarantee is per-partition only; partitioning by aggregate ID is the entire remedy for keeping one aggregate's events in order, and cross-aggregate ordering is neither guaranteed nor needed.
• Version-guarded drop is a legitimate idempotency shortcut for full-state projections and a data-loss bug waiting to happen for delta projections — know which kind of event you're consuming before you write that check.
• Each projection owns its own offset: they can legitimately lag each other, rebuild independently from offset 0, and one projection's failure never blocks another's — that isolation is a direct consequence of per-projection offset ownership, not a bolted-on feature.

---

Re-authored/Deepened for this guide.

I just read a lesson on **CQRS & Event Sourcing in Depth — Optimistic Concurrency, Projection Ordering & Multi-Projection Offsets (Deep Dive)** (System Design) and want to truly understand it. Explain CQRS & Event Sourcing in Depth — Optimistic Concurrency, Projection Ordering & Multi-Projection Offsets (Deep Dive) from first principles using ONE vivid real-world analogy and a visual mental model — draw it as ASCII art or a clear step-by-step diagram — with a concrete example using real numbers. Then ask me one question to check I got the mental picture, and wait for my reply. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Teach me **CQRS & Event Sourcing in Depth — Optimistic Concurrency, Projection Ordering & Multi-Projection Offsets (Deep Dive)** interactively. Ask me ONE guiding question at a time, wait for my answer, and adapt to my confusion — build the idea with me step by step instead of explaining it all at once. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Quiz me on **CQRS & Event Sourcing in Depth — Optimistic Concurrency, Projection Ordering & Multi-Projection Offsets (Deep Dive)** with 5 questions, easy to tricky, ONE at a time. Tell me if each answer is right; at the end, explain clearly what I got wrong and why. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Help me remember **CQRS & Event Sourcing in Depth — Optimistic Concurrency, Projection Ordering & Multi-Projection Offsets (Deep Dive)** for the long term: give the one-sentence intuition, a memorable hook/mnemonic, a tiny worked example, and 3 active-recall flashcards (Q -> A). If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.