Saga in Depth — Orchestrated vs Choreographed, Pivot Transactions, Semantic Locks & Coordinator Crash (Deep Dive)

Two ways to drive the same saga

A saga is already familiar from earlier pages: a business operation that spans services becomes a chain of local transactions T1…Tn, each with a compensating transaction Ci that semantically undoes it if a later step fails (see Microservices Patterns — Saga, Transactional Outbox, Event-Driven Architecture). What none of the existing worked examples show end-to-end is the choreographed version of that same flow, the exact moment a coordinator process dies mid-saga, and the wiring decisions — sync vs event-driven per leg, dedup keys, partition keys — that make either style survive production. That is this page's job. It assumes you already know what a saga and a compensation are.

Orchestrated vs choreographed: who decides the next step

Orchestrated: a central coordinator holds the whole plan. It sends a command to a participant, waits for the reply, and decides — and durably logs — what runs next, including which compensations fire in reverse order on failure. You can point at one component and ask "what state is this saga in?" and get an answer. The cost: the orchestrator is a new piece of infrastructure to build, deploy, and keep available, and it tends to accumulate business logic it should not own (the "god coordinator" anti-pattern).

Choreographed: there is no coordinator. Each service reacts to an event, does its local transaction, and publishes the next event. Order commits and emits OrderCreated; Inventory is subscribed, reacts, commits, and emits StockReserved or StockReservationFailed; Payment reacts to whichever one arrives. No single component knows the whole flow — the plan lives implicitly in the union of every service's event subscriptions. That buys looser coupling (participants do not even need to know each other exist, only the event schema) at the cost of traceability: reconstructing "what happened to order #7731" means grepping event logs across four services instead of reading one saga log.

Traced: a choreographed saga end-to-end

Order O-9042, 2 units of SKU-77, $140 total. Four services subscribe only to events, never to each other directly. Follow the event chain forward, then the failure case and its compensation chain backward.

Now replay it with SKU-77 out of stock. Step 2 fails: Inventory Svc cannot reserve, so it publishes a failure event instead of a success event — and every downstream reaction runs in reverse, each service compensating its own earlier step because nobody else can:

Notice what choreography does not give you here: nobody ever calls Payment Svc, because Payment Svc never ran — the compensation chain only ever needs to unwind services whose forward step actually committed, and in choreography each service is responsible for knowing its own compensation trigger (the specific failure event it must listen for), not for looking up the whole saga's history. That is exactly the "flow is emergent" trade-off: adding a fifth participant later means finding every place a relevant event is already published and subscribing to it — there is no single file to open and extend, unlike the orchestrator's command table.

Pivot ordering: put the cheap, likely failure first

Every saga has a pivot: the step after which the saga can only go forward, never compensate (see the existing compensatable / pivot / retriable classification for the full taxonomy). The ordering question this page adds: which step should you place as, or just before, the pivot? Order the compensatable prefix by how often it fails and how cheap that failure is to undo, and put the pivot on the transaction that is both least likely to fail for a business reason and most expensive/irreversible to compensate.

The rule of thumb: rank compensatable steps by expected failure frequency × compensation cost and clear the high-frequency/high-cost ones first, while they are still cheap to undo. Stock and courier/slot availability fail for ordinary business reasons constantly; a card that has already passed basic authorization rarely fails to capture. So reserve-inventory (compensatable, a fast decrement) belongs before capture-payment, and capture-payment sits at or near the pivot — never the reverse. This is not a universal law (a hard-hold-only-flow, where you authorize but do not capture until the very end, sidesteps the question by making the money movement itself reversible for longer — that is the TCC-style middle ground); the rule is: whichever ordering you pick, put the step most likely to fail for a mundane reason as early and as cheap-to-undo as possible, and never place an irreversible side effect (shipping, a non-refundable third-party call, an email) anywhere but after the pivot.

Semantic locks and the isolation window

A saga has no cross-step isolation: the moment reserve-inventory commits, its PENDING row is visible to every other transaction in the system, even though the saga overall has not finished. A semantic lock is the fix — a status flag that says "this row is mid-saga, treat it specially" — and it has a window: the interval between the reservation commit and the confirm-or-release that closes it. Anything that reads the row during that window must know to wait, skip, or special-case it.

reserveInventory(sku, qty, sagaId):
  UPDATE inventory
     SET status = 'PENDING', reserved_by = :sagaId, reserved_at = now()
   WHERE sku = :sku AND status = 'AVAILABLE' AND qty_on_hand >= :qty
  -- rowsAffected == 0  =>  either true out-of-stock, or someone else's saga
  --                        already holds the semantic lock on this sku
  -- rowsAffected == 1  =>  lock window is now OPEN (status=PENDING)

-- window closes one of two ways:
confirmReservation(sku, sagaId):  UPDATE inventory SET status='RESERVED'   WHERE sku=:sku AND reserved_by=:sagaId
releaseReservation(sku, sagaId):  UPDATE inventory SET status='AVAILABLE' WHERE sku=:sku AND reserved_by=:sagaId

Without the status predicate, a second saga's read during the window sees plain available stock and double-books it — the classic lost update. With it, the second saga's own UPDATE ... WHERE status = 'AVAILABLE' simply matches zero rows and fails cleanly, which is the semantic-lock countermeasure: encode the in-flight state in the row itself so every other saga's own compensatable-step logic naturally backs off, instead of trusting readers to remember to check a separate lock table. Note what this SQL actually does: it flips a whole-row status flag, it does not decrement qty_on_hand. That makes it a coarse, whole-row reservation rather than a fine-grained quantity decrement — while one saga's reservation is PENDING, any other saga trying to reserve any quantity of the same SKU matches zero rows and fails, even if qty_on_hand is far larger than both requests combined. Two related countermeasures for cases where you cannot serialize through one row: commutative updates (a stock adjustment expressed as +2/-2 nets out correctly regardless of interleaving, unlike a "set to absolute value" write, and lets concurrent sagas draw down the same SKU independently instead of single-threading through one in-flight reservation at a time) and re-read the value immediately before acting, rather than trusting a value read at the start of a long saga.

Coordinator crash: after PaymentCaptured, before the next command fires

The existing crash traces on this guide replay a crash during a compensation (mid C3). The more dangerous gap is a crash on the forward path, because it creates an ambiguous step: the participant's local transaction genuinely committed, but the coordinator never got to record that fact or act on it.

1. Orchestrator sends CapturePayment(idemKey=K7731) to Payment Service.
2. Payment Service commits locally — the card is charged — and replies PaymentCaptured.
3. The orchestrator process dies before it appends T2=DONE to the saga log and before it decides the next command. From the log's point of view, T2 is still STARTED: charged in reality, unknown on paper.
4. A fresh orchestrator instance restarts (or a supervisor promotes a standby) and scans the log for open sagas. It finds saga-7731 with T2 STARTED and no terminal record.

The unsafe move is to guess. Re-issuing CapturePayment blind risks a double charge if the first call actually landed; firing RefundPayment blind risks refunding money that was never captured, or refunding a step that is about to succeed on its own. The safe move is to reconcile against the idempotency key before doing anything:

onRestart():
  for step in sagaLog.openSteps():          // state == STARTED, no terminal record
    actual = participant(step).queryByIdempotencyKey(step.idemKey)
    if actual == DONE:
      sagaLog.append(step, "DONE")          // reconcile the log, do NOT refire
      advance(step.saga)                    // resume forward from here
    elif actual == NOT_FOUND:
      reissue(step)                         // safe: same idemKey, participant dedupes any race
    else:  // actual == FAILED
      sagaLog.append(step, "FAILED")
      beginCompensation(step.saga)

This only works because CapturePayment was called with an idempotency key the participant itself persists and can be queried by — the key, not the saga log, is the actual source of truth for "did this really happen." The saga log tells you where to look; it is not allowed to be the only place the answer lives, precisely because it can go missing at the worst possible instant (right after step 2 above). Whichever recovery direction reconciliation lands on, the resumed step must itself be idempotent — a replayed CapturePayment must be a no-op if the key was already seen, and a replayed RefundPayment must be a no-op if the refund already posted.

Per-leg design: choosing the wire for each hop

A production saga rarely uses one transport for every hop. Decide leg by leg:

Idempotency dedup, concretely. Two mechanisms, pick by durability need:

-- Fast, best-effort (fine for low-stakes steps, e.g. "send confirmation email"):
SET dedupe:{event_id} 1 NX EX 86400
-- SETNX-style: succeeds only the first time this event_id is seen; process only on success.
-- Risk: a Redis eviction/restart before the TTL forgets a key you already relied on.

-- Durable, transactional (use for money-moving or stock-moving steps):
INSERT INTO saga_step_log(saga_id, step, idem_key) VALUES (:sagaId, 'T2', :idemKey);
-- UNIQUE(saga_id, step) constraint: a duplicate insert throws; catch it, treat as "already applied",
-- and commit it in the SAME local transaction as the business write it guards.

The partition-key lever. If saga events for order O-9042 could land on different partitions, PaymentCaptured and a later RefundRequested for that same order could be picked up by different consumer instances and processed out of order. Partition by the saga/aggregate id (order_id), not by event type and not randomly, so every event for one saga instance is delivered, in publish order, to one consumer:

producer.send(topic="saga-events", key=order_id, value=event)
-- same key -> same partition -> one consumer -> in-order processing for this saga instance

(General partitioning/ordering mechanics are covered in the Event-Driven Architecture deep dives; this is that lever applied specifically to keeping one saga's own event chain from reordering itself.)

The exactly-once boundary. No broker delivers exactly-once across independent systems end-to-end — "exactly-once" producer/consumer configs (Kafka idempotent producers, transactional writes) guarantee that within the broker and one datastore, not for the external side effect a consumer triggers (you cannot make a call to a payment gateway transactional with a Kafka offset commit). What you actually get, and what you should design for, is at-least-once delivery + an idempotent consumer = effectively-once side effects, with the dedup check committed in the same local transaction as the business write — which is exactly the Transactional Outbox pattern's job on the publish side, mirrored by a dedup table on the consume side.

Retry hygiene. Bounded exponential backoff with jitter; a dead-letter path after N attempts so a poison event does not spin forever holding a semantic lock open; and never retry a step of unknown outcome without first checking its idempotency key state (this is the same reconciliation discipline as the coordinator-crash recovery above — a retry is just a smaller-scale version of "did this already happen?").

Relay crash and the outbox trade-off. Whichever transport carries an event, it usually starts from an outbox row written in the same local transaction as the step. If the relay dies: a polling relay just resumes scanning unsent rows on restart — simple, but it adds continuous read load and a latency floor equal to the poll interval. A CDC relay (tailing the WAL/binlog, e.g. Debezium) resumes from its last checkpointed log offset — near-zero added DB load and much lower latency, but it is another connector to operate, and if it stays down longer than your WAL/binlog retention window, the tail falls off the log and you lose events it never got to ship.

Trade-off ledger: orchestrated vs choreographed

Pitfalls

• Trusting "exactly-once" broker settings to remove consumer-side dedup. They cover the broker-to-datastore hop, not an external side effect (a gateway charge, a shipping API call) — you still need an idempotency key at that boundary.
• Refiring an ambiguous step without reconciling first. On coordinator restart, a step logged STARTED with no terminal record must be resolved against the participant's own idempotency-key record, never guessed forward or backward.
• Partitioning by event type instead of aggregate id. Scatters one saga's own events across consumers, reordering PaymentCaptured against a later RefundRequested for the same order.
• Hot-key skew. Partitioning by aggregate id concentrates a single very-high-traffic order/account id on one partition/consumer — fine for saga ordering, but watch for it becoming a throughput bottleneck.
• Choreography sprawl with no timeout. A dropped event leaves a semantic lock open indefinitely with no coordinator watching the clock; every choreographed step needs its own timeout-and-escalate path.
• Silent event-schema drift. A choreographed consumer that reacts to a field the publisher quietly renamed fails at the worst time, with no central contract to have caught it at deploy time.

Choosing: the judgment layer

Orchestrated vs choreographed is not "simple vs complex" — it's a bet on where you want the coupling and the visibility to live. Choose orchestration once branching or step count grows past a handful, or whenever audit/observability of "why did this saga do X" is a hard requirement (payments, regulated flows). Choose choreography only for short, linear, 2–4-step flows where the participating teams value not depending on a shared coordinator more than they value single-place traceability — and budget for per-service timeouts and event-schema discipline, because nothing else will catch drift for you.

Per-leg sync vs EDA follows the same logic at hop granularity: sync where a human is waiting for an answer and the callee is reliably fast; EDA where the producer must stay up regardless of a consumer's health, or where one event naturally fans out to several reactors. Most real sagas are a mix — sync for the user-facing first hop, EDA for everything that follows — and that mix is a design decision per leg, not a single guide-wide default.

Takeaways

• Choreography removes the coordinator but not the plan — it just makes the plan implicit in a chain of event subscriptions, which trades traceability for decoupling.
• Order compensatable steps by failure-frequency × compensation-cost, cheapest and most-likely-to-fail first; the pivot belongs on the step least likely to fail for a business reason, so routine failures never become charge-then-refund.
• A saga log tells you where a crashed coordinator left off, but only the participant's own idempotency-key record tells you whether an ambiguous step actually happened — reconcile against that before resuming or compensating.
• Per leg, pick transport, delivery semantics, dedup mechanism, and partition key deliberately: at-least-once plus an idempotent, aggregate-id-partitioned consumer is the real "exactly-once," not a broker setting.

Related pages

• Saga — A Worked Orchestration Example — the baseline orchestrated-saga walkthrough this page assumes and builds on.
• 2PC vs Saga vs TCC — Distributed Transactions — how the TCC-style hold-only alternative referenced in the pivot-ordering section compares to 2PC and Saga.
• Distributed IDs & Transaction Protocols — Snowflake Failure Modes, 2PC Blocking, TCC's Three Problems & Saga Isolation (Deep Dive) — TCC's own failure modes and saga isolation guarantees, alongside the semantic-lock isolation window covered here.
• Idempotency & “Exactly-Once Is a Myth” — the dedup mechanics behind the idempotency-key reconciliation used throughout coordinator-crash recovery and retry hygiene.
• Transactional Outbox — Solving the Dual-Write Problem — the full mechanism behind the relay/outbox trade-off in the per-leg design section.

Re-authored/Deepened for this guide. Builds on this guide's existing Saga, Transactional Outbox and Event-Driven Architecture pages (compensatable/pivot/retriable classification, saga-log recovery, semantic locks). Sources: Hector Garcia-Molina & Kenneth Salem, "Sagas" (ACM SIGMOD, 1987) — compensation model and concurrent-access countermeasures; Chris Richardson, Microservices Patterns (Manning, 2018), ch. 4–5 — orchestration vs choreography, pivot classification; Martin Kleppmann, Designing Data-Intensive Applications (O'Reilly, 2017) — delivery semantics and the exactly-once boundary; Debezium and Apache Kafka documentation — CDC log-tailing and partition-key ordering guarantees.