Saga — A Worked Orchestration Example

From definition to a real flow

You’ve seen what a Saga is (2PC vs Saga vs TCC): a sequence of local transactions, each with a compensating transaction that undoes it if a later step fails. Let’s walk a concrete orchestration — a central coordinator drives the steps.

The flow: place an order

  Orchestrator drives 3 local transactions, each in its own service/DB:

   1. Order Service     → CREATE order (status=PENDING)     compensate: CANCEL order
   2. Inventory Service → RESERVE items                     compensate: RELEASE items
   3. Payment Service   → CHARGE card                       compensate: REFUND

  Happy path:  1 → 2 → 3 → mark order CONFIRMED
  Payment fails at step 3:  run compensations in REVERSE → RELEASE items → CANCEL order

Step order is not cosmetic. Reserve (cheap, invisible to undo) comes before charge because the charge is the saga’s pivot — once real money moves, you are past the point of no return: a failure before the pivot aborts backward with internal-only compensations, while charging first and refunding on a failed reservation would move real money and claw it back — an externally visible inconsistency (and often a fee). Everything after the pivot must be retriable.

Orchestrator logic (pseudocode)

 placeOrderSaga(cart):
   order = orderSvc.create(cart)            # step 1
   try:
       inventorySvc.reserve(order)          # step 2
       try:
           paymentSvc.charge(order)         # step 3
       except PaymentError:
           inventorySvc.release(order)      # compensate 2
           orderSvc.cancel(order)           # compensate 1
           return FAILED
   except InventoryError:
       orderSvc.cancel(order)               # compensate 1
       return FAILED
   orderSvc.confirm(order); return OK

What makes it correct

• Each step commits locally (no distributed lock) — so it scales, unlike 2PC.
• Compensations run in reverse order and must be idempotent (a retry of “release items” mustn’t double-release).
• Between steps the system is in a visible intermediate state (order PENDING) — design for it; this is eventual consistency, not isolation.

Takeaways

• A Saga = forward local transactions + reverse compensations on failure.
• Orchestration (central driver, shown here) is easier to reason about than choreography (services reacting to events) for complex flows.
• The events that trigger each step should be emitted reliably via the Transactional Outbox.

What the worked example hides — production concerns

The pseudocode above illustrates the core idea, but a production orchestrator must survive realities that the happy-path drawing omits:

• Durable saga log. The orchestrator itself can crash between steps. It must persist every state transition (step started, step succeeded, compensation invoked) to its own store before acting on the next step. On restart it replays the log and resumes or completes compensations. Without this, a crash turns a partial saga into an orphan order or a phantom reservation.
• Idempotency keys. Network timeouts make every step request ambiguous: did the service receive the call and commit, or did it never arrive? Each saga step carries a unique idempotency key so the service can recognize a retry and return the prior result instead of executing twice. This is how a retry of “charge card” avoids a double charge and a retry of “release items” avoids a double release.
• Timeouts and retries. A step that hangs is indistinguishable from one that is slow. The orchestrator must apply a per-step timeout and retry with backoff, treating a timeout as a failure that triggers compensation. Retry policies differ by step: inventory release can be retried aggressively; payment charge should be retried cautiously because the downstream may have already processed it.
• Compensation failures. Compensations can also fail. The saga log records the failed compensation and alerts an operator; the system cannot declare the saga clean until all compensations complete or are manually resolved. Designing compensations to be safe-to-retry is the most important implementation detail.
• Isolation and visible inconsistency. Between step 2 (reserve) and step 3 (charge), the inventory is reserved but no money has moved. A concurrent reader may see reserved stock that does not correspond to a paid order. The business must accept these intermediate states and design queries to account for them, for example by filtering out PENDING orders from inventory availability.
• Orchestration vs choreography. Orchestration centralizes the flow, which is easier to trace and change for complex sagas but creates a single point of control and coupling to the orchestrator. Choreography distributes the flow through events, which scales better for simple, stable flows but is harder to debug and can produce emergent loops. Choose orchestration when the flow has many branches, human approvals, or strict ordering; choose choreography when the steps are loosely coupled and rarely change.

Sources: Designing Data-Intensive Applications (Kleppmann, ch. 9) for saga semantics and 2PC trade-offs; Building Microservices (Newman) and Azure Architecture Center saga guidance for orchestration patterns; Google SRE Book for idempotency and retry discipline.

Re-authored from-scratch for this guide. Diagrams adapted from Karan Pratap Singh’s System Design (MIT); patterns follow Azure Architecture Center / microservices.io / DDIA conventions.

Concretely: recovering a crash mid-compensation

Make the saga log a real state machine — PENDING → RESERVED → CONFIRMED on the happy path, or … → COMPENSATING → FAILED when a step at or before the pivot aborts — and write every transition durably before the next action. Now trace a crash during backward recovery: the charge (the pivot) is declined, the orchestrator writes COMPENSATING, calls release(reserve_id), and dies before cancel(order_id). On restart it replays the log, sees COMPENSATING with the release already recorded done, skips the completed release, and resumes at cancel. Because release and cancel are keyed by (sagaId, step), even a release that was in flight at crash time is safe to re-drive — the inventory service no-ops the duplicate. This is why the log records each compensation’s outcome, not merely that “compensation started”: resume has to know precisely which undo already happened.

Steps after the pivot (e.g. a confirm/ship step) are retriable: they are retried with backoff until they succeed and are never compensated backward — if a step can fail for a business reason, it must be placed before the pivot.