The Saga Pattern A Solution

A saga keeps a multi-service business operation atomic without a distributed lock by breaking it into a sequence of local ACID transactions — each commits independently in its own service and database — and undoing any already-committed step with an explicit compensating transaction the moment a later step fails. There is no global rollback to fall back on, because each local commit is already durable and visible; the saga instead semantically reverses what was done. That single trade — swapping one distributed transaction for many local ones plus hand-written inverses — is the whole pattern, and every hard part below follows from it.

How the mechanism actually works

Each forward step Ti has a paired compensation Ci that semantically undoes it (refund, not "un-charge"; cancel order, not "delete row"). A saga log — held by the orchestrator, or reconstructed from the event stream in choreography — records which steps committed so recovery knows exactly how far to unwind.

Two recovery directions exist, and mixing them up is the classic design error. Backward recovery runs Ci … C1 in reverse order to abort. Forward recovery retries a failed step until it succeeds. You cannot always choose backward recovery, which forces the key classification of steps:

• Compensatable — steps that ran before the point of no return and can be reversed (create order, reserve credit).
• Pivot — the go/no-go step. Once it commits, the saga must complete; before it commits, the saga can still abort. (Charging the card is a common pivot.)
• Retriable — steps after the pivot that have no compensation and therefore must be designed to eventually succeed (send confirmation email, decrement a counter). They are retried, never rolled back.

This ordering — all compensatable steps first, then the pivot, then only retriable steps — is what makes an eventually-consistent saga tractable. Put a non-compensatable side effect (an email) before the pivot and you can no longer honestly abort.

Traced example: an order that fails at inventory

Customer places order O-8842 for 2 units of SKU-19, total $250. The orchestrator drives the steps in the correct order — all compensatable steps first, then the pivot, then only retriable steps: T1 create order → T2 reserve stock → T3 charge $250 (the pivot) → T4 confirm/ship (retriable). Step 2 fails because only 1 unit is in stock — and because that failure lands before the pivot, the saga aborts backward and no money ever moves:

The pivot (T3 charge $250) and the retriable post-pivot step (T4 confirm/ship) never run — the saga aborted before reaching the point of no return. Two things stand out. First, because the pivot runs only after stock is reserved, a stock failure aborts before any money moved — you never refund a pivot. Contrast the anti-pattern: had we charged the card first and only then tried (and failed) to reserve stock, $250 would have really moved and we would have had to refund it — a genuine, externally-visible window of inconsistency. That is precisely why we reserve before we charge: the pivot belongs after every compensatable step it depends on. Second, T2 was not compensated: it failed, so it never committed, and only committed steps are compensated. And a step after the pivot — T4 confirm/ship — is likewise never compensated: once the pivot commits, a failing post-pivot step is retried forward until it succeeds, never rolled back. Concretely, if T4's "your order has shipped" email bounces, the saga retries the send forward — it does not abort a paid, shipped order to un-send a notification. This is exactly why non-compensatable side effects like email must sit after the pivot: they are retriable, not reversible.

The isolation problem — the caveat nobody mentions first

Local ACID transactions give you A, C, and D. Sagas throw away the I. Because each step commits immediately, its intermediate state is visible to other transactions before the saga finishes — the exact anomalies isolation was invented to prevent:

• Dirty reads. While the saga sits between T1 and the abort, a reporting job counts order O-8842 as a live PENDING order — revenue dashboards briefly include an order that C1 is about to cancel. In a charge-first saga (the anti-pattern above) it is worse: a fraud-analytics job could read a captured payment that a compensation is about to refund, and act on money that will vanish.
• Lost updates. Two sagas read the same order and each writes its own version; one clobbers the other because neither held a lock across steps.

You manage this with saga-specific countermeasures (from Garcia-Molina's original work and Richardson's Microservices Patterns):

• Semantic lock — a compensatable step sets a flag on every record it creates or touches (the order is born status = PENDING at T1). Other transactions must check the flag and wait, skip, or handle the in-flight record explicitly. The flag is cleared when the saga completes — by the last step on success, or by the compensation on abort. This is a hand-built application-level lock, not a database lock.
• Commutative updates — design operations so order doesn't matter (a balance +50 then -50 nets out regardless of interleaving), removing the anomaly instead of guarding it.
• Reread value / version check — before writing, re-read the row and verify it hasn't changed (optimistic concurrency) to catch lost updates.
• By-value / risk-tiering — route high-value requests through stricter concurrency control and let low-value ones ride the fast eventually-consistent path.

Choreography vs orchestration

These are the two ways to drive the sequence. In choreography there is no coordinator: each service subscribes to events and reacts (Order emits OrderCreated → Payment reacts and emits PaymentCaptured → Inventory reacts…). In orchestration a central saga orchestrator issues commands ("Payment: charge $250") and decides the next step from each reply.

The honest decision rule is not "simple vs complex" alone — it is about coupling, visibility, and cyclic dependencies:

• Choose choreography for short sagas (2–4 steps) where you want no single point of control and services already speak events. Cost: the workflow logic is smeared across services, there is no one place to see saga state, and it is easy to create hidden cyclic event dependencies that are miserable to debug.
• Choose orchestration as step count grows or when you need auditability, timeouts, and explicit failure handling. Cost: the orchestrator is a new component to build and operate, and you must resist putting business logic into it (it should sequence, not decide domain rules).

Pitfalls a working engineer hits

• Non-idempotent compensations. After an orchestrator crash the saga log may say "refund in progress" and the recovery retries C2 — issuing a second $250 refund. Every step and every compensation must be idempotent, keyed by (sagaId, step). Illustrative guard (pseudocode):
  

  void refundPayment(sagaId, paymentId, amountCents):
      if compensationLog.has(sagaId, "C2"): return   // already done — no-op on retry
      gateway.refund(paymentId, amountCents)
      compensationLog.record(sagaId, "C2")

  Why the naive version ("just call refund") is wrong: at-least-once delivery and crash-retries mean it will run twice, double-refunding real money.
• Uncompensatable side effects placed before the pivot. Sending an email or calling a third-party API with no reversal, then trying to abort. Move all such effects after the pivot and make them retriable.
• Countervailing / new-order races. A compensation that "restores stock" while a concurrent saga just reserved it can over-count. Use commutative updates or semantic locks, not blind re-adds.
• The compensation itself fails. C2's refund call times out. You cannot compensate a compensation — you must retry it to death and alert; design compensations to be retriable and monitored, with a dead-letter path for human intervention.
• Missing timeouts. In choreography a dropped event silently stalls a saga forever with a semantic lock held. Every saga needs a timeout that triggers abort.
• Lost saga log = orphaned state. If the log isn't written in the same local transaction as the step (outbox pattern), a crash between committing the step and recording it leaves state the recovery can't see.

When to use it — and when NOT to

Reach for a saga when a single business operation must update data owned by multiple services/databases, you cannot (or won't) run a distributed transaction across them, and the business can tolerate a brief window of inconsistency plus explicit compensation. Concrete signals: microservices with database-per-service, long-lived operations (checkout, booking, onboarding), or steps that span systems you don't control (payment gateways, shipping APIs).

Named alternative — 2PC / distributed ACID transaction (XA). 2PC gives you real isolation and an automatic atomic rollback: no compensations to write, no dirty reads. What it costs: a coordinator holds locks across all participants for the whole transaction, so throughput collapses under contention, tail latency spikes, and a coordinator crash after "prepare" leaves resources blocked (the blocking problem). Many cloud datastores and message brokers simply don't support XA. Sagas buy availability, low coupling, and long-lived operations at the price of lost isolation and hand-written, idempotent compensation logic.

• Choose a saga when steps cross service/database boundaries, availability and loose coupling matter more than strict isolation, and the flow may be long-lived.
• Prefer 2PC when all participants sit inside one transaction manager (e.g., a single RDBMS or XA-capable resources), correctness demands true isolation, transactions are short, and contention is low.
• Prefer neither — collapse the boundary — when the data that must change atomically really belongs to one aggregate: keep it in a single service and use one plain local transaction. A saga is a symptom that you crossed a consistency boundary; sometimes the fix is redrawing the boundary, not orchestrating across it.

Takeaways

• A saga = sequence of local commits + explicit compensations; it trades away isolation to gain availability and cross-service atomicity, and every difficulty flows from that trade.
• Classify steps as compensatable → pivot → retriable and order them that way; the pivot is the point of no return that decides whether you can abort (backward) or must finish (forward).
• Compensations and steps must be idempotent and durably logged (outbox), or crash-retries will double-apply real effects.
• Guard the visible intermediate state with semantic locks / commutative updates; choose orchestration for auditable multi-step flows and choreography only for short, loosely-coupled ones — and use 2PC or a single service when true isolation is cheaper than compensating.

Re-authored/Deepened for this guide. Sources: Hector Garcia-Molina & Kenneth Salem, "Sagas" (ACM SIGMOD, 1987) — the original long-lived-transaction paper and compensation model; Chris Richardson, Microservices Patterns (Manning, 2018) and microservices.io — saga orchestration/choreography, pivot & retriable classification, and countermeasures for lost isolation; Martin Kleppmann, Designing Data-Intensive Applications (O'Reilly, 2017) — 2PC, the blocking problem, and eventual consistency trade-offs.