Introduction to Saga Pattern

Introduction to the Saga Pattern

Imagine placing an order on an e-commerce site. Behind that one click, three separate services must all succeed: Orders creates the order, Payments charges the card, and Inventory reserves the stock. In a monolith with one database, you'd wrap all three writes in a single ACID transaction: either everything commits or everything rolls back. But in microservices, each service owns its own database. There is no shared transaction that spans all three. So what happens when Payments succeeds but Inventory says "out of stock"? You've charged the customer for something they'll never receive.

The tempting fix is a distributed transaction using two-phase commit (2PC), where a coordinator locks all three databases until everyone agrees to commit. This works, but it's poison for high-throughput systems: it holds locks across network hops, blocks if the coordinator crashes mid-flight, and couples the availability of every service to every other. The Saga pattern is the answer most large systems actually reach for. Instead of one big atomic transaction, a saga is a sequence of local transactions, each committing independently, with a matching compensating transaction that semantically undoes it if a later step fails.

How it works, precisely

A saga breaks a business operation into a series of steps T1, T2, ... Tn. Each Ti is a normal local ACID transaction inside one service's own database and commits immediately. For each Ti you define a compensating transaction Ci that semantically reverses its effect. If step Tk fails, the saga runs C(k-1), C(k-2), ... C1 in reverse order to unwind the work already done.

Crucially, compensation is semantic, not physical. You can't "roll back" a charge that already committed and possibly cleared; instead C issues a refund — a new forward transaction that offsets it. This means sagas provide ACD (Atomicity-ish via compensation, Consistency, Durability) but sacrifice Isolation: intermediate states are visible to other transactions. That lost isolation is the source of most saga hazards.

There are two coordination styles:

• Choreography — no central brain. Each service emits an event when its local transaction commits, and the next service subscribes and reacts. Orders publishes OrderCreated, Payments listens and publishes PaymentCompleted, and so on. Decentralized and low-overhead, but the workflow logic is scattered across services and hard to trace.
• Orchestration — a central orchestrator (a saga execution coordinator) explicitly tells each service what to do via commands and awaits replies. The workflow lives in one place, is easy to reason about and to add steps to, but the orchestrator is a component you must build and keep available.

Two properties are non-negotiable: every step and every compensation must be idempotent (messages get redelivered), and the saga state must be durably persisted so it survives a crash and resumes.

A worked scenario with numbers

Say the checkout service handles 2,000 orders/sec at peak. A 2PC across Orders, Payments, and Inventory would hold row locks in all three databases for the full round-trip. If each service call averages 40 ms and there are three sequential phases, locks are held ~120 ms per order. At 2,000 QPS that's ~240 concurrent locked transactions per shard fighting over hot rows (popular SKUs, the same coordinator table) — lock contention collapses throughput long before the CPUs are busy, and a coordinator crash leaves rows locked until timeout.

With an orchestrated saga, each local transaction commits in ~15 ms and releases its locks immediately; the orchestrator carries the state between steps. No cross-service locks exist. The trade you accept: for the brief window (tens to hundreds of ms, sometimes seconds if a service is slow) between the payment charge and the stock reservation, the system is in an inconsistent-but-visible state — money is taken but stock isn't confirmed. When stock reservation fails, the orchestrator fires C2 refund and C1 cancel order. The customer briefly saw a pending charge, then a refund — eventual consistency, not atomic consistency.

Trade-offs: when to use, when not

Use a saga when a business operation spans multiple services (each with its own database), the steps are long-lived or high-throughput, and the domain tolerates eventual consistency with compensations. This describes most microservice workflows: order fulfillment, travel booking (flight + hotel + car), account provisioning, money transfers between ledgers.

Prefer alternatives when:

• Two-phase commit (2PC) — choose it only when you truly need strict serializable isolation across a small number of resources that all speak XA, latency is not critical, and volume is low (e.g., a nightly financial reconciliation). Its cost is locking and coordinator-blocking.
• A single service / shared database — if the data naturally belongs together and you're inventing a distributed transaction to glue two services back, the real fix is often to merge them and use one local ACID transaction. Don't reach for a saga to paper over a bad service boundary.
• Transactional outbox + event-driven — sagas are frequently built on top of an outbox (write the event and the state change in the same local transaction), not an alternative to it.

Choreography vs orchestration: choreography suits 2–3 steps with simple flow; beyond that, cyclic event dependencies become impossible to trace. Orchestration scales to complex, branching workflows and is usually the senior-interview default for anything non-trivial.

Pitfalls an interviewer probes

• Lost isolation / dirty reads. Because intermediate steps are visible, another transaction can read data that later gets compensated away. Countermeasures: semantic locks (mark a record PENDING so others treat it cautiously), commutative updates, or a reread-value check before acting. Expect the interviewer to ask "what if someone reads the reserved-but-not-paid inventory?"
• Compensations can fail too. A refund service can be down. You need retries with backoff, and compensations must eventually succeed (backward recovery) or you must design steps so a stuck saga can retry forward (forward recovery / "retriable" transactions). Some steps are pivot transactions — once passed, the saga can only go forward, never back.
• Non-idempotent handlers. At-least-once messaging redelivers; a double-charge or double-refund is a classic bug. Every handler needs a dedup key.
• Not everything is compensatable. "Send the shipment" or "email the customer" can't be undone. Order the saga so irreversible steps come last, after all cancellable steps have succeeded.
• Choreography sprawl. Asked to add a step, the candidate who chose choreography must touch many services and risks event cycles — a signal they under-weighted observability.

Key takeaways

• A saga replaces one impossible distributed ACID transaction with a sequence of local transactions plus compensating transactions, trading strict isolation for eventual consistency and far better throughput than 2PC.
• Choose orchestration (central coordinator, easy to trace) for complex workflows and choreography (event-driven, decentralized) only for short simple flows.
• The hard parts are all consequences of lost isolation: dirty reads, failed compensations, non-idempotent handlers, and irreversible steps — mitigate with semantic locks, idempotency keys, retries, and ordering irreversible steps last.
• A saga is not a default: if two services are only ever changed together, fix the boundary and use one local transaction instead.