Build a Saga Orchestrator & Reconciliation — Money Correctness Across Services

Saga Orchestrator & Reconciliation — Exactly-once Effects Across Services

Every LLD ledger kata ends at the edge of one process: sum(postings) == 0 is an invariant a single database transaction can enforce for you, atomically, for free. The moment money has to move across two services that don't share a database — reserve inventory in one, charge a card at a payment provider in another — that free atomicity is gone, and it does not come back. You cannot open a two-phase-commit transaction across Stripe. This lab builds the thing that stands in for the transaction you can't have: an orchestrated saga with idempotent steps, compensations, a durable log, crash recovery, and a reconciliation job that catches the money that leaked anyway. You'll ship it, break it (charge succeeds, booking doesn't; coordinator dies mid-flight), measure the exact window where the books are wrong, and watch compensation and replay pull them back to a machine-checkable invariant — in Java and Go. The gap this fills: money-correctness beyond single-process atomicity is cited in every system-design answer and built in almost none.

1. The Trap — a booking is two writes to two owners

A booking for seat SEAT-12A at $50.00 must do two things that live in two different services with two different databases:

Inventory service — decrement the seat, so nobody else can take it.
Payment service — charge the customer's card through an external provider.

The obvious code writes both in sequence: inventory.reserve(); payment.charge();. It is correct exactly as long as nothing fails between the two calls — which is to say, it is correct in the demo and wrong in production. There is no transaction wrapping the two, because one of them isn't even your database; it's a third party across the internet. So every partial failure is a money bug with a name:

Charged, not reserved — the charge succeeds, then the reserve fails (or the process dies before it runs). The customer paid and got nothing. This is the one that ends up on Hacker News.
Reserved, not charged — the seat is decremented, then the charge fails. You're holding inventory hostage against money you never collected; at scale this quietly sells out an event that isn't actually sold.
Charged, reserved, but the coordinator died before it recorded success — everything actually worked, but nothing knows it did. A retry or a human "fixing" it double-charges.

The trap is believing any of these is rare enough to ignore. Each is just "a network blip landed in the one-instruction gap between two writes," and at a few thousand bookings a second that gap is hit continuously. The naive two-call sequence has no vocabulary to even describe which of these happened, let alone undo it.

Scope for this lab. Two services, in-memory, idempotent. One orchestrator that owns the booking. A durable saga log that outlives the orchestrator process. Steps: reserve (a hold) → charge → confirm the hold. Compensations, in reverse: refund → release. A reconciliation job that compares the two services after the fact. Out of scope: real network transport, exactly-once delivery (impossible; we buy exactly-once effect with idempotency instead), and multi-currency FX.

2. Scope it like a senior — the questions that reshape the design

Can you 2PC across the payment provider? No. The provider exposes a synchronous charge API, not an XA resource you can prepare and hold a lock on. This single fact is why saga exists at all — name it out loud before anyone asks.
Is each step idempotent, and keyed by what? Every step and every compensation must be safe to run twice, because recovery will run them twice. The key is derived from the booking id (bk-2:pay, bk-2:refund) so a replay hits the same key and no-ops. Without this, replay is just a second charge.
Where does saga state live? In a durable log that is not the coordinator's memory — a table, a WAL, a stream. If it lives in the coordinator's heap, a coordinator crash loses the one record that knows a charge is outstanding, and the money is unrecoverable except by reconciliation.
Orchestration or choreography? Orchestration (one coordinator drives the steps) for this lab: the money path is short, needs a single owner of "what state is this booking in," and must be debuggable at 3am. Choreography (services react to each other's events) is the alternative — Movement 7 defends the choice.
Roll forward or roll back on failure? Both, depending on where you failed. A hard business failure (card declined) rolls back via compensation. A transient failure after an irreversible step (charge already succeeded, then the coordinator died) rolls forward via replay — you finish the booking rather than un-charging a good payment.
Who owns the invariant? The reconciliation job. Compensation handles the failures the saga saw; reconciliation handles the ones it didn't — a compensation that itself crashed, a lost refund, a bug. It is the backstop that makes "the books are eventually correct" a checked property, not a hope.

3. Reason to the design — from two calls to a recoverable saga

Simplest thing that could work: reserve(); charge(); in sequence, no state, no undo. That's the trap. It has no memory of what it did, so it can neither undo a partial failure nor resume an interrupted one.

First iteration — add compensations. Give every forward step an inverse: reserve↔release, charge↔refund. Now on a step failure the orchestrator can walk backwards and undo what already succeeded. This alone kills the "charged, not reserved" and "reserved, not charged" bugs for failures the orchestrator is alive to see. It does nothing for a coordinator crash, because a dead coordinator runs no compensation.

Second iteration — persist the saga state before you need it. Write each step's completion to a durable log as it happens: RESERVED, then CHARGED, then CONFIRMED, then COMPLETED. The log is separate from the coordinator, so when the coordinator dies mid-saga, a fresh one reads the log, sees "we got to CHARGED," and resumes from there. This is what turns "a crash loses money" into "a crash costs a few seconds." The log is the saga; the coordinator is just a replaceable worker that advances it.

Third iteration — make every step idempotent. Recovery replays steps. If replaying charge charges again, recovery is worse than the crash. So each step is keyed and checks "did I already do this key?" before acting. Idempotency is not an optimization here — it is the precondition that makes replay safe, which is what makes the durable log useful, which is what makes crash recovery possible. The three ideas only work as a set.

The backstop — reconciliation. Even with all of the above, a compensation can crash halfway, or a bug can skip one. So a separate job periodically joins the two services on the booking key and asks: is there any live charge with no COMPLETED booking behind it? Every such row is an orphan charge — real money owed back — and the job flags and repairs it. This is the only mechanism that catches the failures the saga machinery itself got wrong.

4. Build it — Java

Two idempotent in-memory services, a durable saga log kept deliberately separate from the coordinator, an orchestrator that advances the log and compensates on failure, and a reconciliation job. The failAt hook lets us inject the two failure shapes in Movement 5. Reserve is modeled as a hold plus a later confirm so the phrase "fails after charge but before reserve confirms" is a real, injectable point, not hand-waving.

import java.util.*;
import java.util.concurrent.*;

public class SagaDemo {

    // ---------- Idempotent in-memory services ----------
    static class Inventory {
        private final Map<String,Integer> stock = new HashMap<>();
        private final Map<String,Integer> holds = new HashMap<>();
        private final Set<String> confirmed = new HashSet<>();
        Inventory(String item, int qty){ stock.put(item, qty); }

        synchronized void hold(String key, String item, int qty){
            if (holds.containsKey(key) || confirmed.contains(key)) return; // idempotent
            int a = stock.getOrDefault(item, 0);
            if (a < qty) throw new RuntimeException("out_of_stock");
            stock.put(item, a - qty);
            holds.put(key, qty);
        }
        synchronized void confirm(String key){
            if (confirmed.contains(key)) return;                          // idempotent
            if (!holds.containsKey(key)) throw new RuntimeException("no_hold");
            confirmed.add(key);
        }
        synchronized void release(String key, String item){              // compensation
            Integer q = holds.remove(key);
            confirmed.remove(key);
            if (q == null) return;                                        // idempotent
            stock.merge(item, q, Integer::sum);
        }
        synchronized int available(String item){ return stock.getOrDefault(item, 0); }
    }

    static class Payment {
        static class Entry { final String booking, type; final long cents;
            Entry(String b, String t, long c){ booking=b; type=t; cents=c; } }
        private final List<Entry> ledger = new ArrayList<>();
        private final Set<String> applied = new HashSet<>();             // idempotency keys
        synchronized void charge(String key, String booking, long cents){
            if (applied.contains(key)) return;                           // idempotent
            applied.add(key);
            ledger.add(new Entry(booking, "CHARGE", cents));
        }
        synchronized void refund(String key, String booking, long cents){
            if (applied.contains(key)) return;
            applied.add(key);
            ledger.add(new Entry(booking, "REFUND", -cents));
        }
        synchronized long net(){ long s=0; for (Entry e: ledger) s += e.cents; return s; }
        synchronized long netFor(String b){ long s=0; for (Entry e: ledger) if (e.booking.equals(b)) s += e.cents; return s; }
        synchronized long chargeCount(String b){ long n=0; for (Entry e: ledger) if (e.type.equals("CHARGE") && e.booking.equals(b)) n++; return n; }
        synchronized Set<String> chargedBookings(){ Set<String> s=new LinkedHashSet<>(); for (Entry e: ledger) if (e.type.equals("CHARGE")) s.add(e.booking); return s; }
        synchronized void forceCharge(String booking, long cents){ ledger.add(new Entry(booking,"CHARGE",cents)); } // seed drift
    }

    // ---------- Durable saga log (a table, not coordinator memory) ----------
    enum State { STARTED, RESERVED, CHARGED, CONFIRMED, COMPLETED, COMPENSATING, FAILED }
    static class SagaRecord {
        final String bookingId, item; final int qty; final long cents;
        State state = State.STARTED;
        SagaRecord(String b,String i,int q,long c){ bookingId=b; item=i; qty=q; cents=c; }
    }
    static class SagaLog {
        private final Map<String,SagaRecord> rows = new ConcurrentHashMap<>();
        void put(SagaRecord r){ rows.put(r.bookingId, r); }
        void setState(String id, State s){ rows.get(id).state = s; }     // durable write
        SagaRecord get(String id){ return rows.get(id); }
        Collection<SagaRecord> all(){ return rows.values(); }
    }

    static class CrashError extends RuntimeException { CrashError(String m){ super(m); } }

    // ---------- Orchestrator ----------
    static class Orchestrator {
        final Inventory inv; final Payment pay; final SagaLog log;
        Orchestrator(Inventory inv, Payment pay, SagaLog log){ this.inv=inv; this.pay=pay; this.log=log; }

        void run(String bookingId, String item, int qty, long cents, String failAt){
            SagaRecord r = log.get(bookingId);
            if (r == null){ r = new SagaRecord(bookingId,item,qty,cents); log.put(r); }
            try { execute(r, failAt); }
            catch (CrashError ce){ throw ce; }        // coordinator dies; log persists; no compensation here
            catch (RuntimeException ex){ compensate(r, failAt); } // any step failure -> roll back (compensation may itself crash)
        }
        private void execute(SagaRecord r, String failAt){
            String id = r.bookingId;
            if (r.state.ordinal() < State.RESERVED.ordinal()){
                inv.hold(id+":inv", r.item, r.qty); log.setState(id, State.RESERVED);
            }
            if (r.state.ordinal() < State.CHARGED.ordinal()){
                pay.charge(id+":pay", id, r.cents); log.setState(id, State.CHARGED);
            }
            if ("crashAfterCharge".equals(failAt)) throw new CrashError("died after charge, before confirm");
            if (r.state.ordinal() < State.CONFIRMED.ordinal()){
                if ("confirm".equals(failAt) || "confirmThenCrashDuringCompensation".equals(failAt)) throw new RuntimeException("confirm_failed");
                inv.confirm(id+":inv"); log.setState(id, State.CONFIRMED);
            }
            log.setState(id, State.COMPLETED);
        }
        private void compensate(SagaRecord r){ compensate(r, null); }
        private void compensate(SagaRecord r, String failAt){
            String id = r.bookingId; log.setState(id, State.COMPENSATING);
            if ("confirmThenCrashDuringCompensation".equals(failAt))
                throw new CrashError("died mid-compensation: COMPENSATING persisted, refund/release not yet run");
            pay.refund(id+":refund", id, r.cents);     // reverse order, each idempotent
            inv.release(id+":inv", r.item);
            log.setState(id, State.FAILED);
        }
        void recover(){                                 // a fresh coordinator resumes from the log
            for (SagaRecord r: log.all()){
                if (r.state == State.COMPENSATING){      // crashed mid-rollback: FINISH the rollback, never roll forward
                    compensate(r);                        // idempotent: re-running refund()/release() is safe
                } else if (r.state != State.COMPLETED && r.state != State.FAILED){
                    try { execute(r, null); } catch (RuntimeException ex){ compensate(r); }
                }
            }
        }
    }

    // ---------- Reconciliation job ----------
    static List<String> reconcile(Payment pay, SagaLog log, boolean repair){
        List<String> orphans = new ArrayList<>();
        for (String b : pay.chargedBookings()){
            if (pay.netFor(b) <= 0) continue;           // fully refunded -> no exposure
            SagaRecord r = log.get(b);
            if (r == null || r.state != State.COMPLETED){ // charge with no completed booking = orphan
                orphans.add(b);
                if (repair) pay.refund(b+":recon-refund", b, pay.netFor(b));
            }
        }
        return orphans;
    }

    public static void main(String[] args){
        System.out.println("== Scenario 1: step fails after charge -> compensation reconciles ==");
        Inventory inv1 = new Inventory("SEAT-12A", 1);
        Payment pay1 = new Payment(); SagaLog log1 = new SagaLog();
        new Orchestrator(inv1, pay1, log1).run("bk-1","SEAT-12A",1,5000,"confirm");
        System.out.println("  final state      = " + log1.get("bk-1").state);
        System.out.println("  net(bk-1)        = " + pay1.netFor("bk-1") + "  (charge 5000 then refund -5000)");
        System.out.println("  seats available  = " + inv1.available("SEAT-12A"));
        System.out.println("  INVARIANT sum==0 : " + (pay1.net()==0) + ", no orphan charge held.");

        System.out.println();
        System.out.println("== Scenario 2: coordinator crash mid-saga -> replay reconciles (exactly-once effect) ==");
        Inventory inv2 = new Inventory("SEAT-12A", 1);
        Payment pay2 = new Payment(); SagaLog log2 = new SagaLog();
        try { new Orchestrator(inv2, pay2, log2).run("bk-2","SEAT-12A",1,5000,"crashAfterCharge"); }
        catch (CrashError ce){ System.out.println("  [coordinator crashed]: " + ce.getMessage()); }
        System.out.println("  --- read-committed window (charged, not confirmed) ---");
        System.out.println("  net(bk-2)        = " + pay2.netFor("bk-2") + "  <- money already moved");
        System.out.println("  saga state       = " + log2.get("bk-2").state + "     <- booking NOT complete: INCONSISTENT");
        // fresh coordinator, same durable stores:
        new Orchestrator(inv2, pay2, log2).recover();
        System.out.println("  --- after recovery replay ---");
        System.out.println("  final state      = " + log2.get("bk-2").state);
        System.out.println("  charge count     = " + pay2.chargeCount("bk-2") + "     <- replay did NOT double-charge");
        System.out.println("  net(bk-2)        = " + pay2.netFor("bk-2"));
        System.out.println("  seats available  = " + inv2.available("SEAT-12A"));

        System.out.println();
        System.out.println("== Scenario 3: reconciliation job vs seeded drift ==");
        // reuse pay2/log2 (holds one legit COMPLETED booking bk-2), then inject orphan charges:
        pay2.forceCharge("bk-ORPHAN-A", 3000); // a charge whose compensation never ran
        pay2.forceCharge("bk-ORPHAN-B", 7000); // another lost-refund orphan
        List<String> found = reconcile(pay2, log2, false);
        System.out.println("  bookings with a live charge = " + pay2.chargedBookings());
        System.out.println("  discrepancies detected      = " + found.size() + "  -> " + found);
        System.out.println("  (bk-2 not flagged: it maps to a COMPLETED saga)");
        reconcile(pay2, log2, true); // repair
        System.out.println("  after repair: net(bk-ORPHAN-A)=" + pay2.netFor("bk-ORPHAN-A")
                + ", net(bk-ORPHAN-B)=" + pay2.netFor("bk-ORPHAN-B") + "  (orphans refunded to 0)");
        System.out.println("  re-run detects              = " + reconcile(pay2, log2, false).size() + "  (clean)");

        System.out.println();
        System.out.println("== Scenario 4: coordinator crash DURING compensation -> recovery finishes the rollback ==");
        Inventory inv4 = new Inventory("SEAT-12A", 1);
        Payment pay4 = new Payment(); SagaLog log4 = new SagaLog();
        try { new Orchestrator(inv4, pay4, log4).run("bk-4","SEAT-12A",1,5000,"confirmThenCrashDuringCompensation"); }
        catch (CrashError ce){ System.out.println("  [coordinator crashed]: " + ce.getMessage()); }
        System.out.println("  --- mid-compensation window (COMPENSATING persisted, refund/release NOT yet run) ---");
        System.out.println("  saga state       = " + log4.get("bk-4").state + "     <- neither COMPLETED nor FAILED: rollback interrupted");
        System.out.println("  net(bk-4)        = " + pay4.netFor("bk-4") + "  <- charge still outstanding, refund not yet applied");
        System.out.println("  seats available  = " + inv4.available("SEAT-12A") + "     <- hold not yet released");
        // fresh coordinator, same durable log:
        new Orchestrator(inv4, pay4, log4).recover();
        System.out.println("  --- after recovery finishes the rollback ---");
        System.out.println("  final state      = " + log4.get("bk-4").state + "     <- FAILED, never rolled forward to COMPLETED");
        System.out.println("  net(bk-4)        = " + pay4.netFor("bk-4") + "  (charge 5000 then refund -5000)");
        System.out.println("  seats available  = " + inv4.available("SEAT-12A"));
        System.out.println("  INVARIANT sum==0 : " + (pay4.net()==0) + ", COMPENSATING record was finished, never rolled forward.");
    }
}

Compile and run: javac SagaDemo.java && java SagaDemo. The three key moves are visible in execute: it advances the durable log after each real effect (so the log never claims a step it didn't do), each step is guarded by an idempotency key and the log's ordinal (so replay skips completed steps), and a CrashError escapes without compensating — modeling a coordinator that simply stops existing, leaving the log as the only record.

4b. Build it — Go

Same design, same booking ids, same numbers. A returned crashError stands in for the coordinator dying; every other error path runs compensation inside Run.

package main

import (
	"errors"
	"fmt"
	"sync"
)

// ---------- Idempotent in-memory services ----------

type Inventory struct {
	mu        sync.Mutex
	stock     map[string]int
	holds     map[string]int
	confirmed map[string]bool
}

func NewInventory(item string, qty int) *Inventory {
	return &Inventory{
		stock:     map[string]int{item: qty},
		holds:     map[string]int{},
		confirmed: map[string]bool{},
	}
}
func (s *Inventory) Hold(key, item string, qty int) error {
	s.mu.Lock()
	defer s.mu.Unlock()
	if _, ok := s.holds[key]; ok {
		return nil
	} // idempotent
	if s.confirmed[key] {
		return nil
	}
	if s.stock[item] < qty {
		return errors.New("out_of_stock")
	}
	s.stock[item] -= qty
	s.holds[key] = qty
	return nil
}
func (s *Inventory) Confirm(key string) error {
	s.mu.Lock()
	defer s.mu.Unlock()
	if s.confirmed[key] {
		return nil
	} // idempotent
	if _, ok := s.holds[key]; !ok {
		return errors.New("no_hold")
	}
	s.confirmed[key] = true
	return nil
}
func (s *Inventory) Release(key, item string) { // compensation
	s.mu.Lock()
	defer s.mu.Unlock()
	q, ok := s.holds[key]
	delete(s.holds, key)
	delete(s.confirmed, key)
	if !ok {
		return
	} // idempotent
	s.stock[item] += q
}
func (s *Inventory) Available(item string) int {
	s.mu.Lock()
	defer s.mu.Unlock()
	return s.stock[item]
}

type entry struct {
	booking, kind string
	cents         int64
}
type Payment struct {
	mu      sync.Mutex
	ledger  []entry
	applied map[string]bool
}

func NewPayment() *Payment { return &Payment{applied: map[string]bool{}} }
func (p *Payment) Charge(key, booking string, cents int64) {
	p.mu.Lock()
	defer p.mu.Unlock()
	if p.applied[key] {
		return
	} // idempotent
	p.applied[key] = true
	p.ledger = append(p.ledger, entry{booking, "CHARGE", cents})
}
func (p *Payment) Refund(key, booking string, cents int64) {
	p.mu.Lock()
	defer p.mu.Unlock()
	if p.applied[key] {
		return
	}
	p.applied[key] = true
	p.ledger = append(p.ledger, entry{booking, "REFUND", -cents})
}
func (p *Payment) Net() (s int64) {
	p.mu.Lock()
	defer p.mu.Unlock()
	for _, e := range p.ledger {
		s += e.cents
	}
	return
}
func (p *Payment) NetFor(b string) (s int64) {
	p.mu.Lock()
	defer p.mu.Unlock()
	for _, e := range p.ledger {
		if e.booking == b {
			s += e.cents
		}
	}
	return
}
func (p *Payment) ChargeCount(b string) (n int) {
	p.mu.Lock()
	defer p.mu.Unlock()
	for _, e := range p.ledger {
		if e.kind == "CHARGE" && e.booking == b {
			n++
		}
	}
	return
}
func (p *Payment) ChargedBookings() []string {
	p.mu.Lock()
	defer p.mu.Unlock()
	seen := map[string]bool{}
	var out []string
	for _, e := range p.ledger {
		if e.kind == "CHARGE" && !seen[e.booking] {
			seen[e.booking] = true
			out = append(out, e.booking)
		}
	}
	return out
}
func (p *Payment) ForceCharge(booking string, cents int64) { // seed drift
	p.mu.Lock()
	defer p.mu.Unlock()
	p.ledger = append(p.ledger, entry{booking, "CHARGE", cents})
}

// ---------- Durable saga log ----------

type State int

const (
	Started State = iota
	Reserved
	Charged
	Confirmed
	Completed
	Compensating
	Failed
)

func (s State) String() string {
	return [...]string{"STARTED", "RESERVED", "CHARGED", "CONFIRMED", "COMPLETED", "COMPENSATING", "FAILED"}[s]
}

type SagaRecord struct {
	BookingID, Item string
	Qty             int
	Cents           int64
	St              State
}
type SagaLog struct {
	mu   sync.Mutex
	rows map[string]*SagaRecord
}

func NewSagaLog() *SagaLog { return &SagaLog{rows: map[string]*SagaRecord{}} }
func (l *SagaLog) Put(r *SagaRecord) {
	l.mu.Lock()
	defer l.mu.Unlock()
	l.rows[r.BookingID] = r
}
func (l *SagaLog) SetState(id string, s State) {
	l.mu.Lock()
	defer l.mu.Unlock()
	l.rows[id].St = s
}
func (l *SagaLog) Get(id string) *SagaRecord {
	l.mu.Lock()
	defer l.mu.Unlock()
	return l.rows[id]
}
func (l *SagaLog) All() []*SagaRecord {
	l.mu.Lock()
	defer l.mu.Unlock()
	out := make([]*SagaRecord, 0, len(l.rows))
	for _, r := range l.rows {
		out = append(out, r)
	}
	return out
}

type crashError struct{ msg string }

func (e crashError) Error() string { return e.msg }

// ---------- Orchestrator ----------

type Orchestrator struct {
	inv *Inventory
	pay *Payment
	log *SagaLog
}

// returns crashError if the coordinator "dies"; nil otherwise (step failures compensate internally)
func (o *Orchestrator) Run(bookingID, item string, qty int, cents int64, failAt string) error {
	r := o.log.Get(bookingID)
	if r == nil {
		r = &SagaRecord{BookingID: bookingID, Item: item, Qty: qty, Cents: cents, St: Started}
		o.log.Put(r)
	}
	if err := o.execute(r, failAt); err != nil {
		if ce, ok := err.(crashError); ok {
			return ce // coordinator dies; log persists; no compensation here
		}
		return o.compensate(r, failAt) // any step failure -> roll back (compensation may itself crash)
	}
	return nil
}
func (o *Orchestrator) execute(r *SagaRecord, failAt string) error {
	id := r.BookingID
	if r.St < Reserved {
		if err := o.inv.Hold(id+":inv", r.Item, r.Qty); err != nil {
			return err
		}
		o.log.SetState(id, Reserved)
	}
	if r.St < Charged {
		o.pay.Charge(id+":pay", id, r.Cents)
		o.log.SetState(id, Charged)
	}
	if failAt == "crashAfterCharge" {
		return crashError{"died after charge, before confirm"}
	}
	if r.St < Confirmed {
		if failAt == "confirm" || failAt == "confirmThenCrashDuringCompensation" {
			return errors.New("confirm_failed")
		}
		if err := o.inv.Confirm(id + ":inv"); err != nil {
			return err
		}
		o.log.SetState(id, Confirmed)
	}
	o.log.SetState(id, Completed)
	return nil
}
func (o *Orchestrator) compensate(r *SagaRecord, failAt string) error {
	id := r.BookingID
	o.log.SetState(id, Compensating)
	if failAt == "confirmThenCrashDuringCompensation" {
		return crashError{"died mid-compensation: COMPENSATING persisted, refund/release not yet run"}
	}
	o.pay.Refund(id+":refund", id, r.Cents) // reverse order, each idempotent
	o.inv.Release(id+":inv", r.Item)
	o.log.SetState(id, Failed)
	return nil
}
func (o *Orchestrator) Recover() { // a fresh coordinator resumes from the log
	for _, r := range o.log.All() {
		if r.St == Compensating { // crashed mid-rollback: FINISH the rollback, never roll forward
			o.compensate(r, "") // idempotent: re-running refund/release is safe
		} else if r.St != Completed && r.St != Failed {
			if err := o.execute(r, ""); err != nil {
				o.compensate(r, "")
			}
		}
	}
}

// ---------- Reconciliation job ----------

func reconcile(pay *Payment, log *SagaLog, repair bool) []string {
	var orphans []string
	for _, b := range pay.ChargedBookings() {
		if pay.NetFor(b) <= 0 {
			continue
		} // fully refunded -> no exposure
		r := log.Get(b)
		if r == nil || r.St != Completed { // charge with no completed booking = orphan
			orphans = append(orphans, b)
			if repair {
				pay.Refund(b+":recon-refund", b, pay.NetFor(b))
			}
		}
	}
	return orphans
}

func main() {
	fmt.Println("== Scenario 1: step fails after charge -> compensation reconciles ==")
	inv1 := NewInventory("SEAT-12A", 1)
	pay1 := NewPayment()
	log1 := NewSagaLog()
	(&Orchestrator{inv1, pay1, log1}).Run("bk-1", "SEAT-12A", 1, 5000, "confirm")
	fmt.Println("  final state      =", log1.Get("bk-1").St)
	fmt.Println("  net(bk-1)        =", pay1.NetFor("bk-1"), " (charge 5000 then refund -5000)")
	fmt.Println("  seats available  =", inv1.Available("SEAT-12A"))
	fmt.Println("  INVARIANT sum==0 :", pay1.Net() == 0, ", no orphan charge held.")

	fmt.Println()
	fmt.Println("== Scenario 2: coordinator crash mid-saga -> replay reconciles (exactly-once effect) ==")
	inv2 := NewInventory("SEAT-12A", 1)
	pay2 := NewPayment()
	log2 := NewSagaLog()
	if err := (&Orchestrator{inv2, pay2, log2}).Run("bk-2", "SEAT-12A", 1, 5000, "crashAfterCharge"); err != nil {
		fmt.Println("  [coordinator crashed]:", err)
	}
	fmt.Println("  --- read-committed window (charged, not confirmed) ---")
	fmt.Println("  net(bk-2)        =", pay2.NetFor("bk-2"), " <- money already moved")
	fmt.Println("  saga state       =", log2.Get("bk-2").St, "    <- booking NOT complete: INCONSISTENT")
	(&Orchestrator{inv2, pay2, log2}).Recover() // fresh coordinator, same durable stores
	fmt.Println("  --- after recovery replay ---")
	fmt.Println("  final state      =", log2.Get("bk-2").St)
	fmt.Println("  charge count     =", pay2.ChargeCount("bk-2"), "    <- replay did NOT double-charge")
	fmt.Println("  net(bk-2)        =", pay2.NetFor("bk-2"))
	fmt.Println("  seats available  =", inv2.Available("SEAT-12A"))

	fmt.Println()
	fmt.Println("== Scenario 3: reconciliation job vs seeded drift ==")
	pay2.ForceCharge("bk-ORPHAN-A", 3000)
	pay2.ForceCharge("bk-ORPHAN-B", 7000)
	found := reconcile(pay2, log2, false)
	fmt.Println("  bookings with a live charge =", pay2.ChargedBookings())
	fmt.Printf("  discrepancies detected      = %d  -> %v\n", len(found), found)
	fmt.Println("  (bk-2 not flagged: it maps to a COMPLETED saga)")
	reconcile(pay2, log2, true) // repair
	fmt.Printf("  after repair: net(bk-ORPHAN-A)=%d, net(bk-ORPHAN-B)=%d  (orphans refunded to 0)\n",
		pay2.NetFor("bk-ORPHAN-A"), pay2.NetFor("bk-ORPHAN-B"))
	fmt.Println("  re-run detects              =", len(reconcile(pay2, log2, false)), " (clean)")

	fmt.Println()
	fmt.Println("== Scenario 4: coordinator crash DURING compensation -> recovery finishes the rollback ==")
	inv4 := NewInventory("SEAT-12A", 1)
	pay4 := NewPayment()
	log4 := NewSagaLog()
	if err := (&Orchestrator{inv4, pay4, log4}).Run("bk-4", "SEAT-12A", 1, 5000, "confirmThenCrashDuringCompensation"); err != nil {
		fmt.Println("  [coordinator crashed]:", err)
	}
	fmt.Println("  --- mid-compensation window (COMPENSATING persisted, refund/release NOT yet run) ---")
	fmt.Println("  saga state       =", log4.Get("bk-4").St, "    <- neither COMPLETED nor FAILED: rollback interrupted")
	fmt.Println("  net(bk-4)        =", pay4.NetFor("bk-4"), " <- charge still outstanding, refund not yet applied")
	fmt.Println("  seats available  =", inv4.Available("SEAT-12A"), "    <- hold not yet released")
	(&Orchestrator{inv4, pay4, log4}).Recover() // fresh coordinator, same durable log
	fmt.Println("  --- after recovery finishes the rollback ---")
	fmt.Println("  final state      =", log4.Get("bk-4").St, "    <- FAILED, never rolled forward to COMPLETED")
	fmt.Println("  net(bk-4)        =", pay4.NetFor("bk-4"), " (charge 5000 then refund -5000)")
	fmt.Println("  seats available  =", inv4.Available("SEAT-12A"))
	fmt.Println("  INVARIANT sum==0 :", pay4.Net() == 0, ", COMPENSATING record was finished, never rolled forward.")
}

Run: go run saga.go (clean under go vet). The r.St < Charged guards are the exact analogue of Java's ordinal check — the log's monotonically-advancing state is what makes replay skip work already done.

5. Break it by running it — measure the inconsistent window, then the recovery

Both languages print byte-identical numbers. Here is the Java run, annotated:

== Scenario 1: step fails after charge -> compensation reconciles ==
  final state      = FAILED
  net(bk-1)        = 0  (charge 5000 then refund -5000)
  seats available  = 1
  INVARIANT sum==0 : true, no orphan charge held.

== Scenario 2: coordinator crash mid-saga -> replay reconciles (exactly-once effect) ==
  [coordinator crashed]: died after charge, before confirm
  --- read-committed window (charged, not confirmed) ---
  net(bk-2)        = 5000  <- money already moved
  saga state       = CHARGED     <- booking NOT complete: INCONSISTENT
  --- after recovery replay ---
  final state      = COMPLETED
  charge count     = 1     <- replay did NOT double-charge
  net(bk-2)        = 5000
  seats available  = 0

== Scenario 3: reconciliation job vs seeded drift ==
  bookings with a live charge = [bk-2, bk-ORPHAN-A, bk-ORPHAN-B]
  discrepancies detected      = 2  -> [bk-ORPHAN-A, bk-ORPHAN-B]
  (bk-2 not flagged: it maps to a COMPLETED saga)
  after repair: net(bk-ORPHAN-A)=0, net(bk-ORPHAN-B)=0  (orphans refunded to 0)
  re-run detects              = 0  (clean)

== Scenario 4: coordinator crash DURING compensation -> recovery finishes the rollback ==
  [coordinator crashed]: died mid-compensation: COMPENSATING persisted, refund/release not yet run
  --- mid-compensation window (COMPENSATING persisted, refund/release NOT yet run) ---
  saga state       = COMPENSATING     <- neither COMPLETED nor FAILED: rollback interrupted
  net(bk-4)        = 5000  <- charge still outstanding, refund not yet applied
  seats available  = 0     <- hold not yet released
  --- after recovery finishes the rollback ---
  final state      = FAILED     <- FAILED, never rolled forward to COMPLETED
  net(bk-4)        = 0  (charge 5000 then refund -5000)
  seats available  = 1
  INVARIANT sum==0 : true, COMPENSATING record was finished, never rolled forward.

Scenario 1 — a step fails after the charge, compensation reconciles. The card is charged, then the confirm step throws. Without a saga this is a $50 orphan charge on a booking the customer never got. With the saga, the orchestrator catches the failure and walks the compensations in reverse: refund (net for bk-1 goes +5000 → 0) then release (the seat comes back, available == 1). The measured invariant — sum(ledger) == 0 — is now a fact the code checked, not a claim.

Scenario 2 — the coordinator crashes, and we measure the exact window where the books are wrong. After the charge commits, the coordinator dies (throws CrashError) before confirming. The two printed lines are the money bug caught in the act: net(bk-2) == 5000 (the customer's $50 has moved) while saga state == CHARGED (the booking is not COMPLETED). That is the read-committed window — anyone querying now sees a charge with no finished booking. Then a fresh coordinator, pointed at the same durable log, calls recover(): it reads "we reached CHARGED," skips the reserve and charge (idempotent + ordinal guard), runs the pending confirm, and reaches COMPLETED. The load-bearing measurement is charge count == 1: the replay did not charge a second time. That is exactly-once effect — not because delivery was exactly-once (it wasn't; the charge step ran twice logically), but because idempotency collapsed the second attempt to a no-op.

Scenario 3 — the reconciliation backstop. See the next movement.

Scenario 4 — the coordinator crashes a second time, now mid-compensation, and recovery still finishes the job. This is the failure the first three scenarios never touch: not "a step failed," not "the coordinator died before confirming," but the coordinator dies while undoing its own damage. The confirm step throws (same as Scenario 1), compensate() persists COMPENSATING to the log — and then, before the refund or the release runs, the coordinator dies. The mid-compensation snapshot is the money bug in its cruelest form: net(bk-4) == 5000 (still charged), available == 0 (still held), and the saga state is neither COMPLETED nor FAILED — it's stuck exactly where a naive recovery routine ("if not COMPLETED, run it forward") would wrongly roll it forward and re-confirm a booking whose charge was supposed to be reversed. The code guards against exactly that: recover() checks for COMPENSATING first and routes it back into compensate(), never into execute(). Because refund and release are keyed and idempotent, and neither had run yet, a fresh coordinator finishes them cleanly: refund brings net to 0, release restores the seat, and the log lands on FAILED — never COMPLETED. The invariant this proves is the one that was previously only narrated: a record crashed in COMPENSATING is always finished, never rolled forward, and sum(ledger) == 0 is the checked receipt.

6. The reconciliation job — catch the drift the saga missed

Compensation and replay only fix failures the orchestrator participated in. Two failure modes escape it entirely: a compensation that itself crashed halfway (refund issued, release skipped, or vice-versa) and a charge written by a buggy or retried path that never got a saga record at all. Both look the same from the outside — a live charge with no COMPLETED booking behind it: an orphan charge, real money the company owes back and doesn't know it.

The reconciliation job seeds two such orphans directly into the payment ledger (bk-ORPHAN-A $30, bk-ORPHAN-B $70) alongside the one legitimately-COMPLETED booking from Scenario 2. It then joins the payment side against the saga log on the booking key and classifies every charge:

bk-2 — has a COMPLETED saga → matched, not flagged.
bk-ORPHAN-A, bk-ORPHAN-B — a live charge, no completed saga → discrepancy.

The run reports discrepancies detected = 2 — it caught both injected orphans and correctly left the good booking alone. In repair mode it issues a keyed refund for each (net → 0), and the re-run reports 0: the drift is gone and the job is idempotent (the :recon-refund key means running it twice can't double-refund). This is the same shape as the standalone ledger-reconciliation lab, scoped down to the two-service saga: a hash-join on the correlation key, four buckets collapsed to "matched vs orphan," bounded auto-repair.

The scaling note worth saying out loud: this in-memory scan is O(charges). At real volume you partition by (service, date) and hash-join the two sides — 2M charges join in seconds, not the hours a nested-loop would take — but the correctness argument is identical: reconciliation is the only component that observes both services and can therefore be the sole owner of the cross-service invariant.

7. Defend under drilling

"Isn't 'exactly-once' just a guarantee you turn on?" No — and saying so is the fastest way to fail the question. Exactly-once delivery is impossible over an unreliable network (you can guarantee at-most-once or at-least-once, never both). What you build instead is exactly-once effect: at-least-once delivery (retries, replay) plus idempotent handlers keyed so that duplicates collapse to no-ops. The charge step in Scenario 2 logically ran twice; the effect happened once because the second run matched an existing idempotency key. Idempotency + a durable log is the mechanism; "exactly-once" is the observable result, not a flag.
"Why not just 2PC across the two services?" Because one of them is a payment provider you don't control and it exposes no XA prepare/commit — you cannot make Stripe hold a lock in your transaction. Even where 2PC is available, it takes a synchronous coordinator lock across all participants for the duration; a participant crash between prepare and commit leaves resources locked (the blocking problem) and pins availability to the least-available node. Saga trades 2PC's isolation for availability: it accepts a visible, temporary inconsistency (the window in Scenario 2) and buys correctness back with compensation + reconciliation instead of a global lock.
"What is the read-committed window, exactly, and who sees it?" It's the interval between an irreversible step committing (the charge) and the saga reaching a terminal state (COMPLETED or FAILED). During it, a reader that joins the two services sees a charge with no finished booking — an intermediate state a single-DB transaction would never expose. You cannot eliminate the window (there's no global commit point); you manage it: keep it short, make sure nothing downstream treats "charged" as "booked," and let reconciliation catch any saga that dies inside it and never recovers.
"Orchestration vs choreography — which did you pick and why?" Orchestration: one coordinator owns the saga state and drives reserve → charge → confirm. It's the right call here because the money path is short, needs a single queryable "what state is this booking in," and must be debuggable when it's on fire. Choreography — each service emits an event the next reacts to, no central driver — scales the org better (teams evolve their step independently, no central bottleneck) but the saga's state is now smeared across N services' event logs, and "why did booking X stall" becomes a distributed forensics exercise. Rule of thumb: orchestrate money paths and anything needing a clear owner; choreograph high-fan-out, loosely-coupled workflows where no single team should own the whole thing.
"How is TCC different from what you built?" TCC (Try-Confirm-Cancel) is a saga variant where every participant exposes three operations and the Try reserves resources in a pending state that the resource itself holds — e.g., Try authorizes the card (holds funds) without capturing, Confirm captures, Cancel voids the authorization. The difference from compensation-based saga: TCC's Try never creates a visible committed effect that needs undoing (the auth is a hold, not a charge), so there's no orphan-charge window — Cancel voids a hold that never moved money. It's stronger, but it requires every participant to support a three-phase reserve/confirm/cancel contract; a plain "charge now" API can't do TCC, only compensation. My reserve step is actually TCC-shaped (hold → confirm/release); my charge step is compensation-shaped (charge → refund) because the provider only offers charge/refund. Real systems mix both, per-participant.
"The coordinator crashes during compensation — now what?" Same answer as during the forward path: compensations are idempotent and the log records COMPENSATING, so a fresh coordinator re-runs them and the keys make the already-done ones no-op. And if compensation is permanently stuck (refund API down for hours), reconciliation is the floor: the orphan charge sits flagged until the refund path recovers, but it is never silent.

8. Trade-off — saga vs 2PC vs TCC, orchestration vs choreography

Approach	Atomicity / isolation	Availability & blocking	Requires of participants	Reach for it when
2PC (XA)	True atomic commit; no visible intermediate state	Coordinator + participant locks held across the round trip; a crash between prepare and commit blocks resources; availability = least-available node	Every participant must be an XA resource (prepare/commit/rollback)	All participants are your own DBs, in one trust/latency domain, and you need real isolation (rare across services)
Saga (compensation)	No isolation; a visible read-committed window; correctness restored by compensations + reconciliation	No global lock; each step commits locally; highly available	Each forward step needs a defined inverse (refund, release); steps must be idempotent	Steps cross service/trust boundaries (esp. a 3rd-party API), and you can define a business inverse for each
TCC (Try-Confirm-Cancel)	No committed effect until Confirm, so no orphan window; Cancel voids a hold, not a charge	Resources held in "pending" between Try and Confirm/Cancel (a soft reservation, not a DB lock)	Every participant must expose a 3-phase reserve/confirm/cancel contract	Participants support reservations (auth-then-capture, seat holds) and you want to avoid the visible-charge window entirely

Coordination style	Where saga state lives	Coupling / org fit	Debuggability	Reach for it when
Orchestration (this lab)	One coordinator + one durable log	Central driver knows all steps; tighter coupling to the flow, looser between services	High — one place to query "what state is this booking in"	Money paths, short flows, anything needing a clear owner and 3am debuggability
Choreography	Smeared across N services' event logs	Services only know their own step + the events they emit/consume; loosest coupling, best org scaling	Low — reconstructing a stalled flow is distributed forensics	High-fan-out, loosely-coupled workflows where no single team should own the whole chain

The judgment call: across a payment provider, 2PC is off the table, so the real choice is saga vs TCC, decided by what the participant supports — TCC where a hold/auth exists, compensation where only charge/refund does. And orchestration vs choreography is decided by who needs to own the flow — money almost always wants a single owner, so orchestrate it.

9. Acceptance criteria — you're done when

Every forward step has an idempotency key and a defined compensation; running any step or compensation twice changes nothing (the :pay/:refund/:recon-refund keys prove it).
Saga state is persisted to a log that outlives the coordinator process, and a fresh coordinator can resume a half-finished saga from that log alone.
A step failure after the charge leaves sum(ledger) == 0 and no held inventory (Scenario 1 measured, not asserted).
A coordinator crash after the charge is recovered by replay with charge count == 1 — no double-charge (Scenario 2).
You can point at the exact read-committed window (charged, not confirmed) and say who can observe it and why it can't be eliminated, only bounded.
A reconciliation job independently detects orphan charges (2 seeded, 2 caught, good booking untouched), repairs them idempotently, and re-runs clean.
You can defend saga vs 2PC vs TCC and orchestration vs choreography as choices driven by participant capabilities and flow ownership — not as fashions.

10. You can now defend

Why single-process atomicity doesn't cross a service boundary, and why you can't 2PC across a payment provider — so a partial failure is a money bug with three named shapes.
That "exactly-once" is an effect assembled from at-least-once delivery + idempotency + a durable log, not a guarantee you switch on — and you've watched a charge step run twice logically and once in effect.
The precise read-committed window a saga exposes, who sees it, and why compensation handles the failures you saw while reconciliation is the mandatory backstop for the ones you didn't.
Saga vs 2PC vs TCC and orchestration vs choreography, as trade-offs of isolation-for-availability and ownership-for-org-scaling — decided by what participants support and who must own the flow.

Re-authored/deepened for this guide. Reference code compiled and executed before publishing: Java (javac SagaDemo.java && java SagaDemo, JDK) clean; Go (go vet saga.go, go run saga.go) clean. Both languages produce byte-identical scenario numbers (compensation → net 0; crash+replay → charge count 1; reconciliation → 2 orphans caught then 0) — reproduced from the runs, not asserted from memory. Sources: Hector Garcia-Molina & Kenneth Salem, "Sagas" (ACM SIGMOD, 1987) — the original long-lived-transaction-with-compensations paper; Chris Richardson, Microservices Patterns — the orchestration/choreography saga pattern and the outbox that makes the internal event stream reliable; Martin Kleppmann, Designing Data-Intensive Applications, ch. 9 — why distributed transactions/2PC block and why exactly-once needs idempotency, not magic. See also the guide's idempotent payment API and ledger reconciliation labs for the single-service pieces this lab composes across services.

🎯 STRICT STANDOUT — Saga Orchestrator & Reconciliation (Hands-On H4)

Why this lab: Single-DB ledger atomicity stops at the process boundary. You cannot 2PC across a card network. Sagas provide recoverable multi-step money flows with compensations; reconciliation is the mandatory backstop for what the orchestrator never saw.

1. Runnable contract (K11)

Steps e.g. Reserve → Charge → Confirm; each step + compensation is idempotent under a stable key (bookingId:reserve, bookingId:charge, bookingId:refund, …).
Durable saga log outlives the coordinator; crash recovery resumes or compensates from log alone.
On step failure after charge: compensate (refund, release) → net money effect 0 where business requires abort.
Reconciliation job: join payments vs saga log on correlation key; classify matched vs orphan; repair with keyed refund; re-run clean.

Hand-run (Scenario 1 — charge then fail): charge succeeds, confirm fails → compensation refund → ledger net 0; inventory not held.

Hand-run (Scenario 2 — coordinator crash): crash after charge, before success recorded → new coordinator replays; charge step runs twice logically, once in effect (idempotent provider key) → charge count 1.

Hand-run (recon): seed orphan charges ($30, $70) + one good booking → job reports 2 discrepancies; repair; second run 0 (idempotent repair keys).

2. Failure under crash / dual-write (K12)

Charged, not reserved / reserved, not charged / both done, log missing — three named partial-failure shapes.
Read-committed window: temporary charged-not-confirmed visibility; cannot eliminate without 2PC; bound with timeouts + recon.
Compensation itself crashes halfway → only recon may see orphan charge or stuck hold.
Choreography without a single owner → harder money audit trail; dual-write between services without outbox loses events.

3. When-NOT (K13)

Single database / single service → local transaction, not saga cosplay.
Provider supports true auth/hold + capture (TCC-like) → prefer hold/capture over charge+refund when available.
Don’t claim “exactly-once delivery”; claim exactly-once effect via idempotency + log.

4. Complexity / cost

Happy path latency = sum of steps; failure path adds compensation RTTs.
Recon scan O(charges) naive; production partitions by time/key and hash-joins.

5. Hostile panel (≥3)

Why not 2PC across Stripe? — provider is not an XA resource; locks would not span their API.
What is the inconsistent window? — name charged-not-confirmed and who can observe it.
Why recon if saga is correct? — bugs, lost logs, partial compensations, manual ops paths.
Orchestration vs choreography for money? — money wants a single owner/audit story → usually orchestrate.

6. Drills / acceptance

[ ] Compensation restores net-zero money after mid-flight abort (measured).
[ ] Crash+replay does not double-charge (charge count 1).
[ ] Recon catches seeded orphans and is idempotent on re-run.
[ ] Every step lists compensation + idempotency key.

Sources: Garcia-Molina & Salem, Sagas (SIGMOD 1987); Richardson Microservices Patterns (orchestration/outbox); DDIA ch.9 (distributed TX / exactly-once).

🔩 Production depth — Saga + Reconciliation

Spine: money · private lab · not a tool list

Production angle

Across services you rarely have 2PC. You have steps + compensations and a reconciliation net that admits humans and jobs will fail mid-flight. A saga that cannot answer “are we net-zero on money?” is choreography theater.

Worked failure: charge succeeded, book failed

Step 1: charge provider OK.
Step 2: ledger post fails; orchestrator dies.
No compensate; customer charged; books empty → recon must find orphan charge and refund or post.

Fix shape: durable saga state machine; each step/compensate idempotent; terminal states include needs_recon; periodic recon pages on age.

Invariants

No silent half-done money movement.
Compensate is safe to retry.
Business invariant (e.g. net ledger 0 for canceled flow) is checked, not assumed.

When-NOT

Single-DB local transaction — use it. Saga for everything is complexity cosplay. Prefer 2PC only where a real coordinator exists and latency allows.

Related: Ledger · Payment API · Job queue · Payment gateway · 2PC vs Saga theory.

🤖 Don't fully get this? Learn it with Claude

Stuck on Build a Saga Orchestrator & Reconciliation — Money Correctness Across Services? Open Claude, copy a block below, and it'll teach you this exact concept — visually and interactively.

🎨 Explain it visually

Build the mental picture, not memorization.

I just read a lesson on **Build a Saga Orchestrator & Reconciliation — Money Correctness Across Services** (Hands-On Builds) and want to truly understand it. Explain Build a Saga Orchestrator & Reconciliation — Money Correctness Across Services from first principles using ONE vivid real-world analogy and a visual mental model — draw it as ASCII art or a clear step-by-step diagram — with a concrete example using real numbers. Then ask me one question to check I got the mental picture, and wait for my reply. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

🤔 Walk me through it (interactive)

Socratic — adapts to where you're stuck.

Teach me **Build a Saga Orchestrator & Reconciliation — Money Correctness Across Services** interactively. Ask me ONE guiding question at a time, wait for my answer, and adapt to my confusion — build the idea with me step by step instead of explaining it all at once. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

🧪 Quiz me & fix my gaps

Active recall exposes what you missed.

Quiz me on **Build a Saga Orchestrator & Reconciliation — Money Correctness Across Services** with 5 questions, easy to tricky, ONE at a time. Tell me if each answer is right; at the end, explain clearly what I got wrong and why. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

🧠 Make it stick

Intuition + hook + flashcards for long-term memory.

Help me remember **Build a Saga Orchestrator & Reconciliation — Money Correctness Across Services** for the long term: give the one-sentence intuition, a memorable hook/mnemonic, a tiny worked example, and 3 active-recall flashcards (Q -> A). If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

📝 My notes

← Design a Subscription Billing Syst