Knowledge Guide
HomeHands-On BuildsSD Design Labs

Design Ledger Reconciliation

Design Ledger Reconciliation (Against an External Settlement Source)

You already built a double-entry ledger whose core promise is that every transaction's postings sum to exactly zero — a machine-checkable invariant that catches bugs inside your own database. Reconciliation is the escalation that invariant can't cover on its own: your ledger can be perfectly self-consistent and still be wrong, because the money doesn't actually move until a bank or card network agrees it moved, and that agreement arrives asynchronously, sometimes a full day later, sometimes contradicting what you already recorded. This is the staff-level differentiator Stripe interviews specifically probe for — not "can you build a ledger," but "what do you do when your ledger and the bank's ledger disagree, and how do you stop that disagreement from becoming silent, permanent drift." This lab is the HLD worksheet version of that problem: no dual-code reveal, a design workshop with a traced worksheet, a rubric, and a model answer, exactly like the room you'll sit in.

1. The Trap — the bank says yes, then no

Payout po_88213, $4,250.00, submitted to a payout processor for a marketplace seller. The processor's synchronous API call returns 200 OK within 300ms, so your service marks the payout SETTLED and, on the strength of that, releases the seller's held balance and shows them "paid" in the dashboard. That's the obvious design: trust the synchronous acknowledgment, because it's the only signal you have at request time.

The next morning, the processor's T+1 settlement file lands — the actual authoritative record of what cleared the banking rails overnight. Row 41,902 of that file lists po_88213 as FAILED, reason account_closed. The synchronous ack you trusted yesterday was provisional, not final; it meant "we accepted your request to attempt this payout," not "this payout succeeded." Your ledger now says a seller was paid $4,250.00 that, per the bank of record, never actually left your account. If nothing reconciles this, three things are true simultaneously and none of them are visible to each other: your ledger's sum(postings) == 0 invariant still holds (it's self-consistent), your dashboard still shows the seller "paid," and $4,250.00 sits in a state that doesn't match reality — discovered, if ever, by an accountant weeks later trying to explain why the bank statement is $4,250.00 higher than the books predict.

It runs the other direction just as often: the synchronous call times out or returns an ambiguous 5xx, your service marks the payout FAILED (conservatively, to avoid double-paying), refunds the hold — and the next day's file shows it actually SETTLED at the bank. Now you've told a seller their payout failed when it didn't, and (worse) you may have double-released funds you'd already committed. Either direction is the same underlying fact: a synchronous ack is not a source of truth, and the only way to know what actually happened is to check your books against the bank's — which means building a system whose entire job is finding and resolving disagreements between two independently-written ledgers, at scale, every single day.

2. Scope it like a senior

Before reaching for a matching job, pin down what "reconciled" means — each answer reshapes the design:

For the rest of this lab: daily T+1 batch files, correlation id echoed by the bank, all four exception buckets in scope, auto-resolve within tolerance / manual for the rest, reconciled before next business-day open, corrections always via reversing entry.

3. Reason to the design

Simplest thing that could work: trust the synchronous ack and never check it again. That's the trap in movement 1 — it fails not because the ack is usually wrong (it's usually right), but because "usually right" is not a property you can build a payments company on, and there is no mechanism to even detect the cases where it wasn't.

First iteration — a payment-intent state machine. Stop treating "settled" as a single fact set once. Model the lifecycle explicitly: INITIATED → SUBMITTED → PROVISIONALLY_SETTLED / PROVISIONALLY_FAILED (from the synchronous ack — provisional, not final) on next file arrival: RECONCILED if the file agrees with the provisional state, or DISPUTED if it doesn't. This buys you the vocabulary to even talk about a mismatch — "provisional" versus "reconciled" is now a real, queryable distinction instead of a single overloaded settled boolean. But a state machine alone doesn't produce the events the reconciliation job needs to consume, and it doesn't do the matching — two more pieces are still missing.

The missing piece: reliably getting the internal side of the comparison out of the database. The naive way to notify a reconciliation pipeline that "this payout was submitted" is: commit the DB write, then separately publish an event. That two-step is the classic dual-write problem — the DB commit and the publish are not the same operation, and anything that can happen between them (a crash, a network blip, a slow GC pause) can commit one and lose the other. The fix is the outbox pattern: write the business row and an outbox_event row in the exact same local DB transaction, so they are atomic by construction — either both land or neither does — and a separate relay process (poll the outbox table, or read it via CDC) is the only thing that ever publishes, retrying until it gets a durable ack. This is what makes the reconciliation job's internal-side data trustworthy at all; without it, "internal-only" and "external-only" buckets (movement 4d) can't be told apart from "we just lost the event."

The core mechanism: an async job that joins two logs on a shared key. With the state machine giving you a queryable provisional state and the outbox guaranteeing the internal event stream is complete, reconciliation itself is: for each settlement file, join its rows against the internal payment intents awaiting settlement, keyed by correlation_id, and classify every result into exactly one of four buckets — matched, internal-only, external-only, or amount-mismatch. Movement 4 turns that one sentence into a worksheet with real numbers; movement 5 shows precisely what breaks if you skip the outbox or trust either side blindly.

4. Build it — the design worksheet

This is the movement an LLD kata would spend on dual Java+Go code. A reconciliation design lab spends it on a worksheet: pin the requirements, do the arithmetic, sketch the data model and the matching algorithm, draw the path, then show the rubric.

a) Requirements (from movement 2)

b) Back-of-envelope estimation — arithmetic visible

  1. Scale target: 2,000,000 transactions/day, settling across 40 distinct (bank × currency) combinations — roughly one settlement file per combination per day, so ~40 files/day.
  2. File size: each row carries a correlation id, amount, currency, status, fee, and a timestamp — call it ~200 bytes/row with formatting overhead. Total daily settlement volume: 2,000,000 × 200 bytes ≈ 400MB/day, averaging ~10MB per file — trivial to download and parse; this is not an I/O-bound design.
  3. The naive matching approach — nested-loop join: for each of the 2,000,000 internal payment intents awaiting settlement, scan all 2,000,000 external rows looking for a correlation-id match. That's 2,000,000 × 2,000,000 = 4×1012 comparisons. Even at an optimistic 108 comparisons/sec in memory, that's 4×104 seconds ≈ 11+ hours — roughly 4× over the 3-hour SLA, and that's before the job has even partitioned by bank or touched a disk-backed join.
  4. The fix — hash-join on correlation_id: build an in-memory hash map keyed by correlation_id from one side (2,000,000 entries × ~150 bytes payload ≈ 300MB — fits comfortably in a single worker's memory), then do a single O(1) lookup per row on the other side. Total work is O(N+M) ≈ 4,000,000 hash operations. At a conservative 1,000,000 hash ops/sec, that's ~4 seconds; even 10× slower (100,000 ops/sec, accounting for string-key overhead and disk-backed page faults) that's ~40 seconds — inside the 3-hour SLA with roughly 250× headroom.
  5. Why this generalizes cleanly: partition the join by (bank, settlement_date) — each bank's file only contains that bank's own transactions, so partitions never need to see each other's data. Each partition's hash map is independently sized (typically far smaller than the 2,000,000-row worst case above) and the partitions run in parallel across workers, which is exactly the lever movement 7 reaches for when volume grows 10×–100×.

The naive-vs-hash-join gap here is the same shape as any interview BOTE: name the naive number, show it blows the SLA, then find the one algorithmic change (indexing by the join key instead of scanning) that turns an infeasible number into a comfortable one.

c) Data model

payment_intent(
  id                TEXT PRIMARY KEY,     -- the correlation_id, echoed to the bank
  amount_cents      BIGINT,
  currency          TEXT,
  state             TEXT,                -- INITIATED|SUBMITTED|PROVISIONALLY_SETTLED|
                                          -- PROVISIONALLY_FAILED|RECONCILED|DISPUTED
  created_at        TIMESTAMP
)

ledger_entry(                            -- from the double-entry ledger kata
  id, payment_intent_id, account_id, amount_cents, created_at
)                                         -- append-only; NEVER edited (movement 5)

outbox_event(                            -- written in the SAME txn as payment_intent/ledger_entry
  id, aggregate_id, event_type, payload, created_at, published_at NULLABLE
)

settlement_file_row(                     -- one row per bank-file line, as ingested
  id, file_id, bank, correlation_id, amount_cents, currency,
  status, fee_cents, settled_at, ingested_at
)

reconciliation_exception(
  id, payment_intent_id NULLABLE, settlement_row_id NULLABLE,
  bucket            TEXT,                -- fee-deduction|rounding|timing|unexplained
  delta_cents       BIGINT,
  status            TEXT,                -- open|auto-resolved|manual-resolved
  resolution_note   TEXT,
  reversing_entry_id NULLABLE
)

d) The matching algorithm — join internal ledger vs external file on correlation_id

Per (bank, settlement_date) partition:

  1. Load the file's settlement_file_rows and build a hash map keyed by correlation_id.
  2. Load internal payment_intents in that partition with state ∈ {SUBMITTED, PROVISIONALLY_SETTLED, PROVISIONALLY_FAILED} into a second hash map, same key.
  3. For every external row, probe the internal map (O(1)):
    • Found, amounts match exactly, status agrees (e.g. provisional SETTLED, file says SETTLED) — matched: transition the intent to RECONCILED, remove it from the internal map.
    • Found, status disagrees (provisional SETTLED, file says FAILED, or vice versa — the movement-1 trap) — status mismatch: transition to DISPUTED, always routed to the unexplained bucket (movement 4e) — a status contradiction is never auto-resolved.
    • Found, amounts differ but status agreesamount mismatch: bucket by the size and shape of the delta (movement 4e: fee-deduction, rounding, or unexplained).
    • Not found in the internal mapexternal-only: the bank has a record you don't — bucketed as timing (most common: your event hasn't landed yet) unless it ages out (movement 4e).
  4. Whatever remains unconsumed in the internal map after all external rows are processed — internal-only: you have a record the bank's file doesn't (yet) confirm — also bucketed as timing initially.

e) The exception buckets — and how each resolves

BucketTriggerResolution
fee-deductionAmount mismatch exactly equal to a known fee-schedule entry (e.g. bank settles net-of-fee, ledger recorded gross)Auto-resolve: post an explicit fee-expense leg so the transaction's postings still sum to zero, mark RECONCILED. Never silently absorb the delta — it must appear as its own posting.
rounding|delta| ≤ a configured tolerance (e.g. 1 cent per leg, or half a minor unit on FX conversion)Auto-resolve: post a small, capped "rounding adjustment" leg. Monitor the auto-resolved rounding total per day — a tolerance that's quietly absorbing real errors shows up as a rounding total that's suspiciously large or one-directional.
timingInternal-only or external-only, within a grace window (e.g. 2 settlement cycles)Hold, re-check automatically against the next file. Auto-closes to matched once the other side shows up; escalates to unexplained if the grace window expires with no match.
unexplainedStatus mismatch (movement 4d), or an amount mismatch beyond fee/rounding tolerance, or a timing hold that aged outNever auto-resolved. Opens a manual-review ticket. The eventual fix, decided by a human, is always a new reversing entry referencing the original transaction id — the original postings are left untouched for audit (movement 5, design flaw 2).

f) High-level diagram

The path a payment takes from internal commit to reconciled (or bucketed) outcome:

Internal ledger plus outbox feeds a reconciliation job that also ingests the external settlement file; the job hash-joins on correlation_id and routes matches to reconciled, mismatches into fee-deduction, rounding, timing, and unexplained exception buckets
Internal ledger plus outbox feeds a reconciliation job that also ingests the external settlement file; the job hash-joins on correlation_id and routes matches to reconciled, mismatches into fee-deduction, rounding, timing, and unexplained exception buckets

Rubric — what a strong answer hits

Model answer: compare your worksheet against Designing a Payment System for the end-to-end payment architecture this reconciliation job sits inside, Transactional Outbox — Solving the Dual-Write Problem for the outbox mechanics used in movement 3, and What Is the Outbox Pattern, and How Does Change Data Capture Work for the CDC alternative used in movement 6 — use them to check what you missed, not as a first read. Pair this lab with Design a Double-Entry Ledger, whose sum-to-zero invariant and reversing-entry rule this design leans on directly.

5. Break it — run the design as a test and watch it fail

Design flaw 1 — no outbox, the dual-write problem, as a test. Build the reconciliation feed the naive way: the payment service commits a DB transaction marking po_88213 as SUBMITTED, then, as a second and separate step, calls publish("payment.submitted", ...) to notify the async pipeline the reconciliation job reads from. Kill the process (simulate a pod eviction, a network blip to the broker, a slow GC pause) in the gap between the commit and the publish call. Expected result if events were reliable: the reconciliation job's internal-side dataset contains every SUBMITTED payment intent. Actual result: the DB write is durable — po_88213 really is SUBMITTED — but the event was never published, so nothing downstream ever learns about it. At a realistic 0.1% crash-in-the-gap rate across 2,000,000 submissions/day, that's ~2,000 lost events/day, each one surfacing tomorrow as an unexplained "external-only" row (the bank settled a transaction the reconciliation job's internal map never saw coming) — a symptom that looks like a bank-side anomaly but is actually a hole in your own event pipeline. Run it the other direction and it's worse: publish succeeds, then the enclosing transaction rolls back on a later validation failure — now a phantom event exists for a payment intent that was never actually created.

The fix, traced against the same scenario: write the payment_intent row and the outbox_event row in the exact same local DB transaction (movement 3). The commit is atomic across both by construction — there is no gap between "the intent exists" and "the event that says so exists," because they're the same write. A separate relay process (or a CDC reader tailing the DB's write-ahead log) polls outbox_event rows where published_at IS NULL and retries publishing until it gets a durable ack, then stamps published_at. Kill the process at any point now, and on restart the relay simply finds the same unpublished row still sitting in the outbox table and retries — the event is never silently lost, only ever delayed.

Design flaw 2 — the naive "trust our ledger" auto-fixer, as a test. Build a first-cut reconciliation job that, on any mismatch, directly UPDATEs the payment_intent's state and the account balance to whatever it currently believes is correct — no reversing entry, no ticket, no audit trail. Feed it the movement-1 scenario: internal state PROVISIONALLY_SETTLED (seller's balance already credited $4,250.00), file says FAILED. The naive job silently flips the state to FAILED and decrements the balance in place. Expected result if the fix were sound: an auditor re-deriving the ledger from its append-only journal can explain every posting. Actual result: two failure modes, both real. (1) If the settlement file itself later turns out to be wrong (a stale re-send, a bank-side processing error, corrected the following day) you've now introduced an unexplained, undocumented edit into a previously-correct ledger — there is no posting recording why the balance moved, so the append-only journal from the double-entry kata no longer explains its own state. (2) Even when the bank was right, mutating a balance in place without a matching opposite-leg posting breaks the sum(postings) == 0 invariant outright — you've reintroduced the exact vanishing/appearing-money bug the double-entry ledger kata was built specifically to prevent, just one layer up, inside the tool meant to guard against it.

The fix, traced against the same scenario: never edit po_88213's original postings. Instead, post a new reversing entry — a debit/credit pair that references the original transaction id, moves the $4,250.00 back out of the seller's balance, and lands the intent in DISPUTED pending manual review (the "unexplained" bucket, since a status contradiction is never auto-resolved — movement 4d). The journal now shows exactly what happened: the original posting, the disagreement, and the correction, in that order, with nothing rewritten. Combine this with an idempotency key on the reversing entry itself (e.g. reversal:po_88213:v1) so a job re-run after a crash can't double-post the same correction — the same "make the retry safe" principle from the double-entry kata's idempotent posting, applied to corrections instead of original transactions.

6. Optimise — with trade-offs

The batch-file, hash-join, outbox-backed design in movement 4 is one point on a spectrum. A strong answer names the neighboring points and says explicitly why it picked one:

DecisionOption AOption BWhen A winsWhen B wins
Reconciliation cadenceBatch (daily T+1 settlement file, this lab's baseline)Real-time (per-transaction webhook/event from the bank as it settles)The realistic default — most acquirers and card networks only expose a daily batch file; detection latency is bounded by file cadence (up to 24h), which is acceptable for most exception types (fee/rounding/timing)When the processor genuinely offers real-time settlement webhooks (some modern PSPs do) — catches status-mismatch disputes (movement 4d) within seconds instead of a day, valuable specifically for the "bank says failed, we said settled" case where a seller is waiting
Exception handlingAuto-resolve (fee-deduction, rounding, within a tolerance)Manual review (unexplained, status mismatch)High-volume, well-understood, bounded-magnitude mismatches — the only way reconciliation scales past a few hundred exceptions/day without an ops team drowning in ticketsAnything outside a known, tight tolerance or involving a status contradiction — auto-resolving these to "look clean" is exactly the design flaw traced in movement 5; a human must decide before a reversing entry is posted
Reliable internal-event deliveryTransactional Outbox (app writes an outbox row in the same txn, a relay publishes)CDC (tail the DB's write-ahead log directly, e.g. Debezium on Postgres)Simpler to reason about and debug — the outbox table is just a table you can query; no dependency on DB-internal log formats or extra infrastructure (Kafka Connect, a CDC connector)Once outbox-table write volume itself becomes a bottleneck, or you want zero extra application-level writes — CDC derives events straight from the WAL with no outbox table at all, at the cost of coupling to the database's internal replication/log format and needing dedicated CDC infrastructure
Reliable internal-event deliveryOutbox or CDC (either, both are "commit locally, deliver asynchronously")2PC (a distributed transaction spanning the DB and the message broker)Almost always — outbox/CDC decouple the DB commit from delivery, so a broker outage never blocks a payment from committingRarely justified in practice: 2PC needs broker-side XA/distributed-transaction support (uncommon at internet scale), blocks the DB commit on the broker being reachable, and serializes throughput behind a coordinator — the classic reason outbox/CDC displaced 2PC for this exact problem

7. Defend under drilling

Q1 — "The bank says success, you say failure — how do you resolve it?"
That's a status mismatch (movement 4d) — by design, it is never auto-resolved, regardless of direction. It transitions the payment intent to DISPUTED and routes straight to the unexplained bucket for manual review. The resolution is always a reversing entry decided by a human: if the bank's file is authoritative (the usual case), reverse whatever the provisional state assumed and notify the affected party; if the file itself is later found to be in error (a stale re-send, a bank-side data issue), escalate to the bank and hold the intent in DISPUTED until a corrected file arrives — you never silently trust either side by default.

Q2 — "How do you guarantee you don't lose an event (the dual-write problem)?"
The transactional outbox (movement 3, traced as a failing test in movement 5): write the business row and an outbox_event row in the same local DB transaction, so they commit atomically together. A separate relay (or CDC reader) is the only thing that ever publishes, polling for unpublished rows and retrying until it gets a durable ack. A crash between "DB write" and "publish" simply can't happen anymore, because there is no longer a gap — both are the same write.

Q3 — "How do you correct a wrong posting once you've found one?"
You never edit it. You post a new reversing entry that references the original transaction id, with its own debit/credit legs that keep the ledger's sum(postings) == 0 invariant intact (movement 5, design flaw 2). The original posting stays exactly as it was, wrong-looking and all — that's the audit trail. This is a direct extension of the double-entry ledger kata's "reversal-only corrections" rule, now applied to bugs the bank's file surfaces instead of bugs your own code introduces.

Q4 — "A settlement file re-sends a row you already processed, or duplicates one — now what?"
The reconciliation job's ingestion step must be idempotent on (file_id, correlation_id) — upsert settlement_file_rows keyed on that pair rather than blind-inserting, so a re-sent or duplicated row overwrites its own prior version instead of creating a second one the matching step would double-count. The same idempotency-key discipline covers the resolution side too: a reversing entry is keyed (e.g. reversal:po_88213:v1) so a job re-run after a crash can't double-post the same correction — the exact "make retries safe" principle from the ledger kata, now protecting corrections instead of original postings.

Q5 — "How do you scale matching to millions of transactions a day?"
Movement 4b's arithmetic: a naive nested-loop join over 2,000,000 × 2,000,000 rows is ~4×1012 comparisons — over 11 hours, blowing any reasonable SLA. Switching to a hash-join keyed on correlation_id drops that to O(N+M) — a few seconds. Partitioning by (bank, settlement_date) (movement 4b, point 5) means each partition's hash map is independently sized and every partition can run in parallel across workers, since one bank's file never needs to see another bank's data.

Escalation — what breaks at 10×–100× this volume? At 20,000,000 transactions/day, the naive nested-loop number becomes absurd (~4×1014 comparisons) but the hash-join answer barely moves — O(N+M) scales linearly, and 40,000,000 hash operations at 1,000,000 ops/sec is still under a minute per partition, run in parallel across more workers. What actually starts to hurt at this scale is the exception-handling side, not the join: a fixed manual-review-team headcount can't absorb a proportional rise in unexplained-bucket volume, so the auto-resolve tolerance (movement 4e) and the timing grace window both need to be tuned tighter with real monitoring (are auto-resolved totals trending up in a way that suggests a real, systemic error rather than genuine rounding?) rather than loosened to "make the queue smaller." This is also where the batch-vs-real-time trade-off (movement 6) gets revisited: at high enough volume, moving even a subset of transactions to a processor's real-time settlement webhook (where available) shrinks the unexplained-bucket dwell time from a day to seconds for the highest-value transactions, without re-architecting the whole batch pipeline.

The pairs_with move: if you're asked this cold in an interview, say the LLD-to-HLD escalation out loud — "single-process double-entry ledger (build it first) guarantees internal self-consistency, but self-consistency isn't the same as agreeing with the outside world — here's the async system that checks it against a bank's own record, and why every correction still has to be a reversing entry, never an edit" — that's exactly the arc this lab walked movements 1 through 5.

8. You can now defend


Re-authored/Deepened for this guide. Related theory: Designing a Payment System · Transactional Outbox — Solving the Dual-Write Problem · What Is the Outbox Pattern, and How Does Change Data Capture Work · Idempotency & "Exactly-Once Is a Myth". Paired LLD kata: Design a Double-Entry Ledger.

🤖 Don't fully get this? Learn it with Claude

Stuck on Design Ledger Reconciliation? Open Claude, copy a block below, and it'll teach you this exact concept — visually and interactively.

🎨 Explain it visually

Build the mental picture, not memorization.

I just read a lesson on **Design Ledger Reconciliation** (Hands-On Builds) and want to truly understand it. Explain Design Ledger Reconciliation from first principles using ONE vivid real-world analogy and a visual mental model — draw it as ASCII art or a clear step-by-step diagram — with a concrete example using real numbers. Then ask me one question to check I got the mental picture, and wait for my reply. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.
🤔 Walk me through it (interactive)

Socratic — adapts to where you're stuck.

Teach me **Design Ledger Reconciliation** interactively. Ask me ONE guiding question at a time, wait for my answer, and adapt to my confusion — build the idea with me step by step instead of explaining it all at once. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.
🧪 Quiz me & fix my gaps

Active recall exposes what you missed.

Quiz me on **Design Ledger Reconciliation** with 5 questions, easy to tricky, ONE at a time. Tell me if each answer is right; at the end, explain clearly what I got wrong and why. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.
🧠 Make it stick

Intuition + hook + flashcards for long-term memory.

Help me remember **Design Ledger Reconciliation** for the long term: give the one-sentence intuition, a memorable hook/mnemonic, a tiny worked example, and 3 active-recall flashcards (Q -> A). If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

📝 My notes