Knowledge Guide
HomeHands-On BuildsSD Design Labs

Design a Payment Gateway End-to-End

Design a Payment Gateway (End-to-End)

You've already built the idempotency store that stops a single charge endpoint from double-billing under a retry race. Now zoom out: that store is one component inside a much bigger machine, and the machine has a much bigger way to lose money than a retry race — it can lose the record that a charge ever happened at all. This lab is the HLD worksheet for the flagship fintech system-design problem: design a Stripe-shaped payment gateway that accepts a charge, moves money through a card network it does not control, and never loses track of where the money is — even when a process crashes mid-charge, the card network times out, or a merchant asks "did you actually pay me?" six months later. No dual-language code kata here — a design workshop with a model answer, a rubric, and a defend-under-pushback round, exactly like the room you'll sit in for a payments-team system-design interview.

1. The Trap — call the card network inline, and lose money on the first hiccup

The obvious first design: POST /charge arrives, the request handler validates the request, calls the card network's API synchronously, waits for the authorization response, writes the result to the database, and returns 200 to the client — one straight-line function, no queues, no workers. It passes the demo. Trace what happens the first time it meets production conditions:

None of these three failures required anything exotic — a slow but honest downstream call is the normal case for a card network, not an edge case, and a synchronous straight-line design turns "normal downstream slowness" directly into cascading timeouts, double charges, and silently lost money-state. The fix is not "make the card network faster" (you don't control it) — it's "never let the customer-facing request depend on how long the card network takes to answer." That single architectural move is what the rest of this lab builds.

2. Scope it like a senior

Before drawing boxes, pin down what this gateway actually has to guarantee — each answer reshapes the design that follows:

Locked in for the rest of this lab: 10,000 TPS peak, asynchronous accept-then-notify, multi-PSP, zero-data-loss on the attempt record, idempotent at the edge, refund/dispute states modeled but not fully worked.

3. Reason to the design

Simplest thing that could work: the synchronous straight line from Movement 1. It fails for the three reasons already traced — and notice none of the three failures is about correctness of the card-network call itself; all three are about coupling the client's request lifetime to the card network's response time. That's the bottleneck to remove, not the charge logic.

Decouple accept from charge. Split the operation into two phases: (1) accept the request, validate it, durably persist it in a PENDING state, and return immediately — this phase is entirely under your control and can be made fast and reliable; (2) a background worker picks up the persisted intent and does the slow, unreliable part (talking to the card network) on its own time, with its own retry policy, completely off the customer's request thread. The client gets a payment_intent_id back in milliseconds and later learns the outcome by polling that id or receiving a webhook. This single split removes Movement 1's failure #1 and #2 entirely: no request thread ever blocks on the card network, and there's no window where two client retries can both reach the network before either is recorded, because idempotency is claimed at the fast, synchronous accept step — before any worker exists to race.

The remaining failure: your own process can still crash mid-worker. Splitting accept from charge doesn't remove Movement 1's failure #3 — it just moves it from "in the request handler" to "in the worker." The worker can still call the PSP, get a success response, and die before writing that down. The fix here is the same durability principle applied one level deeper: the worker must durably record which PSP-facing operation it is about to attempt (with its own idempotency key) BEFORE making the network call — a transactional outbox — so that if it crashes after the call succeeds but before marking its own record complete, the next worker that picks up the same outstanding intent re-sends the same idempotency key to the PSP and gets back the PSP's cached result instead of charging again. This is traced precisely, with the exact crash window, in Movement 5.

Money movement gets its own ledger, separate from the operational status. payment_intent.status answers "what state is this charge in for API/UX purposes" — it's a mutable row that gets updated in place. The actual movement of money is a different kind of fact: it must never be edited or deleted once written, because it's the thing you reconcile against the bank and, ultimately, a legal financial record. That's why the design has two separate stores doing two separate jobs: a mutable state machine for operational status, and an append-only, double-entry ledger for money movement (built in depth in the ledger kata — this lab uses it as a component, not a thing to re-derive). Finally, since your side and the bank's side can each independently be told a wrong story by a flaky network, a periodic reconciliation job diffs your ledger against the PSP's own settlement report and surfaces any mismatch as an exception, rather than trusting either side blindly (Movement 7, Q4).

4. Build it — the design worksheet

This is the movement an LLD kata would spend on dual Java+Go code. A flagship system-design lab spends it on a worksheet: pin the requirements, do the arithmetic, sketch the API and data model, draw the path, and name the one invariant that has to hold no matter what.

a) Requirements (from Movement 2)

b) Back-of-envelope estimation — arithmetic visible

  1. Scale target: peak 10,000 TPS (flash-sale spike); steady-state average 500 TPS (a ~20× peak-to-average ratio is normal for a payments platform — most days are ordinary, some hours are Black Friday).
  2. Daily transaction volume (at average): 500 × 86,400s ≈ 43.2 million transactions/day.
  3. Writes per transaction: the state machine (Movement 4d) advances through ~4 rows written over a charge's life (Pending → Authorized → Captured → Settled), plus one double-entry ledger pair (2 immutable rows) ≈ 6 DB writes/transaction.
  4. Sustained write rate: 500 TPS × 6 ≈ 3,000 writes/sec; at the 10,000 TPS peak: 10,000 × 6 = 60,000 writes/sec — this is the number that forces sharding (Movement 4d), not the inbound request rate itself.
  5. Storage: a payment_intent row ≈ 1KB (ids, amount, currency, status, PSP refs, idempotency key, timestamps, metadata); a ledger pair ≈ 2 × 300B = 600B. ≈ 1.6KB/transaction × 43.2M/day ≈ 69GB/day raw. With 3× replication and index overhead (≈1.5×): ≈ 310GB/day — a multi-year retention plan is a real storage-capacity conversation, not an afterthought.
  6. Bandwidth (sanity check, not the bottleneck): a ~2KB request+response at the edge × 10,000 TPS peak ≈ 20MB/s — trivial next to the 60,000 writes/sec figure; the write path, not network bandwidth, is what you design around.

c) API sketch

POST /v1/charges
Idempotency-Key: 5c1e0e2a-6b7a-4b1e-9c1a-2f7b6a0e9e11   <- ties directly to the LLD kata's atomic claim
{
  "amount_cents": 4999,
  "currency": "USD",
  "payment_method_token": "pm_tok_abc123",
  "merchant_id": "m_9f21",
  "capture_method": "automatic"
}

→ 202 Accepted (NOT 200 -- the final result is not known yet)
{
  "payment_intent_id": "pi_7ad2e9",
  "status": "pending",
  "created_at": "2026-07-11T10:03:22Z"
}

GET /v1/charges/{payment_intent_id}          -> poll for the current status
POST https://merchant.example.com/webhooks   -> pushed by the gateway, HMAC-signed:
{ "type": "payment_intent.succeeded", "payment_intent_id": "pi_7ad2e9", "status": "captured" }

The 202 instead of 200 is the API surfacing the architecture's real contract: "accepted for processing," not "here is the final answer" — a client that expects the card-network result synchronously is expecting the exact thing Movement 1 proved doesn't scale. The webhook push itself needs its own retry/backoff/signature-verification design if the merchant's endpoint is slow or down — that mechanism is built in full in the webhook-delivery kata; this lab treats the dispatcher as a component, the same way it treats the ledger.

d) Data model & state machine

payment_intent (mutable, one row per charge attempt, updated in place as it advances):

id (ULID, PK)  merchant_id  idempotency_key (UNIQUE)
amount_cents  currency  status  psp_name  psp_reference
created_at  updated_at  version   -- version: optimistic-concurrency guard on status transitions

State machine (the only transitions a strong answer should draw — anything else is a bug):

PENDING --(worker claims + calls PSP)--> AUTHORIZED --(capture)--> CAPTURED --(settlement batch)--> SETTLED
   |                                         |                        |
   +-----------(PSP declines)-----------> FAILED               (post-capture)--> REFUNDED

ledger_entry (append-only, immutable, double-entry — full mechanism in the ledger kata): entry_id, payment_intent_id (FK), account_id, direction (debit|credit), amount_cents, currency, created_at — never UPDATEd, only ever appended.

Partitioning: shard both tables by hash(merchant_id), keeping one merchant's full history co-located so their dashboard/reporting queries never scatter-gather across every shard — at the cost of a single very high-volume ("whale") merchant becoming a hot shard, mitigated by sub-sharding that one merchant's key range separately once it's measured, not pre-emptively.

e) High-level diagram & the one invariant that must hold

The card-network call is drawn deliberately on the far side of a hard line — that line is the entire design:

Charge pipeline from client through edge, PENDING persistence, a queue, a PSP worker pool, the card network, the ledger, and a webhook dispatcher, split by a dashed line into a synchronous client-facing half and an asynchronous background half
Charge pipeline from client through edge, PENDING persistence, a queue, a PSP worker pool, the card network, the ledger, and a webhook dispatcher, split by a dashed line into a synchronous client-facing half and an asynchronous background half

f) Bottlenecks & scaling

Rubric — what a strong answer hits

Model answer: compare your worksheet against Designing Payment System, the existing guide page that walks the full RESHADED treatment for this exact problem — use it to check what you missed, not as a first read.

5. Break it — run the design as a test and watch it fail

Design flaw 1 — synchronous PSP call in the request path, quantified. Suppose the average card-network authorization takes 500ms and the gateway is at its 10,000 TPS peak. By Little's Law (requests in flight = arrival rate × time in system): 10,000 × 0.5s = 5,000 requests concurrently parked waiting on the card network at any instant — before a single one of them has done anything else. A typical app server's thread/connection pool (low hundreds to low thousands) is exhausted by that alone; new, unrelated requests now queue behind slow-PSP requests, health checks start failing under load, the load balancer marks instances unhealthy, and the remaining healthy instances absorb even more load — a cascading failure fleet-wide, triggered by ONE slow downstream dependency the design chose to sit in the critical path. The fix: the accept/charge split from Movement 3 — the request handler's only synchronous work is the fast local PENDING write; the 500ms card-network call happens on a worker whose pool size is sized to the PSP's OWN throughput limit, completely decoupled from the client-facing tier's capacity.

Design flaw 2 — double-capture without idempotency. Client sends a charge, the card network is genuinely slow, the client's own timeout fires before any response arrives, and the client retries the identical request. Without an idempotency key tying the retry to the original attempt, the handler has no way to recognize "this is a resend of an in-flight charge" versus "this is a brand-new charge" — it processes both, and the card is charged twice for one purchase. This is exactly the check-then-act race the idempotent-payment-api LLD kata reproduces at the code level (50 concurrent retries against a naive store → 50 real charges) and fixes with a single atomic claim. The fix, at the gateway's edge: claim the Idempotency-Key atomically at accept time — the very first synchronous step, before PENDING is even written — so a retry is recognized and replayed before it can ever spawn a second worker job.

Design flaw 3 — money-state lost on crash without an outbox. Trace the exact window: a PSP worker calls the card network; the card network processes the charge and returns success; the worker process crashes (deploy, OOM, host failure) before it commits AUTHORIZED to the database. Money has already moved — the card network has no undo — but your system's only record still says PENDING. Worse, if the retry logic has no durable memory that the PSP call was ever sent, it may re-send a fresh charge to the PSP, risking a second, independent double-charge — this time triggered by your OWN system's retry, not the client's. This is the classic dual-write problem: two operations (call an external system; commit your own state) that must both happen or neither, with no distributed transaction spanning your database and the PSP.

The fix — a transactional outbox. In the SAME transaction that writes the PENDING row, also write an outbox_event row carrying its own idempotency key — this key, not the payment_intent id, is what actually gets sent to the PSP as its Idempotency-Key. A worker reads an unprocessed outbox row, calls the PSP with that key, and only then marks the row processed. If the worker crashes after the PSP call succeeds but before marking the row processed, the NEXT worker picks up the same unprocessed row and resends the same key — and because the PSP is itself idempotent on that key (the exact mechanism proved in the LLD kata, now applied to the PSP-facing hop instead of the client-facing one), it returns the original charge's result instead of charging again. The outbox makes "did I already tell the PSP to do this?" durable across a crash; the PSP-side idempotency key makes the inevitable resend safe rather than a second charge.

6. Optimise — with trade-offs

ApproachCustomer-facing latencyFailure blast radiusComplexityBest for
Synchronous PSP call in the request path (Movement 1's trap)Bound to the PSP's own latency — 200ms–2s, unbounded tail during a PSP incidentFleet-wide — one slow dependency exhausts the shared thread/connection pool (Movement 5, flaw 1)Lowest — one function, no queue/workersNever at real scale; acceptable only for a single-box demo/prototype.
Async accept-then-worker (this lab's design)Sub-50ms — bounded by the local PENDING write onlyContained to the PSP-worker pool and queue depth, never touches the client-facing tier's capacityHigher — queue, worker pool, webhook dispatcher, polling endpointAny production payment path at meaningful scale.
Single-PSP integrationn/aA PSP outage is a total platform outage — no fallbackLowest — one code path, one webhook format, one set of retry semanticsEarly-stage, low volume, before a PSP's reliability or fee structure has actually hurt you.
Multi-PSP routing (a routing layer that translates one canonical charge into each PSP's own API shape)Slightly higher — a routing decision per chargeA single PSP outage degrades capacity, doesn't take the platform down — traffic fails over to a healthy PSPMaterially higher — every PSP has its own semantics, retry behavior, and settlement report format to reconcile againstAny platform where volume is high enough that one PSP's outage or fee structure is a real business risk (Stripe/Adyen-scale platforms run this).
Strong consistency for balance reads (read straight from the ledger's primary / synchronous ledger write in the capture path)Adds the ledger write to the synchronous capture pathLedger becomes a single-writer-per-account bottleneck under load (Movement 4f)Lower — one source of truth, no staleness to reason aboutThe authoritative ledger itself, and any payout/withdrawal decision — never let a merchant withdraw against a stale, too-high balance.
Eventual consistency for balance reads (a denormalized, async-updated read replica/cache the dashboard reads from)Dashboard can lag the true ledger by secondsReads scale horizontally, never contend with the ledger's write pathLower operationally, but requires explicitly tracking stalenessDisplay-only balance UIs and analytics — anything that isn't gating an actual money-movement decision.

The judgment call to say out loud: the async accept/charge split and per-PSP idempotency are not optional performance tuning — skipping either one reopens a correctness bug this lab traced (Movement 5), not just a latency regression. Multi-PSP routing and the consistency-for-reads choice, by contrast, are genuine judgment calls: pick single-PSP and strong-everywhere while volume is low and simplicity is worth more than resilience/scale, and revisit both the moment either one has actually cost you an incident or a client complaint — not preemptively.

7. Defend under drilling

Q1 — "The PSP call times out — did it go through?"
A timeout on the worker's connection to the PSP tells you the RESPONSE didn't arrive in time — it tells you nothing about whether the PSP's backend actually processed the request; the request may well have landed and been processed even though the acknowledgment never came back. The safe move is never "assume failure and retry with a fresh request" — it's "retry the SAME idempotency key" (the outbox key from Movement 5). The PSP, idempotent on that key by contract, either returns the charge that already happened or genuinely processes it for the first time — either way, exactly once. If retries themselves keep failing, the intent sits in a distinct "PSP status unknown" sub-state rather than being guessed at, and the reconciliation job (Q4) is the backstop that resolves it definitively against the PSP's own settlement report within its SLA.

Q2 — "Zero data loss — how?"
Two mechanisms, together: the transactional outbox (Movement 5, flaw 3) durably commits "intent to call the PSP" in the SAME transaction as the PENDING row, before any network call happens — a crash after that point loses nothing, because the next worker resumes from the durable intent, not from in-memory state; and every state-machine transition is itself a committed row, never held only in memory. "Zero data loss" means zero loss of the RECORD of what was attempted and what happened — it does not mean the system is never ambiguous (Q1 can absolutely happen); it means every ambiguous state is durably recorded as ambiguous and resolved by reconciliation, never silently dropped.

Q3 — "Double-charge prevention?"
Two independent, deliberately redundant layers: the client-facing Idempotency-Key on POST /v1/charges (the LLD kata's exact mechanism) dedups a client-side retry before it ever reaches a PSP worker a second time; the outbox's own idempotency key on the PSP-facing call separately dedups an internal retry (a worker crash-and-resume, Movement 5 flaw 3) so even your own system's resend can't double-charge. Either layer alone misses a direction of retry; both together cover the client side and the internal side.

Q4 — "How do you reconcile with the bank?"
The PSP/card network publishes its own settlement report (a daily batch file or API) listing every transaction it actually processed and settled, independent of what your system believes happened. A scheduled reconciliation job diffs that report against your internal ledger three ways: (a) a PSP-reported transaction with no matching internal record — a hole, rare given the outbox design but not impossible (a bug, a manual fix, a migration gap); (b) an internal CAPTURED/SETTLED record with no matching PSP settlement — you think you got paid and didn't; (c) a matched pair with a mismatched amount (partial capture, currency-conversion rounding). Every mismatch lands in an exceptions queue for resolution, and "zero discrepancies today" is a metric you actively watch, not an assumption you make.

Q5 — "10,000 TPS — how does this scale?"
Walk Movement 4b's own numbers: the client-facing tier scales trivially — it does one fast local write and returns, with no PSP call in that path at all, so it isn't coupled to card-network latency regardless of fleet size. The real pressure is the ~60,000 writes/sec figure, which forces sharding payment_intent/ledger_entry by merchant_id (Movement 4d) and lets the PSP-worker pool scale independently of the API tier (they're decoupled by the queue — 10× more workers needs zero change to the client-facing fleet). The two things that do NOT just scale by adding boxes (Movement 4f): PSP rate limits — the PSP itself won't accept more than its published cap no matter how many workers you run, so the queue is what absorbs the mismatch — and the ledger's single-writer-per-account requirement, since sharding spreads different merchants apart but one hot merchant's own account is still one logical writer regardless of shard count.

Escalation — what breaks at 100×? At 1,000,000 TPS the 60,000 writes/sec figure becomes 6,000,000/sec — on the order of 100× more shards purely for write throughput, which makes the whale-merchant hot-shard problem (Movement 4d) and ledger single-writer contention (Movement 4f) bite for far more merchants, not just the one or two outliers you'd handle by hand at today's scale. At that point the honest answer shifts from "shard harder" to "stop treating every merchant's ledger the same way" — give the small number of genuinely hot accounts their own dedicated, pre-sharded write path (sequenced, batched commits) while the long tail of ordinary merchants stays on the shared sharded design.

The pairs_with move: say the LLD→HLD escalation out loud if asked cold — "single-endpoint idempotent charge API (built it first) is correct but only solves one race at the edge; here's everything that has to exist around it once the charge has to survive a worker crash, a slow card network, and a bank that keeps its own independent record" — that's exactly the arc Movements 1 through 5 walked.

8. You can now defend


Re-authored/Deepened for this guide. See also: Design an Idempotent Payment API (the LLD kata this lab's edge-idempotency and outbox-key mechanisms build on), Design a Double-Entry Ledger (the money-movement store used as a component here), and Designing Payment System (the model-answer RESHADED walkthrough this worksheet checks against).

🤖 Don't fully get this? Learn it with Claude

Stuck on Design a Payment Gateway End-to-End? Open Claude, copy a block below, and it'll teach you this exact concept — visually and interactively.

🎨 Explain it visually

Build the mental picture, not memorization.

I just read a lesson on **Design a Payment Gateway End-to-End** (Hands-On Builds) and want to truly understand it. Explain Design a Payment Gateway End-to-End from first principles using ONE vivid real-world analogy and a visual mental model — draw it as ASCII art or a clear step-by-step diagram — with a concrete example using real numbers. Then ask me one question to check I got the mental picture, and wait for my reply. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.
🤔 Walk me through it (interactive)

Socratic — adapts to where you're stuck.

Teach me **Design a Payment Gateway End-to-End** interactively. Ask me ONE guiding question at a time, wait for my answer, and adapt to my confusion — build the idea with me step by step instead of explaining it all at once. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.
🧪 Quiz me & fix my gaps

Active recall exposes what you missed.

Quiz me on **Design a Payment Gateway End-to-End** with 5 questions, easy to tricky, ONE at a time. Tell me if each answer is right; at the end, explain clearly what I got wrong and why. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.
🧠 Make it stick

Intuition + hook + flashcards for long-term memory.

Help me remember **Design a Payment Gateway End-to-End** for the long term: give the one-sentence intuition, a memorable hook/mnemonic, a tiny worked example, and 3 active-recall flashcards (Q -> A). If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

📝 My notes