Knowledge Guide
Home › Hands-On Builds › Fintech Labs

Design an Idempotent Payment API

Design an Idempotent Payment/Payout API

Every payment API interview and every real Stripe/Razorpay outage post-mortem eventually reaches the same sentence: the network timed out, the client retried, and the customer got charged twice. You already know what an idempotency key is — a client-supplied token on the request. This lab builds the part that's actually hard: the server-side store that makes "charge exactly once, no matter how many times the request arrives" true under concurrency, not just on paper. You'll ship a version that looks correct, watch it double-charge under a real race, then fix it with one atomic operation — in both Java and Go — and be able to defend every follow-up a payments-team interviewer throws at you.

1. The trap

You implement idempotency keys exactly the way the docs describe: client sends Idempotency-Key: abc123 with a charge request; your server looks the key up in a store; if it's not there, you charge and save the result; if it is there, you return the saved result instead of charging again. You test it by hand — send the same key twice, sequentially — and it works: one charge, one stored result, the second call returns the cached response. Ship it.

Then production happens. A merchant's checkout page is slow under load; their client has a 2-second timeout and retries on timeout — but the original request didn't fail, it was just slow, so now two requests carrying the same Idempotency-Key are in flight at the same time, possibly landing on two different instances behind your load balancer. Run that scenario 50 times concurrently against the naive implementation above and here is what actually happens (real output, reproduced below in Movement 5, not illustrative):

50 concurrent callers, same Idempotency-Key:
  provider.chargeCount()  = 50   <- every single one of them charged the customer
  distinct charge ids     = 50   <- 50 separate, real charge IDs from the payment processor
  rejected callers        = 0

Fifty charges for one logical payment. Nothing crashed, nothing threw an exception, no log line screamed — the code you wrote is doing exactly what you told it to do: check, then act, then remember. The bug isn't in any single line; it's in the gap between "check" and "remember", and that gap is wide enough to drive fifty concurrent requests through. This lab is about seeing that gap and closing it with one atomic operation instead of two separate ones.

2. Scope it like a senior

Before writing a store, pin down what "idempotent" has to mean here — a candidate who jumps straight to "I'll cache the response" is solving a weaker problem than the one that actually ships:

Answering these first is what separates "I added a cache" from "I know exactly what has to be atomic, why, and what I do when someone tries to break it."

3. Reason to the design

Attempt 0 — no idempotency key at all. Retry the charge call → charge again. Obviously wrong, but worth stating precisely: the reason a plain POST /charge isn't idempotent is that "create a new charge" has no natural identity — unlike PUT/DELETE, which are idempotent by construction because they're keyed by the resource's own ID. An Idempotency-Key header is exactly a client-supplied identity for an otherwise identity-less write. That's the whole mechanism in one sentence.

Attempt 1 — check-then-act (the trap). GET the key from a shared store. Miss → call the payment provider → PUT the result back keyed by the idempotency key. This is the version almost everyone ships first, and it is not wrong on paper — it's wrong under concurrency, because "check" and "act-then-remember" are two separate operations with an unguarded window between them, during which the provider call itself is slow (a real network round-trip to a card network, tens to hundreds of milliseconds). Any second caller with the same key that arrives inside that window also sees a miss, because nothing has been written yet. The store being "thread-safe" (a ConcurrentHashMap, a database with row-level locking on individual statements) does not help — each individual read and each individual write is safe, but the sequence read → (slow external call) → write as a whole is not one atomic unit. This is precisely the classic TOCTOU (time-of-check-to-time-of-use) bug, wearing a payments costume.

Attempt 2 — claim the key atomically, BEFORE charging. The fix is to collapse "is this key new?" and "mark it as mine" into a single atomic operation, so there is no window for a second caller to slip through:

Whoever's claim succeeds is now the only caller that will ever call the payment provider for this key — everyone else, by construction, sees "already claimed" and takes a different path (Movement 4). The provider call itself still happens outside the lock/transaction (you never want to hold a database row lock for the full duration of a slow external network call) — but that's now safe, because no one else can be racing to make the same call; they're all waiting on or replaying the winner's eventual result.

The remaining two design fixes, both closing a specific hole:

4. Build it — milestones

Attempt each milestone yourself — the reference implementation is positioned as the reveal, after you've tried.

Reference implementation — Java (JDK 8, compiles clean with javac, zero warnings)

The fake processor (PaymentProvider.java) — the sleep is what widens the race window enough to make M2's bug reproduce reliably, not just occasionally:

import java.util.concurrent.atomic.AtomicInteger;

/** A fake downstream payment processor (stands in for a card network / bank
 *  rail call). chargeCount lets tests observe how many times money actually
 *  MOVED -- that's the number this whole kata is about, not just how many
 *  HTTP calls returned 200. The sleep simulates real network latency; it is
 *  exactly what widens the race window in NaiveIdempotencyStore below. */
public final class PaymentProvider {
    private final AtomicInteger chargeCount = new AtomicInteger(0);

    public String charge(long amountCents, String currency) {
        try {
            Thread.sleep(20); // simulate a real network round-trip to the card network
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
        }
        int n = chargeCount.incrementAndGet();
        return "ch_" + n; // fake charge id
    }

    public int chargeCount() { return chargeCount.get(); }
}

The stored record (IdempotencyRecord.java):

/** What gets stored PER idempotency key: which request it belongs to
 *  (requestHash -- this is what catches same-key-different-body), which
 *  stage it's in, and, once COMPLETED, the exact response to replay on
 *  every future retry. */
final class IdempotencyRecord {
    enum Status { IN_PROGRESS, COMPLETED }

    volatile Status status;
    final int requestHash;
    volatile String chargeId; // set only once status == COMPLETED

    IdempotencyRecord(int requestHash) {
        this.status = Status.IN_PROGRESS;
        this.requestHash = requestHash;
    }
}

M1/M2 — the naive store (the trap, on purpose) (NaiveIdempotencyStore.java):

import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;

/** THE BUG this kata is about: check-then-act. get() is a read; the actual
 *  charge happens OUTSIDE any exclusion; put() is a second, later write.
 *  Two concurrent callers with the SAME key can both pass the "have I seen
 *  this key?" check before either one has written a result -- both then
 *  call the provider. Using a ConcurrentHashMap here does NOT fix this: the
 *  map's internal locking only makes each individual get()/put() call safe,
 *  it does nothing to make "check, then (slow) act, then write" atomic as a
 *  whole. That gap is the whole vulnerability. */
public final class NaiveIdempotencyStore {
    private final Map<String, IdempotencyRecord> store = new ConcurrentHashMap<String, IdempotencyRecord>();
    private final PaymentProvider provider;

    public NaiveIdempotencyStore(PaymentProvider provider) {
        this.provider = provider;
    }

    public String charge(String key, long amountCents, String currency, int requestHash) {
        IdempotencyRecord existing = store.get(key);                    // 1. CHECK
        if (existing != null && existing.status == IdempotencyRecord.Status.COMPLETED) {
            return existing.chargeId;                                    // (rarely the one that wins the race)
        }
        String chargeId = provider.charge(amountCents, currency);        // 2. ACT -- money moves here, unguarded
        IdempotencyRecord rec = new IdempotencyRecord(requestHash);
        rec.status = IdempotencyRecord.Status.COMPLETED;
        rec.chargeId = chargeId;
        store.put(key, rec);                                             // 3. WRITE -- too late, others already passed step 1
        return chargeId;
    }
}

M3/M4 — the fix (AtomicClaimStore.java):

import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;

/** THE FIX: claim the key ATOMICALLY, before charging -- the moral
 *  equivalent of a database `INSERT ... ON CONFLICT DO NOTHING` on a
 *  UNIQUE(idempotency_key) column, or a Redis `SETNX idem:{key} IN_PROGRESS`.
 *  ConcurrentHashMap.putIfAbsent is a single atomic compare-and-set: exactly
 *  one caller can ever be the one whose insert "wins" a given key, no matter
 *  how many threads race in at once. Whoever wins is the ONLY caller that
 *  will ever call the provider for this key; everyone else either gets 409'd
 *  or replays the winner's stored result. */
public final class AtomicClaimStore {

    public static final class DuplicatePayloadException extends RuntimeException {
        private static final long serialVersionUID = 1L;
        public DuplicatePayloadException(String key) {
            super("Idempotency-Key '" + key + "' was already used with a different request payload");
        }
    }

    public static final class InProgressException extends RuntimeException {
        private static final long serialVersionUID = 1L;
        public InProgressException(String key) {
            super("a request for Idempotency-Key '" + key + "' is already in flight");
        }
    }

    private final Map<String, IdempotencyRecord> store = new ConcurrentHashMap<String, IdempotencyRecord>();
    private final PaymentProvider provider;
    private final boolean waitForInFlight; // true: block+poll for the in-flight duplicate; false: throw 409 immediately

    public AtomicClaimStore(PaymentProvider provider, boolean waitForInFlight) {
        this.provider = provider;
        this.waitForInFlight = waitForInFlight;
    }

    public String charge(String key, long amountCents, String currency, int requestHash) {
        IdempotencyRecord claim = new IdempotencyRecord(requestHash);
        IdempotencyRecord winner = store.putIfAbsent(key, claim); // THE ATOMIC CLAIM -- one CAS, full stop

        if (winner == null) {
            // We won: we are the ONLY caller that will ever charge for this key.
            String chargeId = provider.charge(amountCents, currency);   // slow call happens AFTER the claim, not before
            synchronized (claim) {
                claim.chargeId = chargeId;
                claim.status = IdempotencyRecord.Status.COMPLETED;
                claim.notifyAll();
            }
            return chargeId;
        }

        // We lost: someone else already owns this key.
        if (winner.requestHash != requestHash) {
            throw new DuplicatePayloadException(key); // same key, different body -> reject, never charge
        }
        synchronized (winner) {
            while (winner.status == IdempotencyRecord.Status.IN_PROGRESS) {
                if (!waitForInFlight) {
                    throw new InProgressException(key); // 409 -- don't block this caller's thread
                }
                try {
                    winner.wait(2000);
                } catch (InterruptedException e) {
                    Thread.currentThread().interrupt();
                }
            }
        }
        return winner.chargeId; // idempotent replay: identical response, provider is NEVER called again for this key
    }
}

All four classes above compile clean under javac (JDK 8), zero warnings. (A demo harness — not shown, same shape as the terminal output in Movement 5 — drives the break-it scenario below.)

Reference implementation — Go (1.21+, go build ./... and go vet ./... clean)

The store is its own library package (idempotency) — a demo/test binary imports it, exactly like the DB or Redis client it stands in for would be imported in real code.

The fake processor (idempotency/provider.go):

// Package idempotency builds an idempotent payment charge API in stages:
// a naive check-then-act store (the trap) -> an atomic-claim store (the fix).
package idempotency

import (
	"fmt"
	"sync/atomic"
	"time"
)

// PaymentProvider is a fake downstream payment processor (stands in for a
// card network / bank rail call). ChargeCount lets the break-it test
// observe how many times money actually MOVED, not just how many calls
// returned success. The sleep simulates real network latency -- it's what
// widens the race window in NaiveStore below.
type PaymentProvider struct {
	count int64
}

func (p *PaymentProvider) Charge(amountCents int64, currency string) string {
	time.Sleep(20 * time.Millisecond) // simulate a real network round-trip to the card network
	n := atomic.AddInt64(&p.count, 1)
	return fmt.Sprintf("ch_%d", n)
}

func (p *PaymentProvider) ChargeCount() int64 {
	return atomic.LoadInt64(&p.count)
}

M1/M2 — the naive store (idempotency/naive_store.go):

package idempotency

import "sync"

// Record is what gets stored PER idempotency key: which request it belongs
// to (RequestHash -- catches same-key-different-body), which stage it's in,
// and, once Completed, the exact response to replay on every future retry.
type Record struct {
	Completed   bool
	RequestHash uint64
	ChargeID    string
}

// NaiveStore is THE BUG this kata is about: check-then-act. The map itself
// is protected by a mutex, so individual reads/writes are memory-safe -- but
// the DECISION ("have I seen this key?") and the ACT (calling the slow
// provider) happen across two SEPARATE lock acquisitions, with nothing
// synchronized in between. Two concurrent callers with the same key can
// both see "not present" before either one has written a result.
type NaiveStore struct {
	mu       sync.Mutex
	records  map[string]*Record
	provider *PaymentProvider
}

func NewNaiveStore(p *PaymentProvider) *NaiveStore {
	return &NaiveStore{records: make(map[string]*Record), provider: p}
}

func (s *NaiveStore) Charge(key string, amountCents int64, currency string, requestHash uint64) string {
	s.mu.Lock()
	existing, ok := s.records[key] // 1. CHECK
	s.mu.Unlock()
	if ok && existing.Completed {
		return existing.ChargeID // (rarely the one that wins the race)
	}

	chargeID := s.provider.Charge(amountCents, currency) // 2. ACT -- money moves here, unguarded

	s.mu.Lock()
	s.records[key] = &Record{Completed: true, RequestHash: requestHash, ChargeID: chargeID} // 3. WRITE -- too late
	s.mu.Unlock()
	return chargeID
}

M3/M4 — the fix (idempotency/atomic_store.go):

package idempotency

import (
	"errors"
	"sync"
)

// ErrDuplicatePayload: same Idempotency-Key, different request body -> reject.
var ErrDuplicatePayload = errors.New("idempotency key reused with a different request payload")

// ErrInProgress: a duplicate arrived while the original is still in flight,
// and this Store is configured to fail fast (409) instead of blocking.
var ErrInProgress = errors.New("request for this idempotency key is already in flight")

// Store is THE FIX: claim the key ATOMICALLY, before charging -- the moral
// equivalent of a database `INSERT ... ON CONFLICT DO NOTHING` on a
// UNIQUE(idempotency_key) column, or a Redis `SETNX idem:{key} IN_PROGRESS`.
// The claim ("is this key present? if not, insert it") happens while
// holding ONE mutex for the ENTIRE decision -- so exactly one goroutine can
// ever win a given key, no matter how many race in at the same instant.
type Store struct {
	mu       sync.Mutex
	cond     *sync.Cond
	records  map[string]*Record
	provider *PaymentProvider
	wait     bool // true: block the duplicate caller until the winner finishes; false: return ErrInProgress (409) immediately
}

func NewStore(p *PaymentProvider, wait bool) *Store {
	s := &Store{records: make(map[string]*Record), provider: p, wait: wait}
	s.cond = sync.NewCond(&s.mu)
	return s
}

func (s *Store) Charge(key string, amountCents int64, currency string, requestHash uint64) (string, error) {
	s.mu.Lock()
	existing, ok := s.records[key]
	if !ok {
		// THE ATOMIC CLAIM: still holding s.mu, so no other goroutine can
		// observe "key absent" until AFTER we've inserted our own record.
		rec := &Record{Completed: false, RequestHash: requestHash}
		s.records[key] = rec
		s.mu.Unlock()

		chargeID := s.provider.Charge(amountCents, currency) // slow call happens WITHOUT the lock held

		s.mu.Lock()
		rec.Completed = true
		rec.ChargeID = chargeID
		s.cond.Broadcast() // wake up anyone parked in the wait loop below
		s.mu.Unlock()
		return chargeID, nil
	}

	// Lost the claim: someone else already owns this key.
	if existing.RequestHash != requestHash {
		s.mu.Unlock()
		return "", ErrDuplicatePayload // same key, different body -> reject, never charge
	}
	for !existing.Completed {
		if !s.wait {
			s.mu.Unlock()
			return "", ErrInProgress // 409 -- don't block this caller
		}
		s.cond.Wait() // atomically unlocks s.mu while parked, re-locks on wakeup
	}
	chargeID := existing.ChargeID
	s.mu.Unlock()
	return chargeID, nil // idempotent replay: identical response, provider NEVER called again for this key
}

The full module (idempotency/ package above + cmd/demo/main.go, which imports it and runs the concurrent break-it scenario + a -race-checked test file in the idempotency package) passes go build ./... and go vet ./... clean.

5. Break it — the test that fails (and the one that doesn't)

Break-it test: 50 concurrent callers, same Idempotency-Key, same body. This is the exact scenario from Movement 1 — a slow original request plus a client-side retry, or two app-server instances handling the same duplicate delivery. Fire 50 threads/goroutines at the naive store, then at the atomic-claim store; the only thing that changes between the two runs is which store they call. Real output, both languages, both reproduced independently:

-- Java (IdempotencyDemo) --
=== Break-it test 1: NaiveIdempotencyStore (check-then-act) ===
50 concurrent callers, same Idempotency-Key:
  provider.chargeCount()  = 50   <- how many times money actually moved
  distinct charge ids     = 50   [ch_1 .. ch_50, all distinct]
  rejected callers        = 0

=== Fix: AtomicClaimStore (atomic claim BEFORE charging) ===
50 concurrent callers, same Idempotency-Key:
  provider.chargeCount()  = 1    <- exactly once, every run
  distinct charge ids     = 1    [ch_1]
  rejected callers        = 0    (all 49 duplicates blocked, then replayed ch_1)

-- Go (same shapes, cmd/demo importing the idempotency package) --
NaiveStore:  provider.ChargeCount() = 50   distinct charge ids = 50
Store (fix): provider.ChargeCount() = 1    distinct charge ids = 1

Fifty threads, one logical payment, fifty real charges from the naive store — every single run, not a rare flake, because the 20ms sleep in PaymentProvider.charge widens the check-then-act gap wide enough that all 50 callers reliably pass the "have I seen this key?" check before any of them has written a result. Switch only the store to the atomic-claim version and the count is exactly 1, every run, because the claim (putIfAbsent / the mutex-guarded map insert) makes "check and reserve" one indivisible step — there is no window left for a second caller to slip through.

A subtlety worth naming precisely, because an interviewer will probe it: this is a business-logic race (a TOCTOU bug), not a data race in the formal sense. Go's race detector (go test -race) does not flag the naive store — every individual map read and write is properly mutex-guarded, so there's no unsynchronized memory access for the detector to catch. The bug is entirely in the sequencing of otherwise-safe operations. The only way to catch it is to assert on the domain invariant — chargeCount — exactly like this test does:

$ go test -race -v ./...
=== RUN   TestNaiveStore_DoubleCharges
    idempotency_test.go:31: BUG REPRODUCED: naive store charged the provider
        50 times for ONE logical payment (want 1)
--- PASS: TestNaiveStore_DoubleCharges (0.02s)
=== RUN   TestStore_ExactlyOnceUnderConcurrency
--- PASS: TestStore_ExactlyOnceUnderConcurrency (0.02s)
=== RUN   TestStore_RejectsMismatchedPayload
--- PASS: TestStore_RejectsMismatchedPayload (0.02s)
PASS
ok  	idempotencylab/idempotency	1.588s

("PASS" on TestNaiveStore_DoubleCharges here means the test's assertion — "the naive store's bug reproduces" — held, not that the store behaved correctly; read the log line, not just the pass/fail color.)

Guard test: same key, different body. Call charge("k", $10.00, ...), then charge("k", $20.00, ...) with the same key. Both languages: the second call throws/returns DuplicatePayloadException / ErrDuplicatePayload, and provider.chargeCount() stays at 1 — the mismatched retry never reaches the payment provider at all.

6. Optimise — with trade-offs

ApproachSafe under concurrency / multi-instance?Durable across a crash/restart?LatencyUse when
Per-process cache (a local HashMap, no shared store)No — a retry landing on a different instance sees nothing; this is the failure mode Movement 2 calls out explicitlyNo — lost on restart/deployFastest (no I/O)Never, for anything beyond a single-box prototype. A cache alone can never be the idempotency store for a fleet.
DB check-then-act (SELECT then INSERT, this lab's M1/M2 trap)No — the exact race this lab reproduces; a shared durable store does NOT fix a non-atomic check-and-write sequenceYesTwo round tripsNever — this is the pattern to recognize and reject, not adopt.
DB unique constraint / INSERT ... ON CONFLICT (M3/M4's mechanism, DB-backed)Yes — atomic across every instance by construction, since they all hit the same table/constraintYes — survives restarts, the DB is your source of truthOne round trip for the claim; the DB is usually already in your critical path for the transaction record anywayThe default choice: you already have a relational store for the ledger/transaction, and correctness matters more than shaving milliseconds.
Redis SET ... NX EXYes — same atomic claim, sub-millisecondDepends — plain Redis can lose keys on failover unless persistence (AOF) is configured; treat as "durable enough for a bounded TTL window," not a ledgerSub-millisecond, much faster than a DB round tripHigh-QPS charge paths where the DB round trip itself is the bottleneck, and you can accept "very likely, not absolutely guaranteed" durability for the dedup window specifically (the actual money-movement record still lives in the durable ledger).
Block the duplicate (wait()/sync.Cond until the winner finishes, this lab's default)Correct either wayn/aDuplicate caller's latency = the winner's full charge latencySynchronous APIs where the caller is already blocking on a response anyway — returning the real result beats a client-side retry loop.
409 the duplicate immediatelyCorrect either wayn/aDuplicate caller returns instantly, empty-handedHigh-fanout or webhook-heavy systems where you don't want N requests parked holding a thread/connection each — push the wait onto the caller's own retry/backoff policy instead.
TTL length: 1 hour vs 24 hours (Stripe's default)n/an/an/aShort TTL risks a legitimately slow client retrying after expiry and double-charging via a "new" key that's really a duplicate — pick a TTL that comfortably exceeds your p99.9 client retry window, not just your typical one.

The judgment call to say out loud: a coarse per-process cache is never sufficient the moment you have more than one instance — that's not a performance trade-off, it's a correctness gap. Between DB-unique-constraint and Redis-SETNX, the real trade is durability vs. latency: reach for Redis only once you've measured that the DB round trip on the claim path is your bottleneck, and you're comfortable that "dedup window" durability (bounded by the TTL, backed by AOF/replication) is weaker than your ledger's own durability guarantee — which is fine, because the ledger, not the idempotency key, is your legal record of the money movement.

7. Defend under drilling

8. You can now defend

Re-authored/Deepened for this guide. Reference code compiled and executed before publishing: Java (javac, JDK 8) clean with zero warnings; Go (go build ./..., go vet ./..., go test -race ./...) clean, including a dedicated race-detector run confirming the naive store's bug is a business-logic race rather than a data race. The 50-concurrent-caller break-it numbers (50 charges naive vs. 1 charge fixed) were reproduced independently in both languages, not asserted from memory. See also: What Are Idempotency Keys and How to Implement Them Safely for Payments, Exactly-Once Is a Myth — Idempotency & Dedup, and Designing Payment System for the full HLD this key mechanism sits inside.

🤖 Don't fully get this? Learn it with Claude

Stuck on Design an Idempotent Payment API? Open Claude, copy a block below, and it'll teach you this exact concept — visually and interactively.

🎨 Explain it visually

Build the mental picture, not memorization.

I just read a lesson on **Design an Idempotent Payment API** (Hands-On Builds) and want to truly understand it. Explain Design an Idempotent Payment API from first principles using ONE vivid real-world analogy and a visual mental model — draw it as ASCII art or a clear step-by-step diagram — with a concrete example using real numbers. Then ask me one question to check I got the mental picture, and wait for my reply. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.
🤔 Walk me through it (interactive)

Socratic — adapts to where you're stuck.

Teach me **Design an Idempotent Payment API** interactively. Ask me ONE guiding question at a time, wait for my answer, and adapt to my confusion — build the idea with me step by step instead of explaining it all at once. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.
🧪 Quiz me & fix my gaps

Active recall exposes what you missed.

Quiz me on **Design an Idempotent Payment API** with 5 questions, easy to tricky, ONE at a time. Tell me if each answer is right; at the end, explain clearly what I got wrong and why. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.
🧠 Make it stick

Intuition + hook + flashcards for long-term memory.

Help me remember **Design an Idempotent Payment API** for the long term: give the one-sentence intuition, a memorable hook/mnemonic, a tiny worked example, and 3 active-recall flashcards (Q -> A). If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

📝 My notes

Design a Double-Entry Ledger →