Idempotency & “Exactly-Once Is a Myth”

"Exactly-once delivery" does not exist — so stop chasing it

Over an unreliable network (fallacy #1), a sender that gets no ack can't know if the message was lost or the ack was lost. So you get one of two guarantees: at-most-once (don't retry — may lose) or at-least-once (retry — may duplicate). There is no exactly-once delivery. The practical answer the whole industry uses:

at-least-once delivery + idempotent processing = effectively-once. Stop trying to deliver exactly once; make duplicates harmless.

What makes an operation idempotent

Doing it twice has the same effect as once. PUT user.email = x, DELETE id=7, SET balance = 100 are naturally idempotent. balance += 50, "append", "send email", "charge card" are not — a retry doubles them. Two ways to fix the non-idempotent ones:

• Idempotency key: the client generates a unique key per logical operation and sends it on every retry. The server records "key → result"; if it sees the key again, it returns the stored result without re-doing the work (the diagram). Stripe's API works exactly this way.
• Natural dedup: a unique constraint (e.g. a transfer id) so the second insert fails harmlessly.

Cost and contract of idempotency keys

Idempotency keys are not a free default. They need a durable lookup store — typically the same database as the business transaction — and a TTL at least as long as the client's retry window (Stripe uses 24 hours). The key must be unique per logical operation and stable across retries; if the client changes the key on every retry, the server sees each attempt as a new operation and duplicates it.

When a natural unique constraint exists, prefer it. A bank transfer with a client-generated transfer_id can rely on a UNIQUE(transfer_id) index instead of a separate idempotency-key table: cheaper, simpler, and no TTL to manage. Use an explicit idempotency-key store only when the operation has no natural unique handle (e.g., "charge $50 to this card"). If the client cannot generate a stable key, idempotency keys are not a fix at all.

At scale the key store is a hot path, not a footnote: at 5k charges/s with a 24 h TTL and ~200 B/key, a fully-retained working set is 5,000 × 86,400 × 200 B ≈ 86 GB — so shard it, expire keys at the retry-window boundary, and prefer a natural UNIQUE constraint (no separate table, no TTL) whenever the domain gives you one.

// server, processing a charge with an idempotency key — atomically
if (store.putIfAbsent(key, PENDING) != null)   // someone already has this key
    return store.awaitResult(key);             // return the first attempt's result
Result r = chargeCard(amount);                 // do the real work once
store.put(key, r);
return r;

The crash while PENDING — the in-doubt case

The code above survives the concurrent-retry race (that's what the atomic putIfAbsent buys), but not a crash. Interleaving: attempt 1 reserves PENDING; chargeCard() succeeds at the processor; the process dies before store.put(key, r). The key is now permanently PENDING — every retry hits awaitResult(key) and, with no surviving writer to ever complete it, blocks forever. Worse: blindly re-executing after some timeout would double-charge, because the card was charged — you just never recorded it. The charge is in-doubt: you cannot know locally which side of the crash you are on.

• Lease the reservation: store PENDING with leased_until = now + 2× the charge's own timeout (the same lifecycle idea as the outbox relay), instead of an unbounded claim that a dead process holds forever.
• Expired lease = IN-DOUBT, not retryable-by-default: for an external side effect, the recovery path must reconcile with the downstream — query the processor by the same idempotency key ("did a charge with key abc123 succeed?"), record a found result as the answer, and re-execute only on a confirmed "no such charge."
• The general principle: an idempotency layer over an external effect is only as safe as your ability to interrogate that effect's outcome — which is exactly why Stripe both accepts Idempotency-Key headers and exposes retrieval by that same key.

Contrast: for a purely local effect committed under the same DB transaction (the natural-UNIQUE-constraint variant above), the whole problem vanishes — the effect and its record commit atomically, so no in-doubt window can exist. That atomicity, not the saved TTL bookkeeping, is the real reason to prefer the natural constraint.

Where this shows up everywhere

Payment APIs, message-queue consumers (Kafka redelivers on failure → consumers must be idempotent), webhook receivers, and every retry from the Designing-for-Failure lesson. "Kafka exactly-once" is really idempotent producers + transactional offsets — effectively-once, not magic.

Pitfalls

• The dedup check-then-act is itself a race — use an atomic putIfAbsent / unique constraint (the counter++ lesson again).
• Key scope & TTL: keys must be unique per logical op and retained long enough to catch retries.
• Same key, different body: store a hash of the request alongside the key; if a retry's body hash differs, reject with a 422 rather than returning the stored result — returning it would hand the caller an answer to a question they didn't ask (Stripe rejects exactly this way, as an idempotency_error).
• Retrying a non-idempotent op without a key = double charge — the classic production incident.

Takeaways

• Exactly-once delivery is impossible; aim for at-least-once + idempotency = effectively-once.
• Make non-idempotent ops safe with an idempotency key (atomic dedup) or a natural unique constraint.
• Mandatory wherever you retry: payments, queue consumers, webhooks.

---

Re-authored for this guide; idempotency-key diagram hand-authored as SVG. Follows the Stripe idempotency docs and Kafka delivery-semantics. See also: The 8 Fallacies (#1), Designing for Failure (retries), Kafka, (Concurrency) Race Conditions, Ticketmaster.

Production judgment

Idempotency is a business identity for an intent (“this checkout”), not a transport feature. If the client generates a new key on every retry, you built a double-charge machine with extra headers.

Production smell: logs show 200 OK twice for one checkout with two different keys generated by a flaky mobile client. Metric to watch: unique business intents vs unique charges in the ledger for the same window.

Decision language: “We are at-least-once on the wire; we are exactly-once on money via ledger keys.” That sentence is the staff answer.

I just read a lesson on **Idempotency & “Exactly-Once Is a Myth”** (System Design) and want to truly understand it. Explain Idempotency & “Exactly-Once Is a Myth” from first principles using ONE vivid real-world analogy and a visual mental model — draw it as ASCII art or a clear step-by-step diagram — with a concrete example using real numbers. Then ask me one question to check I got the mental picture, and wait for my reply. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Teach me **Idempotency & “Exactly-Once Is a Myth”** interactively. Ask me ONE guiding question at a time, wait for my answer, and adapt to my confusion — build the idea with me step by step instead of explaining it all at once. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Quiz me on **Idempotency & “Exactly-Once Is a Myth”** with 5 questions, easy to tricky, ONE at a time. Tell me if each answer is right; at the end, explain clearly what I got wrong and why. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Help me remember **Idempotency & “Exactly-Once Is a Myth”** for the long term: give the one-sentence intuition, a memorable hook/mnemonic, a tiny worked example, and 3 active-recall flashcards (Q -> A). If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.