hard Exactly-Once is a Myth — Idempotency & Dedup

Why exactly-once delivery cannot exist

Consider any sender that must do two things: apply an effect (charge a card, enqueue a job) and durably record that it did. These are two separate steps, so a crash can land between them. When the sender restarts it finds no “done” record and cannot tell which world it is in:

• the effect already happened (crash was after applying, before recording) → if it retries, that is a duplicate;
• the effect never happened (crash was before applying) → if it does not retry, that is a loss.

Add an unreliable network and the receiver has the mirror problem: a lost ack is indistinguishable from a lost message. There is no observation either side can make to collapse the ambiguity. Hence you can pick at-most-once (never retry — may lose) or at-least-once (retry — may duplicate), but exactly-once delivery is impossible. What you can build:

at-least-once delivery + idempotent effect = effectively-once. You stop trying to deliver exactly once and instead make a duplicate harmless.

The trap inside the fix: the dedup race

“Effectively-once” hinges on a dedup check: have I already processed this idempotency key? The naive implementation is check-then-act — and that is itself a race. Two duplicate deliveries of the same key K land on two workers at the same time:

Both reads ran before either write, so both passed the “never seen it” check. This is the same read-modify-write hazard as counter++. The dedup must be atomic, not read-then-write.

Make the dedup atomic

Let the database serialize the contenders on the key itself:

-- one atomic step: the DB rejects the second inserter
INSERT INTO processed(idempotency_key) VALUES (:k)
  ON CONFLICT (idempotency_key) DO NOTHING;   -- key has a UNIQUE constraint
-- if a row was inserted, this worker won the key: do the effect (same txn).
-- if 0 rows inserted, a duplicate already owns it: skip / return stored result.

Equivalents: a UNIQUE constraint on the key (the second INSERT fails with a unique violation), or an atomic conditional write (DynamoDB PutItem with attribute_not_exists, Redis SET key val NX, or a compare-and-set). The rule: exactly one writer can win the key, decided by the storage layer — never by a prior SELECT. Commit the key row and the effect in the same transaction so they cannot diverge.

Pitfalls

• Recording the key in a separate step from the effect re-opens the crash window — keep them in one transaction, or make the effect naturally idempotent (e.g. a transfer keyed by a unique transfer id).
• Non-transactional key store (Redis) + transactional effect (DB) can drift: you marked the key done but the DB write rolled back, or vice versa. Prefer the effect’s own DB for the key, or a careful two-phase approach.
• Storing the key but not the response: a retry after success should return the original result, so persist the response alongside the key.

The judgment layer

At-least-once + idempotent vs at-most-once. At-least-once (retry until acked, dedup on the consumer) is the default for anything that must not be lost — payments, orders, state changes — and the cost is that every consumer must be idempotent and pay for a dedup store. At-most-once (fire-and-forget, no retry) is legitimate when a lost message is cheaper than a duplicate and volume is huge — metrics samples, best-effort telemetry, cache warmers — where deduping every event would cost more than the occasional gap is worth.

Idempotency-key store + TTL trade-off. Keys must be retained at least as long as the maximum retry window (client retries + queue redelivery + replays). Too short a TTL and a late retry arrives after the key expired → the duplicate slips through and re-applies the effect. Too long and the store grows without bound. Size the TTL to the real worst-case redelivery horizon (often hours to a day), not the happy-path latency. Store choice trades durability for speed: a DB unique index is durable and co-transactional with the effect but adds write load; Redis SETNX + TTL is fast and self-expiring but is not in the same transaction as the effect, so you must reason about the ordering carefully.

Takeaways

• Exactly-once delivery is impossible (crash between effect and record + lossy acks); aim for at-least-once + idempotency = effectively-once.
• Dedup by SELECT-then-act is a race — make it atomic with a UNIQUE key / ON CONFLICT / conditional write, and commit the key with the effect.
• Persist the original response so a post-success retry returns it instead of re-doing work.
• Set the key TTL to the maximum redelivery window, and pick the key store (DB vs Redis) by whether it must be transactional with the effect.

Synthesized from the Two Generals / FLP intuition, Kleppmann, Designing Data-Intensive Applications (Ch. 11, “exactly-once”), the Stripe idempotency-key API design, and Kafka’s idempotent-producer / transactional-offset semantics. Re-authored/Deepened for this guide.