Event Delivery Semantics & Dead Letter Queues

What “delivered” guarantees — and poison that cannot be processed

Networks drop packets; consumers crash mid-handler; brokers redeliver. Async messaging must define both the delivery contract and what happens when a message never succeeds. This page is the operations-minded core: ack timing, the three semantics, a charge-card redelivery trace, and DLQ discipline.

The three semantics

Semantic	Guarantee	Typical use	Cost
At-most-once	May lose; never duplicate	Metrics, best-effort logs	Loss under crash
At-least-once	No silent loss; may duplicate	Default for business events	Idempotent consumers required
Exactly-once (effectively)	Effect once	Ledger-ish paths inside a txn boundary	Usually = at-least-once + dedupe; broker EOS does not cover arbitrary side effects (SMS, HTTP)

Ack-before vs ack-after processing

When you ack	Crash window	Resulting semantic
Ack before processing	Crash after ack → message gone	At-most-once (loss possible)
Ack after successful processing	Crash after process before ack → redelivery	At-least-once (duplicate possible)

Two Generals framing: the broker and consumer can never be simultaneously certain of delivery over an unreliable network without an unbounded protocol. Practical systems pick loss or duplication and design for it — almost always duplication + idempotency for money-adjacent work.

Worked trace: charge-card redelivery

Queue message: ChargeCard{paymentId=P1, amount=40.00, idempotencyKey=K1}.

1. Consumer receives message, charges card successfully, then crashes before ack.
2. Broker redelivers after visibility timeout / uncommitted offset.
3. Without idempotency: second charge → customer billed twice.
4. With idempotency store keyed by K1: second attempt returns original success; one capture.

Same pattern for Kafka: process then commit offset; crash between process and commit → reprocess after rebalance.

Ordering

Global total order is expensive and rarely needed. Prefer per-entity order (Kafka partition key = orderId / rideId). Unrelated entities can proceed in parallel. Single-queue FIFO with one consumer is a scalability ceiling.

Retries, DLQs, and ordering — the hidden trade

Sidelining a message breaks the very per-key ordering you partitioned for: a replayed DLQ message for orderId=O arrives after every later event for O. Concretely: events e1, e2, e3 for O sit on one partition; e1 poisons and is parked to the DLQ after N attempts; e2 and e3 process normally; replaying e1 later means the consumer sees e2, e3, e1 — any consumer applying deltas or last-writer-wins without a version/sequence guard regresses state. Two survival strategies:

• Version-guarded full-state apply: make events carry full state plus a version, and guard the apply so a stale replayed e1 is a no-op.
• Key-level quarantine for delta events: if events are deltas, block the key — park all subsequent events for that key to the DLQ too — so replay restores order at the cost of latency for that key.

Retry-topic tiers (retry-5m, retry-30m) have the same property: they trade ordering for main-topic liveness.

Kafka vs classic queue (ops differences)

• Queue (SQS/Rabbit work queue): competing consumers; message deleted/acked and gone; DLQ via maxReceiveCount; limited replay of history.
• Kafka log: retained offsets; consumer groups; replay by resetting offsets; "DLQ" often a side topic you produce to after N failures; poison does not block other partitions if isolated.

Dead Letter Queues — mechanism and ops

A poison message (bad schema, permanent business reject, handler bug) that fails forever will either block a partition/consumer or burn retry budget. After max-receive-count / max attempts, move to a DLQ (side queue or topic) so the main path keeps flowing.

• Classify: transient (infra) vs permanent (poison). Only permanent (or exhausted transient) should DLQ.
• Alert on DLQ depth and age — silent DLQ is silent data loss of business intent.
• Replay procedure: fix code or data → reprocess DLQ messages in controlled rate → verify idempotent → purge or archive.
• Do not auto-replay blindly into a still-broken consumer (loop of doom).

When NOT to force at-least-once

• High-volume telemetry where loss is acceptable and cheaper.
• You cannot build idempotency — then at-least-once is unsafe; redesign the effect or use a transactional outbox + unique constraints.

Drill ladder

1. Q: Ack-before-process: what semantic? A: At-most-once (crash loses work).
2. Q: Why is broker "exactly-once" not enough for email send? A: Side effect outside the broker txn; duplicates still possible at the mail API unless deduped.
3. Q: DLQ depth rising after a deploy — first actions? A: Inspect sample payloads/errors; roll back or fix consumer; do not mass-replay until fixed; check if schema change broke compat.

Takeaways

• At-least-once + idempotent consumers is the pragmatic business default.
• Ack timing chooses loss vs duplication.
• DLQ quarantines poison; alert and rehearse replay.
• Order per key; Kafka and queues differ on retention and replay.

---

Re-authored for this guide. Cross-link Idempotency & Exactly-Once Is a Myth, Kafka internals, Transactional Outbox. Diagram hand-authored.

I just read a lesson on **Event Delivery Semantics & Dead Letter Queues** (System Design) and want to truly understand it. Explain Event Delivery Semantics & Dead Letter Queues from first principles using ONE vivid real-world analogy and a visual mental model — draw it as ASCII art or a clear step-by-step diagram — with a concrete example using real numbers. Then ask me one question to check I got the mental picture, and wait for my reply. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Teach me **Event Delivery Semantics & Dead Letter Queues** interactively. Ask me ONE guiding question at a time, wait for my answer, and adapt to my confusion — build the idea with me step by step instead of explaining it all at once. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Quiz me on **Event Delivery Semantics & Dead Letter Queues** with 5 questions, easy to tricky, ONE at a time. Tell me if each answer is right; at the end, explain clearly what I got wrong and why. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Help me remember **Event Delivery Semantics & Dead Letter Queues** for the long term: give the one-sentence intuition, a memorable hook/mnemonic, a tiny worked example, and 3 active-recall flashcards (Q -> A). If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.