Transactional Outbox — Solving the Dual-Write Problem

The problem: two writes that cannot be atomic

Placing an order must (1) save the order in the database and (2) publish an OrderPlaced event to a broker so Inventory, Email, and Analytics react. Those are two systems with no shared transaction. A crash between them produces one of two disasters:

• DB commits, publish never happens — silent missing event; inventory never reserves.
• Publish succeeds, DB rolls back — phantom event; consumers act on an order that does not exist.

This is the dual-write problem. It is not fixed by "try/catch and hope" or by a distributed transaction across Postgres and Kafka in most real fleets.

The fix: outbox row in the same local transaction

Insert the business change and an outbox row in one local DB transaction. Both commit or both roll back — no dual-write gap. A separate relay reads unsent outbox rows, publishes to the broker, then marks them sent.

• Polling publisher — SELECT … FOR UPDATE SKIP LOCKED on pending rows. Simple; adds DB load; latency ≈ poll interval.
• CDC log tailing — Debezium/etc. tails WAL/binlog of outbox (or of business tables). Lower poll load; more moving parts.

Crash timeline (the part people skip)

t	What happens	Outbox state	Broker
1	BEGIN; INSERT order; INSERT outbox(id=E9, status=pending); COMMIT	pending	—
2	Relay reads E9, publishes to Kafka successfully	pending	E9 delivered
3	Relay crashes before mark-sent	pending	E9 present
4	Relay restarts, reads E9 again, publishes again	pending → sent	E9 duplicate

Therefore publishing is at-least-once. Consumers must be idempotent (dedupe on event id or business key). The outbox does not buy exactly-once end-to-end; it buys no silent loss and no phantom without a commit.

Mark-sent must be conditional and safe

-- Wrong: blind update races two relays
UPDATE outbox SET status = 'sent' WHERE id = 'E9';

-- Better: only pending → sent, single winner
UPDATE outbox SET status = 'sent', sent_at = now()
 WHERE id = 'E9' AND status = 'pending';
-- check rowcount == 1; if 0, another worker won or already sent

Use leasing (SKIP LOCKED), partition by id, or a single-threaded relay per shard to limit double-publish windows — but still assume duplicates can occur after publish-before-mark.

Ordering

If consumers need per-aggregate order, either: (1) relay publishes in outbox insertion order for that aggregate key, or (2) put ordering keys on the broker (Kafka partition key = orderId). Concurrent relays without care can reorder — prefer ordered poll per partition or CDC stream order.

When NOT to use an outbox

• No external event consumers — pure single-DB app; skip the pattern.
• Broker is system of record (event-sourced write path appends to log first) — different design; dual-write shape inverted.
• You can keep side effects in the same DB transaction (no message bus) — do that.
• 2PC/XA across DB and broker — rare, operationally heavy; outbox is the usual microservice choice.

Failure / operability

• Relay down — outbox backlog grows; business commits succeed but downstream freezes. Alert: oldest pending age, pending count.
• Poison payload — unparseable row blocks a naive single-threaded poller. Isolate bad rows; DLQ the publish side.
• Table bloat — delete or archive sent rows on a schedule; index status+id.
• Clock / multi-region — one writer region for an aggregate's outbox to avoid split-brain emits.

Decision defensibility

Why not "publish then write DB"? Phantoms on rollback. Why not "write DB then publish in the request thread"? Loss on crash after commit — same dual-write. Why not inbox-only on consumer? Consumer inbox helps idempotent receive; it does not create the missing event if the producer never published. Outbox is the producer-side half of reliable messaging; inbox/idempotency is the consumer-side half.

Drill ladder

1. Q: DB committed order, relay never ran. User impact? A: Order exists; no downstream reactions until relay catches up — lag, not permanent loss if outbox row exists.
2. Q: Why consumers still need idempotency with outbox? A: Publish-then-crash-before-mark causes redelivery.
3. Q: Poll every 5s vs CDC — one trade-off. A: Poll: simple, +latency and DB load. CDC: fresher, more infra and failure modes.
4. Q: Dispatcher is down for an hour — walk the blast radius. A: Business commits keep succeeding (no user-facing failure); outbox backlog and oldest-pending age grow; downstream projections/sagas freeze. On restart the relay drains in insertion order, so consumers see a burst of stale-but-ordered events — customer-visible lag, not loss. Alert on oldest-pending age, not just count.

Takeaways

• Cannot atomically write DB + broker → write event to outbox in the same transaction, then relay.
• Relay implies at-least-once publish → idempotent consumers.
• Mark-sent is conditional; monitor backlog age; this is how sagas and CQRS projections receive reliable facts.

---

Re-authored for this guide. Sources: Richardson, Microservices Patterns (transactional outbox); Kleppmann on dual writes; Debezium outbox routing docs. Diagram hand-authored.

I just read a lesson on **Transactional Outbox — Solving the Dual-Write Problem** (System Design) and want to truly understand it. Explain Transactional Outbox — Solving the Dual-Write Problem from first principles using ONE vivid real-world analogy and a visual mental model — draw it as ASCII art or a clear step-by-step diagram — with a concrete example using real numbers. Then ask me one question to check I got the mental picture, and wait for my reply. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Teach me **Transactional Outbox — Solving the Dual-Write Problem** interactively. Ask me ONE guiding question at a time, wait for my answer, and adapt to my confusion — build the idea with me step by step instead of explaining it all at once. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Quiz me on **Transactional Outbox — Solving the Dual-Write Problem** with 5 questions, easy to tricky, ONE at a time. Tell me if each answer is right; at the end, explain clearly what I got wrong and why. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Help me remember **Transactional Outbox — Solving the Dual-Write Problem** for the long term: give the one-sentence intuition, a memorable hook/mnemonic, a tiny worked example, and 3 active-recall flashcards (Q -> A). If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.