Event-Driven Architecture A Promising Solution

Event-driven architecture works because the producer writes a fact — an immutable record that something happened — to a shared broker and then forgets about it, while any number of consumers read that fact on their own schedule; the broker's durability and its offset bookkeeping are what let a fire-and-forget write fan out to services the producer has never heard of, without the producer blocking on any of them.

That single mechanism — write a fact, let readers pull it independently — is the whole idea. Everything hard about EDA (ordering, duplicates, eventual consistency, the dual-write problem) falls out of the fact that the write and the reads are now separated in time and across process boundaries. This page traces one real event end to end, then confronts each of those hard parts head-on.

Bus vs. queue: the distinction that matters

The old version of this page called the event bus "also known as the message queue." That conflation hides the single most important design decision in EDA, so let's be precise:

• Point-to-point queue (e.g. an AWS SQS queue, a RabbitMQ work queue): a message is delivered to one consumer and then deleted. It is a work-distribution primitive — hand a task to exactly one of N identical workers. If two services both need the message, a plain queue cannot help you; the first reader consumes it.
• Pub-sub broker / retained log (e.g. Apache Kafka, Google Pub/Sub, a RabbitMQ topic exchange): a message is published to a topic and delivered to every interested subscriber. In a log like Kafka the message is not deleted on read — it stays for a retention window (say 7 days), and each consumer tracks its own offset (its bookmark) into the log. A new consumer can join tomorrow and replay from the beginning.

EDA's fan-out — one photo-upload event triggering notifications, feed updates, and moderation — is a pub-sub property. A point-to-point queue gives you decoupling and async, but not fan-out. Calling them the same thing is the mistake that gets a design review sent back. The rest of this page assumes a retained log (Kafka-style), because that is what makes replay, multiple consumer groups, and offset-based ordering possible.

Tracing one event: PhotoUploaded

User u_812 uploads a photo. The Upload service publishes one event to the photo.uploaded topic. Three unrelated consumer groups react. Here is the actual byte-level trace, with real offsets and keys.

The event (JSON payload the producer writes):

topic:     photo.uploaded
partition: key = "u_812"  -> hash -> partition 2
offset:    2  (the 3rd record ever written to partition 2)

{
  "event_id":   "evt_9f3a",     // unique — used for dedup
  "event_type": "PhotoUploaded",
  "user_id":    "u_812",
  "photo_id":   "ph_55",
  "occurred_at":"2026-07-02T10:04:12Z",
  "version":    1
}

Step-by-step trace of what each actor does:

Notice what the decoupling bought and cost: the producer finished in 3 ms regardless of the fact that moderation took 900 ms and was three records behind. The producer has no idea notif-svc or mod-svc exist. Adding a fourth consumer (say, analytics) tomorrow requires zero changes to the producer — it just subscribes and starts reading. That is the payoff. The cost is that at step 1 the photo is uploaded but not yet reflected in feeds or notifications: the system is eventually consistent, and every consumer above can crash, retry, and re-deliver — which is where the hard parts begin.

Delivery semantics: the trade-off you cannot avoid

The naive mental model is "the event gets delivered." In a distributed system, "gets delivered" is a choice among three guarantees, and you cannot have the strongest one for free:

The practical reality: you run at-least-once and make your consumers idempotent. In the trace above, if notif-svc processes offset 2, sends 340 pushes, then crashes before committing offset 3, it will re-read offset 2 on restart and send 340 pushes again. The fix is a dedup key:

// Consumer side — idempotency via the event's own id
seen = redis.setnx("proc:" + event.event_id, 1)  // atomic
if (!seen) return;          // already handled — skip, still commit offset
redis.expire("proc:" + event.event_id, 7*24*3600) // match retention
handle(event)
commitOffset()

Why the naive version is wrong: a consumer that just does handle(event); commitOffset() with no dedup key double-sends on every crash-retry and every consumer-group rebalance. Redelivery is not an edge case in at-least-once systems — it is the contract. Idempotency is not optional polish; it is the price of admission.

The dual-write problem (the bug that eats EDA teams)

Here is the trap almost every team walks into. The Upload service needs to (a) save the photo row to its own database and (b) publish PhotoUploaded to the broker. The tempting code:

// WRONG — two independent writes, no shared transaction
db.insert(photo);           // write 1: local Postgres
broker.publish(event);      // write 2: Kafka

These are two separate systems with no shared commit. Four things can happen, and two of them corrupt your system:

• Both succeed — fine.
• DB write fails, publish never runs — fine (nothing happened).
• DB commits, then the process crashes before publish — the photo exists but no consumer ever hears about it. Feeds and moderation silently never run. Ghost record.
• Publish succeeds, then the DB transaction rolls back — consumers act on a photo that does not exist. Phantom event.

You cannot fix this by reordering the two lines — you just move which failure you get. There is no ordering of two non-atomic writes that is safe. The standard fix is the Transactional Outbox: write the event into an outbox table in the same local DB transaction as the business row, so they commit atomically. A separate relay process (or Change-Data-Capture tailing the DB log, e.g. Debezium) reads the outbox and publishes to the broker, marking rows as sent.

BEGIN;
  INSERT INTO photos(id, user_id, ...) VALUES ('ph_55','u_812',...);
  INSERT INTO outbox(event_id, type, payload, sent)
         VALUES ('evt_9f3a','PhotoUploaded', '{...}', false);
COMMIT;               -- both or neither

-- relay, decoupled, at-least-once to the broker:
for row in SELECT * FROM outbox WHERE sent=false:
    broker.publish(row.payload)     // may double-send on relay crash
    UPDATE outbox SET sent=true WHERE event_id=row.event_id

The outbox turns two unsafe writes into one atomic DB transaction plus one idempotent relay — and the relay double-sending is fine precisely because consumers are already idempotent. This is why the two hard parts (delivery semantics and dual-write) are really one problem viewed from both ends.

Pitfalls

• Assuming order you don't have. Kafka guarantees order only within a partition. If PhotoUploaded and PhotoDeleted for the same photo land on different partitions (because you keyed by event_id instead of photo_id), a consumer can process the delete before the upload. Key by the entity whose order matters (photo_id or user_id), so all its events share a partition.
• The invisible failure. In request/response, a 500 is right in the caller's face. In EDA a consumer can crash, lag, or dead-letter silently — the producer never knows. Without consumer-lag dashboards and dead-letter-queue alerts, you find out from an angry user, not a stack trace.
• Eventual consistency leaking to the UI. The trace showed the photo uploaded (3 ms) but not yet in the feed (feed-svc still catching up). If the UI reads the feed immediately after upload and shows nothing, users think it broke. You must design the read path for the lag (optimistic UI, read-your-writes routing) rather than pretend the system is synchronous.
• Poison messages. One malformed event that always throws can block a partition forever under naive retry (the consumer re-reads the same offset, fails, re-reads…). Always cap retries and route failures to a dead-letter queue.
• Schema drift. Producer adds a required field; old consumers choke. The version field and a schema registry with compatibility rules exist for exactly this — an event is a public API contract with services you don't control.
• Debugging across the seam. A single user action now spans five services and no call stack connects them. Without a propagated correlation/trace id on every event, root-causing is archaeology.

When to use it — and when NOT to

Reach for EDA when the concrete signals line up: one action must fan out to many independent reactions (the photo-upload case), consumers should scale and fail independently, you want an audit/replay log of what happened, or the producer genuinely should not wait for downstream work. If you can't name a real fan-out or a real need to decouple deploys, you probably don't need it yet.

vs. synchronous request/response (REST/gRPC): Direct calls give you an immediate answer, a strong consistency story, and a stack trace when things break — you know now whether it worked. EDA trades all of that for decoupling and fan-out: the producer learns nothing about success, and correctness becomes eventual. Choose request/response when the caller needs the result to proceed (a checkout must know the card was charged). Choose EDA when the reactions are fire-and-forget and multiple.

vs. a point-to-point message queue: A plain work queue (SQS, RabbitMQ work queue) gives you async and load-leveling with far less operational weight than a retained log — no partitions, offsets, or consumer-group coordination to reason about. But it delivers each message to one worker and deletes it: no fan-out, no replay, no new-consumer-joins-later. Choose a queue when it's one producer handing tasks to a pool of identical workers. Choose a pub-sub log when multiple different services each need every event, or you need replay.

vs. orchestration (a Saga coordinator / workflow engine): Pure event choreography (services reacting to each other's events with no central brain) is beautifully decoupled but the end-to-end business flow exists nowhere — you can't point at the code that describes "place order." An orchestrator (Temporal, a Saga coordinator) makes the flow explicit and easy to reason about at the cost of re-introducing a coupling point. Choose choreography for loosely related reactions; choose orchestration when a multi-step business transaction needs coordinated compensation and a single place to see its state.

What EDA costs in every case: eventual consistency you must design around, mandatory idempotency, the dual-write problem, ordering caveats, and a whole new class of observability (lag, DLQs, distributed tracing). If your system is a modest CRUD app with synchronous needs, that operational tax buys you nothing — and "we might scale someday" is not a signal, it's a guess.

Takeaways

• The mechanism is write a durable fact, let readers pull it on their own offsets — separating write from reads in time is what enables decoupling and fan-out, and is also the source of every hard part.
• A pub-sub broker / retained log is not a point-to-point queue: fan-out, replay, and multiple consumer groups are log properties, not queue properties. Picking the wrong one is a design error.
• You run at-least-once, so duplicates and redelivery are the contract, not an edge case — consumers must be idempotent (dedup on event_id), and ordering only holds within a partition (key by the entity).
• Never do two independent writes (DB + broker). Use the transactional outbox so the fact commits atomically with the state change; the relay's re-sends are safe because consumers are already idempotent.

Re-authored and deepened for this guide. Sources: Sam Newman, Building Microservices, 2nd ed. (event-driven collaboration, choreography vs. orchestration); Chris Richardson, Microservices Patterns and microservices.io (Transactional Outbox, Saga, event sourcing); Martin Kleppmann, Designing Data-Intensive Applications (logs, delivery semantics, dual writes, exactly-once); Apache Kafka documentation (partitions, offsets, consumer groups, transactions); Debezium documentation (change-data-capture / outbox relay).