Introduction to Messaging System

The mechanism

A messaging system works by placing a durable buffer — the broker — between producer and consumer, so a producer's send() returns the instant the broker has persisted the message, the consumer reads it later at its own pace, and a message that is delivered but not acknowledged is redelivered rather than lost. That one move — persist, then hand off under acknowledgement — is what buys decoupling along three axes:

• Time — producer and consumer need not be running at the same moment; the broker holds the message across a consumer outage.
• Rate — a fast producer no longer overruns a slow consumer; the backlog absorbs the difference.
• Identity — neither side needs the other's address, count, or protocol; they agree only on the message and the topic/queue name.

The problem it solves

Picture a log-aggregation service that must store and index ~300 log entries/second from many sources. Wire the sources directly to it and three things break: (1) a traffic spike above what one instance can process drops or crashes it; (2) every source is now coupled to the aggregator's protocol, data format, and address; (3) if the aggregator restarts, in-flight logs are simply gone. Inserting a broker converts all three into a single, well-understood buffering-plus-redelivery problem.

Worked example: absorbing a spike

The consumer cluster steadily drains 500 messages/second. Baseline load is 300/s (backlog stays empty). Now a burst of sources pushes 2000/s for five seconds. Without a broker the aggregator can only take 500/s, so it drops 1500/s or falls over. With a broker, the excess piles up as backlog instead — and the newest message's wait time is backlog ÷ drain rate:

After the spike, suppose the sources go quiet at ~100/s for a while — an explicit new assumption; bursts are often followed by a lull rather than an instant return to baseline. The queue then drains at a net 400/s and the 7500-message backlog clears in 7500 ÷ 400 ≈ 19 seconds — zero messages lost. If arrivals instead return straight to the 300/s baseline, the net drain is only 200/s and the same backlog takes 7500 ÷ 200 = 37.5 seconds.

The trade the mechanism is making: the broker converts dropped messages into added latency (15s of staleness at the peak). That trade is only sound when the spike is bounded. If arrivals stay above 500/s indefinitely, the backlog grows without limit — a broker cannot fix an under-provisioned consumer, it can only ride out a temporary imbalance. Sizing the consumer for the sustained rate is still mandatory; the broker just buys headroom for the peaks.

Two delivery topologies: queue vs pub/sub

Once messages sit in a broker, the question is who gets to read each one. There are two answers, and they exist for opposite reasons.

Queue (point-to-point / competing consumers). Messages sit in one line; each message is handed to exactly one consumer and removed once acked. Add consumers and they compete for the line — total throughput rises because the work is split. This is the model for a task/work queue: 4 workers draining an order-processing queue each handle a disjoint quarter of the orders. What you cannot do is have two consumers both react to the same message.

Publish-subscribe (topic / fan-out). Messages are grouped by topic; every subscriber to a topic gets its own copy of every message. This is the model for event broadcast: one order.placed event is delivered independently to the billing service, the search-index updater, and the confirmation-email sender — none of them consumes the others' copy. Adding subscribers does not split load; it multiplies it (N copies, N independent cursors).

The broker underneath both is the same durable buffer; queue vs pub/sub is purely a policy on how many consumers may observe each message.

Delivery semantics — the trade-off that bites

The acknowledgement loop from the first diagram has a subtle consequence: when a consumer acks decides what guarantee you get. Trace one message where the consumer crashes:

1. Broker delivers log #42 to consumer A.
2. A writes #42 to disk and updates the index — processing succeeds.
3. A crashes before sending the ack.
4. The broker's ack timeout expires; #42 is still marked unacknowledged.
5. The broker redelivers #42 to consumer B → #42 is now processed twice.

This is at-least-once delivery, and it is the practical default of almost every broker. The three regimes:

• At-most-once — ack before processing. A crash after the ack but before the write loses the message. Fast, lossy; fine for metrics you can afford to drop.
• At-least-once — ack after processing (above). Never loses, but redelivery causes duplicates. The default.
• Exactly-once — the effect of each message applied once. Not free: it needs an idempotent consumer (dedup by message id, or upsert by key) or a broker+consumer transaction (e.g. Kafka transactions). Usually you get exactly-once effects by making at-least-once consumers idempotent, not by a magic broker flag.

Why the naive assumption is wrong: teams routinely assume messages arrive once and write non-idempotent consumers ("insert row", "charge card"). Under at-least-once, a single dropped ack double-charges the customer. The fix is to key every effect on the message id so a replay is a no-op.

When to use it — and when not

Messaging vs. synchronous RPC

A broker is not free: every hop adds a broker write plus a consumer poll to end-to-end latency, introduces eventual consistency, forces you to handle duplicates, and adds a stateful component to operate and monitor.

• Choose messaging when the work can be deferred, load is spiky, the producer must not block on the consumer, the consumer may be temporarily down, or several independent systems must react to the same event (fan-out) or a stream must be replayed.
• Prefer synchronous RPC (HTTP/gRPC) when the caller needs the result now to continue (a read path, a request/response), low latency matters more than durability, and the extra hop, eventual consistency, and duplicate-handling are not worth it. Don't put a broker on your login endpoint.

Queue vs. pub/sub

• Choose a queue when each message represents a unit of work that exactly one worker should do, and you scale throughput by adding competing consumers. Cost: no fan-out — other systems can't also see the message.
• Choose pub/sub when multiple independent consumers must each react to the same event. Cost: no automatic load-splitting within a subscriber, and N copies / N cursors to store and track.

Log-based brokers like Kafka deliberately blur the line: a topic is partitioned, consumers within one group split the partitions (queue-like competing consumers), while different groups each get the full stream (pub/sub-like fan-out) — and because the log is retained, a new consumer can replay from the start. Classic brokers like RabbitMQ keep queue and exchange/fan-out as distinct primitives and delete on ack.

Push vs pull consumers

Brokers deliver messages, but the direction of that delivery matters. In a push model the broker sends messages to the consumer as soon as they arrive (or as fast as the consumer permits). In a pull model the consumer asks the broker for the next batch at its own pace, using a cursor or offset. Neither is universally better; they shift control and risk to opposite ends.

Push. RabbitMQ (via basic.consume) and ActiveMQ push messages to workers; on AWS the push counterpart is SNS delivering to HTTP endpoints or triggering Lambdas. The broker owns flow control: it tracks in-flight acks and stops sending when the consumer's prefetch window (RabbitMQ's QoS setting) is full. The consumer gets low latency but can be overwhelmed if the broker ignores its capacity. Push systems usually need backpressure — a signal from consumer to broker to slow down — or they drop messages.

Pull. Kafka consumers poll partitions and remember their own offset. SQS is also pull, despite often being lumped with RabbitMQ: consumers call the ReceiveMessage API (long polling waits up to 20 s for a message), in-flight messages are hidden by a visibility timeout and reappear if not deleted — the consumer can never be flooded because it sets its own poll rate. The consumer owns flow control: it decides how many records to fetch and how fast to process them. A slow consumer simply polls less often; it cannot be flooded by the broker. The cost is a small minimum latency (the poll interval) and the operational burden of keeping commits — offset commits in Kafka, DeleteMessage calls in SQS — in sync with side effects.

Pulsar is a hybrid: the broker pushes into a client-side receive queue bounded by consumer-issued permits, so delivery looks like push but the consumer still owns flow control — which is why it fits neither column cleanly.

Worked trace. A consumer that crashes after receiving a pushed message but before processing it loses the message if it acked early, or duplicates it if the broker redelivers. A pull consumer that crashes after processing but before committing its offset will re-fetch and reprocess from the last committed offset — the same at-least-once problem, but the failure surface is the offset commit, not the broker's redelivery timer. That is why Kafka's exactly-once story is really about making offset commits atomic with the records the consumer produces back into Kafka (the consume-transform-produce loop) — external side effects such as a DB write, an email, or a charge sit outside that transactional boundary and still need the idempotent-consumer pattern above — not about eliminating duplicates in the network.

Retention, TTL, and replay policy

Once a message is in the broker, how long does it live? The answer splits the world again. Queue brokers usually delete a message on ack; if you need it gone sooner, you set a TTL (time-to-live) and the broker drops it when it expires. Log brokers keep every message for a retention window — time-based (7 days), size-based (100 GB per partition), or compacted (keep the latest record per key forever).

The choice changes what you can recover from:

• Ack-and-delete + TTL: good for jobs that must run once. If a consumer is down longer than the TTL, the message is lost.
• Time/size retention: good for replay and debugging. If you ship a bug in your analytics consumer, you can redeploy and reprocess the last N days.
• Compacted log: good for keyed state (e.g., user profile updates). Older values for the same key are garbage-collected, so the log shrinks to the latest state per key while still allowing replay.

When retention bites. A team runs Kafka with 1-day retention and discovers their consumer was silently failing for two days. The messages are gone; they cannot reconstruct the lost state. Retention is not an infinite safety net — size it to your mean-time-to-detect plus redeploy time. Conversely, an SQS queue with a 14-day maximum retention can hold a failed job for two weeks, but after that the message is deleted regardless of whether it was ever processed.

Saga pattern: messaging as a transaction coordinator

Sometimes a business operation spans several services, each with its own database. You cannot wrap them in one ACID transaction without a single lock manager, so you use a saga: a sequence of local transactions coordinated by messages, where each step publishes the event that triggers the next, and a failure triggers compensating transactions that undo earlier steps.

Consider an e-commerce checkout. Order the steps by how painful each is to undo — cheapest-to-compensate first, hardest last:

1. Reserve inventory (inventory service) → publishes InventoryReserved. Compensation is trivial: release the reservation — an internal write no customer ever sees.
2. Charge payment (payment service) → publishes PaymentCharged. This is the pivot: once money moves, undoing it means a customer-visible refund that leaks card-processing fees and can itself fail — so it runs only after every easily-compensable step has succeeded.
3. Create shipping label (shipping service) → publishes OrderShipped.

If the charge fails after inventory was reserved, the payment service publishes PaymentFailed; the inventory service listens and releases the reservation. That release is the compensating transaction — and because payment ran last, no money ever moved, so there is nothing to refund. Invert the order (charge first, reserve second) and an out-of-stock item forces a real refund: visible on the customer's statement, fee-leaking, and itself a fallible operation. That is the ordering rule: put the hardest-to-compensate step last. The saga does not give atomic isolation — a reader can see inventory reserved before payment is charged — but it guarantees that the system ends in a consistent state, either completed or fully compensated.

Why messaging is the natural spine. Each service reacts to an event, performs its local work, and emits the next event. The broker durably holds those events, so a crashed service resumes from where it left off. The orchestration can be choreography (every service listens and reacts) or orchestration (a central saga manager sends commands and tracks state). Choreography is looser and scales better; orchestration is easier to reason about when the flow is long or has many branches.

The trap. Compensations are themselves messages, so they are subject to at-least-once delivery. A compensation that runs twice must be a no-op (idempotent reservation release). If it runs zero times because the compensation message is lost, you have dangling reserved inventory. Design every compensating action to be idempotent and alarm on unprocessed saga timeouts.

Pitfalls

• Assuming exactly-once. The default is at-least-once. Non-idempotent consumers double-write or double-charge on any redelivery. Make effects idempotent (dedup on message id / upsert on key).
• Unbounded backlog. Broker storage is finite. If consumers are chronically slower than producers, the queue fills until the broker blocks producers or drops messages. The broker only absorbs bounded spikes — alarm on consumer lag, and size consumers for the sustained rate.
• Poison messages / head-of-line blocking. One un-processable message that always fails is redelivered forever, stalling the queue or partition behind it. Cap retries and route failures to a dead-letter queue.
• Ordering illusions. Order is guaranteed only within a single queue/partition consumed serially. Add competing consumers or partitions and per-key order can break. Partition by the key whose order matters (e.g. by account id).
• Failure hidden by decoupling. A producer's send() succeeds even when no consumer exists or every consumer is failing — the bug surfaces far downstream and late. Monitor consumer lag and DLQ depth, not just producer-side errors.

Takeaways

• The broker is a persist-then-ack buffer. That single mechanism buys time-, rate-, and identity-decoupling — at the price of added latency, an operational component, and duplicate handling.
• Buffering converts dropped messages into added latency; it is sound only for bounded spikes. A permanently overloaded consumer just grows the backlog forever — size it for the sustained rate.
• Queue = one message, one consumer (split work); pub/sub = one message, every subscriber (broadcast events). Pick by whether consumers compete or all independently care.
• Default delivery is at-least-once: make consumers idempotent, or you will process duplicates.

Re-authored / Deepened for this guide. Synthesizes the messaging-system fundamentals in Grokking the System Design Interview (DesignGurus), Martin Kleppmann's Designing Data-Intensive Applications (O'Reilly, ch. 11 “Stream Processing”), and the delivery-guarantee, acknowledgement, and consumer-group documentation of Apache Kafka and RabbitMQ.

Idempotent consumer pattern

The way to survive at-least-once delivery is to key every side effect on a unique dedup_id so a replay becomes a no-op. The subtlety: some writes are naturally idempotent (an insert/upsert keyed by the event id), but an additive effect like a wallet credit is not — balance = balance + X applied twice double-credits, which is exactly the duplicate-charge bug. Additive effects must therefore be gated on the dedup insert inside one transaction, so the credit fires only the first time the event is seen.

-- Pattern A — naturally idempotent write (upsert keyed by event id):
INSERT INTO processed_events (dedup_id, payload, processed_at)
VALUES ('evt-42', '{...}', NOW())
ON CONFLICT (dedup_id) DO NOTHING;    -- replay inserts 0 rows: a no-op

-- Pattern B — NON-idempotent additive effect (balance += X double-credits
-- on replay), so gate it on the dedup insert in ONE transaction:
WITH seen AS (
  INSERT INTO processed_events (dedup_id) VALUES ('evt-99')
  ON CONFLICT (dedup_id) DO NOTHING
  RETURNING dedup_id                  -- returns a row ONLY the first time
)
UPDATE wallets
SET balance = balance + ?
WHERE id = ? AND EXISTS (SELECT 1 FROM seen);   -- skipped on replay

Both patterns guarantee that the observable state changes at most once, even when the broker redelivers the message.

Saga choreography trace

OrderSvc          InventorySvc          PaymentSvc           ShippingSvc
   | createOrder        |                    |                    |
   |  OrderCreated      |                    |                    |
   |------------------->|                    |                    |
   |                    | reserve stock      |                    |
   |                    |  InventoryReserved |                    |
   |                    |------------------->|                    |
   |                    |                    | charge card (pivot)|
   |                    |                    |  PaymentCharged    |
   |                    |                    |------------------->|
   |                    |                    |                    | ship
   |                    |                    |                    | OrderShipped

Compensation path (charge fails AFTER inventory was reserved):
PaymentSvc publishes PaymentFailed
InventorySvc consumes it and releases the reservation (idempotent)
OrderSvc consumes it and marks order CANCELLED -- no money ever moved

Each service reacts only to events it owns; no central orchestrator is required, but the event schema must include enough context for every compensation step.