Designing a Distributed Message Queue — Partitions, ISR & Delivery Semantics, Traced

Building the thing every other design uses

Almost every system design in this guide reaches for "add a message queue." This page designs the queue itself — which is where you discover that the properties everyone assumes (durability, ordering, exactly-once) are not free, and that each one is bought by giving up something specific.

Requirements worth stating

• Producers publish, consumers subscribe; both scale independently.
• Durable — an acknowledged message survives broker failure.
• High throughput — hundreds of thousands of messages/second.
• Configurable retention — days, so consumers can replay.
• Ordering guarantees, at least somewhere.

Two messaging models

Point-to-point: each message is consumed by exactly one consumer, then deleted — a work queue. Publish-subscribe: each message goes to every subscriber, and the message is retained independently of who has read it.

The design that generalizes both is a retained log with per-consumer-group offsets. Messages are appended to a log and kept for the retention period; each consumer group tracks its own read position. Point-to-point is then one consumer group; pub-sub is several groups reading the same log. That is a genuinely important unification: the broker stops managing per-message state (which is what makes traditional queues hard to scale) and manages a cursor per group instead — the same trick as the chat sync cursor.

Storage: why a write-ahead log and not a database

The obvious implementation is a table of messages with a "consumed" flag. It fails at scale for a specific reason: random reads and writes plus per-message mutation, on the highest-volume path in your infrastructure.

Instead, each partition is an append-only log segmented into files. Writes are sequential appends — the fastest thing a disk does, and roughly comparable to memory throughput for streaming writes. Reads are sequential scans from an offset. Nothing is mutated; deletion happens by dropping whole old segments. Consumers do not remove messages, they advance an offset.

This buys three things at once: throughput (sequential I/O), replay (rewind the offset and reprocess — a superpower for bug recovery and for adding a new consumer to existing data), and simple durability (the log is the write-ahead log). The cost: you store messages for the full retention window whether or not anyone reads them, and a slow consumer's problem shows up as an offset lag rather than as a growing queue.

Partitions: the unit of parallelism and of ordering

A topic is split into partitions, each an independent log on (potentially) a different broker. This is the central design decision, and it makes one trade that must be understood clearly:

Ordering is guaranteed within a partition, and nowhere else. There is no total order across a topic, because that would require a single serialization point and destroy the parallelism partitions exist to provide. So if you need events for one entity in order, they must go to the same partition — which you arrange by partitioning on a key (hash(user_id) % partitions). Order per user, parallelism across users.

Consequences worth knowing before you rely on them:

• Partition count is hard to increase. Adding partitions changes hash(key) % n, so a key's messages start landing in a different partition than its history — breaking per-key ordering across the change. This is why partition counts are over-provisioned up front.
• Decreasing partitions is worse. The log in a removed partition still holds unread data, so the operation is usually not supported at all; you migrate to a new topic instead.
• A hot key is a hot partition. One celebrity user's traffic lands on one broker, and no amount of extra partitions helps, because the key pins it.

Replication and the ISR: where durability actually lives

Each partition has a leader and followers. Producers write to the leader; followers replicate. The set of replicas currently caught up is the in-sync replica set (ISR). A follower that falls too far behind is removed from the ISR so it cannot hold up writes.

The producer's acks setting decides when a write is acknowledged:

• acks=0 — do not wait at all. Fastest, and a message can be lost before any broker has it. Fine for lossy telemetry; never for anything you would miss.
• acks=1 — wait for the leader only. The failure traced in the diagram: the leader acknowledges and dies before replicating, a follower is promoted, and the acknowledged message is gone while the producer believes it is durable. This is the dangerous middle setting, because it looks safe and usually is.
• acks=all — wait for every in-sync replica. An acknowledgement now means the data survives leader loss. The cost is latency equal to the slowest ISR member.

The subtlety that catches people: acks=all is only as strong as the ISR is large. If followers fall behind and get evicted, the ISR can shrink to just the leader, at which point "all in-sync replicas" means one machine — and acks=all silently degrades to acks=1. The guard is a minimum-ISR floor: if the ISR drops below it, reject the write rather than acknowledge something a single disk failure can erase. That is a deliberate choice of unavailability over silent data loss, and it is the correct one for a durable queue.

Delivery semantics: the three options, honestly

• At-most-once — send, do not retry. Messages can be lost, never duplicated. Choose for high-volume metrics where a missing sample is invisible.
• At-least-once — retry until acknowledged. Nothing is lost; duplicates happen (a broker acknowledgement that never reached the producer means the producer re-sends). This is what almost everything actually uses.
• Exactly-once — the interesting case, because true exactly-once delivery across a network is impossible: the sender cannot distinguish "message lost" from "acknowledgement lost", so it must either risk losing or risk duplicating.

What real systems provide is exactly-once processing, built from at-least-once delivery plus one of two mechanisms: producer idempotence (each producer stamps a sequence number so the broker discards a duplicate append) and transactional commits (the message's effect and the consumer's offset advance commit atomically, so reprocessing cannot double-apply). Both narrow the guarantee to a scope you control.

The practical advice is the one people skip: make consumers idempotent and stop trying to buy exactly-once from the broker. A consumer whose handler is keyed by message ID turns at-least-once into effectively-once at a fraction of the cost and complexity, and it survives failure modes the broker's transactions do not cover (such as your handler writing to an external system that has never heard of the transaction).

Consumer groups and rebalancing

Consumers in a group divide the partitions between them: each partition is owned by exactly one consumer in the group, which is what preserves per-partition ordering while allowing parallel consumption. Group membership is coordinated by a broker acting as group coordinator (older designs used ZooKeeper for this).

Rebalancing reassigns partitions when a consumer joins, leaves, or stops heartbeating. It is necessary and it is disruptive: in the simple protocol, all consumers stop consuming while the assignment is recomputed — a stop-the-world pause. Two consequences follow:

• Parallelism is capped by partition count. More consumers than partitions means idle consumers; you cannot scale consumption past the partition count, which is another reason to over-provision partitions.
• Slow processing causes rebalance storms. If a consumer's handler takes longer than the poll interval, the coordinator assumes it died and rebalances — which slows everyone, causing more timeouts, causing more rebalances. The fix is smaller batches or a longer timeout, not more consumers.

The extras that get requested

• Batching — the core throughput/latency dial. Larger batches amortize network and disk overhead across more messages, raising throughput and raising per-message latency. There is no setting that improves both.
• Message filtering — ideally broker-side by a tag or header, so consumers do not download and discard. Filtering on the payload requires deserializing on the broker, which puts consumer logic in your storage tier — usually the wrong trade.
• Delayed / scheduled messages — a log is ordered by arrival, not by due time, so delays need a separate structure: per-delay-interval topics, or a time-indexed store the broker drains into the real topic.
• Dead-letter queues — after N failed attempts, move the message aside. Without this, one poison message blocks its partition forever, which is the most common production stall.

Which setting, when

Decision	Option	Choose when	Cost / breaks when
Durability	acks=all + min-ISR	Payments, orders, anything you cannot lose	Latency tracks the slowest replica; writes rejected when ISR shrinks
Durability	acks=1	Logs, clickstream where rare loss is tolerable	Leader failure loses acknowledged messages — silently
Durability	acks=0	Metrics samples, fire-and-forget telemetry	Anything with business meaning
Semantics	At-least-once + idempotent consumer	Nearly always — the pragmatic default	Requires a dedupe key and somewhere to store it
Semantics	Transactional exactly-once	Stream processing wholly inside one system	External side effects; throughput and complexity cost
Ordering	Key-partitioned	Per-entity order with cross-entity parallelism	Hot keys pin to one partition; repartitioning breaks history
Ordering	Single partition	Total order genuinely required	Throughput capped at one broker, one consumer
Throughput	Large batches	Bulk pipelines, analytics ingestion	Latency-sensitive paths — messages wait to fill a batch

Pitfalls

• Assuming topic-wide ordering. The single most common wrong assumption; order exists per partition only.
• acks=all without a min-ISR floor. Believing you have replication while the ISR has quietly shrunk to one.
• Committing the offset before processing succeeds. Converts at-least-once into at-most-once and loses messages on any handler failure — usually discovered as unexplained gaps.
• No dead-letter queue. One unparseable message halts its partition indefinitely.
• More consumers than partitions and expecting more throughput.
• Long handlers inside the poll loop, triggering rebalance storms that look like broker instability.
• Retention shorter than your worst outage. If a consumer is down longer than the retention window, the data it never read is deleted — unrecoverable, and the metric that would have warned you is consumer lag approaching retention, which few teams alert on.

Cost model — what dominates the bill

A message queue's cost is replicated storage plus cross-zone network, and the replication factor multiplies both.

Rough BOTE: 500,000 messages/second at 1 KB each is 500 MB/s ≈ 43 TB/day of ingress. With a 7-day retention that is ~300 TB stored, and at replication factor 3 that is ~900 TB provisioned. On SSD-backed storage at roughly $0.10/GB-month, ~900 TB is on the order of $90,000/month — which is why retention is the most consequential setting in the whole system, and why tiered storage (recent segments on SSD, older segments on object storage at ~$0.02/GB-month) is now standard: it can cut that line by 4–5× for the cold majority.

Then the network. Replication factor 3 across three availability zones means every message crosses a zone boundary twice. At 500 MB/s, that is ~1 GB/s of inter-zone traffic; at typical $0.01/GB inter-zone rates that is roughly $26,000/month in network charges alone. Add consumer fan-out: each additional consumer group re-reads the full stream, so five groups means 5× the egress on the read side.

Dominant line items: replicated retained storage; inter-zone replication network; then read egress multiplied by the number of consumer groups.

Levers: shorten retention (largest single effect, bounded by your recovery requirements); tiered storage to object storage for older segments; compression, which cuts storage and both network legs at once and is nearly free given batching already groups messages; and rack/zone-aware consumer placement so consumers read from a local replica rather than across a zone.

Operability: the fingerprints of a sick queue

Consumer lag growing linearly is the base case — consumers cannot keep up — but the shape matters: lag on one partition while others are flat is a hot key or a poison message, and adding consumers will not help either. Lag sawtoothing with periodic collapse to zero is a rebalance storm: consumers pause, lag builds, they resume and catch up, then a timeout triggers another rebalance.

ISR shrinking on one broker is a leading indicator of the durability problem, and it is the metric most worth alerting on, because it silently weakens acks=all before any data is lost. Under-replicated partitions that never recover mean a follower cannot catch up — often a slow disk — and it will stay out of the ISR, leaving that partition one failure from loss.

The nastiest fingerprint is consumer lag approaching the retention window: past that point the broker deletes data the consumer has not read, and the loss is permanent and unannounced. It should be an alert, not a dashboard. Watch also for throughput dropping while CPU and disk look idle, which usually means batch sizes collapsed (a producer config change) so per-message overhead now dominates, and duplicate processing after every deploy, which means offsets are committed on a timer rather than after successful processing.

Signals worth having: consumer lag per partition (not just per topic), ISR size and under-replicated partition count, lag-versus-retention headroom, rebalance frequency, average batch size, dead-letter queue arrival rate, and per-partition throughput skew.

---

Authored for this guide to cover designing a distributed message queue (Alex Xu Vol. 2, ch. 19 — not present in the Vol. 1 PDF); acks/ISR failure-trace diagram hand-authored as SVG. Complements this guide's Kafka topic (partitions, offsets, consumer groups, RabbitMQ vs Kafka vs ActiveMQ) by designing the broker's durability and delivery guarantees from first principles.

I just read a lesson on **Designing a Distributed Message Queue — Partitions, ISR & Delivery Semantics, Traced** (System Design) and want to truly understand it. Explain Designing a Distributed Message Queue — Partitions, ISR & Delivery Semantics, Traced from first principles using ONE vivid real-world analogy and a visual mental model — draw it as ASCII art or a clear step-by-step diagram — with a concrete example using real numbers. Then ask me one question to check I got the mental picture, and wait for my reply. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Teach me **Designing a Distributed Message Queue — Partitions, ISR & Delivery Semantics, Traced** interactively. Ask me ONE guiding question at a time, wait for my answer, and adapt to my confusion — build the idea with me step by step instead of explaining it all at once. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Quiz me on **Designing a Distributed Message Queue — Partitions, ISR & Delivery Semantics, Traced** with 5 questions, easy to tricky, ONE at a time. Tell me if each answer is right; at the end, explain clearly what I got wrong and why. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Help me remember **Designing a Distributed Message Queue — Partitions, ISR & Delivery Semantics, Traced** for the long term: give the one-sentence intuition, a memorable hook/mnemonic, a tiny worked example, and 3 active-recall flashcards (Q -> A). If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.