Messaging patterns

In distributed systems, messaging patterns define how components communicate via asynchronous messages. Kafka implements all of the classic patterns, but it does so with a small set of primitives — topics, partitions, producers, and consumer groups — so the same pattern can look very different from the same pattern in a traditional queue broker. This page walks each pattern through the Kafka lens: which primitive you use, what the config looks like, and the operational trap you are most likely to hit in production.

The primitives Kafka gives you

Every pattern below is just a different way of combining these six primitives.

1. Point-to-Point — one consumer group

Idea: each record is processed by exactly one worker. In Kafka you do not create a "queue"; you create a topic and have all workers join the same consumer group. The group coordinator divides partitions among the members, so each partition is consumed by only one member at a time.

Config sketch

// producer
props.put("bootstrap.servers", "kafka:9092");
props.put("key.serializer", "StringSerializer");
props.put("value.serializer", "StringSerializer");
Producer<String, String> producer = new KafkaProducer<>(props);
producer.send(new ProducerRecord<>("email-jobs", userId, payload));

// consumer — all instances share this group.id
props.put("group.id", "email-workers");
props.put("enable.auto.commit", "false");
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Arrays.asList("email-jobs"));
while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(200));
    for (ConsumerRecord<String, String> r : records) {
        sendEmail(r.value());
        // commit exactly this record: offset + 1 = "next record to read"
        // (at-least-once — a crash before this line reprocesses only r)
        consumer.commitSync(Collections.singletonMap(
            new TopicPartition(r.topic(), r.partition()),
            new OffsetAndMetadata(r.offset() + 1)));
    }
}

Why not plain commitSync() in the loop? The no-arg commitSync() commits the position after the entire last poll(), for all assigned partitions — so calling it per record silently commits records you have not processed yet. Trace it with a 3-record batch at offsets 10–12: process #10, no-arg commit writes offset 13, crash before #11 — on restart the group resumes at 13 and records 11–12 are never processed. With the default max.poll.records=500 that loss window is up to 499 records, and your "manual commit after processing" is actually at-most-once for the rest of the batch. The targeted commitSync(Map<TopicPartition, OffsetAndMetadata>) above commits exactly offset + 1, which is honest at-least-once per record. The cheaper variant: one no-arg commitSync() after the for-loop — a crash mid-batch then reprocesses from the batch start, still at-least-once, at the cost of more duplicates.

What the trace looks like

Topic email-jobs has 6 partitions. Three consumers in group email-workers start up. The coordinator assigns partitions 0–1 to consumer A, 2–3 to consumer B, 4–5 to consumer C. A producer sends 6 records with evenly distributed keys; each lands in a different partition. Each record is consumed by exactly one consumer. If consumer B crashes, the coordinator reassigns partitions 2–3 to the surviving members (a rebalance). No record is delivered to two consumers simultaneously because ownership is exclusive per partition.

Kafka gotcha: partitions are the ceiling

You cannot have more concurrent consumers than partitions. If email-jobs has 6 partitions, the 7th consumer in the group joins but receives no partitions and sits idle. Scaling horizontally is therefore a two-step decision: add consumers and add partitions ahead of time. Also, if one message is much slower than the others and that partition is owned by one consumer, that single slow message blocks every other message in the same partition even if other consumers are idle. Use keys only when per-key order matters; otherwise let Kafka round-robin across partitions so slow messages do not serialize the whole topic.

2. Publish-Subscribe — independent consumer groups

Idea: every interested service gets its own independent copy of the event stream. In Kafka you do not fan out inside the broker to per-subscriber queues; you simply have multiple consumer groups read the same topic. Each group stores its own offsets, so one group’s lag does not affect another.

Config sketch

// analytics service
props.put("group.id", "analytics-events");
consumer.subscribe(Arrays.asList("user-registered"));

// welcome-email service
props.put("group.id", "welcome-email");
consumer.subscribe(Arrays.asList("user-registered"));

// fraud service
props.put("group.id", "fraud-detection");
consumer.subscribe(Arrays.asList("user-registered"));

Trace

A single record {userId: 42, plan: "pro"} is appended to partition 3 of user-registered. The analytics group reads it at offset 15,001, the welcome-email group reads it at offset 8,203, and the fraud group reads it at offset 31,994. Each group advances its offset independently. If the analytics service is down for an hour, its offsets fall behind, but the other two groups keep processing in real time.

Kafka gotcha: retention, not delivery

Kafka does not know whether a subscriber is "listening" the way RabbitMQ does. If a consumer group is offline longer than the topic’s retention window (default 7 days), its offsets may point to records that have already been deleted. When the group comes back, it either starts from the latest offset (loses data) or from the earliest offset (reprocesses everything). For critical subscribers, monitor consumer lag and size retention so the slowest subscriber can catch up.

3. Request-Reply — reply topic + correlation ID

Idea: a requester sends a message and expects a response later. Kafka has no built-in request-reply primitive, but the pattern is easy to build: the requester writes to a request topic and includes a reply topic and a correlation ID in the message headers. The replier reads the request, does the work, and writes the response to the reply topic with the same correlation ID.

Config sketch

// requester
String replyTopic = "report-replies-" + UUID.randomUUID().toString();
correlationId = UUID.randomUUID().toString();
ProducerRecord<String, String> req = new ProducerRecord<>(
    "report-requests", orderId, payload);
req.headers().add("reply-to", replyTopic.getBytes(StandardCharsets.UTF_8));
req.headers().add("correlation-id", correlationId.getBytes(StandardCharsets.UTF_8));
producer.send(req);

// wait for the matching reply
consumer.subscribe(Arrays.asList(replyTopic));
ConsumerRecords<String, String> records = consumer.poll(timeout);
for (ConsumerRecord<String, String> r : records) {
    String replyCorr = new String(r.headers().lastHeader("correlation-id").value());
    if (correlationId.equals(replyCorr)) {
        return r.value();
    }
}

// replier
consumer.subscribe(Arrays.asList("report-requests"));
for (ConsumerRecord<String, String> r : records) {
    String replyTopic = new String(r.headers().lastHeader("reply-to").value());
    String corr = new String(r.headers().lastHeader("correlation-id").value());
    String result = generateReport(r.value());
    ProducerRecord<String, String> reply = new ProducerRecord<>(replyTopic, result);
    reply.headers().add("correlation-id", corr.getBytes(StandardCharsets.UTF_8));
    producer.send(reply);
}

Trace

The web API sends a report request at t0 and starts a 30-second consumer poll. The report worker receives it at t1, generates a PDF by t3, and writes the reply to the dedicated reply topic. The API consumer sees the reply at t4, matches the correlation ID, and returns the PDF URL to the caller.

Kafka gotcha: do not reply to the request topic

Using the same topic for requests and replies creates an infinite loop: the replier writes a "reply" record, the same consumer group reads it as a new "request," and replies to it again. Always use a separate reply topic. Also, transient reply topics accumulate if the requester crashes before deleting them; either use a short retention (e.g., 1 hour) or a fixed, partitioned reply topic keyed by correlation ID. And note the named alternative: if the caller genuinely needs a tight synchronous reply on the request path (sub-100 ms, one caller waiting), a direct RPC (gRPC/HTTP) is the right tool — Kafka request-reply is for async, long-running, or replayable request/response, not low-latency call-and-wait.

4. Fan-Out/Fan-In — partitioned scatter-gather

Idea: split one big job into independent sub-tasks, process them in parallel, then merge the results. In Kafka the dispatcher writes sub-task records to a partitioned topic; workers in a consumer group pick them up. The fan-in side can be another Kafka consumer, a Kafka Streams aggregator, or a stateful service that collects responses keyed by the original job ID.

Config sketch

// dispatcher: one search query → N shard tasks, keyed by SHARD id so they spread
String queryId = UUID.randomUUID().toString();
for (int shard = 0; shard < shardCount; shard++) {
    ProducerRecord<String, String> task = new ProducerRecord<>(
        "search-tasks", queryId + "-" + shard, json(queryId, shard, query));
    producer.send(task);
}

// workers: same consumer group, each processes one shard
props.put("group.id", "search-workers");
consumer.subscribe(Arrays.asList("search-tasks"));
for (ConsumerRecord<String, String> r : records) {
    ShardResult result = searchShard(r.value());
    // results keyed by the ORIGINAL queryId so they converge for the aggregator
    producer.send(new ProducerRecord<>("search-results", result.queryId(), result));
}

Trace

A search request arrives with query "distributed systems." The dispatcher fans it into 12 shard tasks keyed by shard id (q7-0 … q7-11), so they spread across the partitions of search-tasks and up to 12 workers process them in parallel. Each worker writes its shard result to search-results keyed by queryId=q7, so all 12 results land in one partition, arriving in production order for the aggregator. The aggregator reads search-results, buffers results for q7 until it has 12 (or a timeout), merges them, and returns the final ranked list.

The keying is the whole pattern. Keying the tasks by queryId would collapse the scatter into a single partition — same key, same partition, one consumer-group member owning it — so one worker processes all 12 tasks serially while eleven idle: zero parallelism. Spread the tasks (shard-id key, or an explicit null key so the partitioner distributes unkeyed records across partitions) and converge only the results on the job id. The trade-off is stated order: spread tasks have no cross-partition ordering, which is fine here because shard tasks are independent by construction.

Kafka gotcha: partial results need a timeout

If one worker is slow or crashes, the aggregator waits forever unless you cap it. Typical choices: return partial results after 95% of shards respond, or fail the whole query after a strict deadline. Also, because Kafka consumers process whole partitions, a poison shard task that crashes every worker will stall the entire partition until it is moved to a dead-letter topic.

5. Dead Letter Queue — a dead-letter topic

Idea: messages that cannot be processed successfully after retries are isolated so they do not block the main stream. Kafka has no broker-level DLQ, so you implement it in the consumer: catch the exception, send the bad record (with original headers and a failure reason) to a dedicated <topic>-dlq topic, and commit the original offset so the consumer can move on.

Config sketch

props.put("max.poll.records", "1");     // easier to reason about per-record retries

for (ConsumerRecord<String, String> r : records) {
    try {
        // retries are APPLICATION logic — Kafka has no consumer config for them
        for (int attempt = 1; ; attempt++) {
            try { process(r); break; }
            catch (Exception e) {
                if (attempt == 3) throw new PoisonMessageException(r, e);
                Thread.sleep(1000L * attempt);  // backoff before the next attempt
            }
        }
        consumer.commitSync();
    } catch (PoisonMessageException e) {
        ProducerRecord<String, String> dlq = new ProducerRecord<>(
            "orders-dlq", r.key(), r.value());
        dlq.headers().add("original-topic", r.topic().getBytes(StandardCharsets.UTF_8));
        dlq.headers().add("original-partition", String.valueOf(r.partition()).getBytes());
        dlq.headers().add("original-offset", String.valueOf(r.offset()).getBytes());
        dlq.headers().add("failure-reason", e.getMessage().getBytes(StandardCharsets.UTF_8));
        dlq.headers().add("failed-at", Instant.now().toString().getBytes());
        dlqProducer.send(dlq);
        consumer.commitSync(); // commit so the consumer advances past the poison record
    }
}

No config will do this for you: Kafka has no consumer config for application-level processing retries. retry.backoff.ms is real, but it paces the client's own broker-request retries (fetches, offset commits) — it never re-runs an exception thrown by your handler. If a record should be attempted three times before dead-lettering, that retry loop is your code, as above.

Trace

A record in orders has malformed JSON. The consumer tries to parse it three times, fails each time, and produces it to orders-dlq with headers describing the original partition/offset and the parse error. The main consumer commits its offset and continues. An alert fires because orders-dlq is non-empty; an engineer inspects the record, fixes the producer, and replays or deletes the DLQ record.

Kafka gotcha: ordering vs. DLQ

Committing the offset of a poison record and moving it to the DLQ means later records in the same partition are processed before the bad one is fixed. If order matters, the DLQ pattern alone is not enough: you need to block the partition until the bad record is resolved, or design your semantics so that later records do not depend on the failed one. Also, never catch Throwable and blindly send to DLQ — a transient database outage would dump thousands of good records into the DLQ.

6. Transactional Outbox — killing the dual-write

Idea: the hardest reliability bug in event-driven systems is the dual write — a service that must both commit a row to its database and publish an event to Kafka. Done as two independent operations, a crash between them corrupts state: publish-then-commit can emit an event for a transaction that later rolls back (a phantom event); commit-then-publish can lose the event entirely if the producer call fails after the DB has already committed. Neither ordering is safe, because the database and the broker share no transaction.

The outbox pattern removes the dual write. In the same local database transaction that writes the business row, the service also inserts a row into an outbox table (event type, payload, aggregate id). Because both writes are in one transaction, they commit or roll back together — there is no window where one exists without the other. A separate relay — a polling publisher, or a change-data-capture connector such as Debezium tailing the DB log — reads new outbox rows and publishes them to Kafka, marking each as sent.

Kafka gotcha: the outbox gives atomicity, not exactly-once

Teams adopt the outbox expecting "exactly-once" and are surprised by duplicates. The relay is at-least-once: if it crashes after publishing a row but before marking it sent, it republishes on restart. The outbox guarantees atomicity of the write and no lost or phantom events — it does not make delivery exactly-once. Pair it with idempotent consumers (dedup on the outbox row / event id), the same harmless-duplicate discipline every Kafka pipeline needs. When a workflow spans several services rather than one write, this composes into a saga (each step emits an event that triggers the next, with compensating actions on failure) — choreography via events for loose coupling, or a central orchestrator when the flow is complex enough to need one place to reason about it.

Decision table: which Kafka primitive for which pattern?

Putting it together: a single event through three patterns

Consider user-registered again. One record can flow through all three read-side patterns at once:

1. Pub/sub: analytics, welcome-email, and fraud groups each read the event independently.
2. Point-to-point within a group: inside the fraud group, exactly one of three fraud workers processes the event.
3. DLQ: if the welcome-email worker cannot parse the event after retries, it sends the record to user-registered-dlq and continues.

The producer did not change; the topic did not change. The only difference between pub/sub and point-to-point is whether consumers share a group.id. That is the central insight: Kafka is not a queue broker and not a pub/sub broker — it is a distributed log, and the same log supports both semantics depending on how consumers organize themselves.

Interview traps

• "Kafka is pub/sub only." False. A single consumer group gives you point-to-point load balancing.
• "More consumers always mean more throughput." Only up to the partition count; beyond that, consumers idle.
• "Kafka has a built-in DLQ." Not at the broker level. You build it in the consumer, and the commit/ordering semantics are your responsibility.
• "A consumer group gives you exactly-once for free." False. Kafka gives at-least-once by default. Exactly-once processing inside a Kafka pipeline needs idempotent producers, transactions, and careful consumer-side idempotency for any external side effects — a consumer group alone does not give exactly-once delivery over the network.

Interactive scenario

Play with the kafka simulator and predict the outcome before each step.