hard Kafka Partitions: parallelism & ordering

A topic isn't one log — it's N independent ordered logs, and that split is the whole mechanism

A Kafka topic is physically split into partitions. Each partition is its own ordered, append-only log: records are appended at the tail and each gets a monotonically increasing offset that never changes once assigned. Splitting a topic this way gives partitions two jobs at once: they are the unit of parallelism — within one consumer group, exactly one consumer reads a given partition at a time — and the unit of ordering — Kafka guarantees order only within a single partition, never across the topic as a whole. Everything else about partitions (replication, placement, sizing) follows from those two jobs.

How a record picks its partition

The producer's partitioner decides, per record:

• With a key (e.g. userId): the default partitioner hashes the key with murmur2 and computes hash(key) % numPartitions. Because the hash is deterministic, every record with the same key always lands on the same partition — which is what gives you ordering for that key. This is the mechanism traced in the diagram above: user-42 always routes to partition 2.
• Without a key: since Kafka 2.4 (KIP-480) the default partitioner is sticky — it fills one partition's batch at a time and periodically switches, which produces bigger, more efficient batches than the older strict round-robin behavior. Either way, un-keyed records get no cross-partition ordering guarantee — they are deliberately spread for load balance, not correctness.

Traced: scaling the consumer group against 6 fixed partitions

Topic orders has 6 partitions and one consumer group, orders-svc. Kafka's group coordinator enforces a simple rule: each partition is assigned to exactly one consumer in the group at a time.

Consumers in group	Assignment	Effect
3	2 partitions each (C1, C2, C3)	Balanced — every consumer pulls its weight
4	2, 2, 1, 1	6 doesn't divide evenly by 4 — some consumers do half the work of others
7	6 consumers get 1 each, 1 gets none	The 7th consumer is provisioned but permanently idle

Adding or removing a group member triggers a rebalance — the coordinator reassigns partitions (briefly pausing the ones that move) using an assignor (range, round-robin, or the incremental cooperative-sticky assignor, which avoids the older stop-the-world pause). No assignor can invent partitions that don't exist — it can only redistribute what's there, which is why consumer #7 above is unavoidably idle.

Pitfalls

• Too few partitions caps parallelism permanently. If a topic has 6 partitions, you can never usefully run more than 6 consumers in one group, no matter how much you scale the consumer fleet — the rest sit idle (traced above).
• Too many partitions isn't free either. More partitions means more open file handles and replica connections per broker, more replication and controller-metadata traffic, longer leader-election/failover time when a broker fails (every partition it led needs a new leader), and smaller, more scattered producer batches that can raise end-to-end latency. Size partitions for the throughput and consumer scale-out you actually expect, not "as many as possible."
• Partition count isn't freely reversible. Kafka lets you increase a topic's partition count but never decrease it. Worse, increasing it changes the divisor in hash(key) % numPartitions for every key — a key that used to land on partition 2 may land somewhere else once the partition count changes, breaking the "same key, same partition" guarantee for new records going forward (old records stay where they were written). This silently breaks per-key ordering and co-location assumptions — including a Kafka Streams application's assumption that a topic is still correctly co-partitioned for a join.

Judgment layer

• Sizing partition count. Pick a partition count at least as large as the maximum number of consumers you realistically expect to scale to, with headroom — but resist wildly over-provisioning, since overhead (open files, replication fan-out, failover time) grows with total partition count across the cluster. The named alternative is a single-partition topic: it gives a true total order for the entire topic with zero key-placement decisions to make, at the cost of capping throughput to whatever one consumer can process — multiple partitions trade away that whole-topic order for horizontal scale. Because adding partitions later reshuffles the key space, the practical rule is to size generously up front rather than grow into it.
• Choosing the partition key. A high-cardinality, evenly-distributed key (e.g. userId) spreads load close to uniformly across partitions — the common, usually-correct choice. No key (sticky/round-robin) gives the best possible load balance but sacrifices all cross-message ordering, so it only fits streams where per-record order truly doesn't matter. A low-cardinality or skewed key (e.g. tenantId with one dominant customer, or country) creates a hot partition: that one partition — and the single consumer reading it — becomes the throughput ceiling no matter how many total partitions or consumers exist (see Shard Keys: many-key skew vs a single hot key, and Rate-Limit Hot Keys, for the same failure mode elsewhere). The trade-off is always ordering guarantee vs. load spread — pick the key that gives the ordering you actually need and the flattest distribution available within that constraint.

Takeaways

• A partition is an ordered, append-only log with monotonic offsets — the unit of both parallelism (one consumer per partition per group) and ordering (guaranteed only within a partition).
• Max useful consumers in a group = partition count; extra consumers are idle, and too few partitions is a hard parallelism ceiling.
• The producer's key choice is the ordering/load-balance trade-off: same key always same partition (ordered, but risks a hot partition), no key spreads evenly (balanced, but zero cross-record order).
• Changing partition count later reshuffles hash(key) % N for every key — size generously up front rather than relying on painless growth.

---

Re-authored for this guide from Apache Kafka's documentation on partitions and consumer groups (group coordinator, partition assignors), KIP-480 (Sticky Partitioner), and standard Kafka operational guidance on partition sizing. Diagrams hand-authored as SVG for this guide. See also: Kafka Internals — Partitions, Offsets & Consumer Groups; Message Ordering; Shard Keys: many-key skew vs a single hot key; Rate-Limit Hot Keys.

I just read a lesson on **Kafka Partitions: parallelism & ordering** (System Design) and want to truly understand it. Explain Kafka Partitions: parallelism & ordering from first principles using ONE vivid real-world analogy and a visual mental model — draw it as ASCII art or a clear step-by-step diagram — with a concrete example using real numbers. Then ask me one question to check I got the mental picture, and wait for my reply. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Teach me **Kafka Partitions: parallelism & ordering** interactively. Ask me ONE guiding question at a time, wait for my answer, and adapt to my confusion — build the idea with me step by step instead of explaining it all at once. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Quiz me on **Kafka Partitions: parallelism & ordering** with 5 questions, easy to tricky, ONE at a time. Tell me if each answer is right; at the end, explain clearly what I got wrong and why. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Help me remember **Kafka Partitions: parallelism & ordering** for the long term: give the one-sentence intuition, a memorable hook/mnemonic, a tiny worked example, and 3 active-recall flashcards (Q -> A). If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.