Kafka Internals — Partitions, Offsets & Consumer Groups
How Kafka stores and delivers
Kafka is a distributed append-only commit log. Three ideas explain its scaling and ordering:
- Partitions: a topic is split into independent ordered logs — the unit of parallelism AND ordering. Each message gets a monotonic offset.
- Ordering is per-partition only. Send related messages with the same key (e.g.
userId) → same partition → ordered. - Consumer groups: each partition goes to exactly one consumer in the group → add consumers to scale, up to #partitions. Each group tracks its own committed offset and can replay from any offset.
- Replication: each partition has a leader + in-sync follower replicas; a follower takes over if the leader dies.
| Kafka (log) | RabbitMQ (queue) | |
|---|---|---|
| Model | Pull by offset; retained | Push; deleted after ack |
| Replay | Yes (re-read offset) | No |
| Best for | Streaming, high volume, event sourcing | Task queues, complex routing |
Takeaways
- Partition = ordered log + parallelism unit; offset = position; order is per-partition (use a key).
- Consumer group: one consumer per partition; each group has its own offset and can replay.
Re-authored from-scratch; diagram hand-authored (SVG) for this guide.
🤖 Don't fully get this? Learn it with Claude
Stuck on Kafka Internals — Partitions, Offsets & Consumer Groups? Open Claude, copy a block below, and it'll teach you this exact concept — visually and interactively.
🎨 Explain it visually
Build the mental picture, not memorization.
I just read a lesson on **Kafka Internals — Partitions, Offsets & Consumer Groups** (System Design) and want to truly understand it. Explain Kafka Internals — Partitions, Offsets & Consumer Groups from first principles using ONE vivid real-world analogy and a visual mental model — draw it as ASCII art or a clear step-by-step diagram — with a concrete example using real numbers. Then ask me one question to check I got the mental picture, and wait for my reply. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.
🤔 Walk me through it (interactive)
Socratic — adapts to where you're stuck.
Teach me **Kafka Internals — Partitions, Offsets & Consumer Groups** interactively. Ask me ONE guiding question at a time, wait for my answer, and adapt to my confusion — build the idea with me step by step instead of explaining it all at once. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.
🧪 Quiz me & fix my gaps
Active recall exposes what you missed.
Quiz me on **Kafka Internals — Partitions, Offsets & Consumer Groups** with 5 questions, easy to tricky, ONE at a time. Tell me if each answer is right; at the end, explain clearly what I got wrong and why. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.
🧠 Make it stick
Intuition + hook + flashcards for long-term memory.
Help me remember **Kafka Internals — Partitions, Offsets & Consumer Groups** for the long term: give the one-sentence intuition, a memorable hook/mnemonic, a tiny worked example, and 3 active-recall flashcards (Q -> A). If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.