hard Kafka Consumer-Group Rebalancing

A rebalance is the coordinator re-dealing partitions whenever the group's membership changes

Every partition of a topic must be owned by exactly one consumer within a consumer group at a time. The group coordinator (a broker) tracks membership through heartbeats each consumer sends on heartbeat.interval.ms; if a member's heartbeats stop arriving for longer than session.timeout.ms, or a member explicitly joins or leaves, the coordinator triggers a rebalance — it recomputes which consumer owns which partitions using the group's configured assignor, and hands out the new assignment. The part that actually matters for your throughput is how much processing has to pause while that recomputation happens, and that is entirely determined by which rebalance protocol the group uses.

• EAGER (the original protocol) — on any membership change, every consumer in the group revokes all of its partitions (its onPartitionsRevoked callback fires for every partition it owned), then all members rejoin. The coordinator waits for every member to check back in before it computes and hands out a brand-new full assignment. Until that round trip completes, no partition in the group is being consumed — including partitions whose ownership never actually changed.
• COOPERATIVE / incremental (via CooperativeStickyAssignor) — the coordinator first figures out the target assignment, then only revokes the partitions that actually need to move to a different consumer. Partitions staying with their current owner are never revoked and keep being processed straight through the rebalance. This typically takes two short rebalance rounds (revoke only what must move, then hand it to its new owner) instead of one full stop-the-world round.

Both protocols answer the same question — "who owns which partition now?" — but eager answers it by pausing everyone and cooperative answers it by pausing only the partitions that changed hands.

Traced: C3 joins a 2-consumer, 6-partition group

Topic has 6 partitions (P0–P5); the group starts with C1 owning {P0,P1,P2} and C2 owning {P3,P4,P5}. A new instance, C3, starts up and sends a JoinGroup request.

Rebalance storms & static membership

A rebalance storm is repeated, cascading rebalances that keep the group from ever settling — usually caused by a flapping member: a consumer whose processing occasionally exceeds max.poll.interval.ms (a separate liveness check from the heartbeat thread, introduced so a slow-but-alive consumer doesn't block the group forever) gets treated as dead and kicked out, rejoins, gets reassigned partitions, and repeats. A session.timeout.ms set too aggressively low has the same effect: normal GC pauses or brief network jitter look like a crash, triggering a rebalance that didn't need to happen. Kafka's own default was raised from 10s to 45s (KIP-735) specifically because 10s produced too many false-positive rebalances in production.

Static membership (group.instance.id set to a stable, per-instance value) fixes a different failure mode: routine rolling restarts. Without it, every restarted consumer is a brand-new dynamic member, so a rolling deploy of N consumers triggers N rebalances. With a static ID, the coordinator recognizes the returning process as the same member reconnecting within its session window and simply restores its old assignment — no rebalance fires at all for a planned bounce.

Pitfalls

• Mixing eager and cooperative assignors across a group's clients is unsafe — a rolling upgrade to CooperativeStickyAssignor must go through Kafka's documented two-phase migration; the protocols don't safely interoperate mid-rollout.
• Tuning session.timeout.ms alone doesn't fix a rebalance storm caused by slow processing — that's a max.poll.interval.ms / max.poll.records problem, since the heartbeat thread is separate from the poll thread and can keep beating while the processing loop is stuck.
• Static membership isn't automatic failure tolerance — if the instance doesn't come back within the session/registration window, the coordinator still treats it as gone and rebalances; it only suppresses rebalances for prompt, planned bounces.
• Cooperative still pauses the moving partitions — it shrinks the blast radius, it doesn't eliminate rebalance cost entirely.

When to use eager vs. cooperative — and the trade-off

• Prefer COOPERATIVE (CooperativeStickyAssignor) when the group is large, partition counts are high, or scaling/deploy events are frequent — anywhere a group-wide stall has a real latency or lag-buildup cost. It costs an extra rebalance round trip (revoke-what-moves, then assign) versus eager's single round, and a mandatory two-phase rollout if migrating an existing group.
• EAGER is acceptable for small, stable groups where rebalances are rare and brief enough that a full stop-the-world pause is a non-issue — and it remains the simpler mental model with one rebalance round instead of two.
• Static membership is not a substitute for choosing an assignor — it targets a different cause (planned restarts triggering unnecessary rebalances at all) and combines with either protocol. Use it whenever consumers restart routinely (rolling deploys, autoscaling that recycles instances with the same identity).

Session / heartbeat tuning: fast failure detection vs. false rebalances

Lowering session.timeout.ms detects a genuinely crashed consumer sooner — its partitions sit un-consumed for less time, which matters when consumer lag has a latency SLA. But push it too low and ordinary jitter (a GC pause, a slow disk flush, a momentary network blip) crosses the threshold and fires an unnecessary rebalance, which now stalls the whole group under eager, or at least the moving partitions under cooperative — for a failure that wasn't real. Raising the timeout trades slower failover for fewer false positives. There is no value that is simply "correct"; it is set against how tolerant the workload is of (a) a slow real failover vs (b) an occasional unnecessary pause, which is exactly why Kafka's own shipped default moved from 10s to 45s once the false-positive cost became clear in practice.

Takeaways

• A rebalance fires on any group membership change — join, explicit leave, or heartbeats missing past session.timeout.ms — and the coordinator recomputes partition ownership.
• Eager revokes every partition group-wide before reassigning (full stall); cooperative revokes only the partitions that must move (scoped stall).
• Rebalance storms usually trace back to max.poll.interval.ms (slow processing) or too-tight session.timeout.ms (false failure detection) — diagnose which one before tuning either.
• Static membership (group.instance.id) eliminates rebalances on planned restarts entirely — a different, complementary fix to picking an assignor.

Sources: Apache Kafka documentation — consumer group protocol, session.timeout.ms/heartbeat.interval.ms/max.poll.interval.ms, ConsumerPartitionAssignor, and static membership (group.instance.id, KIP-345); KIP-429 (Incremental Cooperative Rebalancing); KIP-735 (session timeout default change); Neha Narkhede/Gwen Shapira/Todd Palino, Kafka: The Definitive Guide, ch. 4 (Consumers). Re-authored for this guide; both diagrams hand-authored as SVG. See also: Kafka Internals (partitions & consumer groups), Kafka Durability Under Broker Failure.