hard Kafka Durability Under Broker Failure
Durability is not "Kafka" — it is three knobs agreeing with each other
A produced record survives a broker crash only if enough replicas had it before the crash, and Kafka gives you exactly three levers that decide "enough": acks (how many replicas the producer waits for), min.insync.replicas (the floor on how many must be reachable for a write to be accepted at all), and unclean.leader.election.enable (what happens when that floor is breached anyway). Get all three right together and you can lose a broker with zero data loss. Get one wrong and the other two don't save you.
The replication unit is the partition. Each partition has one leader (all reads/writes go through it) and N–1 followers that pull the leader's log and replay it. The ISR (in-sync replica set) is the subset of replicas — leader included — that have fully caught up within replica.lag.time.max.ms; a follower that falls behind is temporarily dropped from the ISR until it catches back up.
- acks=0 — fire-and-forget. The producer doesn't even wait for a response. Fastest, but a network blip, a leader crash, or a rejected request all look identical: silence. Any of them can lose data with the producer none the wiser.
- acks=1 — the leader acks after writing to its own log, before followers have replicated. If B1 crashes in the gap between "wrote locally" and "B2/B3 caught up," the client was told the write succeeded, but it never left B1 — gone.
- acks=all (a.k.a.
acks=-1) — the leader acks only after every current ISR member has the record. As long as at least one ISR member survives a crash, the record survives with it. This is the durability setting; it costs the extra round trip to the slowest ISR member.
Notice acks=all promises durability relative to whatever the ISR happens to be right now. If the ISR has shrunk to one replica, acks=all is only as strong as that one replica. That is exactly the gap min.insync.replicas closes:
min.insync.replicas is a floor, not a wait-count. With acks=all, if the ISR's size is below this floor when a produce request arrives, the broker doesn't accept a weaker write — it rejects the request outright withNotEnoughReplicas(orNotEnoughReplicasAfterAppend). Produce calls fail; existing data is untouched.
And unclean.leader.election.enable answers the last question: what if the leader dies and no in-sync replica is left to promote — only a stale, out-of-sync one? Default false: the partition goes offline rather than promote a replica that's missing committed records — safety over availability. Set to true: Kafka promotes the stale replica anyway so the partition stays writable, but every record the stale replica never received is permanently gone, and any conflicting entries on other replicas get truncated to match the new leader.
Traced: RF=3, min.insync.replicas=2, acks=all, losing brokers one at a time
| Step | Event | ISR after | Producer-visible outcome |
|---|---|---|---|
| t0 | Steady state: B1 leader, B2/B3 followers, all caught up | {B1,B2,B3} | acks=all writes commit on all 3 → ACK |
| t1 | B1 (leader) crashes | {B2,B3} | Controller promotes B2 (it was in-sync, so it already has every acked record) — no data lost, brief write pause during election |
| t2 | B3 also goes down | {B2} | ISR size 1 < min.insync.replicas=2 → new produce requests get NotEnoughReplicas — writes blocked, nothing lost (B2 still safely holds everything already committed) |
| t3 | B2 dies too; only stale B1 (which missed B2/B3's last records) rejoins first | {} then {B1} if unclean=true | unclean=false (default): partition stays OFFLINE, zero loss, unavailable until an in-sync replica returns. unclean=true: B1 is elected anyway — partition is writable again but the un-replicated committed tail is permanently lost. |
Pitfalls
- "acks=all" alone is not a durability guarantee — it's only as strong as the ISR at that instant. Pair it with
min.insync.replicas > 1, or a shrunk ISR can silently make acks=all behave like acks=1. - Setting min.insync.replicas = RF (e.g. 3 of 3) means any single broker outage — even a planned rolling restart — blocks all writes to that partition. That's usually too strict; RF=3, min.insync=2 is the common sweet spot (tolerates exactly one broker down).
- NotEnoughReplicas looks like an outage but isn't data loss — treat it as "the cluster is protecting you," not a bug to silently retry into a lower acks setting.
- unclean.leader.election.enable=true is a standing risk, not a one-time decision — it silently trades correctness for availability on every future failure, not just the one you were thinking about when you flipped it.
- Replication ships mistakes too. A bad write with acks=all replicates to every ISR member just as faithfully as a good one — this durability model has nothing to say about application-level corruption.
When to use each setting — and the trade-off
- acks=all + min.insync.replicas=2 (RF=3) — the standard choice for anything you can't afford to lose: financial events, orders, audit logs. Costs one extra network round-trip per write versus acks=1.
- acks=1 — metrics, logs, telemetry: losing the last few records on a rare leader crash is acceptable; you want lower write latency and higher throughput.
- acks=0 — only for genuinely disposable data (best-effort counters) where even knowing about a failure isn't worth the round trip.
- unclean.leader.election.enable=true — reach for it only when the topic can tolerate silent data loss in exchange for staying writable during a correlated multi-broker outage (e.g. a non-critical internal event bus). Leave it false everywhere durability matters.
Trade-off vs. a named alternative: majority-quorum replication (Raft / Multi-Paxos)
Kafka's ISR is not a majority quorum, and that's the key thing to carry into an interview. A Raft-style log (as used by etcd, CockroachDB's ranges, or a Kafka topic running under KRaft's own metadata log) always requires a strict majority of nodes to agree before electing a leader or committing an entry — with 3 nodes, that's fixed at 2, and a minority can never take over, unclean or not. Kafka's ISR, by contrast, can legitimately shrink to a single member (as in the t2/t3 trace above) and still be considered "in sync" — the safety net at that point is entirely the min.insync.replicas floor and the unclean.leader.election.enable flag, both of which are operator-configurable escape hatches. That flexibility is why Kafka can keep serving reads/writes in more partial-failure shapes than a strict-majority system would allow — but it also means Kafka can be configured (via unclean=true) into states a majority-quorum system structurally forbids. Choose Kafka's model when you want that tunable durability/availability dial per topic; choose a majority-quorum store when you need "never elect a leader without proof of the latest committed data" as a hard, un-overridable invariant.
Takeaways
- acks sets how far a write must travel before the client hears "OK"; min.insync.replicas sets the floor on the ISR size acks=all is allowed to trust; unclean.leader.election.enable decides safety (offline) vs availability (data loss) when that floor is breached anyway.
- RF=3 with min.insync.replicas=2 tolerates exactly one broker failure with zero data loss under acks=all; losing a second broker blocks writes (not loses them) until the ISR recovers.
- Data is only actually lost when a replica that never had the record becomes leader — which requires both the ISR being wiped out and unclean election being enabled.
- Kafka's ISR is a tunable durability dial, not a majority quorum — that's more flexible than Raft/Paxos but puts the CP-vs-AP choice explicitly in your config file.
Sources: Apache Kafka documentation — replication, acks, min.insync.replicas, and unclean.leader.election.enable broker/producer configs; Confluent's Kafka replication & durability guides; Neha Narkhede/Gwen Shapira/Todd Palino, Kafka: The Definitive Guide, ch. 6 (Reliability). Re-authored for this guide; both diagrams hand-authored as SVG. See also: Kafka Internals (partitions & consumer groups), What is Replication, The Consistency Spectrum.
🤖 Don't fully get this? Learn it with Claude
Stuck on Kafka Durability Under Broker Failure? Open Claude, copy a block below, and it'll teach you this exact concept — visually and interactively.
Build the mental picture, not memorization.
I just read a lesson on **Kafka Durability Under Broker Failure** (System Design) and want to truly understand it. Explain Kafka Durability Under Broker Failure from first principles using ONE vivid real-world analogy and a visual mental model — draw it as ASCII art or a clear step-by-step diagram — with a concrete example using real numbers. Then ask me one question to check I got the mental picture, and wait for my reply. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.
Socratic — adapts to where you're stuck.
Teach me **Kafka Durability Under Broker Failure** interactively. Ask me ONE guiding question at a time, wait for my answer, and adapt to my confusion — build the idea with me step by step instead of explaining it all at once. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.
Active recall exposes what you missed.
Quiz me on **Kafka Durability Under Broker Failure** with 5 questions, easy to tricky, ONE at a time. Tell me if each answer is right; at the end, explain clearly what I got wrong and why. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.
Intuition + hook + flashcards for long-term memory.
Help me remember **Kafka Durability Under Broker Failure** for the long term: give the one-sentence intuition, a memorable hook/mnemonic, a tiny worked example, and 3 active-recall flashcards (Q -> A). If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.