What is Heartbeat
Background
In a distributed system, work and data are spread across many servers. For requests to be routed correctly and for failed capacity to be replaced quickly, every server (or a coordinator) needs an up-to-date view of which servers are actually alive. Waiting for a request to simply time out against a dead server is slow and wasteful — the system needs a way to notice a failure before a client feels it, so it can stop routing to the dead server and start recovery (replacing it, re-electing a leader, rebalancing data) as soon as possible.
A heartbeat is the simplest building block for that: a lightweight "I'm still here" signal, sent on a fixed schedule, whose absence is treated as evidence of failure.
How it works
Two parameters drive every heartbeat design:
- Heartbeat interval (Δ) — how often a node sends its "I'm alive" signal.
- Timeout (T) — how long a watcher waits after the last received heartbeat before declaring the sender dead.
T is always set to some multiple of Δ (commonly 3–5×) so that ordinary network jitter or a single dropped packet doesn't trigger a false failure. Who does the watching splits into two topologies:
- Central monitor — every node sends its heartbeat to one coordinator (or a small monitoring cluster). The coordinator holds the single source of truth about who is alive.
- Gossip / peer-to-peer — there is no dedicated watcher; each node heartbeats a random subset of peers every interval, and liveness information propagates peer-to-peer until the whole cluster converges on the same view.
The real ambiguity: crash, or just slow?
A missing heartbeat is not proof of death. The sender could have crashed — or it could be alive but pausing (a long GC, a CPU-starved host, a congested link, a delayed packet still in flight). Over an asynchronous network with no upper bound on message delay, no timeout value can distinguish "dead" from "arbitrarily slow" with certainty. Every heartbeat-based failure detector is therefore unreliable by construction: it can produce false positives (declaring a live-but-slow node dead), or, if tuned conservatively to avoid those, it detects real failures more slowly.
It's worth being precise about what this is not. This is not the classical Two Generals' Problem, which is about the impossibility of two parties reaching guaranteed, common agreement on a single decision over a channel that can silently lose messages — no number of acknowledgements is ever quite enough for both sides to be certain the other committed. Heartbeat ambiguity is a different, narrower problem, studied under the theory of unreliable failure detectors (Chandra & Toueg's 1996 formalization): the question isn't "did we agree?" but "is that node up?", and the fix isn't more rounds of messaging — it's accepting an explicit trade-off between detection speed and false-positive rate, tuned via T and Δ.
Worked trace
Take a node heartbeating a central monitor every Δ = 1 s, with a monitor timeout of T = 3 s measured from the last heartbeat it actually received.
- t = 0.0, 1.0, 2.0 s — heartbeats sent and received normally; the monitor's countdown resets to 3 s each time.
- t = 2.4 s — the node crashes (or pauses long enough to be indistinguishable from a crash) — after its t = 2.0 s heartbeat already landed.
- t = 3.0 s — the heartbeat due at this tick never arrives. The monitor doesn't know that yet; it's still just counting down from its last successful receipt at t = 2.0 s.
- t = 5.0 s — the monitor's countdown (2.0 + T = 2.0 + 3 = 5.0) expires with no heartbeat received. The node is declared dead and marked for replacement/rerouting.
So detection fired 5.0 − 2.4 = 2.6 s after the actual failure — noticeably less than the full T = 3 s, because the failure happened partway through an interval rather than right at the last successful heartbeat. In general, for a failure landing at a random point after the last received heartbeat, the delay from actual failure to detection ranges between T − Δ (crash lands just before the next heartbeat was due) and T (crash lands right after the last heartbeat was sent) — it is never more than T, but it can be noticeably less.
Detection time by timeout setting (Δ = 1 s)
| Timeout T | Detection latency after last received heartbeat | Detection latency after actual failure instant (range) | False-positive risk |
|---|---|---|---|
| 2 s | exactly 2 s | 1 s – 2 s | Higher — a single slow heartbeat (GC pause, packet delay) can trip it. |
| 3 s | exactly 3 s | 2 s – 3 s | Moderate — tolerates one dropped heartbeat plus some jitter. |
| 6 s | exactly 6 s | 5 s – 6 s | Low — comfortably survives multiple missed beats or a GC pause. |
The two middle columns matter for different reasons: "after last received heartbeat" is the number the monitor's timer actually implements (T, deterministic); "after actual failure instant" is what a user or SRE experiences, and it's always somewhat less than T because failures don't conveniently happen right when the timer resets.
When to use it — and when not to
Central monitor vs. gossip (choosing a heartbeat topology)
- Prefer a central monitor when the cluster is small-to-medium, you already have (or need) a coordinator for other jobs (scheduling, leader election), and you want one consistent, easily-queried view of liveness. Trade-off: the monitor is a single point of failure and a scaling bottleneck — O(n) heartbeats converge on one place.
- Prefer gossip (each node heartbeats a random subset, spreading liveness peer-to-peer) when the cluster is large, there's no natural coordinator, or you need liveness information to keep flowing even while parts of the system are partitioned. Trade-off: convergence is probabilistic and slower — it takes several gossip rounds for a failure to become globally visible, and you're trading a crisp global view for resilience and horizontal scalability.
Heartbeating vs. genuinely different liveness mechanisms
Heartbeating is not the only way to answer "is that thing alive?" — and it isn't always the right one:
- Lease / TTL-based liveness (etcd, ZooKeeper sessions) — the node holds a lease that a coordination service auto-expires if not renewed. Prefer this over raw heartbeating when the absence of a heartbeat needs to automatically and atomically release something (a lock, leadership, a registration) rather than just flip a status flag — the expiry and the side effect are one operation, avoiding a race where you've "declared dead" but the resource is still held.
- LB-level synchronous health checks (HTTP/TCP probes) — the load balancer actively polls each backend. Prefer this over heartbeating when what matters is "can this instance actually serve requests right now" (its dependencies, disk, queue depth) rather than "is the process still running" — an active probe exercises the real request path, while a heartbeat thread can keep beating on a node whose request-handling is actually wedged.
- TCP keepalive — the transport layer alone detects a dead peer connection. Prefer this (or use it as a cheap first line of defense) when all that's needed is reclaiming a broken socket/connection, not application-level liveness of a whole node or service — it's nearly free but tells you nothing about whether the process itself is healthy versus just the one connection.
- Stick with heartbeating when you need custom, application-defined liveness semantics (e.g., "alive AND caught up on replication") across a fleet you control, at a cadence and timeout you can tune — the price is that you own the crash-vs-slow trade-off explicitly, as above, rather than delegating it to infrastructure.
L0 · a heartbeat is a periodic liveness signal whose absence past a timeout is treated as evidence — never proof — of failure
L1 · ② Failure — "P99 detection latency is too high, fix it"
Trap: "Cut the timeout to 1s so we catch failures faster."
Bar: Shrinking T directly raises the false-positive rate — a GC pause or one congested hop now reads as death. The real fix replaces the fixed threshold with a phi-accrual detector: model each link's own inter-arrival distribution and emit a continuous suspicion level φ instead of a binary alive/dead flag, so sensitivity adapts per-node instead of one global timeout. phi-accrual & the cost of a false positive
L2 · ④ Time/Lifecycle — "the dead node's heartbeat reappears after you promoted a replacement"
Trap: "It was declared dead already, ignore it — the status flag flipped."
Bar: The node wasn't dead, it was suspended (GC pause) — it can resume mid-write still believing it holds leadership, and letting it write again silently corrupts state. Every promotion must bump a monotonically increasing fencing token, and storage/quorum members reject any write carrying a stale token, so the resurrected node's writes are refused even though its process is alive. split-brain, fencing & safe failover
L3 · ⑤ Adversary/Edge — "A can't reach B, but C can reach both — who's dead?"
Trap: "Whichever node declares failure first is right — trust the local view."
Bar: In an asymmetric partition A sees B as dead while B sees itself as fine — there is no single ground truth, only conflicting local views, and trusting whoever speaks first is exactly how split-brain happens. The fix is requiring quorum agreement (a majority of voting members) before any promotion or reconfiguration, so heartbeat suspicion is only an input to consensus, never the decision itself. fencing tokens, sloppy quorums & BFT consistency
L4 · ③ Scale — "cluster grew from 10 to 10,000 nodes, heartbeat traffic exploded"
Trap: "Just raise the interval / heartbeat less often to cut the traffic."
Bar: All-to-one heartbeating is O(N) fan-in on one coordinator, all-to-all is O(N²) messages — both saturate long before 10k nodes, and heartbeating less often only trades bandwidth for slower detection, it doesn't fix the topology. Gossip (SWIM-style) fixes it structurally: each round every node pings a small random subset of peers and piggybacks liveness state, holding per-node traffic to O(1) while suspicion spreads cluster-wide in O(log N) rounds. gossip / SWIM dissemination
L5 · ⑥ Cost/Simplicity — "prove your failure detector is correct"
Trap: "It's correct once T is tuned large enough — false positives go to zero."
Bar: They can't, and that's the point — over an asynchronous network with no bound on message delay, no timeout can prove "dead" rather than "arbitrarily slow" (the result underlying Chandra & Toueg's unreliable failure detectors). The honest engineering move is picking an explicit completeness/accuracy point for your SLA and routing the final decision through quorum-backed consensus, not shipping a detector that claims a certainty it structurally cannot have. why consensus is hard — FLP, quorums, Paxos vs Raft
The floor keeps dropping: now do it with Byzantine nodes that lie about their own liveness, clock skew that invalidates your interval math across data centers, or the monitor/coordinator itself needing a failure detector watching it.
Self-locate: died at L1 → mid-level; L4+ → staff signal.
Facing any new concept? Hit it with the six: concurrent? failing? at 100×? over time? adversarial? worth the cost? — that's the interviewer's whole playbook.
Sources
- Chandra, T. D., & Toueg, S. (1996). "Unreliable Failure Detectors for Reliable Distributed Systems." Journal of the ACM, 43(2).
- Gray, J. (1978), and Akkoyunlu, Ekanadham & Huber (1975) — the Two Generals' Problem: the impossibility of guaranteed common agreement over a channel that can silently lose messages (distinct from, and not to be confused with, Lamport, Shostak & Pease's 1982 Byzantine Generals Problem, which concerns malicious/arbitrary faults).
- "Grokking the System Design Interview" (Educative) — Heartbeat pattern overview.
- etcd documentation — Lease API (TTL-based liveness).
- Apache ZooKeeper documentation — Sessions and ephemeral nodes.
🤖 Don't fully get this? Learn it with Claude
Stuck on What is Heartbeat? Open Claude, copy a block below, and it'll teach you this exact concept — visually and interactively.
Build the mental picture, not memorization.
I just read a lesson on **What is Heartbeat** (System Design) and want to truly understand it. Explain What is Heartbeat from first principles using ONE vivid real-world analogy and a visual mental model — draw it as ASCII art or a clear step-by-step diagram — with a concrete example using real numbers. Then ask me one question to check I got the mental picture, and wait for my reply. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.
Socratic — adapts to where you're stuck.
Teach me **What is Heartbeat** interactively. Ask me ONE guiding question at a time, wait for my answer, and adapt to my confusion — build the idea with me step by step instead of explaining it all at once. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.
Active recall exposes what you missed.
Quiz me on **What is Heartbeat** with 5 questions, easy to tricky, ONE at a time. Tell me if each answer is right; at the end, explain clearly what I got wrong and why. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.
Intuition + hook + flashcards for long-term memory.
Help me remember **What is Heartbeat** for the long term: give the one-sentence intuition, a memorable hook/mnemonic, a tiny worked example, and 3 active-recall flashcards (Q -> A). If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.