Heartbeat

A heartbeat lets a monitor infer that a process is alive not by asking it, but by requiring it to keep proving liveness — the process emits a tiny "I'm still here" message on a fixed interval, and the monitor concludes failure only from the absence of those messages past a timeout, because over an asynchronous network you can never positively distinguish a crashed node from a slow or partitioned one — you can only wait and give up.

That single fact — you detect failure by waiting, not by observing it — drives every design decision. Two knobs control the mechanism: the interval Δt (how often a beat is sent, e.g. 1 s) and the timeout T (how long silence is tolerated, usually a small multiple of Δt, e.g. 3·Δt). Shrink them and you detect crashes faster but wrongly accuse healthy-but-slow nodes; grow them and you stop the false accusations but a truly dead node keeps receiving traffic longer. This is the core detection-time vs. accuracy trade-off, formalised by Chandra & Toueg as completeness (every real crash is eventually detected) versus accuracy (no live node is wrongly suspected) — no timeout-based detector can maximise both.

Two topologies exist. Centralized: every node beats to one monitor (simple, but the monitor is a bottleneck and single point of failure). Kubernetes is the canonical centralized example: every kubelet heartbeats a Lease object (in the kube-node-lease namespace, renewed every ~10 s) to the API server, and a central node controller convicts the node after a grace period — the monitor bottleneck is mitigated by making it an HA control plane rather than by gossip. Peer / gossip: each node beats to a random subset of peers and suspicion spreads epidemically (no central bottleneck, scales to thousands, but membership state is only eventually consistent). Cassandra, Akka Cluster and Consul (via Serf/SWIM) use variants of the peer model.

Worked trace: one node, one monitor

Monitor M watches node S. Δt = 1 s, timeout T = 3 s (declare dead after 3 s of silence). S beats at t = 0,1,2,3,4 then crashes at t ≈ 4.4 s. M keeps a lastSeen timestamp and, on its own clock, checks now − lastSeen > T.

Note the two costs baked in: detection latency is ≈ 2.6 s (crash at 4.4, detected at 7.0), and traffic sent to S between 4.4 and 7.0 is silently lost or must be retried elsewhere.

Why the naive version is wrong. A tempting simplification is "missed one beat ⇒ dead" (T = Δt). But real inter-arrival times jitter from network queueing, TCP retransmits and scheduler stalls. A single delayed packet then evicts a perfectly healthy node, triggering a needless failover — and if the node is a shard leader, a spurious leader election. That is why T is always a multiple of Δt, and why the better answer below stops using a hard threshold at all.

The GC-pause false positive, and the φ-accrual fix

The classic killer of fixed timeouts: a JVM (or Go runtime) stop-the-world garbage-collection pause of 2–4 s freezes everything, including the heartbeat sender thread. The node is perfectly healthy — it just couldn't emit a beat — yet a 3 s timeout convicts it. Under load these pauses cluster, so the whole cluster can start flapping (nodes marked down, marked up, rebalancing, marked down again). A hard binary threshold has no way to say "this silence is a bit unusual but not yet alarming."

The φ-accrual failure detector (Hayashibara et al., 2004; used by Cassandra and Akka) replaces the yes/no timeout with a continuously rising suspicion level φ. It records the recent history of inter-arrival times, fits a distribution (mean μ, std σ), and reports

φ(now) = −log₁₀( P(next beat arrives later than the elapsed silence) )

φ = 1 means ~10% chance the silence is normal; φ = 8 means ~10⁻⁸. The application picks a threshold (Cassandra's default phi_convict_threshold = 8) and — crucially — φ self-tunes: on a jittery WAN link where beats are naturally late, μ and σ grow, so a 2 s gap yields a low φ and no conviction; on a rock-steady LAN the same 2 s gap yields a high φ and fast conviction. Same code, adapts to each link.

Worked numbers, μ = 1000 ms, σ = 200 ms (modelled normal):

Pitfalls

• Timeout tuned tighter than your worst GC/scheduler pause. A 3 s timeout with 4 s STW pauses guarantees false positives under load. Rule of thumb: T ≥ (max realistic pause) + a few RTTs, or use φ-accrual and let it adapt.
• The heartbeat succeeds while the real work is broken. A lightweight liveness ping keeps flowing from a dedicated thread even though the request-serving thread pool is deadlocked or the disk is full. Prefer a shallow health check that touches the real path (accept a connection, touch the DB) over a bare "process is running" beat — the distinction between liveness and readiness (Kubernetes) exists precisely for this.
• Symmetric partition ⇒ both sides declare the other dead. If A and B can't hear each other, each may promote itself. Heartbeats alone give you failure suspicion, not agreement — you still need a quorum/fencing token (ZooKeeper, Raft) before acting, or you get split-brain and double writes.
• Sender and monitor clocks / the sender's own timer drift. Compare elapsed time on the monitor's single clock (now − lastSeen); never trust a timestamp inside the beat across machines without NTP discipline.
• Beat storms at scale. N nodes all beating a central monitor is O(N) load on one box and O(N²) in naive all-to-all peer mode. Gossip/SWIM bounds this to O(1) messages per node per period.
• Cascading rebalances. One false conviction of a data node can trigger re-replication of terabytes, whose extra load slows other nodes into their timeouts. Add hysteresis / a suspicion state before hard conviction.

When to use it — and when to reach for something else

Reach for a heartbeat when nodes need continuous, low-latency liveness signals about each other — leader/follower health, shard-owner tracking, load-balancer backend pools, session keep-alive. Concrete signal: you must detect a dead peer within seconds and take automated action (reroute, fail over, re-replicate), and you control both ends of the connection.

Binary timeout vs. φ-accrual. Binary is trivial to implement and reason about — choose it on a controlled LAN with predictable latency and tolerant timeouts. Choose φ-accrual when link quality varies (WAN, cloud AZs, GC-heavy runtimes) and false positives are expensive; you pay a little history-tracking state and lose the crisp "dead at exactly T" guarantee in exchange for adaptivity.

Centralized vs. gossip/SWIM. Centralized monitoring is fine up to a few dozen nodes and when you already have a coordinator. Past that, prefer gossip-style peer heartbeats (SWIM: ping + indirect ping via k relays) — it removes the single monitor, spreads load evenly, and scales to thousands, at the cost of only eventually consistent membership and harder debugging.

Named alternatives and the trade-off

• ZooKeeper / etcd ephemeral nodes (session + lease). Instead of you polling, the coordinator holds a session with its own heartbeat; when it lapses, the ephemeral znode vanishes and watchers fire. Gain: failure detection plus consensus and fencing in one primitive — no split-brain. Cost: a whole coordination service to run, extra hop of latency, and everyone now depends on ZK's availability. Choose this when a failure must trigger a coordinated decision (leader election, lock release), not just a local reroute.
• TCP keepalive. Free, built into the socket. Cost: OS defaults are ~2 hours and coarse; it detects a dead connection, not a dead application (a hung process holds the socket open). Fine as a backstop; useless for second-scale application-level failure detection.
• Request-driven / lazy detection (detect failure only when a call fails, e.g. circuit breakers). Gain: zero background traffic, no idle cost. Cost: you learn of the failure only at the moment a user request hits the dead node — too late for proactive rerouting. Pair a circuit breaker with heartbeats; don't replace one with the other.

Choose a heartbeat when you need proactive, sub-10-second liveness and you own the endpoints; prefer ZooKeeper/etcd leases when the detection must drive cluster-wide agreement; prefer gossip/SWIM when N is large; fall back to TCP keepalive / circuit breakers when failures are rare and reactive detection is acceptable.

Takeaways

• Heartbeats detect failure by the absence of expected messages — in an async system that is the only signal available, so detection is always "wait then give up," never instantaneous.
• The interval/timeout pair is a direct detection-time vs. accuracy dial: faster detection buys more false positives, and no timeout detector escapes this (Chandra–Toueg completeness vs. accuracy).
• Fixed timeouts break under GC/scheduler pauses and variable-latency links; φ-accrual replaces the hard threshold with an adaptive suspicion level and is the production default (Cassandra, Akka).
• Heartbeats give suspicion, not agreement: combine with quorum/fencing (ZooKeeper, Raft) before acting, or a symmetric partition turns into split-brain.

Re-authored / deepened for this guide. Sources: Grokking the System Design Interview (Design Gurus) heartbeat pattern; Naohiro Hayashibara et al., "The φ Accrual Failure Detector" (SRDS 2004) and Apache Cassandra's FailureDetector / phi_convict_threshold; Tushar Chandra & Sam Toueg, "Unreliable Failure Detectors for Reliable Distributed Systems" (JACM 1996) for completeness vs. accuracy; Das, Gupta & Motivala, "SWIM: Scalable Weakly-consistent Infection-style Membership" (DSN 2002); Wei Chen et al. on the quality of service of failure detectors; and Kubernetes liveness/readiness probe semantics.