hard Health Probes: liveness, readiness & startup

The orchestrator can't see inside your process, so it asks

Kubernetes (and every serious orchestrator) can watch that a container's process is running, but it cannot tell whether that process is actually healthy or actually able to serve. So it asks the container simple questions on a fixed schedule and acts on the answers. The whole design turns on one fact: each probe triggers a different remedy. A failed liveness answer means “you are broken — restart me.” A failed readiness answer means “don't send me traffic right now” — but do not restart. A startup probe means “I'm still booting — hold the other two.” Match the remedy to the fault and the system self-heals; mismatch them and it self-destructs.

The three probes and what each one does

Probe	Question	On failure	Use it for
Liveness	Are you wedged / dead?	kubelet restarts the container	deadlock, wedged event loop, unrecoverable in-process state a restart would cure
Readiness	Can you serve traffic right now?	removed from the Service endpoints — no traffic; not restarted	warmup, cache priming, a transient not-ready state, a dependency blip
Startup	Have you finished booting?	gates liveness & readiness until it first succeeds	slow-booting apps (JVM warmup, large caches) so a long boot isn't killed

Every probe shares five knobs. initialDelaySeconds waits before the first check. periodSeconds is how often it runs. timeoutSeconds is how long a single check may take before it counts as a failure. failureThreshold is how many consecutive failures trip the action, and successThreshold is how many consecutive passes clear it (must be 1 for liveness and startup; readiness may require more to damp flapping).

Traced example: how long until a wedged pod is restarted?

Configure a liveness probe with periodSeconds: 10, failureThreshold: 3, timeoutSeconds: 1. The pod runs happily, passing a probe every 10s. Then at t=95s a deadlock wedges the request handler. Trace it:

Time	Probe	Consecutive fails	Action
t=80s, t=90s	pass	0	steady state
t=95s	—	—	app wedges (still running, just stuck)
t=100s	fail	1	wait
t=110s	fail	2	wait
t=120s	fail	3 ≥ threshold	kubelet kills & restarts the container

Detection lag ≈ periodSeconds × failureThreshold = 10 × 3 = 30s (plus up to one extra period of scheduling jitter). Tighten the knobs and you detect faster — but you also kill sooner on a false alarm, which is exactly the trap below.

Pitfalls that take down production

1. Liveness too aggressive → restart loops under load

Under a traffic spike, a healthy pod gets slow — GC pauses, saturated CPU, a full work queue. If the liveness probe is tight (short timeout, low threshold) or, worse, does real work (hits the database), a slow-but-alive pod fails it and gets restarted. Now there are fewer replicas, so the survivors get more load, get slower, fail liveness, and get restarted too. The probe designed to heal the system is now amplifying the overload into a cascading restart loop. Liveness must test only “am I fundamentally wedged,” with generous timeouts — never “am I fast.”

2. Readiness that checks a shared dependency → all replicas leave at once

Tempting: make readiness verify the database is reachable, so an unhealthy pod stops taking traffic. But every replica shares that database. When it blips for two seconds, every replica fails readiness simultaneously, so Kubernetes pulls all of them from the Service endpoints. The Service now has zero endpoints — a total outage — and because a shallow blip took out everything at once, there's no capacity left to absorb the recovery. A shared-dependency check turns a minor blip into a self-inflicted, correlated outage.

Judgment: which check goes in which probe

Liveness vs readiness is restart vs de-route, and the two remedies fix different faults. A restart only cures state inside this process — a deadlock, a leaked handle, a corrupted in-memory structure. It does nothing about an external dependency: restarting your pod will not bring the database back, it just adds churn while the real problem persists. So the rule follows mechanically:

• Liveness → only local, cheap, “is this process wedged” checks that a restart would actually cure. Never put a dependency call here — you'd restart a healthy pod because something else is down.
• Readiness → “should I get traffic right now.” This is where warmup and soft dependency awareness live — but design it so a shared dependency's blip can't zero out every replica at once.
• Startup → only to buy slow boots time. Without it you'd either set a huge initialDelaySeconds on liveness (which then also delays detecting a real wedge forever), or watch a slow-booting pod get killed mid-boot. Startup decouples “booting is slow” from “running is broken.”

Deep vs shallow health checks — the real trade-off

A shallow check asks only “is the process up and answering?” A deep check verifies downstream dependencies too. Deep checks give accurate routing — a pod that truly can't serve stops getting traffic — but they couple every replica to the dependency, so a shared-dependency blip fails them all together (pitfall 2). Shallow checks are robust and independent, but they happily route traffic to a pod that will then error on the dependency. The senior move: keep liveness strictly shallow (a restart can't fix a dependency anyway); make readiness deep enough to be useful but not brittle — don't fail readiness on a soft dependency, lean on the dependency's own circuit breaker, and prefer serving degraded (stale cache, reduced feature) over removing every replica.

Named alternative: you could skip probes and rely on the load balancer's own health checks (or a service mesh's outlier detection) to de-route bad backends. That works for de-routing but gives you no restart primitive and no boot-gating — the LB can stop sending traffic to a wedged pod, but only the orchestrator's liveness probe can recycle it. In practice you use both: probes drive restart + endpoint membership; the LB/mesh adds a second, outside-the-node layer of outlier ejection.

Takeaways

• Three probes, three remedies: liveness restarts, readiness de-routes, startup gates the other two. Match the remedy to the fault.
• Detection lag ≈ periodSeconds × failureThreshold — tighten for speed, but a tight liveness probe kills slow-but-alive pods and can cascade into a restart loop under load.
• Never put a shared/external dependency in liveness (a restart can't fix it) and be careful in readiness (one blip can pull every replica at once).
• Liveness shallow always; readiness deep only where it won't create correlated failure — prefer degraded service over zero endpoints.

---

Re-authored for this guide; probe-remedy and detection-lag diagrams hand-authored as SVG. Follows the Kubernetes documentation (liveness/readiness/startup probes) and the Google SRE Workbook (health checking, cascading failures). See also: Designing for Failure (circuit breaker, load shedding), Load Balancing (endpoint health), Rate Limiting. Deepened for this guide.

I just read a lesson on **Health Probes: liveness, readiness & startup** (System Design) and want to truly understand it. Explain Health Probes: liveness, readiness & startup from first principles using ONE vivid real-world analogy and a visual mental model — draw it as ASCII art or a clear step-by-step diagram — with a concrete example using real numbers. Then ask me one question to check I got the mental picture, and wait for my reply. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Teach me **Health Probes: liveness, readiness & startup** interactively. Ask me ONE guiding question at a time, wait for my answer, and adapt to my confusion — build the idea with me step by step instead of explaining it all at once. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Quiz me on **Health Probes: liveness, readiness & startup** with 5 questions, easy to tricky, ONE at a time. Tell me if each answer is right; at the end, explain clearly what I got wrong and why. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Help me remember **Health Probes: liveness, readiness & startup** for the long term: give the one-sentence intuition, a memorable hook/mnemonic, a tiny worked example, and 3 active-recall flashcards (Q -> A). If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.