Service Discovery & Distributed Circuit Breakers — Detection Lag, CAP Stance, Registry Herd & Per-Instance vs Coordinated State (Deep Dive)

Why detection lag and distributed breaker state get their own page

A service registry and a circuit breaker both exist to answer the same question — "is this dependency still good to call, right now?" — but neither answer is instant, and neither is shared. The registry's answer is stale by exactly the sum of its lease timers plus your client's cache window: a number you can derive, not guess. The breaker's answer is worse than stale — it is local: N different callers each form their own independent opinion, with no coordination between them, so "the circuit is open" is really N circuits, open or closed on N different schedules. This page derives both delays with real numbers and shows what a senior engineer changes because of them.

The three-state breaker itself (Closed → Open → Half-Open) is covered in "Circuit Breaker — The State Machine (Closed → Open → Half-Open)" — read that first if the state machine is new to you; this page assumes it and goes distributed. Likewise, registration, discovery styles, and the basic staleness window are covered in "What is Service Discovery Pattern" and its worked example — this page owns the exact arithmetic of that staleness, the registry's own CAP choice, the herd problem discovery creates at scale, and how a fleet of circuit breakers actually behaves once it is more than one instance.

1. Detection lag: how long does a dead address keep getting called?

A caller only stops dialing a dead instance once every layer between it and the truth has caught up: the registry must decide the lease is gone, its eviction process must actually act on that decision, and the caller's own cache must refresh. Each layer runs on its own clock, and the delays add, they don't overlap — that's the whole mechanism.

Take Eureka's real defaults, the most commonly interviewed case:

• Client renewal interval — 30s. Each instance re-sends its heartbeat (renews its lease) every 30 seconds.
• Server lease duration — 90s. The registry only considers a lease expired after 90 seconds of silence — i.e. it tolerates up to 2 missed renewals (a GC pause, a blip) before calling an instance dead. Worst case: the instance dies immediately after a successful renew, so the full 90s elapses before the registry even notices.
• Eviction sweep period — 60s. Marking a lease "expired" and actually removing it from the served instance list are two different steps. Eureka's eviction task itself only runs every 60 seconds, so an already-expired lease can sit in the registry for up to another 60s before a sweep purges it.
• Client registry cache — 30s. Each caller doesn't hit the registry per request; it polls/refreshes its local copy every 30 seconds. Even the instant the registry drops the dead entry, a caller can be looking at a copy that is up to 30s old.

Add the worst case of each stage — they compound, they do not overlap:

Worst case ≈ 90 + 60 + 30 = 180 seconds — a full three minutes before the last caller in the fleet has forgotten a dead address, purely from registry mechanics. Even the typical case — averaging out where in each window the death happened to land — is still on the order of a couple of minutes, because the three delays are independent and rarely all land at their best case simultaneously.

That number is the whole argument for defense in depth: no registry tuning alone gets this under a second. Shrinking the TTLs trades lag for false evictions (a slow GC pause now looks like death) and for heartbeat load on the registry (see §3). Production systems instead bridge the gap at the request layer — independent of the registry's own clock: a client-side connect/read timeout well under a user-visible SLA, an immediate retry against a different instance on failure, and a per-instance circuit breaker that stops routing to a specific dead address within one or two failed calls, long before its lease would ever expire. The registry's job is eventual truth; the breaker's job is to not wait for it.

2. AP or CP: the registry has to pick a side of CAP

During a network partition, a registry node that can't reach its peers has exactly two honest options: keep answering with what it has (possibly stale or incomplete), or refuse to answer until it can prove its data is current. That single decision is the registry's CAP stance, and different registries commit to it deliberately, not by accident.

The intuition an interviewer wants: for pure service discovery, a slightly stale instance list is nearly always the safer failure — the caller retries a dead address once and moves on. A registry that goes silent during exactly the kind of network event that also makes services flaky is often the worse outcome, because now nobody can find anybody. That's why Eureka's designers explicitly chose AP for a system whose only consumer is "who do I call next." Consul/etcd/ZooKeeper earn their CP stance when the same coordination service is also being trusted for things where two different answers is a correctness bug, not an inconvenience.

3. Registry-side thundering herd: poll vs watch

The registry itself is a shared, highly-read service, and its load is driven by one arithmetic fact: registry QPS ≈ (number of clients) × (1 / refresh interval). With 2,000 caller instances each polling every 30s, that's already 2000 / 30 ≈ 67 QPS of steady-state reads — fine on its own, but it is the steady state that is not the danger.

Two designs answer "how does a client learn the list changed":

• Periodic poll (Eureka's default): every client re-fetches on a timer regardless of whether anything changed. Simple, stateless on the server, but load scales linearly with fleet size and is wasted almost all the time — membership rarely changes every 30s.
• Watch / long-poll (Consul/etcd/ZooKeeper watches): the client holds an open request; the server only responds when the data actually changes (push-on-change). Far less wasted traffic in steady state, but now the registry must hold open connections/state per watcher, and a change fans out to every watcher at once — a different kind of spike.

The real herd shows up at the edges, not in steady state: a mass client restart (a rolling deploy, a datacenter failover, every pod restarting after a bad config push) means thousands of clients re-registering and re-fetching in the same few seconds — the registry sees a burst many multiples of its steady-state QPS, right when it's least equipped to absorb it (the same event probably also stressed the network). A watch-based registry has the mirror problem: one change (say, a registry node itself flapping) can fire every open watch simultaneously, producing the identical spike from the other direction.

The fix in both directions is the same: never let every client wake up on the same clock. Add jitter (randomize each client's refresh/reconnect interval by, say, ±20–30%) so a synchronized restart de-synchronizes within a couple of cycles, and use exponential backoff on reconnect/re-register failures so a struggling registry isn't hit harder by the very clients whose requests are already failing. This is the identical damping logic used for retries against any overloaded dependency — the registry is not special, it's just usually the first thing to feel a mass-restart storm because every single instance depends on it to start serving traffic at all.

4. Circuit-breaker state is per-instance — by default, N breakers, not one

A circuit breaker's Closed/Open/Half-Open state normally lives in the process memory of the caller that created it. If orders runs 50 instances and each one calls payments, that is 50 separate breakers, each keeping its own failure count, each deciding independently when to trip and when to try recovery. Nothing shares that state unless you explicitly build a shared store for it.

Traced scenario. payments goes down; all 50 orders breakers see failures and trip to Open within seconds of each other — that part is fine, they're reacting to the same real signal. payments recovers at t=0. Every one of those 50 breakers is running its own independent Open-duration timer (say, 30s, started at slightly different moments as each one tripped). As each timer expires, that instance moves to Half-Open and sends its one trial probe — with no idea any other caller is doing the same thing. If enough of those 30s timers happen to expire close together (likely, since they were all started within the same few-second window when the outage began), dozens of half-open probes can land on the freshly-recovered payments at nearly the same instant. A service that had just enough capacity to come back up now takes a synchronized hit from its entire caller fleet — and can relapse under the exact probes meant to confirm it was safe to reopen.

Two structural fixes, and they compose:

• The half-open concurrency guard (works even with zero coordination): configure the breaker so Half-Open permits only a small, fixed number of concurrent trial calls — often just one — instead of throwing the gate fully open the instant the timer fires. Resilience4j calls this permittedNumberOfCallsInHalfOpenState; Hystrix has the equivalent notion. This alone caps how hard each individual breaker's own recovery test can hit the dependency, but with 50 independent breakers you can still get 50 individually-gentle probes arriving together.
• Coordinated / shared breaker state: put the Open/Half-Open/Closed state and its permit counter in one shared store (Redis, etcd — the same shape as a distributed rate limiter). Now there is genuinely one breaker for the whole fleet: one shared timer, one shared half-open permit, so exactly one caller anywhere probes the recovering dependency at a time, and every other caller gets an instant fast-reject from the shared state, no network call to the dependency at all.

The trade-off is direct: per-instance is free, needs no extra infrastructure, and adds zero latency to the fast path — but you accept N uncoordinated views of reality. Coordinated is consistent — but the shared store is now on your hot path, it must be fast and highly available itself, and it must fail open to the per-instance behavior if it becomes unreachable, or you've built a new single point of failure to protect against the old one.

Two lines worth remembering cold: "trip sooner, not probe sooner" — lowering your error-rate trip threshold so the breaker opens before the dependency is fully saturated does more good than tightening the half-open retry cadence, which just adds more probes to a system already struggling. And: "state is per-instance" — unless you deliberately built a shared store, one caller's breaker being Open tells you nothing about what the other 49 are doing right now.

5. Spike-tolerant thresholds, and why a slow dependency is worse than a dead one

A breaker that trips on a raw error-rate percentage over a window (say, "open if ≥50% of the last 20 calls failed") is deliberately tolerant of a brief spike — one bad deploy blip or a two-second GC pause shouldn't trip the whole circuit. But that same tolerance is dangerous under sustained overload: if a dependency is merely slow rather than erroring outright, the error-rate signal can sit comfortably under the trip threshold (calls are timing out or queuing, not returning HTTP 500) while the damage compounds elsewhere. This is why production breakers combine an error-rate trip with a separate, usually lower, slow-call-rate threshold (resilience4j's slowCallRateThreshold) — sustained degraded latency needs to shed load even when outright failures never cross the error-rate bar.

The reason slow beats dead as a failure mode, precisely, is Little's Law: L = λ × W — the number of requests in flight (L) equals the arrival rate (λ) times the average time each one spends in the system (W).

With a 200-connection/thread pool sized for the normal case (25 in-flight, 8× headroom), the slow scenario fills all 200 slots in 200 ÷ 500 ≈ 0.4 seconds — and every request after that either queues or is rejected, including requests to unrelated endpoints if the pool is shared. The fast-failure scenario keeps L at 2.5, nowhere near the pool ceiling; the caller absorbs the error immediately and moves on. Same arrival rate, wildly different blast radius — because W, not the failure itself, is what decides how many resources the failure holds hostage. This is the mechanical reason behind the maxim "slow is worse than down", and it's the same argument for why a circuit breaker should trip on latency degradation, not only on hard errors — the pool exhaustion in row two happens whether or not a single request ever technically "failed."

Pitfalls

• Self-preservation silently disables the 180s bound. Eureka's self-preservation mode suspends eviction entirely when the renewal rate drops below a threshold (default 85%), because a mass renewal drop usually means "network partition," not "everyone died." That's the right call for availability — but it means the derived worst case (§1) stops applying during exactly the event you'd most want it to hold: dead instances can now persist in the registry far longer than 180s until the partition clears.
• A live lease is not a health check. An instance can keep renewing on schedule while its request-handling threads are wedged — the heartbeat thread and the request path can fail independently. Discovery answers "is this address registered," not "will this call succeed"; that gap is exactly why a per-call circuit breaker is still required even against an instance the registry insists is healthy.
• A coordinated breaker store becomes a new single point of failure. Centralizing breaker state fixes the N-uncoordinated-probes problem but puts a new dependency on every call's critical path. If that store is slow, every caller's request now waits on it; if it's down, the breaker must explicitly fail open to local per-instance behavior, or you've traded a recovering dependency for a dead breaker layer.
• Half-open with no concurrency cap defeats its own purpose. If every instance's timer expires and each one is allowed unlimited concurrent probes the moment Half-Open starts, you haven't tested recovery gently — you've just delayed the same thundering herd by one Open-duration timer.
• An error-rate-only trip misses a slow-but-not-erroring dependency. A window that only counts hard failures can sit under its trip threshold indefinitely while p99 latency (and thread-pool occupancy) climbs — add a slow-call threshold or a bulkhead sized from Little's Law, don't rely on the error-rate trip alone.

Selection & trade-offs: the judgment layer

AP vs CP registry. Default to AP (Eureka-style) for plain service discovery: the only question being answered is "give me a good-enough address," a stale entry costs one retry, and a registry that goes silent during a partition is strictly worse than one that answers with slightly old data. Reach for CP (Consul/etcd/ZooKeeper with quorum reads) when the same store is also arbitrating something where two different answers is a correctness bug, not an inconvenience — leader election, distributed locks, config that must be seen identically everywhere. Don't pay the CP tax (discovery going dark on a minority partition) for a job that only ever needed "a" healthy instance, not "the" consistent view.

Per-instance vs coordinated circuit breaker. Default to per-instance: zero extra infrastructure, zero added latency, and with a half-open concurrency guard the blast radius of uncoordinated probes is usually tolerable. Move to a coordinated/shared breaker when the caller fleet is large enough that N simultaneous half-open probes reliably re-trip a fragile dependency, or when policy genuinely needs to be global (one signal every caller must honor, e.g. protecting a shared internal gateway that has no headroom for even a handful of simultaneous probes). The cost you're accepting is a new dependency in the hot path that must be fast, highly available, and must fail open — never let the safety mechanism become the outage.

Takeaways

• Detection lag is additive across independent clocks, not a single number: lease duration + eviction sweep period + client cache window — Eureka's real defaults compound to a worst case of ~180s (a couple of minutes), which no registry tuning alone gets under a second; bridge it with client-side timeouts, retry-on-a-different-instance, and per-call circuit breaking.
• A registry commits to AP or CP deliberately: AP (Eureka) serves possibly-stale data through a partition because a stale address is cheap; CP (Consul/etcd/ZooKeeper) refuses to answer without quorum because it's also being trusted for decisions where two different answers is a bug.
• Circuit-breaker state is per-instance by default — N callers form N independent opinions, so a recovering dependency can face N uncoordinated half-open probes at once; guard concurrency in half-open, and only add a coordinated shared store once N-uncoordinated-probes is a real, not theoretical, risk.
• Slow beats dead, mechanically: Little's Law (L=λW) shows a slow dependency inflates in-flight work and exhausts a shared thread/connection pool in a fraction of a second, while a fast failure keeps L tiny — trip breakers on latency degradation, not only hard errors.

Related pages

• What is Service Discovery Pattern — the registration/discovery-styles fundamentals this page assumes before deriving the staleness arithmetic
• Circuit Breaker — The State Machine (Closed → Open → Half-Open) — the single-instance state machine this page extends into per-instance vs coordinated fleets
• Introduction to CAP Theorem — the general AP/CP framework behind the registry's own consistency choice in §2
• What is Quorum — the majority-agreement mechanism CP registries (Consul/etcd/ZooKeeper) rely on
• Designing for Failure — Blast Radius, Timeouts, Breakers & Bulkheads — the broader failure-containment toolkit this page's breaker/timeout arguments belong to

Sources: Netflix Eureka wiki and Spring Cloud Netflix Eureka docs (renewal/lease/eviction defaults and self-preservation mode); HashiCorp Consul and Apache ZooKeeper/etcd documentation (quorum-based consistency); resilience4j CircuitBreaker docs (half-open permitted calls, slow-call-rate threshold); Michael Nygard, Release It! (2nd ed., 2018) — the circuit breaker stability pattern; John D. C. Little, "A Proof for the Queuing Formula: L = λW" (1961); Google SRE Book, ch. on cascading failures and load shedding; the AWS Builders' Library, "Timeouts, retries, and backoff with jitter." Re-authored/Deepened for this guide.