The Inner Workings of the Service Discovery Pattern

Service discovery works because every instance publishes its live network location (IP:port) into a shared registry keyed by a logical name, and the registry continuously expires any entry whose owner stops proving it is alive — so a caller that asks for orders-svc gets back the set of instances currently believed healthy instead of a hard-coded address. Two mechanisms make that work: how an address gets in (registration) and how a dead one gets out (health checks). The subtle, career-defining part is the word believed: the registry is always a slightly stale cache of reality, and the size of that staleness is something you can calculate.

Getting in: self- vs third-party registration

There are two ways an instance's address lands in the registry.

• Self-registration. The instance itself calls the registry to register on startup and (ideally) deregister on shutdown — this is what a Netflix Eureka client library does inside your app. Upside: no extra moving parts; the instance controls its own lifecycle. Downside: the registry client is now coupled into every service, and a hard crash (SIGKILL, OOM-kill) skips the graceful deregister entirely, so you always fall back to timeout-based expiry.
• Third-party registration. A separate registrar watches instances and writes their status into the registry — Kubernetes does this with the kubelet plus the EndpointSlice controller; Registrator does it for raw Docker. Upside: registry logic stays out of the app, and the registrar can deregister a crashed instance the moment it detects the death. Downside: the registrar becomes a privileged, critical component you must run, secure, and keep available.

Getting out: heartbeat vs active monitoring

Once registered, an entry is only trustworthy while the owner keeps proving it is alive. There are two directions the proof can flow.

• Heartbeat (push / TTL model). The instance sends a periodic renew signal; the registry passively resets a time-to-live and evicts the entry if the signal lapses. Receiving a beat costs the registry almost nothing — a passive timestamp reset, no outbound work, no timeout to manage — which is why heartbeat scales to huge fleets. The cost: detection lag is bounded by the TTL, and a beat only proves the process is alive, not that it can serve.
• Active monitoring (pull / probe model). The registry itself reaches out — GET /health, a TCP connect, a ping — and marks the instance down on failure. It can validate real readiness (database reachable, migrations done) and detect faster, but now the registry initiates N probes per interval — bearing connection setup, probe concurrency, and timeout management itself — and the probe travels the registry's network path, which may differ from the caller's.

Be precise about the load claim, because it is where interviewers push back: both models cost the registry O(N) messages per interval — N beats in, or N probes out. The asymmetry is in what each message costs. A beat is instance-initiated, and the registry's work is a timestamp reset: no outbound connection, no timeout to manage. An active probe is registry-initiated: it pays connection setup and concurrency management, and a probe to a dead host holds a socket for the full connect-timeout — seconds, not nanoseconds. That constant-factor, structural gap is why central active checking hits a wall far earlier than passive beats — and why Consul delegates health checks to node-local agents instead of probing everything from the servers.

The next diagram shows the topology; then we trace a real heartbeat timeout with numbers.

A traced heartbeat timeout, with real numbers

Take a Netflix Eureka-style setup with its documented defaults: instances renew every 30s (renewalIntervalInSecs=30), a lease dies after 90s without a renew — three missed beats (leaseExpirationDurationInSecs=90) — and a background eviction task sweeps for expired leases every 60s (evictionIntervalTimerInMs=60000). Watch what happens when orders-svc-7 is OOM-killed right after a successful beat.

So the registry keeps advertising a dead IP from t=93s to t=180s ≈ 87 seconds. During that window every caller that dials 10.2.4.9:8080 gets a connection-refused or a socket timeout. And it is worse in the field: callers usually run a client-side cache of the registry (registryFetchIntervalSeconds=30), so even after removal at t=180s a caller can keep serving the dead address from its cache until its next refresh — up to another 30s. The real worst-case a caller uses a dead address is roughly TTL (90) + sweep interval (60) + client cache (30) ≈ 180 seconds, not zero.

What a missed beat means: the AP/CP consequence

A single missed beat is ambiguous. Did the instance die, or did the network between it and the registry partition while the instance stayed perfectly alive? The registry cannot tell, and how it resolves that ambiguity is a CAP decision baked into the product.

• AP registry (Eureka). Missing beats never trigger instant removal, and Eureka's self-preservation goes further: if it sees renewals drop below ~85% of expected in a window, it assumes a partition rather than mass death and stops expiring leases entirely, continuing to serve stale entries. You never get an empty registry, but you do get dead addresses handed out — so callers must defend themselves with request timeouts, retries to a different instance, and circuit breakers.
• CP registry (Consul / etcd / ZooKeeper, via Raft/ZAB). A lapsed session/TTL causes the leader to delete the key through consensus, and reads go through quorum. Removal is decisive — but during a partition the minority side can't reach the leader, so it rejects reads and writes (no stale data, but also no answers), and a live-but-partitioned instance gets wrongly evicted and must re-register on reconnect.

So the same event — one missing beat — buys you a false positive under AP (a dead node lingers, availability preserved) or a false negative risk under CP (a live-but-isolated node is dropped, correctness preserved). Pick the failure you can live with.

Pitfalls

• Heartbeat treated as readiness. A process can happily renew its lease while its DB connection pool is exhausted — the registry says UP and every request 500s. Gate the heartbeat on real health, or add an active /ready probe that checks downstream dependencies.
• Forgetting the client-side cache. Engineers compute the failure window as "TTL" and are shocked when a dead address is still hit 30s after eviction. The caller's cached registry copy is a second staleness layer — short caches plus retry-to-another-instance, not just a fast TTL.
• Tuning the TTL too low to "detect faster." A stop-the-world GC pause or a brief network blip now misses enough beats to evict a healthy instance, which then re-registers — churn, cache invalidations, and mass eviction during any partition. This exact pain is why self-preservation exists; do not disable it without a plan.
• Relying on graceful deregistration for correctness. SIGKILL/OOM-kill runs no shutdown hooks, so self-registered services can never guarantee a clean deregister. Timeout-based expiry must be your source of truth, not the deregister call.
• Per-caller cache divergence. Each client caches the registry independently, so during the failure window some callers route around the dead node while others still hit it — "it works on my node" incidents that are really cache-timing artifacts.

When to use it / when NOT to

These are three independent knobs; a senior engineer sets each against the workload.

Heartbeat vs active monitoring

Choose heartbeat when the fleet is large and the registry must stay cheap — each beat costs it only a passive timestamp reset (both models are O(N) messages per interval, but beats carry no probe connections or timeouts). It costs you TTL-bounded detection lag and can't distinguish liveness from readiness. Prefer active monitoring when you must validate real readiness (dependencies reachable) and want typed, faster detection; it costs N probes per interval and sees only the registry's network path. In practice many systems run both — Kubernetes liveness probes (is the process alive?) and readiness probes (should it get traffic?) are exactly this split.

Self- vs third-party registration

Choose self-registration for simplicity and no extra components when services are trusted to manage their own lifecycle; it couples the registry client into every app and can't deregister on a hard crash. Prefer third-party (Kubernetes, Registrator) to keep registry logic out of the app and to deregister crashed instances promptly; it costs you a privileged registrar you must run and secure.

AP vs CP registry

Choose an AP registry (Eureka) for large fleets of stateless services where a partition should degrade to stale-but-available routing and callers already carry retries and circuit breakers. Prefer a CP registry (Consul/etcd/ZooKeeper) when a wrong member set is catastrophic — leader election, sharded ownership, anything where two nodes both believing they own a shard corrupts data; it costs you availability in minority partitions.

Crisp rule: choose heartbeat + AP registry + client-side retries when you run a large fleet of stateless services and can tolerate a caller occasionally hitting a dead address for a few seconds; prefer active health checks + a CP registry when serving one dead or unauthorized instance is worse than a brief unavailability.

Takeaways

• The registry is only as fresh as its slowest expiry path: detection lag ≈ TTL + eviction sweep + client cache, not zero — you can and should compute it.
• Liveness (can send a beat) is not readiness (can serve a request); check dependencies, not just that the process is up.
• A missed beat is ambiguous (dead vs partitioned): AP registries keep the entry and stay available, CP registries remove it and stay correct — choose the failure mode that fits the workload.
• Deregistration is never guaranteed (hard crashes skip it), so callers must be resilient by default: timeouts, retry-to-another-instance, and circuit breakers.

Sources: Chris Richardson, Microservices Patterns and microservices.io (Service Registry, Self Registration, 3rd Party Registration, Client-side & Server-side Discovery patterns); Netflix Eureka wiki (renewal, lease-expiration, eviction, and self-preservation defaults); HashiCorp Consul documentation (health checks, TTL, and Raft-based consistency); Kubernetes documentation (EndpointSlices, and liveness vs readiness probes). Re-authored/Deepened for this guide.