Service Discovery Pattern An Example

Service discovery works because every running instance keeps a short-lived lease in a shared registry by sending periodic heartbeats; a caller asks the registry for the live addresses of a service, the registry returns only leases that have not expired, and the caller picks one instance itself — so a crashed instance disappears on its own once its lease lapses, with no central actor rewriting configuration.

The version below is deliberately richer than a map put/get. It carries multiple instances per service, heartbeat expiry (TTL), a load-balanced client-side pick, and a client cache — the four things that make discovery actually behave like discovery.

The registry: instance list + lease expiry

The registry is keyed by service name, but each value is now a set of instances, and each instance carries the timestamp of its last heartbeat. heartbeat both registers a new instance and renews an existing one; discover evicts anything older than the TTL before returning addresses. A Clock is injected so the lease logic is testable without sleep.

import java.time.Clock;
import java.util.*;
import java.util.concurrent.ConcurrentHashMap;

public class ServiceRegistry {

    /** One running copy of a service. */
    static final class Instance {
        final String id;            // "orders-i-1"
        final String address;       // "10.0.0.11:8080"
        volatile long lastBeatMs;   // last heartbeat, epoch millis
        Instance(String id, String address, long now) {
            this.id = id; this.address = address; this.lastBeatMs = now;
        }
    }

    static final long TTL_MS = 15_000;   // drop an instance after 15s of silence

    // serviceName -> (instanceId -> Instance)
    private final ConcurrentHashMap<String, ConcurrentHashMap<String, Instance>> table
            = new ConcurrentHashMap<>();
    private final Clock clock;

    ServiceRegistry(Clock clock) { this.clock = clock; }

    /** First call registers; later calls from the same id renew the lease. */
    public void heartbeat(String service, String instanceId, String address) {
        long now = clock.millis();
        table.computeIfAbsent(service, k -> new ConcurrentHashMap<>())
             .compute(instanceId, (k, prev) -> {
                 if (prev == null) return new Instance(instanceId, address, now);
                 prev.lastBeatMs = now;      // renew
                 return prev;
             });
    }

    /** Live addresses only; stale instances are evicted lazily on read. */
    public List<String> discover(String service) {
        ConcurrentHashMap<String, Instance> instances = table.get(service);
        if (instances == null) return List.of();
        long cutoff = clock.millis() - TTL_MS;
        instances.values().removeIf(i -> i.lastBeatMs < cutoff);   // lazy eviction
        List<String> live = new ArrayList<>(instances.size());
        for (Instance i : instances.values()) live.add(i.address);
        return live;
    }
}

The nested ConcurrentHashMap lets many instances of the same service coexist, and lets heartbeat and discover run concurrently without locking. removeIf on values() is the whole eviction engine: no background thread, the dead lease is reaped the next time anyone reads.

The instance: self-registration by heartbeat

Each instance registers itself and then keeps beating on a schedule (here every 5 seconds, comfortably under the 15-second TTL). This is the self-registration variant — the instance is responsible for its own lease.

import java.time.Clock;
import java.util.concurrent.*;

public class Service {
    private final String name;         // "orders"
    private final String instanceId;   // "orders-i-1"
    private final String address;      // "10.0.0.11:8080"
    private final ServiceRegistry registry;
    private final ScheduledExecutorService beat =
            Executors.newSingleThreadScheduledExecutor();

    Service(String name, String instanceId, String address, ServiceRegistry registry) {
        this.name = name; this.instanceId = instanceId;
        this.address = address; this.registry = registry;
    }

    public void start() {
        // register immediately, then renew every 5s (TTL is 15s -> two misses tolerated)
        beat.scheduleAtFixedRate(
            () -> registry.heartbeat(name, instanceId, address),
            0, 5, TimeUnit.SECONDS);
    }

    public void stop() { beat.shutdownNow(); }   // beats cease; lease lapses in <= 15s
}

Because the TTL (15s) is three heartbeat intervals (5s), the instance can miss up to two beats — a GC pause, a slow network — without being wrongly evicted. That ratio is the single most important tuning knob in the whole pattern; more on it in Pitfalls.

The client: cache + load-balanced pick

The caller does three things the naive example skipped: it caches the instance list for a few seconds (so it is not hammering the registry on every call), it load-balances across instances with round-robin, and it fails loudly only when there is genuinely nothing live.

import java.time.Clock;
import java.util.List;
import java.util.concurrent.atomic.AtomicInteger;

public class ClientService {
    private final ServiceRegistry registry;
    private final Clock clock;
    private final AtomicInteger rr = new AtomicInteger();

    private static final long CACHE_TTL_MS = 5_000;
    private String cachedService;
    private List<String> cached = List.of();
    private long cachedAtMs = Long.MIN_VALUE;

    ClientService(ServiceRegistry registry, Clock clock) {
        this.registry = registry; this.clock = clock;
    }

    /** A load-balanced address, refreshing the local cache when it ages out. */
    public String pickInstance(String service) {
        List<String> live = lookup(service);
        if (live.isEmpty())
            throw new IllegalStateException("No live instances for " + service);
        int idx = Math.floorMod(rr.getAndIncrement(), live.size());  // never negative
        return live.get(idx);
    }

    private List<String> lookup(String service) {
        long now = clock.millis();
        boolean stale = !service.equals(cachedService)
                        || now - cachedAtMs > CACHE_TTL_MS;
        if (stale) {
            cached = registry.discover(service);
            cachedService = service;
            cachedAtMs = now;
        }
        return cached;
    }
}

Why the naive version is wrong

Three defects hide in the tempting one-liners:

• Returning a single String (the original discover) makes load balancing impossible — every caller hammers one instance. You must return the list.
• rr.getAndIncrement() % live.size() looks fine until the counter crosses Integer.MAX_VALUE and wraps negative: -1 % 2 == -1, so live.get(-1) throws IndexOutOfBoundsException. Math.floorMod(-1, 2) == 1 stays in range forever.
• No expiry means a crashed instance is served to callers indefinitely. Without the TTL cutoff, discover is just get — it never notices death.

(One honesty note: the cache fields here assume a single caller thread. Share a ClientService across threads and you would guard the cache — e.g. an AtomicReference to an immutable snapshot.)

Trace it with real values

Two instances of orders register at t=0: i-1 at 10.0.0.11:8080 and i-2 at 10.0.0.12:8080. i-2 crashes at t=2s and stops beating; i-1 keeps beating on schedule at t=0, 5, 10, 15. Registry TTL = 15s, client cache TTL = 5s. This is the exact output of running the code above:

The instructive rows are t=7 and t=16. At t=7 the client still routes to the dead i-2: its lease does not expire until t=15 (last beat at t=0 + 15s TTL), so discover honestly still lists it. The failure window is real and bounded — up to TTL + cacheTTL ≈ 20s of possibly routing to a corpse. At t=16 the cutoff finally passes i-2's last beat, removeIf reaps it, and traffic converges on the survivor. Note also that discover returned [.12, .11], not [.11, .12]: ConcurrentHashMap iteration order is unspecified, which matters for round-robin fairness (Pitfalls).

Pitfalls

• Detection lag is the whole game. A crashed instance keeps getting traffic for up to TTL + clientCacheTTL. Shrink the TTL and you reduce the lag but increase false evictions (a GC pause looks like death) and heartbeat load on the registry. There is no TTL that is both instant and safe — which is why production systems pair leases with a fast client-side signal: retry the next instance on connection failure and trip a circuit breaker so you stop picking a corpse before its lease even expires.
• Round-robin over an unordered snapshot is not fair. ConcurrentHashMap.values() has no defined order, and the order can change on every refresh, so a naive index-based RR does not visit instances evenly across refreshes. Sort by instance id, or hold a stable ordered snapshot per fetch, if fairness matters.
• Stale client cache outlives reality. The cache that saves the registry from being hammered is also what routes you to an instance the registry already dropped. Long cache TTLs amplify detection lag; the fix is push invalidation (registry notifies watchers) rather than pure polling.
• Self-registration trusts the instance to tell the truth. A hung instance whose heartbeat thread is alive but whose request threads are wedged will keep renewing its lease while serving errors. A liveness heartbeat is not a readiness check — the registry needs an actual health probe, not just "am I still running."
• The registry is a single point of truth. One node means one failure kills discovery for everyone; that is why real registries (etcd, ZooKeeper, Consul) are replicated via a consensus protocol — introducing split-brain and stale-read trade-offs the toy map does not have.

When to use it — and when not

The example implements client-side discovery with self-registration: the caller queries the registry and chooses an instance itself. Reach for it when these signals line up:

• You run many short-lived, dynamically-scheduled instances (autoscaling, spot nodes, containers) whose addresses change constantly — static config or hand-edited load-balancer pools cannot keep up.
• Callers are your own services and can embed a discovery client, so you get smart, application-aware load balancing (locality, weighting, hedging) for free.
• You want to avoid an extra network hop per request.

Trade-offs vs the named alternatives

• vs Server-side discovery (client hits a load balancer / router — AWS ELB, Kubernetes Service + kube-proxy, an API gateway). You gain a hop and application-aware balancing; the client is dumb and language-agnostic. You pay in a discovery-client library baked into every caller in every language, and balancing logic duplicated everywhere. Choose client-side when callers are homogeneous services you control and per-request latency is precious; prefer server-side when clients are heterogeneous or external, or you want one place to own routing, TLS, and rate limits.
• vs DNS-based discovery (register instances as A/SRV records). You gain the world's most universal client — every language already resolves DNS, no library. You pay with DNS caching/TTL granularity that is coarse and often ignored by resolvers, so failover is sluggish and per-request load balancing is crude. Prefer DNS when the instance set changes slowly and you value zero client code; choose an explicit registry when you need sub-15-second failover and rich metadata (weights, zones, health).
• vs Third-party registration (an orchestrator registers/deregisters instances — Kubernetes registering Pods). You gain instances that no longer have to know about the registry, and deregistration on crash handled by the platform. You pay with a mandatory registrar component and platform lock-in. Prefer it when you already run an orchestrator; choose self-registration when you want instances self-sufficient with no external registrar.

Do not reach for any of this when you have a fixed, small set of long-lived instances — a static config file or a single load balancer with a hand-maintained pool is simpler, has no registry to operate, and no detection-lag failure mode. Discovery earns its complexity only once addresses genuinely churn.

Takeaways

• Discovery is not put/get — it is a lease: instances renew by heartbeat, and discover returns only unexpired leases, so death is detected by absence, not by a message.
• The registry must hold an instance list, the client must load-balance the pick (with Math.floorMod, not %) and usually cache it — omit any one and you lose the property discovery exists to provide.
• Every discovery design trades freshness against load and false evictions; a crashed instance is routed to for up to TTL + cacheTTL, so pair leases with client-side retry and circuit-breaking.
• Pick client- vs server-side and self- vs third-party registration by who your callers are and whether you already run an orchestrator — not by which is fashionable.

Sources: Chris Richardson, Microservices Patterns (Manning, 2018) — Client-Side Discovery, Server-Side Discovery, Self- and Third-Party Registration patterns; Sam Newman, Building Microservices (2nd ed.); the Netflix Eureka + Ribbon and HashiCorp Consul lease/health-check models; and the Kubernetes Service / endpoint-controller docs for the third-party-registration case. Code compiled and traced on OpenJDK; the displayed List.of() targets Java 9+. Re-authored / deepened for this guide.