System Design Examples

Service discovery in real designs (with hostile review)

Use cases alone do not defend a design. Below: where discovery belongs, a traced e-commerce membership change, and a review table that names the first bottleneck, a rejected alternative, and a paid trade-off.

When the pattern earns its keep

• Microservices with churn: many services, rolling deploys, HPA/autoscaling — membership changes faster than config deploys.
• Multi-AZ / multi-cluster: instances appear in new failure domains; clients need zone-aware selection without redeploy.
• Platform-owned discovery: Kubernetes Service/Endpoints + CoreDNS is already server-side discovery; often you should use it rather than bolt on Eureka-in-the-app.

Worked example: e-commerce checkout under scale-out

Services: User, Inventory, Cart, Order, Payment. Order calls Inventory and Payment. Payment runs 8 pods. Black Friday HPA adds 12 pods in 90 seconds.

Step	What happens	If static config	With discovery + health
1	HPA launches 12 Payment pods	Not in anyone's list	Pods register; health = ready
2	Registry / Endpoints updates	—	Watchers push new set (or DNS TTL refreshes)
3	Order resolves Payment	Still 8 endpoints → overload	Load spreads across 20
4	2 old pods drain for deploy	Callers hit terminating pods → 502/timeout	Deregister + connection draining; removed from set

Numbers: Payment capacity ≈ 500 RPS/pod. Peak need 7,000 RPS → need ≥14 healthy pods. With a stale 8-pod list you run at 4,000 RPS capacity while 12 pods idle — artificial outage with spare hardware.

Streaming platform (discovery + server-side LB)

Video edge pods scale with concurrent viewers. A Control-plane registry (or K8s Endpoints) feeds Envoy/HAProxy. Players never see instance IPs; they hit a stable hostname. Discovery failure mode shifts to: "control plane lag" rather than "client config lag."

Hostile design-review table

System slice	First bottleneck under load	Rejected alternative	Trade-off you accept	Failure mode to monitor
E-commerce Order→Payment	Stale client cache under HPA → timeouts + idle capacity	Bake Payment VIP into every Order deploy	Registry/DNS as runtime dependency	Register lag; % traffic to NotReady endpoints
K8s-native fleet	kube-proxy / iptables or IPVS conntrack under huge Service fan-out	Custom Eureka per language on every pod	Platform coupling; less app-level LB sophistication	Endpoint slice storms; DNS NXDOMAIN spikes
Multi-DC with Consul	Cross-DC query latency; WAN gossip delays	Global anycast to wrong region	Stale remote DC members during partition	Serf health; prepared queries fallback
Mesh (Envoy xDS)	xDS push lag / control plane CPU	Each app implements outlier detection alone	Sidecar CPU/mem per pod	CDS/EDS update age; 503 no_healthy_upstream

Real-world anchors (what they actually run)

• Netflix Eureka: client-side discovery at huge scale; apps register and query; proven under streaming load — also operationally heavy if you are not Netflix.
• Kubernetes Service + CoreDNS: default server-side discovery; apps call http://payment; Endpoints track pods.
• Airbnb SmartStack (historical): ZooKeeper + local HAProxy — registry + node-local LB before mesh ubiquity.
• Consul: multi-DC service catalog + DNS; common outside pure-K8s shops.
• Service mesh: discovery folded into sidecar config (xDS); app dials localhost or cluster DNS.

When NOT to use a separate discovery product

• Monolith or few stable services — config/env is enough.
• Already fully on K8s Services and happy with platform LB — adding Eureka duplicates the source of truth (split-brain membership risk).
• No one to own registry HA, backups, and upgrade playbooks.

Operability checklist

• Alert: discovery client error rate, empty instance sets, registry leader elections.
• Dashboards: instances registered vs ready; traffic share of newest cohort after scale-out (should rise within TTL/push SLA).
• Chaos: kill registry follower; kill 30% of backends; verify drain + no black-hole.

Drill ladder

1. Q: After scale-out, RPS/pod on old pods stays high and new pods near zero for 10 minutes. Root cause class? A: Resolution cache / long DNS TTL / clients not watching registry — not "HPA failed."
2. Q: Why can two registries (Eureka + K8s Endpoints) be worse than one? A: Conflicting membership; partial updates; harder incident response — pick one source of truth.
3. Q: Defend server-side discovery for a 40-language fleet. A: One proxy/mesh implements LB and health; apps stay dumb; cost is hop + platform ownership.

I just read a lesson on **System Design Examples** (System Design) and want to truly understand it. Explain System Design Examples from first principles using ONE vivid real-world analogy and a visual mental model — draw it as ASCII art or a clear step-by-step diagram — with a concrete example using real numbers. Then ask me one question to check I got the mental picture, and wait for my reply. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Teach me **System Design Examples** interactively. Ask me ONE guiding question at a time, wait for my answer, and adapt to my confusion — build the idea with me step by step instead of explaining it all at once. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Quiz me on **System Design Examples** with 5 questions, easy to tricky, ONE at a time. Tell me if each answer is right; at the end, explain clearly what I got wrong and why. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Help me remember **System Design Examples** for the long term: give the one-sentence intuition, a memorable hook/mnemonic, a tiny worked example, and 3 active-recall flashcards (Q -> A). If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.