Reverse Proxy in Practice — Fleet Sharding, Health Checks, Zero-Downtime Reloads & Cache Consistency (Deep Dive)

The reconciliation: is every load balancer a reverse proxy?

A reverse proxy is defined by ONE mechanical fact: it terminates the client's connection and opens its own separate connection to whatever backend it decides should handle the request, then relays the response back — the client never talks to the backend directly. Everything else is a choice about how it picks the backend. That definition makes an L7 load balancer a clear reverse proxy, and so is an L4 load balancer running in full-NAT/proxy mode. The exception, derived fully in What is a Proxy Server: DSR and plain packet-forwarding L4 balancers never terminate the client's TCP connection, so they are load balancers but not reverse proxies.

A plain reverse proxy (say, Nginx routing /api to the orders service and /static to a CDN origin) is doing content-based routing: for any given request there is exactly one correct destination, chosen by path, host, or header. A load balancer is a reverse proxy that has been pointed at N interchangeable replicas of the same service and must instead answer "which of these equivalent copies handles this one?" — a distribution problem, not a routing problem — provided it is running in proxy/full-NAT mode. Packet-forwarding L4 balancers solve the same distribution problem without terminating the connection, so they are load balancers but not reverse proxies.

So: the load balancers most common in system-design interviews — L7 HTTP balancers and full-NAT L4 balancers — are reverse proxies, because they terminate and re-originate. Load balancing is then one specialization of the reverse-proxy category: the routing decision is "pick one of N identical things" rather than "pick the one right thing." But the reverse-proxy category does not include DSR or plain packet-forwarding L4 balancers, which forward without terminating.

Sharding the cache fleet: routing a key to ONE predictable node

Reverse proxies that also cache responses (an Nginx/Varnish tier, a CDN edge, a memcached fronting layer) face a second, different distribution problem: given N cache nodes, which node should own a given object's key? Get this wrong and every node ends up caching everything — N times the memory pressure, N times the eviction churn, and a hit ratio no better than a single node's.

The naive approach is node = hash(key) % N. It shards cleanly while N is fixed, but the moment N changes — a node dies, or you scale the fleet from 4 nodes to 5 — the modulus changes for almost every key, so almost every key now maps to a different node than before. The entire fleet's cache goes cold at once, and the origin gets hit with the full traffic spike right when the fleet is already in flux.

The fix is the same ring used for data partitioning: hash both the nodes and the keys onto one circular space (Karger et al.'s consistent hashing, applied to cache-node selection), and each key belongs to the first node found walking clockwise from its hash position. (CARP — the Cache Array Routing Protocol, the classic protocol for proxy-array cache routing — solves the same problem and gets the same minimal-remap property, but via rendezvous/highest-score hashing rather than a ring: it combines the URL hash with each member's hash and picks the highest combined score. More on that family below.) The ring mechanics themselves (hashing nodes onto the circle, virtual nodes for balance, the walk-clockwise rule) are the same ones already derived on the Consistent Hashing lesson — see System Design → System Design Building Blocks — this page only cares about what the ring buys a proxy fleet specifically: when a node is added or removed, only the keys in the arc immediately owned by that node move; every other node's arc, and the keys in it, are untouched.

Read the two rings left to right. On the left, four nodes (A, B, C, D) split the ring into four arcs; key1 hashes into B's arc, so it is cached on B. On the right, a fifth node E has been inserted between B and C. Only the sliver of the ring between B and E changes ownership: key2, which used to fall on C, now falls on E — it is a genuine cold miss the first time it is requested after the reshard. key3, though it is also in what used to be "C's territory," still falls after E's boundary, so it still belongs to C and its cache entry is never disturbed. key1 is untouched entirely. That is the payoff: adding or removing one node out of N remaps roughly 1/N of the keyspace (only the arc adjacent to the changed node), not the whole fleet, so the other N−1 nodes keep serving hits through the resize.

Health checks: how the proxy learns a backend is down

A reverse proxy must decide, continuously and without being told, which backends in its pool are safe to send traffic to. There are two independent mechanisms, and production fleets run both together:

Active checks — the proxy itself opens a connection (or sends a lightweight request, typically GET /health) to each backend on a fixed interval, independent of real traffic. A backend that fails to respond, times out, or returns a non-2xx status on N consecutive probes is marked unhealthy and pulled out of rotation immediately — even if it hasn't served a single real request in that window. This is the only mechanism that catches a backend that is dead but has received no traffic recently.

Passive checks — the proxy observes the real requests it is already routing: connection refused, TLS handshake failure, timeout, or a 5xx response counts as a strike. After k consecutive failures (a small integer, commonly 3–5) within a rolling window, the backend is marked unhealthy without a dedicated probe ever being sent. Passive checks catch failures active checks miss between probe intervals, at the cost of "wasting" a few real user requests on the dying backend before it trips.

The threshold is deliberately not 1: a single dropped packet or one slow GC pause is noise, not a real outage, and pulling a backend out for one blip causes flapping — the backend gets marked down, its load vanishes, it recovers instantly (because the failure was transient), gets marked healthy, gets slammed with the queued traffic, looks unhealthy again, and cycles. Requiring k consecutive failures before marking down, and a separate (often smaller) number of consecutive successes before marking healthy again, is hysteresis — asymmetric thresholds that damp the oscillation.

Coming back from unhealthy is itself staged, not a single flip. A backend that just restarted has a cold cache, an empty connection pool, and JIT-warmup costs (for JIT'd runtimes) — dumping a full share of live traffic on it the instant its first health check passes can knock it straight back over. Slow start (Nginx and Envoy both call it this) ramps the backend's traffic share from near-zero up to its normal weight over a configured window after it rejoins, so it warms up under partial load instead of full load.

Zero-downtime operation of the proxy itself

Everything above assumes the proxy is a fixed point of stability — but the proxy's own config changes too (a new backend added, a route changed, a certificate rotated), and a naive restart to pick that up would drop every in-flight connection at the moment the old process exits and the new one hasn't finished binding yet. Two related mechanisms avoid that.

Graceful config reload. Nginx's classic mechanism: sending the master process SIGHUP makes it re-read its config and spawn a fresh set of worker processes with the new config, while the old workers are told to stop accepting new connections but keep running until every request they already hold finishes (or a grace timeout elapses), then exit. At no point is there a moment with zero workers listening. Envoy's hot restart is the same idea taken further: the new process shares the listening sockets with the old one (via SCM_RIGHTS / a shared listener), so there is no gap where a new connection has nowhere to land, and the old process drains its existing connections before exiting.

t	Event
t0	Operator edits config, sends SIGHUP (Nginx) or triggers hot restart (Envoy).
t0+ε	Master process re-reads config, validates it; if invalid, it aborts the reload and keeps the OLD workers running untouched (config errors never take the proxy down).
t1	New worker pool starts with the new config and begins accepting new connections on the (shared) listening socket.
t1	Old workers stop accepting new connections but keep serving requests already in flight.
t1..t2	Both worker generations run side by side; new traffic goes only to new workers.
t2	Last in-flight request on an old worker finishes (or the grace timeout fires); that old worker exits.
t2	Only new workers remain — reload complete with zero dropped connections and zero listen-socket downtime.

Connection draining is the same idea applied to a single backend being removed from the pool (scale-down, deploy, maintenance) rather than the proxy's own config: the proxy stops routing new requests to that backend immediately, but leaves its existing in-flight requests alone until they complete naturally, only then closing the connection and removing it from rotation. Terminating the backend before its connections drain is exactly the naive-restart mistake, just one layer down — it turns a planned, harmless scale-down into user-visible errors.

Mid-request failure: can the proxy safely retry on another backend?

A backend can die after the proxy has already picked it and forwarded the request. Whether the proxy may transparently retry on a different backend — so the client never even sees an error — depends on exactly one thing: has any byte of the response already been sent to the client?

Before the first response byte goes out, the proxy still fully controls what the client has seen (nothing), so a retry is invisible to the client. Once the proxy has started streaming the response, retrying would mean sending a second, possibly different, response on top of a partial one — corrupting the client's view of the reply. So the retry window closes the instant the first byte is flushed downstream, regardless of how much of the backend's own processing had completed.

The second condition is independent of timing: the request must be idempotent — retrying it on a second backend must not double-apply its effect. A GET is safe by definition. A POST that creates an order is not safe to blindly retry (the first backend may have written the order to its database and then died before the response left it — a retry would create a duplicate order) unless the client supplied an idempotency key the backend can use to detect and collapse the duplicate.

Case	Idempotent?	Response bytes sent?	Safe to retry on another backend?
GET /orders/42, backend TCP RST before any reply	yes	no	Yes — replay on backend 2, client never sees an error.
POST /orders (no idempotency key), backend dies after committing the write but before replying	no	no	No — retrying could create a second order. Surface the error (or require an idempotency key so it CAN be made safe).
POST /orders with an Idempotency-Key header, backend dies before replying	yes (by construction)	no	Yes — backend 2 sees the same key, detects the duplicate, returns the original result instead of re-creating.
GET /report (streaming a large CSV), backend dies mid-stream after 40% of bytes sent	yes	yes	No — the client already has a truncated, half-written file; a silent retry would append a second stream on top of it. The only safe move is to fail the connection and let the client re-request from scratch.

The practical rule an interviewer is listening for: safe transparent retry requires idempotent request semantics AND zero response bytes sent. Either condition failing means the proxy must surface the failure rather than silently paper over it.

Cache-tier consistency: how stale can the answer be?

A caching reverse proxy sits directly in the CAP/PACELC trade-off: when the cached copy might be out of date, does the proxy prioritize availability and latency (serve what it has) or consistency (always confirm freshness first)? Two read strategies sit at opposite ends of that trade-off.

Stale-while-revalidate (an actual Cache-Control directive, also the general pattern): on a cache hit past its freshness window but still within a configured staleness allowance, the proxy returns the stale copy to the client immediately and, in the background, fires an async request to the origin to refresh the cached entry for the next reader. The client gets a fast answer every time, at the cost of possibly seeing data that is up to "staleness window" seconds old — this is the AP-leaning choice, the EL side of PACELC — Else (no partition), favor Latency over Consistency; a system built around it is PA/EL in Abadi's notation. Even under normal operation (no partition), it still trades consistency for lower latency.

Strong read-through: on any cache miss OR any expiry of the freshness window, the proxy blocks the client's request until it has fetched a fresh copy from the origin, populates the cache, and only then replies. The client never sees data older than the configured TTL by even a moment past expiry, but every expiry turns into a full origin round-trip on the critical path — the CP-leaning choice, trading latency for a tighter consistency bound.

Pick stale-while-revalidate for read-heavy, latency-sensitive, eventually-consistent data (product listings, article content, a social feed) where a few seconds of staleness is invisible to the user. Pick strong read-through where the cost of a stale read is concrete and immediate — a price shown at checkout, an inventory count that must not oversell, a permission check.

Pitfalls

• Modulo sharding a cache fleet. Looks correct at fixed N; every scale event (the exact moment you need the cache most) invalidates almost everything and origin traffic spikes right when the fleet is unstable.
• Health-check threshold of 1. One dropped health probe or one 5xx does not mean the backend is down — it causes flapping (down → recover → slammed → down again) rather than protecting the fleet.
• No slow start after recovery. A backend that just rejoined with a cold cache and empty connection pool gets its full traffic share immediately and falls back over.
• Killing a backend without draining it. Removing a backend from the pool and terminating its process in the same step drops every request it still had in flight — the same mistake as a naive proxy restart, one layer down.
• Retrying a non-idempotent write on failover. Retrying a POST with no idempotency key on backend failure can silently double-apply the write (duplicate order, double charge).
• Retrying after bytes have already streamed. Once the response has started, a "helpful" automatic retry corrupts the client's view by appending a second response on top of a partial one.
• Config reload racing an in-flight deploy. Reloading with a syntactically valid but semantically wrong config (points at a backend pool that no longer exists) succeeds at the reload step and only fails at request time — validate the target pool, not just the config syntax.
• A shared dependency in the readiness check. If every node's readiness probe pings the same downstream (one database, one auth service), a single blip there fails every node's check at the same instant and the whole fleet is pulled from rotation at once — a correlated, self-inflicted total outage. A readiness probe should test only what that node needs to serve its own traffic, not a shared backend every node happens to touch.

Judgment layer: selection and trade-offs

Consistent hashing vs modulo hashing for the cache fleet: modulo is simpler and perfectly even, but only while N is fixed — adding or removing a single node remaps roughly (N−1)/N of the keyspace, all but ~1/N of keys. (Special resizes can do better — doubling 4→8 remaps only half — but nobody resizes a cache fleet in powers of two on purpose.) Consistent hashing keeps ~1/N remap cost on resize at the price of slightly uneven load across nodes (fixed by adding virtual nodes per physical node). Use modulo only for fleets that never resize; use consistent hashing for anything elastic.

Consistent hashing vs rendezvous hashing: both bound the remap cost to roughly 1/N on a topology change; rendezvous hashing computes a weight per (key, node) pair and picks the max rather than walking a ring, which avoids maintaining ring state at the cost of an O(N) scan per lookup — this is the family CARP belongs to. See the dedicated Rendezvous vs Consistent Hashing page for the full comparison and when the O(N) scan actually matters.

Active vs passive health checks: active checks alone need a dedicated /health endpoint and probe traffic, but catch a fully idle-but-dead backend that passive checks never would (no real traffic means no strikes accumulate). Passive checks alone are free (no extra endpoint or probe load) but always cost a handful of real user requests before tripping. Production systems run both: active checks for continuous, traffic-independent coverage; passive checks as a fast-acting backstop between probe intervals.

Reverse-proxy health checks vs orchestrator-level readiness/liveness probes (e.g. Kubernetes): an orchestrator's readiness probe decides whether a pod should receive traffic at the service-mesh/endpoint level and can restart the container on liveness failure; the reverse proxy's own health check is a second, independent layer that reacts faster (proxy-local, no control-plane round trip) and covers backends the orchestrator doesn't manage (e.g. a fixed VM pool). Neither replaces the other — the orchestrator handles lifecycle, the proxy handles routing-time exclusion.

Stale-while-revalidate vs strong read-through: covered above — the axis is "can this specific read tolerate being a few seconds old," not a fleet-wide policy. Many real systems mix both per route: strong read-through for a checkout price, stale-while-revalidate for the product description on the same page.

Takeaways

• L7 and full-NAT L4 load balancers ARE reverse proxies — they terminate, re-originate, and relay, with the routing decision specialized to "pick one of N equivalent replicas." But the containment is not universal: DSR and plain packet-forwarding L4 balancers distribute load without ever terminating the client's connection, so they are load balancers that are NOT reverse proxies. State which kind you mean before claiming the subset.
• Consistent hashing on the cache-key ring (or CARP's rendezvous-style highest-score selection) keeps fleet resizes cheap — only ~1/N of keys remap — where modulo hashing would invalidate almost the whole fleet on every resize.
• Health checks need both active probing and passive traffic observation, a consecutive-failure threshold (hysteresis) to avoid flapping, and a slow-start ramp on recovery — and the proxy's own config changes need the identical "drain, don't drop" discipline via graceful reload and connection draining.
• A transparent mid-request retry is only safe when the request is idempotent AND no response bytes have reached the client yet — either condition failing means surface the error instead of silently retrying.

---

Re-authored/Deepened for this guide.

I just read a lesson on **Reverse Proxy in Practice — Fleet Sharding, Health Checks, Zero-Downtime Reloads & Cache Consistency (Deep Dive)** (System Design) and want to truly understand it. Explain Reverse Proxy in Practice — Fleet Sharding, Health Checks, Zero-Downtime Reloads & Cache Consistency (Deep Dive) from first principles using ONE vivid real-world analogy and a visual mental model — draw it as ASCII art or a clear step-by-step diagram — with a concrete example using real numbers. Then ask me one question to check I got the mental picture, and wait for my reply. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Teach me **Reverse Proxy in Practice — Fleet Sharding, Health Checks, Zero-Downtime Reloads & Cache Consistency (Deep Dive)** interactively. Ask me ONE guiding question at a time, wait for my answer, and adapt to my confusion — build the idea with me step by step instead of explaining it all at once. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Quiz me on **Reverse Proxy in Practice — Fleet Sharding, Health Checks, Zero-Downtime Reloads & Cache Consistency (Deep Dive)** with 5 questions, easy to tricky, ONE at a time. Tell me if each answer is right; at the end, explain clearly what I got wrong and why. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Help me remember **Reverse Proxy in Practice — Fleet Sharding, Health Checks, Zero-Downtime Reloads & Cache Consistency (Deep Dive)** for the long term: give the one-sentence intuition, a memorable hook/mnemonic, a tiny worked example, and 3 active-recall flashcards (Q -> A). If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.