Challenges of Load Balancers

Challenges of Load Balancers

A load balancer (LB) sits on the one path every request must cross, which is what makes it simultaneously the most useful and the most dangerous box in the system. Its steady-state challenges — which layer to route at, which algorithm to pick, whether to keep session state, how to survive a node dying — are each owned by a dedicated lesson in this section and are not re-taught here. This page is about the challenge those pages take for granted: the load balancer is not a fixed appliance, it is a stateful process you keep changing — deploying behind, reconfiguring, and rotating certificates on, many times a day — and every one of those changes is a small, self-inflicted failure event. That is the class of problem that bites teams in production long after the routing diagram is correct.

What the sibling lessons already own (so this one points, not repeats)

• L4 vs L7, hardware/software/cloud, DNS/GSLB, where TLS terminates → Load Balancer Types.
• Round-robin vs least-connections vs power-of-two-choices, consistent hashing, hot keys, the keep-alive skew → Load Balancing Algorithms (and the stepped worked trace in Load Balancing Algorithms — Traced).
• Sticky sessions, cookie vs source-IP affinity, externalizing session state → Stateless vs Stateful Load Balancing.
• The LB as a single point of failure, redundant pairs, VRRP failover timing, split-brain and quorum fencing, health-check flapping, the panic / fail-open threshold, N−1 sizing, availability nines → High Availability and Fault Tolerance.
• Capacity sizing (CPU/handshakes, NIC, connection memory), retry storms, load-shedding, and why an L7 LB is not a DDoS shield → Tier Sizing, Retry Storms, Load-Shedding & GSLB.

Everything below is the gap those five leave open: what goes wrong when you operate the LB through change, not when you route through it or when hardware fails under it.

Change-time is the real risk window

A running LB is almost never harmed by steady traffic — it is provisioned for that. It is harmed at the moments an operator or an automated pipeline touches it: a config push, a rolling deploy of the backends behind it, an endpoint list churning as pods come and go, a certificate being swapped. Each of these is a transition, and a transition on a box that terminates live TCP/TLS connections and holds in-flight requests cannot be atomic. The question this page answers is: what breaks during the transition, and what is the cost of getting it wrong?

Take a reload first. NGINX, HAProxy, and Envoy all reconfigure by keeping the old workers alive to finish existing connections while new workers pick up new ones (NGINX spawns fresh workers on SIGHUP and drains the old ones up to worker_shutdown_timeout; Envoy hot-restart runs the old process alongside the new until it drains). That is graceful — until reloads arrive faster than old workers can drain. In a service-mesh sidecar taking an endpoint update every 2 s from service discovery, with long-lived connections that take 30 s to drain, you accumulate up to 30 ÷ 2 = 15 overlapping "shutting-down" worker generations at once, each still holding sockets and TLS session memory. The reload that was supposed to be free has turned config churn into a memory leak. The fix is to rate-limit reloads (debounce endpoint updates) or use an LB that applies routing changes on the data path without a full worker cycle.

Connection draining on deploy — correctness, not politeness

Every backend deploy removes a server from rotation, lets it finish what it is doing, then kills it. "Lets it finish" is connection draining, and it is where a deploy silently drops users. Work a concrete instance. A backend about to be replaced holds 2,000 in-flight connections: 1,950 ordinary HTTP requests at a p99 of ~50 ms, and 50 long-lived WebSocket / streaming / gRPC connections that stay open for the whole user session (minutes to hours). You set a drain timeout of 30 s and stop sending it new traffic.

• The 1,950 short requests all complete far inside the 30 s window — draining works perfectly for them.
• The 50 long-lived connections are still open at 30 s, because their lifetime has nothing to do with the drain timeout. At t = 30 s the LB force-closes them: 50 users dropped per instance.

Now scale it: a rolling deploy across a 20-instance fleet, one instance at a time, force-closes 50 × 20 = 1,000 long-lived connections per deploy. At a routine 10 deploys/day that is 10,000 forced disconnects a day — an error budget being spent entirely on the deploy pipeline, invisible in any steady-state latency graph.

The crossover that makes this a judgment call, not a config value. The naive fix is "set the drain timeout long enough to cover the longest session." If sessions can last 1 hour, correctness demands a 1-hour drain. But a rolling deploy that drains each instance sequentially now takes up to 20 × 1 h = 20 hours per full fleet rollout — deploys become so slow that you cannot ship a security fix in a working day. Drain-timeout and deploy-velocity trade off directly, and no single timeout value satisfies both once connections outlive a deploy. The senior resolution refuses the dichotomy: do not drain long-lived connections at all — signal them to reconnect. Send an HTTP/2 or gRPC GOAWAY, or a WebSocket close with code 1001 "going away," and let the client re-establish against a healthy instance. That converts an hours-long drain into a fast, controlled hand-off — at the price of the next problem.

The reconnect storm you just created

Telling 1,000 long-lived clients to reconnect at once means 1,000 near-simultaneous TCP+TLS handshakes hitting the surviving instances in the same instant — a self-inflicted thundering herd, and TLS handshakes are the single most CPU-expensive thing an edge LB does (the asymmetric key exchange runs once per new connection). Left unmanaged, the deploy you made faster now spikes handshake load precisely when capacity is already reduced by the instance you are replacing.

The control is jittered reconnect: instead of "reconnect now," clients reconnect after a random delay spread over a window. Spreading the same 1,000 reconnections over 30 s turns a 1,000-handshake instantaneous spike into 1,000 ÷ 30 ≈ 33 handshakes/second — a load any provisioned edge absorbs without noticing. This is the same jitter discipline the retry-storm lesson applies to failed requests, here applied to reconnections caused by the deploy itself. (Node-death failover produces a related but distinct herd; that path, and connection-state replication to hide it, is the domain of High Availability and Fault Tolerance.)

TLS certificate rotation at the edge

When the LB terminates TLS, it is the custodian of the certificate — and certificates expire, so rotation is not optional, it is a recurring scheduled operation that most teams automate and then forget. Two failure modes are unique to running this at the edge:

• Rotation is a reload. Loading a renewed cert on NGINX/HAProxy is a config reload, which re-triggers the exact worker-generation behavior above. Rotating certs under high new-connection churn is therefore not free; on a fleet with many SNI certificates it is a routine, load-bearing reload you must rate-control like any other.
• Expiry is a scheduled outage you opted into. A Let's Encrypt certificate is valid for 90 days; the canonical edge outage is an automation that silently stopped renewing, so the cert lapses. The failure shape is specific and worth knowing: existing TLS sessions keep working, but every new handshake is rejected with a certificate-expired error — so the graphs look fine for connected users while new and reconnecting users see a hard failure, and the incident is often first noticed by customers, not dashboards. This is why the standard is to renew at ~one-third of validity remaining (≈ day 60 of 90), leaving a 30-day buffer for the renewal pipeline to fail and be fixed before anything expires.

Observability: the LB averages away the one sick backend

Because the LB fronts the whole fleet, its natural metrics are fleet-wide — and a fleet aggregate can completely hide a single failing backend. The trap is the fleet p99. Suppose the LB spreads traffic evenly across 200 backends, so each takes 1/200 = 0.5% of requests, and one backend goes bad: its own p99 is 900 ms while the other 199 sit at 50 ms.

Trace the pooled percentile. Only 0.5% of all requests are slow — and 0.5% is below 1%, so the 99th percentile of the pooled distribution still falls at ~50 ms. The bad backend is invisible in the fleet p99, and entirely invisible in the mean or median. Yet that same backend's own p99 is 900 ms — glaring, if you are looking at it. The crossover is exact: fleet p99 masks any single backend serving less than 1% of traffic, i.e. any fleet of roughly 100 or more evenly-balanced backends (with 50 backends the bad one carries 2% > 1% and the fleet p99 catches it; with 200 it does not). The operational consequence: alert on per-backend tail latency and error rate, not fleet aggregates, and let outlier detection eject the host — the aggregate tells you the fleet is fine while one host quietly poisons a slice of users.

Key takeaways

• The routing questions (layer, algorithm, affinity, redundancy, sizing) are each owned by a sibling lesson; the operational challenge unique to this page is that the LB is a stateful process you change constantly, and every change — reload, deploy, cert swap — is a non-atomic transition on a box holding live connections.
• Reloads are only free until they outpace drain: a 2 s update cadence against a 30 s drain leaves ~15 overlapping worker generations resident — debounce config/endpoint churn.
• Connection draining is a correctness problem, not a courtesy: a 30 s drain cleanly finishes short requests but force-closes long-lived ones (≈50/instance → 1,000 per 20-instance deploy → 10,000/day at 10 deploys). You cannot fix it by lengthening the timeout without making a full rollout take ~20 h; instead signal reconnect (GOAWAY / WS 1001) and jitter it (1,000 reconnects over 30 s ≈ 33 handshakes/s).
• Cert rotation is a load-bearing reload and cert expiry is a scheduled outage — new handshakes fail while connected users look fine; renew at ~⅓ validity remaining.
• Fleet aggregates hide a sick backend: at ≥100 evenly-balanced backends a single bad host is invisible in fleet p99 — alert on per-backend tail latency and let outlier detection eject it.

Re-derive it live — the numbers, not the concepts

Each of these is the interview form of a trap above, with different numbers — rehearse the derivation, don't look it up.

1. "Endpoint updates arrive every 5 s, connections take 60 s to drain — how many worker generations pile up, and what breaks first?"
Worked answer: a new reload starts every 5 s and each old generation lingers 60 s, so up to 60 ÷ 5 = 12 shutting-down generations are resident at once. What breaks first is memory — each generation still holds its open sockets and TLS session state, so config churn becomes a leak long before CPU notices. Control: debounce the endpoint updates, or use an LB that applies routing changes without a worker cycle.

2. "Sessions last up to 30 min, 12 instances, sequential drain — what's the worst-case rollout time, and what do you do instead?"
Worked answer: a drain long enough to be correct must cover the longest session, so each instance takes up to 30 min and a sequential rollout takes 12 × 30 min = 6 hours — too slow to ship a security fix. So don't drain long-lived connections at all: send GOAWAY (HTTP/2, gRPC) or WebSocket close 1001, and jitter the reconnects over a window so the hand-off doesn't become a handshake spike.

3. "One of 150 backends has p99 = 1 s, the rest 40 ms — what does the fleet p99 show, and what alert catches it?"
Worked answer: the bad host carries 1/150 ≈ 0.67% of traffic, and 0.67% < 1%, so the pooled 99th percentile still falls in the healthy mass: fleet p99 ≈ 40 ms — the sick backend is invisible. The alert that catches it is per-backend p99 (and per-backend error rate), paired with outlier ejection to remove the host automatically; the fleet aggregate can never see any single host under 1% of traffic.