Load Balancing

A load balancer works by publishing one stable virtual IP (VIP) that clients connect to, then forwarding each incoming connection to a healthy backend chosen by a scheduling algorithm — so the client sees a single unchanging address while requests fan out across a pool of servers that can grow, shrink, or fail underneath it.

That indirection buys three things at once: horizontal scale (add servers behind the VIP without touching clients), availability (a dead backend is detected and taken out of rotation), and a seam for operations (rolling deploys, TLS termination, and traffic shifting all happen at the LB). The scheduling algorithm itself — round-robin, least-connections, consistent hashing — is a topic of its own (see Load Balancing Algorithms); this page is about the machinery around it: how the LB decides a backend is alive, how it keeps a user pinned to one server when it must, and where in the stack the balancing actually happens.

A request, traced

Configure the LB with a VIP of 203.0.113.10:443, the least-connections algorithm, and an HTTP health check: GET /healthz every 2 s, mark a backend unhealthy after 3 consecutive non-200 responses and healthy again after 2 consecutive 200s. Watch the pool absorb a backend failure without the client noticing.

Two subtleties fall out of the trace. First, there is a detection window (t=2 to t=6 here — 4 s in this trace, and up to interval×threshold = 2×3 = 6 s in the worst case, when the server dies just after a probe) during which the LB still sends live requests to a broken server — those users get errors until the third probe fails. Tighter intervals shrink the window but raise probe load and flap risk. Second, the freshly re-added web-2 starts at zero connections, so least-connections will slam it with the next several requests — a cold backend stampede. Slow-start ramping (gradually raising a recovered backend's weight) exists precisely to soften this.

Where load balancers sit

To use scale and redundancy at every tier, you balance load at each hop where one layer talks to a pool of the next. Typical placements:

• User → web tier — the public-facing LB behind the VIP (often an L7 reverse proxy doing TLS termination).
• Web tier → application / cache tier — an internal LB spreading requests across stateless app servers.
• Application tier → database — routing reads across replicas and writes to the primary (a role often played by a database proxy like ProxySQL or PgBouncer rather than a generic LB).

Each LB you add is itself something that can fail, which is why the LB tier is deployed as a redundant pair (covered below) rather than a lone box.

L4 vs L7: how deep does it look?

The single biggest choice is which layer the LB operates at, because it decides how much of the request the LB can understand.

L4 (transport-layer) load balancing makes its decision from the TCP/UDP header alone — source and destination IP and port. It never reads the payload, so it cannot tell an HTTP /checkout from /health; it just pins a connection (usually by a hash of the connection 5-tuple — src/dst IP, src/dst port, protocol; you'll also see "4-tuple" when the protocol is fixed) to a backend and shovels bytes. Because it does almost no work per packet, it handles enormous connection rates cheaply, is protocol-agnostic (works for raw TCP, gRPC, databases, game traffic), and can preserve the client's view of a long-lived connection. AWS's Network Load Balancer is the canonical example.

L7 (application-layer) load balancing terminates the TCP/TLS connection, parses the HTTP request, and routes on content: Host header, URL path, cookies, method. That unlocks path-based routing (/api/* to one pool, /static/* to another), request-level retries, header rewriting, response caching, and cookie-based sticky sessions. The cost is a full proxy per request (more CPU, especially for TLS) and being HTTP-shaped. NGINX, HAProxy in HTTP mode, and AWS's Application Load Balancer live here.

Health checks: how the LB knows a server is alive

Ejection is only as good as the probe. There are three depths, and picking too shallow a one is a classic outage:

• L3 (ICMP ping) — the host answers, but the app process may be dead. Nearly useless for detecting application failures.
• L4 (TCP connect) — the port accepts a connection. But an app stuck in a deadlock or GC pause can still accept() the socket while never serving a real response. A TCP-only check keeps routing traffic into a black hole.
• L7 (application check) — GET /healthz and require a 200. This exercises the request path and is the only check that catches "process is up but broken."

The health endpoint should be shallow but honest: verify the server can serve its own traffic (thread pool, warm caches) without transitively pinging every downstream dependency. A /healthz that returns 503 whenever the shared database is slow will cause every backend to fail its check at once — the LB ejects the entire pool and you have converted a slow dependency into a total outage. Separate a liveness check (is this process healthy in isolation?) from a readiness check (should it receive traffic right now?), and never let a shared dependency flip all liveness checks together.

Two knobs govern responsiveness vs. stability: the interval (how often you probe) and the unhealthy/healthy thresholds (consecutive failures/successes to change state). Requiring N consecutive results is what prevents a single dropped packet from flapping a backend in and out of rotation.

Sticky sessions (session affinity)

By default an L7 LB may send a user's successive requests to different backends. That breaks if the server holds per-user state in local memory — a login session, a half-filled cart, an upload in progress — because the next request lands on a server that has never heard of that user. Session affinity pins a client to one backend so its state stays reachable. Two mechanisms:

• Cookie-based (L7) — the LB sets a cookie (e.g. HAProxy's SERVERID, ALB's AWSALB) naming the chosen backend; subsequent requests carrying that cookie are routed back to it. Precise, survives client IP changes, but requires the LB to parse HTTP.
• Source-IP hash (L4) — hash the client IP to a backend. Works for any protocol, but a whole corporate NAT or mobile carrier can collapse to one IP and one backend, and a client's IP changing (Wi-Fi → cellular) loses the affinity.

The trap: affinity fights load balancing. If sticky sessions are long-lived, load skews toward whichever backends caught the heavy users, and it gets worse after a scale-out — new servers sit idle because existing users are still pinned to the old ones. And when a pinned backend dies, those users lose their session outright.

The senior move is to not need affinity: externalize session state to a shared store (Redis, a signed JWT/cookie, a database) so every backend can serve every user. Then the app is genuinely stateless, any algorithm balances evenly, and a dead backend costs nobody their session. Reach for sticky sessions only when you cannot externalize state cheaply, or as a bridge while migrating a legacy stateful app.

DNS load balancing vs. anycast vs. a real LB

"Load balancing" also happens before the request reaches any VIP — at the network edge — and the two edge techniques behave very differently from the reverse-proxy LB above.

DNS load balancing hands out different A/AAAA records for the same hostname — round-robin, weighted, or geo-based (GeoDNS). It is dead simple and distributes clients across regions or data centers, but it is coarse and slow to react: resolvers and browsers cache records for the TTL, so a dead IP can keep receiving traffic for minutes after you pull it, and the resolver has no health awareness — it will happily return the address of a down server. DNS is a good first hop (steer users to the nearest region) but a poor last line of defense.

Anycast announces the same IP address from many locations via BGP; the internet's routing fabric delivers each client to the topologically nearest site. Failover is handled by BGP itself — withdraw the route at a failed site and traffic reconverges to the next-nearest in seconds, with no DNS TTL to wait out. This is how CDNs and large DNS providers (Cloudflare, Google) put one IP in front of hundreds of POPs. The cost is operational: you need to run BGP and control your address space, and long-lived TCP connections can occasionally break if routes shift mid-flow.

These compose rather than compete: GeoDNS or anycast picks the region, and inside that region an L4/L7 LB behind a VIP picks the server — with real per-request health checks that DNS could never provide.

Redundant load balancers (the LB is not a SPOF)

A single load balancer is itself a single point of failure — if it dies, the whole pool behind it goes dark no matter how healthy the backends are. The fix is a pair (or cluster) in active-passive or active-active mode. The two LBs exchange heartbeats and share the VIP: in active-passive, the standby holds the VIP unused (commonly via VRRP / a floating IP) and claims it within a second or two when the active's heartbeat stops; in active-active, both serve traffic and each can absorb the other's share on failure. Managed cloud LBs (ALB/NLB, Google Cloud LB) do this for you — the "single" LB you provision is already a redundant, horizontally-scaled fleet behind one address.

Pitfalls

• TCP-only health checks hide app failures. The port accepts connections while the app is deadlocked or OOM-thrashing; the LB keeps routing into a black hole. Use an L7 /healthz that exercises the request path.
• A dependency-coupled health check ejects the whole pool. If every backend's /healthz pings the same slow database, one slow dependency fails all checks simultaneously and the LB removes every server — a self-inflicted total outage. Keep liveness checks local.
• Retries amplify overload. An LB that retries failed requests onto other backends turns one struggling server into a cascade: the retry storm piles the failed load onto the survivors, which then fail too. Cap retries, use retry budgets, and pair with circuit breaking and load shedding.
• Sticky sessions defeat scale-out. New servers stay idle because existing users are pinned to old ones; load stays skewed until sessions expire, and a dead backend loses its users' sessions. Externalize state instead.
• Ignoring connection draining. Removing a backend (deploy, scale-in) that still has in-flight requests kills those requests. Enable draining so existing connections finish before the server is cut.
• DNS TTL outlives the outage. Relying on DNS round-robin for failover means cached records keep sending users to a dead IP for the full TTL. DNS steers regions; it does not do fast health-based failover.
• Cross-zone traffic cost and TLS CPU. Spreading connections across availability zones can incur inter-AZ data charges, and terminating TLS at an L7 LB is CPU-heavy at scale — both are real budget lines, not footnotes.

When to use which — and the trade-offs

The decision is rarely "should I load balance?" (yes) but "at which layer, and with how much intelligence?"

• Choose L4 when you need raw throughput and low latency, the protocol isn't HTTP (gRPC streaming, databases, MQTT, game servers), you want to preserve the client's connection end-to-end, or you're doing TLS pass-through for end-to-end encryption. You gain speed, cheapness, and protocol independence; you lose content-based routing, per-request retries, and HTTP-aware stickiness.
• Prefer L7 when you need path/host-based routing, header or cookie logic, request-level retries, response caching, WAF integration, or cookie sticky sessions. You gain rich request control; you pay in CPU (a full proxy plus TLS termination per request), added latency, and being HTTP-shaped. Many real deployments stack both: an L4 NLB at the edge for scale and static IPs, fronting L7 proxies for routing.
• Use DNS / GeoDNS to distribute clients across regions or data centers — coarse, global, zero per-request cost. Do not lean on it for fast failover; TTL caching and no health awareness make it slow to react. Prefer anycast when you need one IP globally with second-scale BGP failover (and can run BGP), and prefer a VIP-fronted L4/L7 LB for actual server selection with health checks inside a region.
• Sticky sessions vs. stateless: choose stickiness only when you can't externalize per-user state cheaply; otherwise externalize state and stay stateless — you trade a little infrastructure (Redis) for even load, painless scale-out, and no session loss on failover.

Rule of thumb: global routing → DNS/anycast; regional server selection → L4 for throughput, L7 for smarts; and design the app to be stateless so the balancer's job stays simple.

Takeaways

• An LB publishes one VIP and forwards each connection to a healthy backend chosen by an algorithm — buying scale, availability, and an operational seam behind a stable address.
• Availability lives or dies by the health check: use L7 probes that exercise the request path, keep them local so a shared dependency can't eject the whole pool, and require consecutive results to avoid flapping.
• L4 vs L7 is the core choice — packets by IP:port (fast, protocol-agnostic) vs. parsed HTTP (content routing, retries, sticky cookies, at higher CPU/latency).
• Prefer stateless backends so any request can hit any server; reach for sticky sessions only as a last resort, and let DNS/anycast handle region selection while a VIP LB handles server selection.

Sources: Grokking the System Design Interview (original placement and redundancy framing); NGINX and HAProxy documentation (L4/L7 modes, health checks, cookie affinity, connection draining, slow-start); AWS Elastic Load Balancing docs (ALB vs NLB, cross-zone, connection draining) and Google Cloud Load Balancing docs; Cloudflare Learning Center (anycast, DNS load balancing); Kubernetes docs (liveness vs readiness probes). Re-authored and deepened for this guide with a traced worked example, hand-drawn diagrams, and selection trade-offs; load-balancing algorithms are covered on the next page.