Introduction to Load Balancing

A load balancer works by terminating the client's connection itself, then — per request or per connection — picking one backend from a pool it continuously health-checks, and forwarding the traffic there; because the pool membership and the pick are the load balancer's decision, a dead server is simply skipped and no single server ever sees more than its share.

That single mechanism is what buys you three things at once: availability (traffic routes around a failed node), scalability (add a node to the pool and it starts absorbing load), and performance (spread the work so no box saturates). The load balancer sits in the request path between the caller and the servers, which is also why it becomes the place you centralize concerns like TLS.

Where load balancers sit

You rarely have just one. To get scalability and redundancy at every tier, you balance load at each hop where a many-to-many relationship exists between callers and servers:

• User → web tier — usually an internet-facing L7 balancer (often behind DNS or anycast), also doing TLS termination.
• Web tier → application / cache tier — an internal balancer so any web node can reach any app node.
• App tier → database — typically a specialized proxy (e.g. read-replica routing) rather than a generic HTTP balancer, because the protocol and the read/write asymmetry matter.

Each layer can fail independently, so each layer gets its own pool and its own health checks.

Traced example: round-robin across a burst, with a failure mid-stream

Three identical backends S1, S2, S3. The balancer keeps a rotating index over its healthy list and hands each new request to the next entry. A health check runs every few seconds. Eight requests arrive; between r4 and r5 the checker marks S2 down (two consecutive failed probes), so the healthy list shrinks to [S1, S3] and the rotation continues over the survivors.

(This balancer continues its rotation and skips the dead entry — after r4 hit S1, the next slot was S2, now dead, so r5 falls through to S3. An implementation that instead resets its rotation index when the healthy list changes would send r5 to S1 — same totals, different order.)

Result: S1 served 4, S3 served 3, S2 served 1 and then zero — the client never saw an error, because the balancer stopped choosing S2 the moment it failed the probe. This is the whole value proposition in one trace: even distribution while everything is up, automatic routing-around when something dies.

Where naive round-robin goes wrong: it assumes requests are equal cost. If S1 gets stuck on a slow 30-second query while S3 handles 1 ms cache hits, round-robin keeps piling requests onto the overloaded S1 anyway. That is the motivation for least-connections (route to the backend with the fewest in-flight requests) — in the trace above, once S1 is deep in slow work, least-connections would steer new requests to S3 instead of blindly alternating.

The parts that make it work

The mechanism above only holds up because of a few supporting pieces. Each one exists to solve a concrete failure, not as vocabulary:

Pitfalls

• The balancer becomes the single point of failure. You removed the SPOF from your servers and created a new one. Run at least two balancers (active-active or active-passive) sharing a floating/virtual IP, or you have simply moved the outage.
• Health checks that lie. A shallow TCP-connect check passes while the app returns HTTP 500s, or a /healthz that only checks the process — not its database connection — keeps a brain-dead node in rotation. Probe something that actually exercises the dependency, but not so deep that a slow dependency ejects your entire fleet at once.
• Sticky sessions defeat balancing. Pin users to nodes and load skews — one node melts under whales while others idle, and losing that node drops every session on it. Prefer stateless backends with sessions in a shared store (Redis) so any node can serve any request.
• Idle-timeout mismatch. If the balancer's client-facing idle timeout is shorter than the backend's, it silently drops long-lived connections (WebSockets, streaming, slow reports); clients see mysterious resets. Align timeouts across the whole path.
• Thundering herd after a restart / cold cache. Round-robin floods a freshly added node with full traffic before its cache/JIT warms, spiking its latency. Use slow-start or gradual ramp to ease a new node into rotation.

When to reach for a load balancer — and when not to

Signals that point here: you run more than one instance of a service, you need zero-downtime deploys, you must survive a single node dying, or one box can no longer hold peak traffic. If any of those is true, put a balancer in front.

When NOT to: a single instance that fits your traffic with headroom does not need one — a balancer adds an extra hop, operational surface, and a new failure domain for zero benefit. Add it when you actually have (or are about to have) more than one backend.

Versus the named alternatives

• DNS round-robin (return multiple A records, let clients pick). Gain: nearly free, no extra box in the path. Cost: DNS TTL caching means a dead node keeps receiving traffic for minutes after it fails — there is no per-request health awareness, and distribution is coarse. Choose DNS RR only for crude geographic steering or as a first tier in front of real balancers; choose a real load balancer when you need fast failover and even distribution.
• Client-side load balancing (smart client / service mesh sidecar picks the backend). Gain: no central hop, so lower latency and no shared bottleneck — great for internal service-to-service traffic. Cost: every client needs the routing logic and a way to discover/health-check peers, which is heavy for external or polyglot clients. Choose client-side inside a mesh you fully control; choose a dedicated balancer for internet-facing traffic and untrusted/varied clients.
• L4 vs L7 (which balancer). An L4 balancer routes on IP/port only — cheap, fast, protocol-agnostic, but blind to the request. An L7 balancer reads the HTTP request, so it can route by path/host, terminate TLS, and retry idempotent requests — at the cost of more CPU and the ability to inspect (and be a bottleneck for) every request. Choose L4 for raw throughput or non-HTTP protocols; choose L7 when you need content-based routing, TLS termination, or per-request logic.

Takeaways

• A load balancer's power comes from owning two decisions per request: who is healthy (via health checks) and who gets this one (via an algorithm) — everything else follows from that.
• Round-robin is fair only when requests are equal cost; when they aren't, least-connections (or a latency-aware variant) beats it.
• The balancer is itself a SPOF — always run it redundantly, and keep backends stateless so failover and even distribution stay cheap.
• Match the tool to the traffic: L4 for speed and non-HTTP, L7 for content routing and TLS; DNS round-robin and client-side balancing are alternatives with real, specific trade-offs — not strictly worse options.

Re-authored and deepened for this guide. Synthesized from NGINX and HAProxy load-balancing documentation (algorithms, health checks, slow-start, connection draining), the AWS Elastic Load Balancing developer guide (L4/L7 and TLS termination), the Google SRE book (health checking and load-balancing frontends), and Grokking the System Design Interview (layered placement of load balancers). The round-robin trace and failure scenarios are hand-constructed.