Scalability and Performance

A load balancer scales an application by fanning one queue of client connections across N backends, so aggregate throughput rises roughly linearly with N — but the load balancer itself is a box with a finite packet, connection, and (for L7/TLS) CPU budget, so the real engineering question is how far you can push that box before it becomes the bottleneck, and how much latency the extra hop it inserts actually costs.

Scaling the load balancer itself: vertical vs horizontal

The LB has two independent ceilings. A well-tuned L7 instance (nginx/HAProxy/Envoy class) might sustain on the order of 1M idle keep-alive connections and tens of Gbps of proxied bytes, but only ~10–20k new TLS 1.3 handshakes/second per core with modern ECDSA-P256 certificates (and only ~1–2k/core with RSA-2048 — the private-key op is ~10× heavier) — because the ECDHE handshake plus signature is real asymmetric-crypto CPU work. State the key type before quoting a number: an unqualified "handshakes/core" figure is meaningless. You usually hit the handshake ceiling long before the bandwidth ceiling.

Where the number comes from: a TLS 1.3 server handshake = 1 ECDHE key agreement + 1 server signature + hashing/record overhead, so the crypto ceiling composes as 1/(t_ecdh + t_sign + overhead). Benchmark your own box with openssl speed ecdsap256 ecdhx25519 rsa2048 — the figures vary by CPU generation and TLS stack, so verify on target hardware. Example composition: if X25519 ≈ 30k ops/s/core and ECDSA-P256 sign ≈ 30k ops/s/core, the crypto alone caps you near 15k handshakes/s/core, and stack overhead takes you toward 10k — which is why the range above is 10–20k, and why RSA-2048 (~1.5k signs/s/core) drags the whole handshake to ~1–2k.

• Vertical scaling — add cores/NICs to the existing instance. TLS handshakes parallelize across cores, so doubling cores roughly doubles handshake capacity. Cheap and instant, but capped by the largest instance and gives you no second failure domain.
• Horizontal scaling — run multiple active-active LB instances and spread clients across them with DNS round-robin or (better) BGP anycast, so every instance advertises the same VIP and the network routes each client to the nearest healthy one. This is how you go past a single machine and get HA for free — the tier that distributes load must not itself be a single point of failure.

Worked trace: what the extra hop actually costs

Take an L7 LB that terminates TLS and forwards to backends in the same availability zone. Concrete latencies: client↔LB RTT 40 ms (public internet), LB↔backend RTT 1 ms (same AZ). The interesting cost is the LB→backend leg, and it depends entirely on whether the backend connection is cold (opened fresh) or warm (pulled from a keep-alive pool).

So connection reuse removes 2 ms of wall-clock latency per request — but the bigger win is CPU: it also removes one TLS handshake per request from the backend. At 50,000 req/s, the cold path forces 50,000 handshakes/s of crypto work on your fleet; the warm path forces ~0. That is why keep-alive is the single highest-leverage LB tuning knob.

Where HTTP/2 and QUIC fit: HTTP/2 multiplexes many concurrent requests as streams over one warm connection, so a handful of pooled connections serve thousands of in-flight requests (fewer sockets, fewer handshakes). QUIC (HTTP/3) runs over UDP and fuses the transport + TLS handshake into 1 RTT — or 0-RTT on resumption — and, because each stream has its own delivery, it avoids the TCP head-of-line blocking where one lost packet stalls every multiplexed stream on that connection.

Rate/connection limits and caching

Connection and rate limits protect both the LB and the backends from being overwhelmed. The LB enforces caps keyed on client IP, domain, or URL pattern — for example, a token-bucket limiter of 100 req/s per IP with a burst of 200, or a ceiling of 10k concurrent connections per backend. These caps do double duty: they blunt abusive clients and DoS floods, and they act as a bulkhead so one hot tenant cannot starve everyone else. The key mechanism to understand is that a per-IP limit is nearly useless against a botnet spread across thousands of IPs — that is what layered limits (per-IP + global + per-route) are for.

Caching at the LB lets the LB answer for static assets (images, CSS, JS) directly from its own memory/disk, so those requests never touch a backend and return in sub-millisecond time instead of paying the 1 ms hop plus backend work. Many L7 LBs also offload gzip/brotli compression and TLS, freeing backend CPU. The trade-off is that an LB cache is a single-region, relatively small cache — it is a convenience, not a substitute for a real cache tier or CDN.

Pitfalls

• Keep-alive backfires under fan-out. Each LB instance keeps a warm pool to every backend. With 20 LB instances × 200 pooled connections each × 50 backends you are holding 200k backend sockets open, and backends run out of file descriptors (EMFILE) while mostly idle. Cap pool size per backend and tune idle timeouts.
• Idle-timeout mismatch causes phantom 502s. If the backend closes an idle keep-alive connection at 60 s but the LB still thinks it is warm and forwards a request onto it, the client gets a 502. Rule for the LB→backend keep-alive pool: the LB's upstream idle timeout must be shorter than the backend's, so the LB never reuses a socket the backend has already closed. (The opposite discipline applies on the client side: the LB's client-facing idle timeout must be longer than the longest legitimate quiet period — WebSockets, streams — or the LB kills live connections; see the Intro pitfall.)
• TLS handshakes are the hidden ceiling. A fleet looks fine on bandwidth graphs, then a thundering-herd of cold clients (cache purge, mobile app relaunch) drives new-connection rate up and the LB pegs CPU on handshakes. Enable TLS session resumption / 0-RTT and reuse backend connections.
• DNS round-robin is not load balancing. Clients cache DNS answers past the TTL, so a newly added LB gets no traffic for minutes and a removed one keeps getting hits. It splits clients, not load; use anycast or a real balancing layer for even distribution.
• LB-cache staleness on dynamic content. Caching something that looks static but is per-user (a personalized page, a signed URL) leaks one user's response to another. Cache only truly shared, cacheable responses and honor Cache-Control.

When to use it / when NOT to

Horizontal vs vertical scaling of the LB

Signals that point to horizontal: you are approaching a single instance's handshake/connection ceiling; you need a second failure domain for HA; you run in an elastic cloud where adding instances is a config change. Signals that point to vertical: you are connection/CPU-bound but nowhere near instance limits; ops simplicity matters; or your LB is licensed per-instance and more cores is cheaper than more licenses.

Trade-off: horizontal buys you unlimited headroom and HA, but costs you a distribution layer above the LBs (anycast/DNS), cross-instance state problems (sticky sessions, rate-limit counters, connection pools no longer shared), and more moving parts to operate. Vertical is dead simple and has zero distribution cost, but has a hard ceiling and leaves you one reboot away from a full outage. Choose horizontal when you need to exceed one machine or survive one dying; prefer vertical as the fast first move when a bigger box still fits and a single failure domain is acceptable.

Caching at the LB vs a dedicated CDN / cache tier

Trade-off vs a CDN: LB caching lives next to your origin — cheap, no extra vendor, good for shrinking backend load in one region. A CDN caches at the edge near users, killing the 40 ms internet RTT for static assets, but adds cost, cache-invalidation complexity, and another party in the request path. Choose LB caching for reducing origin/backend load on shared content within a region; prefer a CDN when the bottleneck is geographic latency to static, globally-shared assets. They compose: CDN at the edge, LB cache at the origin.

Takeaways

• Scale the app by fanning connections across backends; scale the LB vertically (more cores → more TLS handshakes) or horizontally (active-active behind anycast/DNS, which also gives HA).
• The extra hop adds ~1–3 ms; connection reuse (keep-alive) turns a cold 3 ms + one TLS handshake into a warm ~1 ms with none — the CPU saving at scale dwarfs the latency saving.
• HTTP/2 multiplexes many streams over one warm connection; QUIC/HTTP-3 collapses transport+TLS setup to 1-RTT (0-RTT on resume) and dodges TCP head-of-line blocking.
• Rate/connection limits are a bulkhead against DoS and noisy neighbors — but per-IP limits alone don't stop distributed floods; layer per-IP + per-route + global.

Re-authored/Deepened for this guide. Sources: HAProxy and NGINX performance/tuning documentation; Cloudflare Learning Center (load balancing, keep-alive, QUIC/HTTP-3); RFC 9114 (HTTP/3) and RFC 9000 (QUIC); Ilya Grigorik, High Performance Browser Networking (O'Reilly) on TCP/TLS handshake costs; AWS ELB/ALB architecture guides. Illustrative latency and capacity numbers are order-of-magnitude for same-AZ deployments, not vendor guarantees.