Load Balancing — Tier Sizing, Retry Storms, Load-Shedding, GSLB & L7-vs-Volumetric (Deep Dive)

The load balancer is a system you have to size, not just configure

Every other Load Balancing lesson in this guide teaches the LB as a routing decision — which algorithm, which layer. This one teaches the LB as infrastructure you must provision, that fails in specific ways, and that other tiers depend on behaving well: how many boxes you actually need, what happens when a slow tier makes everyone retry, how a duplicate write from that retry stops being dangerous, when the fleet should deliberately say no, how a request finds the right region before it ever reaches your LB, and why the LB itself cannot stop a volumetric flood.

1. Tier sizing: turning capacity ceilings into an instance count

An LB instance runs out of capacity along three independent ceilings, and you must size against all three — whichever is worst wins:

• CPU / TLS handshake cost. The expensive part of TLS is the asymmetric key exchange (ECDHE) done once per new connection, not the symmetric bulk encryption after (AES-NI makes that nearly free). So the CPU ceiling scales with new-connection churn (handshakes/sec), not with steady bandwidth.
• NIC bandwidth (Gbps). A hard line-rate ceiling per instance, independent of CPU — a balancer can have idle cores and still be saturated on the wire.
• Memory / max concurrent connections. Every open connection holds socket buffers + TLS session state in RAM. Enough concurrent connections exhausts memory long before CPU or bandwidth do.

Worked example. Target load at peak: 250,000 new TLS handshakes/sec (churn — many short-lived mobile/API clients), 40 Gbps sustained throughput, 2,000,000 concurrent open connections. Each LB instance is an 8-vCPU box with a 10 Gbps NIC and 32 GB RAM.

Ceiling	Per-instance capacity	Basis	Instances needed
CPU / TLS handshakes	60,000 handshakes/sec (6 of 8 cores × ~10,000/core; 2 cores reserved for OS, health checks, L7 parsing)	250,000 ÷ 60,000	5 (rounds up from 4.2)
NIC bandwidth	7 Gbps usable (10 Gbps line rate × ~70% safety margin)	40 ÷ 7	6 (rounds up from 5.7)
Memory / concurrent conns	500,000 conns (24 GB budget ÷ ~48 KB per connection's TCP+TLS state)	2,000,000 ÷ 500,000	4

Since every instance adds capacity on all three axes at once, the fleet only clears every ceiling once it clears the worst one: N = max(5, 6, 4) = 6 instances. Here bandwidth — not CPU, the one people reach for first — is the binding constraint. Add N+1 redundancy so one instance failing mid-peak doesn't drop you below any ceiling: 7 instances provisioned.

The habit this trains: never size off one number (usually average QPS). Compute the instance count each ceiling implies independently, then take the max — and only then add N+1. A plan sized off CPU alone in the example above would stop at 5 instances — enough handshake capacity, but only 5 × 7 = 35 Gbps of NIC headroom against the 40 Gbps target: a 12.5% bandwidth shortfall that only shows up once real peak traffic arrives.

2. Retry storms: how one slow tier becomes an outage

A request rarely crosses one retrying layer — it typically crosses three: the edge/LB tier retries a failed upstream, the service tier retries its own downstream call, and a DB-proxy/connection-pool tier retries a failed query. If each tier independently retries up to 3×, the attempts compound: 3 × 3 × 3 = up to 27× the original request rate can land back on the one dependency that is already struggling — see the diagram above. This is the LB-tier-specific, worst-case multiplicative view; the general feedback-loop version of the same failure (effective load = arrivals ÷ (1 − p), which is what actually happens continuously as retry-probability p climbs) is derived in Traffic Amplification: Fan-out & Retry Storms — this section is the "why does it specifically compound per LB tier" companion to that trace.

The result is a metastable failure: the original trigger (a GC pause, a hot shard) can clear in seconds, but the retried backlog it created is now the actual bottleneck, and the system does not recover on its own — it keeps re-overloading the one place trying to heal.

The fix — three independent controls, not one

• Retry budgets. Cap total retry traffic to a small fraction of normal traffic — e.g. a token bucket that allows retries only up to ~10% of the request rate. Once the budget is spent, further failures fail fast instead of retrying — the fleet degrades to "some errors" rather than "everyone retries into collapse."
• Exponential backoff + jitter. Fixed-interval backoff re-synchronizes every client's next attempt into the same instant (a thundering herd); backoff that grows exponentially and adds randomness spreads retries out in time so they don't recreate the spike that caused the failure.
• Retry at ONE layer only — usually the edge. If the edge LB, the service, and the DB proxy all retry the same failure independently, their retries compound (the 27× above). Pick one layer — typically the outermost, where the client-facing budget is easiest to reason about and enforce — and make every inner tier fail fast and propagate the error upward instead of also retrying.

3. Idempotency keys: how a retried write stops being a double-charge

Every lesson that assumes "clients retry idempotently" skips the mechanism that actually makes a repeated POST safe. Here it is: an idempotency key — a UUID the client generates once per logical operation (not per attempt) and sends on every retry of that same operation, e.g. a header Idempotency-Key: 7e5a... on a POST /charge.

On the server, the write and the key are recorded atomically, in the same transaction: a row keyed by the idempotency key stores "in-flight" before the charge runs, then the result once it completes. When a request arrives with a key that already exists, the server does not re-execute the write — it looks up and replays the stored result.

Traced

Step	What happens
1	Client generates key k1, sends POST /charge {amount:50, key:k1}.
2	Server: no row for k1 → charges the card, commits {k1 → success, charge_id:C1} in one transaction.
3	The response is lost (LB failover, client timeout) before the client sees it.
4	Client retries the same logical request — same key k1 — not a new one.
5	Server: row for k1 already exists → skips the charge, returns the stored result C1.
6	Customer is charged once, regardless of how many times the network made the client retry.

The single most common way to defeat this: generating a new key on every retry attempt. The key must be stable for all retries of one logical action and only change when the user initiates a genuinely new one (a second, separate purchase). For the deeper "why exactly-once delivery is a myth and idempotency is the real answer" argument, see Idempotency & Exactly-Once Is a Myth — this section is the concrete key-and-dedup-store mechanism that page's framing depends on.

4. Load-shedding vs. rate-limiting — deliberately saying no to protect goodput

These are two different controls that are easy to conflate:

• Rate limiting is a per-client policy — a token-bucket contract enforced against one caller's identity, regardless of whether the server is under stress. It fires the same way at 10% load or 100% load; its job is fairness among clients, not fleet survival.
• Load shedding is a server-side, load-dependent decision, made only once the fleet itself is near or at saturation — the server watches its own queue depth / latency / CPU and starts rejecting a slice of all incoming traffic (503 + Retry-After), independent of any single client's quota, to keep serving the requests it can still serve well.

Why shedding beats collapsing: an overloaded system that keeps accepting and queuing everything doesn't fail gracefully — its queues grow, latency blows past every client's timeout, and effectively nothing completes successfully (goodput → 0), even though the server is "trying." A system that sheds the excess the instant it detects saturation returns a cheap, fast 503 for the rejected slice and keeps the accepted slice inside its normal latency budget — bounded, non-zero goodput beats unbounded, zero-goodput collapse.

Shed by value, not randomly. Reject the least-valuable requests first — background sync, free-tier reads, non-critical analytics pings — before checkout, payment, or authentication traffic. This requires the gateway/LB to carry a request-cost or request-tier signal, not just a client identity. Always return Retry-After with jitter so the shed clients don't all retry in lockstep and recreate the exact spike you just shed (tying back to Section 2). The general back-pressure family — bounded queues, flow control, "where does the mismatch accumulate" — is covered in Back-pressure & Flow Control; this section is the LB-specific policy question of who gets shed and why it is a different knob from a client's rate limit.

5. GSLB traced end-to-end: one request, resolver to instance

Before a request ever reaches the regional LB tier sized in Section 1, it has to find the right region. That's Global Server Load Balancing (GSLB): DNS-based steering (or anycast) that answers "which region" before any regional load balancer answers "which instance."

The trace

1. Client in Mumbai resolves app.example.com. Its recursive resolver queries the authoritative GSLB nameserver (rather than a plain DNS server that just hands back a fixed IP).
2. The GSLB nameserver picks a region using live signals: which regions are healthy (continuous health probes), which is closest (geo-IP or EDNS Client Subnet so it doesn't answer based on the resolver's location instead of the client's), and which has load headroom.
3. It returns the VIP for ap-south-1 with a TTL (here 60s). The client's OS caches that answer for the full TTL — every connection during that window goes straight to ap-south-1 without asking GSLB again.
4. The regional VIP hands off to the regional LB tier — the fleet sized in Section 1 — which picks an instance by whatever algorithm fits (least-connections, power-of-two-choices; see Load Balancing Algorithms — Traced).
5. The instance serves the request.

The DNS-TTL staleness limit on failover speed: if ap-south-1 degrades a few seconds after the client cached its answer, GSLB's next answer to a new lookup will steer elsewhere — but this client isn't doing a new lookup. It keeps sending traffic into the dying region until the TTL expires or the application layer forces a fresh resolution. A shorter TTL bounds this tighter but multiplies query load on the GSLB nameservers; a longer TTL is cheaper but slower to fail over. Anycast does not have this bound — convergence happens in BGP routing, independent of any client's DNS cache, which is why it fails over "below the TTL." The deeper mechanics of Geo-DNS/GSLB and anycast (and why CDNs stack all three) live in DNS Load Balancing and High Availability and DNS failover: two layers, one tradeoff — this trace is the single-request walk that ties GSLB into the regional tier the rest of this page is about.

6. The senior caveat: an L7 LB is a weak defense against volumetric floods

An L7 load balancer must terminate every TCP connection and complete every TLS handshake before it can read a single byte of the request it's supposed to be smart about. That means it holds exactly the same SYN backlog, connection table, and NIC bandwidth limits as any other server — a volumetric SYN flood or a UDP reflection/amplification attack doesn't need to understand your routing rules; it only needs to exhaust the LB's own connection state or line rate. The L7 LB is the victim of that attack, not the shield against it — the SYN-backlog exhaustion math and the amplification case study (a 1.35 Tbps memcached flood) are traced in full in What are DDoS Attacks; the point here is narrower and LB-specific: don't budget your L7 tier as your DDoS control.

Real defense sits upstream of the L7 tier: anycast to spread the flood's packets-per-second across many PoPs so no single stack absorbs the full volume, a scrubbing service (a cloud provider's managed DDoS protection) that filters malicious packets before they reach your infrastructure, SYN cookies at the network edge, and connection-rate caps at L3/L4 — all in front of the L7 balancer, so that by the time traffic reaches it, it's already legitimate-looking.

Judgment layer

• Retry at the edge only, vs. retry everywhere. Edge-only retries bound the worst case to 1× the budgeted amount and are trivial to reason about and cap. Retrying at every hop is more forgiving of a single transient blip at any one layer, but the layers compound (Section 2's 27×) and it is much harder to reason about the aggregate load a failure generates. Default to edge-only + a budget; only let an inner tier retry its own call when that call is cheap, provably idempotent, and has its own, separate budget.
• Shed vs. rate-limit. Rate limiting protects fairness among well-behaved clients under normal load and keys off client identity. Load shedding protects the server's own survival under abnormal load and keys off the server's own saturation signal. You need both, and they fire at different times — neither one substitutes for the other.
• L7 routing vs. L3/L4 protection. The L7 LB is a correctness and routing layer: it assumes traffic already made it past a volumetric defense. Dedicated L3/L4 scrubbing and anycast absorption are the actual DDoS shield. Conflating the LB's resilience features (health checks, retries, per-IP limits) with a security control is the mistake a senior engineer is expected to catch in an interview.

Pitfalls

• Sizing an LB tier off one number (usually average QPS) instead of checking all three independent ceilings — CPU/handshake churn, NIC bandwidth, and concurrent-connection memory — and taking the worst.
• Letting every hop retry independently with no shared budget — the multiplicative 3×3×3 storm is a self-inflicted wound, not something the failing dependency did to you.
• Minting a new idempotency key on every retry attempt, which silently defeats the entire mechanism and lets a lost-response retry double-charge.
• Treating a 503 from load shedding as a bug to eliminate rather than a designed control — the alternative is an unbounded queue and a total collapse in goodput.
• Assuming a short DNS TTL means fast failover — client-side caching and app-level connection reuse can make the real failover time closer to the TTL's worst case than its average.
• Answering a "how do you handle DDoS" interview question with "we have a load balancer" — the LB is not a volumetric-attack control.

Takeaways

• Size an LB tier against all three ceilings — CPU/handshakes, NIC Gbps, and connection memory — and provision for the worst one, then add N+1.
• Retry storms are multiplicative across tiers (up to 27× for 3 tiers × 3 retries); fix with a retry budget, jittered backoff, and retrying at one layer only.
• An idempotency key, recorded atomically with the write, is what actually makes "retry safely" true — without it, "idempotent retries" is a claim, not a mechanism.
• Load shedding (server decides, saturation-triggered, protects goodput) and rate limiting (client policy, always-on, protects fairness) are different controls; GSLB gets you to the right region but is bounded by DNS TTL; and the L7 LB is never your DDoS shield — that job belongs to L3/L4 scrubbing and anycast, upstream of it.

---

Re-authored/Deepened for this guide. Sizing arithmetic and retry/GSLB traces hand-authored; synthesizes Google SRE Book (overload handling, retry budgets), Release It! (Nygard — stability patterns, metastable failure), "The Tail at Scale" (Dean & Barroso) framing on cascading load, and standard GSLB/anycast practice (Cloudflare, AWS Route 53 latency/geo routing docs). See also: Load Balancing Algorithms — Traced; Traffic Amplification: Fan-out & Retry Storms; Idempotency & Exactly-Once Is a Myth; Back-pressure & Flow Control; DNS Load Balancing and High Availability; What are DDoS Attacks; High Availability and Fault Tolerance.

I just read a lesson on **Load Balancing — Tier Sizing, Retry Storms, Load-Shedding, GSLB & L7-vs-Volumetric (Deep Dive)** (System Design) and want to truly understand it. Explain Load Balancing — Tier Sizing, Retry Storms, Load-Shedding, GSLB & L7-vs-Volumetric (Deep Dive) from first principles using ONE vivid real-world analogy and a visual mental model — draw it as ASCII art or a clear step-by-step diagram — with a concrete example using real numbers. Then ask me one question to check I got the mental picture, and wait for my reply. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Teach me **Load Balancing — Tier Sizing, Retry Storms, Load-Shedding, GSLB & L7-vs-Volumetric (Deep Dive)** interactively. Ask me ONE guiding question at a time, wait for my answer, and adapt to my confusion — build the idea with me step by step instead of explaining it all at once. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Quiz me on **Load Balancing — Tier Sizing, Retry Storms, Load-Shedding, GSLB & L7-vs-Volumetric (Deep Dive)** with 5 questions, easy to tricky, ONE at a time. Tell me if each answer is right; at the end, explain clearly what I got wrong and why. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Help me remember **Load Balancing — Tier Sizing, Retry Storms, Load-Shedding, GSLB & L7-vs-Volumetric (Deep Dive)** for the long term: give the one-sentence intuition, a memorable hook/mnemonic, a tiny worked example, and 3 active-recall flashcards (Q -> A). If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.