CPU & Server-Count Playbook

The two formulas nobody teaches

This is the dimension almost every guide skips. Two equivalent lenses:

Little’s Law: concurrency N = λ × W (arrival rate × latency).
CPU cores: cores = (QPS × latency_ms) ÷ (1000 × target_utilization)

Worked example

10,000 QPS, each request burns ~10 ms of CPU, target 70% utilization:

• Work = 10,000 × 10 ms = 100,000 ms of CPU work per second = 100 core-seconds/sec (100 cores fully busy).
• ÷ 0.7 utilization ⇒ ~143 cores ⇒ ~18 eight-core servers (+ headroom).
• Cross-check with per-box anchors: 10K QPS ÷ ~1–5K QPS/server ≈ 2–10 servers if requests are cheap; the CPU math catches that 10 ms requests are not cheap.

CPU-bound vs IO-bound — this decides HOW you scale

	CPU-bound	IO-bound
Bottleneck	Computation (encoding, ML, compression)	Waiting on DB / network / disk
Concurrency vs cores	N ≈ cores	N ≫ cores (threads mostly wait)
Scale by	Add cores / machines	Add concurrency (threads/async) + fix the slow dependency

The kitchen analogy (why 70%, not 100%)

You hire 30 chefs for a 20-dish peak so a sudden rush doesn’t halt the kitchen. Run servers at ~70% so latency stays sane under bursts — queues explode as utilization approaches 100%.

Sizing for p99, Not the Mean — the Utilization Knee

The formula above sizes for throughput: enough cores to chew through the work per second. But throughput sizing is silent on latency. A box can have plenty of average headroom and still miss its SLO, because requests do not arrive politely spaced — they clump, and a busy server makes the next arrival wait behind the clump. That waiting is invisible to the core-count math and dominant to your users.

Where the wait comes from (M/M/1 intuition)

Model one server as a single queue with random (Poisson) arrivals and random service times — the classic M/M/1 queue. Let ρ (rho) be utilization: the fraction of time the server is busy. The mean number of requests in the system, and hence the mean time a request spends waiting plus being served, scales as:

relative wait ∝ 1 / (1 − ρ)

The mechanism: for a new request to start, the server must first clear whatever is already queued. As ρ rises, the odds that something is already in front of you rise too — and near ρ = 1 the queue almost never drains, so the wait runs away to infinity. This is not a modeling artifact; it is why the same hardware feels instant at moderate load and falls off a cliff at high load, with no warning in the average CPU graph until it is too late.

The blow-up, tabulated

Utilization ρ	1 / (1 − ρ)	Relative to ρ=0.5	Feel
0.50	2.0×	1.0×	Flat, roomy
0.70	3.3×	1.7×	Still fine — the knee
0.80	5.0×	2.5×	Tail starting to smear
0.90	10.0×	5.0×	p99 hurts
0.95	20.0×	10.0×	Falling off the cliff

Read the shape, not the numbers: from 0.5 to 0.7 the wait barely moves; from 0.8 to 0.95 it quadruples. That elbow around 70–80% is the utilization knee. Below it, latency is flat and boring; above it, latency is a rocket. This is the quantitative reason the kitchen analogy earlier picks 70% and not 100% — and it is why a box sitting at 80% average CPU can already show a p99 several times its mean — the M/M/1 model alone puts p99 at ~4.6× the mean response time (exponential tail: W·ln 100), and real service-time distributions are heavier-tailed than exponential, which is how production p99s reach 10× and beyond. The mean is dragged down by the many fast requests, while the unlucky ones that land behind a clump eat the full 1/(1−ρ) penalty. Averages hide tails; the knee is where the tail escapes.

Re-size against the SLO, not against throughput

So invert the earlier sizing. Instead of "how few cores clear the work," ask "how many cores keep ρ below the knee so p99 meets the SLO." Concretely: pick a target ρ ≈ 0.6–0.7 and divide the busy-core count by it. Reworking the earlier example — 100 core-seconds/sec of actual work — at ρ=0.7 you get ~143 cores (as shown), but if the SLO is tight you drop to ρ=0.6 and provision ~167 cores. Those extra ~24 cores are not waste; they are the p99 headroom, bought deliberately. The core-count formula's target_utilization term is this knee, made explicit.

The coordinated-omission trap

One more hazard: your load test may lie about the tail. Many load generators fire a request, wait for the response, then send the next — a closed loop. When the server stalls, the generator stalls too and simply sends fewer requests, so the slow requests it would have sent during the stall never get measured. This is coordinated omission: the tester and the system-under-test conspire to drop exactly the slow samples that define p99. The reported p99 looks healthy while production melts. The fix is open-loop / rate-scheduled load (send on a fixed schedule regardless of responses, e.g. wrk2-style, or correct for expected send time as HdrHistogram does) so stalled responses still count against the latency budget.

Trade-off: pack tight vs buy headroom

	Pack tight (ρ→0.9+)	Headroom (ρ≈0.6–0.7)
Cost	Cheap — fewer boxes, high efficiency	Costlier — you pay for idle cores
Tail latency	Fragile — p99 lives on the cliff	Flat — p99 tracks the mean
Burst / failure absorption	None — a spike or a lost node tips you over the knee	A node can die and survivors stay below the knee
Use when	Batch / async work with no user-facing SLO	Any latency-SLO'd serving path

The named alternative to headroom-based static sizing is autoscaling: run tight and add boxes when load climbs. But scaling has lag (detect → provision → warm → join the pool, often minutes), and the knee blows up in seconds. So even with autoscaling you keep a headroom buffer to survive the interval before new capacity lands — autoscaling changes how much buffer you standby-fund, not whether you respect the knee.

When server-count math misleads

Trap	Wrong conclusion	Fix
Servers = peak_QPS / QPS_per_box	Ignores headroom, deploys, AZ loss	Add 30–50% headroom; N+1 or N+2 per failure domain
One QPS/box for all work	CPU-bound and I/O-bound share a number	Benchmark the actual handler; cache-hit path vs miss path differ 10–100×
Forgetting dependent services	App tier scales; DB connections explode	Pool sizes and downstream quotas bound app scale

Boundary: 50k peak QPS, 5k QPS/box sustained → 10 boxes raw. With 40% headroom + survive 1 AZ of 3 → ceil(10×1.4×1.5)= 21 boxes, not 10.

---

Formulas are standard/public-domain engineering math. Approach and reference-table format adapted from the System Design Primer (CC BY 4.0), Jeff Dean’s latency numbers, the DesignGurus capacity-estimation guide, and Little’s Law.

I just read a lesson on **CPU & Server-Count Playbook** (System Design) and want to truly understand it. Explain CPU & Server-Count Playbook from first principles using ONE vivid real-world analogy and a visual mental model — draw it as ASCII art or a clear step-by-step diagram — with a concrete example using real numbers. Then ask me one question to check I got the mental picture, and wait for my reply. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Teach me **CPU & Server-Count Playbook** interactively. Ask me ONE guiding question at a time, wait for my answer, and adapt to my confusion — build the idea with me step by step instead of explaining it all at once. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Quiz me on **CPU & Server-Count Playbook** with 5 questions, easy to tricky, ONE at a time. Tell me if each answer is right; at the end, explain clearly what I got wrong and why. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Help me remember **CPU & Server-Count Playbook** for the long term: give the one-sentence intuition, a memorable hook/mnemonic, a tiny worked example, and 3 active-recall flashcards (Q -> A). If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.