Performance Engineering — The USE Method & Flame Graphs

Find the bottleneck before you touch a line

The Make-it-Fast lesson says "measure first." This is the how — the systematic toolkit performance engineers (Brendan Gregg's methodology) use so they fix the real hotspot, not a guess. Two tools do most of the work: the USE method to find the bottlenecked resource, and the flame graph to find the hotspot code.

USE: which resource is the problem?

For every resource (CPU, memory, disk, network, and per-device), check three things:

	Means	Red flag
Utilization	% busy time	sustained near-100% → a lead — confirm with saturation
Saturation	queued/waiting work	a run-queue or I/O queue building → demand > capacity
Errors	failures	retransmits, drops, ECC errors

A resource that's saturated (queueing) is your bottleneck — that tells you where to profile before you dive into code. Note that 100% utilization with no queue is a fully-used resource, not a bottleneck: a batch job pegging its cores with an empty run queue is healthy — that is what you paid for. Utilization points; saturation convicts.

Reading a flame graph

Each box is a function; width = share of CPU time (samples), and boxes stack by call depth (a caller sits below its callees... or above, depending on orientation). You don't read it top-to-bottom — you scan for the widest box. That plateau is where the time goes. In the diagram, dbQuery → seqScan is 58% of the time: add an index and over half the latency disappears. Micro-optimizing the narrow json box would be wasted (Amdahl).

On-CPU vs off-CPU: a CPU flame graph shows time computing; if your service is slow but CPU is idle, you need an off-CPU profile — the time is spent blocked (waiting on I/O, locks, the network). Knowing which to capture is half the skill.

Self-test: what do you profile next?

Case A. A service's p99 latency is high, but overall CPU is only 30%. The on-CPU flame graph shows 15% in JSON serialization and no single wide plateau. What is the right next step?

• Answer: An on-CPU flame graph is the wrong tool here. High latency + low CPU means the time is spent blocked — capture an off-CPU profile (locks, I/O, network waits) instead.

Case B. CPU is saturated and the flame graph shows handleRequest → dbQuery → seqScan as the widest plateau at 58%. What is the first fix?

• Answer: The widest plateau is the hotspot. Add an index so seqScan becomes an index scan; that single change removes the majority of the latency. Re-measure before touching anything else (Brendan Gregg, Systems Performance).

USE in one worked pass

An API is slow and everyone assumes it's the app code. Walk the resource checklist instead of guessing. For the disk: Utilization 95%, Saturation = average I/O wait await 40 ms with a queue depth of 32, Errors none. Read that line: the device is nearly always busy and a deep queue is building — demand exceeds what the disk can drain, so the bottleneck is storage saturation, not CPU. Adding app replicas here does nothing (they all queue on the same disk); the fix is faster storage, less I/O per request, or a cache in front. That is the whole point of USE: it tells you which resource to attack before you ever open a code profiler.

USE on Linux, concretely

The method is only useful if you can run it on a box at 3am. One observable per cell:

Resource	Utilization	Saturation (threshold shape)	Errors
CPU	mpstat / top % busy	vmstat run-queue r > core count, or /proc/pressure/cpu climbing	rare — machine-check (MCE) entries in dmesg
Memory	free — used vs available	swapping / OOM-kills; PSI memory stalls rising	ECC errors (EDAC counters, dmesg)
Disk	iostat %util	aqu-sz growing; await above the device's normal	I/O errors in dmesg / SMART
Network	throughput vs NIC line rate	socket backlog / listen-queue overflows, drops	netstat -s retransmits; interface error counters

PSI (/proc/pressure/*) is the modern one-stop saturation read for CPU, memory, and I/O: the fraction of wall time tasks spent stalled waiting on each resource.

USE vs RED — resources vs requests

USE is a resource-first method (per CPU/disk/NIC/pool). Its complement is RED (Tom Wilkie): per service, watch Rate (requests/s), Errors (failed/s), and Duration (latency distribution). Use RED to answer "is the user experience healthy and which endpoint is degrading?"; use USE to answer "which resource is exhausted underneath it?" A mature setup runs both: RED on the request path defines the SLO and points at the sick service, then USE on that service's box finds the saturated resource. Neither replaces the other — RED without USE tells you something hurts but not why; USE without RED can miss a latency problem that never saturates any single resource (e.g. lock waits, cross-service fan-out).

When continuous profiling is the wrong reflex

Flame graphs are a scalpel, not a monitor. Do not leave a full-frequency profiler running across the whole fleet in production by default: the sampling overhead is real under load, and stack traces can capture arguments/paths that leak PII — profile a representative host on demand (or sample continuously at a low, budgeted rate with a scrubbing policy), not everything all the time. And a flame graph only finds on-CPU cost; if RED says latency is high while USE shows every resource cool, the time is off-CPU (blocked) and no CPU flame graph will ever show it.

Takeaways

• USE (Utilization / Saturation / Errors) per resource finds which resource is the bottleneck.
• Flame graphs find which code: scan for the widest box, fix that, re-measure.
• If CPU is idle but latency is high, profile off-CPU (blocked) time, not on-CPU.

---

Re-authored for this guide; flame-graph diagram hand-authored as SVG. Follows Brendan Gregg's Systems Performance (USE method, flame graphs). See also: Make It Work→Right→Fast, Tail Latency, How a Query Executes.

I just read a lesson on **Performance Engineering — The USE Method & Flame Graphs** (System Design) and want to truly understand it. Explain Performance Engineering — The USE Method & Flame Graphs from first principles using ONE vivid real-world analogy and a visual mental model — draw it as ASCII art or a clear step-by-step diagram — with a concrete example using real numbers. Then ask me one question to check I got the mental picture, and wait for my reply. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Teach me **Performance Engineering — The USE Method & Flame Graphs** interactively. Ask me ONE guiding question at a time, wait for my answer, and adapt to my confusion — build the idea with me step by step instead of explaining it all at once. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Quiz me on **Performance Engineering — The USE Method & Flame Graphs** with 5 questions, easy to tricky, ONE at a time. Tell me if each answer is right; at the end, explain clearly what I got wrong and why. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Help me remember **Performance Engineering — The USE Method & Flame Graphs** for the long term: give the one-sentence intuition, a memorable hook/mnemonic, a tiny worked example, and 3 active-recall flashcards (Q -> A). If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.