Summary

Circuit Breaker — decision summary (not a pep talk)

The Circuit Breaker pattern fails fast and degrades gracefully when a dependency is unwell, so the caller does not burn threads waiting and the dependency gets load relief. States: Closed → Open → Half-Open → Closed (or back to Open). It is a stability tool, not a correctness tool: it does not make dual-writes safe and it does not replace timeouts.

Operating rules you should be able to recite

• Timeout under the breaker. No timeout → pool dies before trip.
• Threshold vs baseline error rate, not vs zero. Normal 0.5% errors + tiny window = constant flap.
• Match cooldown to recovery reality (drain, cache warm), not vanity "faster is better."
• One probe (or tight budget) in half-open — avoid re-crush.
• One breaker per dependency (often per critical operation class).
• Count dependency-unwell signals only (timeouts, connect errors, 5xx, slow-call rate) — not 4xx.
• Fallback must be correct or omit it (error > lie).
• Combine with bulkhead + idempotent retry budget — breaker alone is incomplete.

When to use / when not

Use breaker	Do not use / prefer other tool
Sustained or correlated remote failure	Need authoritative success/fail for money movement without a durable queue
Meaningful fallback or fast error is acceptable	Fallback would violate inventory/payment correctness
Protect shared finite concurrency	Problem is pure overload of self — need load shedding / rate limit first
Dependency recovery benefits from traffic pause	Single rare blip — one retry with jitter is enough

Top production alerts

1. Breaker open rate / time-in-open per dependency (sudden open on critical path).
2. Caller pool saturation + dependency latency p99 (breaker missing or timeout too long).
3. Half-open success ratio (recovery flapping).
4. Fallback invocation rate (users living on degraded mode — treat as SEV if prolonged).

Mini worked recap

100 threads, Payment timeout 300 ms, 200 RPS, Payment hangs. Timeouts alone pin 200 × 0.3 = 60 threads — survivable; add one naive retry per failure and in-flight doubles to 400 × 0.3 = 120 > 100 — the pool dies. The retry amplification, not the timeout, is what finishes the pool off: without a breaker, threads block on waits/retries. With breaker opening after failure budget, remaining calls short-circuit; Payment cools; half-open probe restores Closed when healthy. Net: localized degradation instead of total Order outage.

Drill ladder

1. Q: Name the full resilience stack around a remote call. A: Timeout → bulkhead/concurrency limit → retry (idempotent, bounded, jitter) → circuit breaker → fallback/load-shed.
2. Q: Why is "shared breaker for all outbound HTTP" a design smell? A: One bad host trips the shared breaker and black-holes healthy dependencies.
3. Q: Interviewer: "Breaker open, what does the user see for checkout?" A: Explicit failure or deferred queue — never a silent "success." Money paths need durable retry, not optimistic UI.

Takeaways

• Breaker = evidence-based traffic stop + careful resume; prerequisite is timeout.
• Tune thresholds to baseline; isolate breakers; half-open gently.
• Wrong fallback is worse than a fast error.
• Pair with bulkhead and retry policy; measure time-in-open as a first-class SLI.

Starting numbers by dependency tier (a cheat-sheet, not a law)

Every rule above says "tune it" without a starting point. These are defensible initial configurations to tune from telemetry, not fixed values — the point is that the numbers differ by tier, and by how much:

Dependency tier	Threshold style	Window / min-volume	Trip point	Cooldown	Half-open	Fallback
Money path (payment, ledger)	rate	60 s / ≥20 calls	≥50% errors	30–60 s (match real recovery)	1 probe	durable queue or explicit fail — never a silent success
Read enrichment (recommendations, price-display)	rate	30 s / ≥50 calls	50%	10–30 s	1–2 probes	cached value or omit the feature
High-QPS internal RPC (>50 req/s)	rate (never count)	10 s / ≥100 calls	30–50%	5–10 s, jittered	≥0.5 s jitter spread	degraded read path
Low-QPS / batch / admin (<50 req/s)	count OK	10 s window	~5 failures	longer (recovery-bound)	1 probe	retry later / async

Three of these numbers are not arbitrary and are worth being able to defend:

• Count vs rate splits at ~50 req/s. A fixed count C false-trips once normal-noise failures per window reach it: baseline × λ × window ≥ C. At a 1% baseline, a 10 s window and C = 5, that is λ = 5 / (0.01 × 10) = 50 req/s — so below ~50 req/s a small count is simpler and reacts faster; above it, use a percentage threshold with a minimum-volume floor.
• Half-open jitter ≥ 0.5 s (for ~50 instances). When many instances share a cooldown deadline they probe in unison. To keep the fleet-wide probe rate under a reviving dependency's warm-up capacity C_warm, spread probes over Δ ≥ N × p / C_warm; for N = 50, p = 1, C_warm = 100 req/s that is 50 / 100 = 0.5 s minimum.
• Retry stays "inside", breaker "outside", crossing at ~10% failure. A retry-once policy adds ~f extra load and is safe only while f ≤ the retry budget (~10%); past a ~10% dependency failure rate the budget is spent and the breaker must take over. This is why the money-path and internal tiers pair a bounded idempotent retry under the breaker rather than choosing one.

The discipline is the takeaway: derive the tier's numbers from its traffic (λ), its baseline error rate, its instance count, and its real recovery time — then let production telemetry (trip rate, time-in-open, half-open success, fallback rate) move them.

I just read a lesson on **Summary** (System Design) and want to truly understand it. Explain Summary from first principles using ONE vivid real-world analogy and a visual mental model — draw it as ASCII art or a clear step-by-step diagram — with a concrete example using real numbers. Then ask me one question to check I got the mental picture, and wait for my reply. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Teach me **Summary** interactively. Ask me ONE guiding question at a time, wait for my answer, and adapt to my confusion — build the idea with me step by step instead of explaining it all at once. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Quiz me on **Summary** with 5 questions, easy to tricky, ONE at a time. Tell me if each answer is right; at the end, explain clearly what I got wrong and why. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Help me remember **Summary** for the long term: give the one-sentence intuition, a memorable hook/mnemonic, a tiny worked example, and 3 active-recall flashcards (Q -> A). If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.