Introduction

Circuit Breaker: fail fast so the rest of the system lives

Remote calls fail. They also hang. A hung dependency is worse than a dead one: threads, connections, and memory stay occupied waiting, until the caller is as dead as the callee. The Circuit Breaker pattern (Nygard, Release It!) stops sending traffic to a dependency that has crossed a failure budget, fails fast (or serves a fallback), then carefully probes for recovery.

Electrical analogy: trip the breaker to protect the house wiring. Software analogy: protect the caller's finite concurrency so one bad neighbor cannot take down the process.

Mechanism: three states

State	Behavior	Transition
Closed	Calls flow; failures counted in a sliding window	→ Open when failure rate or slow-call rate exceeds threshold
Open	Calls short-circuit immediately (error or fallback); no load on dependency	→ Half-Open after cooldown timer
Half-Open	Allow a small number of trial calls	Success budget met → Closed; failure → Open again

Critical prerequisites the breaker does not replace: a per-call timeout (without it, the first slow wave exhausts the pool before the breaker opens) and a decision on what "failure" means (timeouts, 5xx, connection refused — not 4xx validation errors).

Worked example with numbers

Order Service → Payment Service. Order has 100 worker threads. Payment p99 is normally 80 ms; timeout set to 300 ms. At 200 RPS to Payment, in-flight ≈ 200 × 0.08 = 16 — fine.

Payment degrades: calls now take ~2.5 s to complete, so every call is cut off by the 300 ms timeout and counted as a failure. Without a breaker:

• Each request holds a thread up to 300 ms.
• In-flight climbs toward 200 × 0.3 = 60, then higher if timeouts queue; under retry storms it saturates 100 threads.
• Unrelated Order endpoints that share the pool start failing.

With a breaker (e.g. open after 50% failures in a 20-call window, cooldown 10 s):

• After ~10–20 failures the breaker opens (~seconds, not minutes).
• Subsequent calls fail in <1 ms with "payment unavailable" or queue-for-later fallback.
• Threads free for paths that do not need Payment.
• After 10 s, one half-open probe; if Payment is healthy, close; if not, open again.

When a circuit breaker is the wrong fix

• Caller must know the exact outcome ("was the card charged?"). Failing open with a guessed success is a correctness bug; failing closed still needs a durable retry path (outbox/queue), not a silent drop.
• Rare transient blips better handled by a single retry with backoff — a tight threshold will flap (open/close thrash).
• Fallback violates business rules (show "in stock" from cache during inventory outage → oversell).
• In-process pure CPU work — use isolation/bulkheads differently; breakers are for remote dependency health.

Prefer: timeouts always; retries only on idempotent ops; bulkheads so one dependency's wait cannot steal all threads; breakers to stop hammering a known-sick dependency.

Failure / operability

• Flapping: threshold below baseline error rate. Detect: open/close rate high. Fix: threshold vs real baseline (if normal is 0.5% errors, do not open on 3 failures).
• Half-open herd: 100 instances each send a probe → re-crush recovery. Fix: single-probe or rate-limited half-open; coordinated if possible.
• Shared breaker for multiple dependencies: one flaky integration trips traffic to a critical one. Fix: one breaker per dependency (and often per endpoint class).
• Counting 4xx as failures: bad client traffic opens the breaker for everyone. Count dependency-unwell signals only.

Decision defensibility

Why not only retries? Retries amplify load on a sick service and need idempotency. Why not only bulkheads? Bulkheads cap concurrency but still allow continuous load at the cap. The breaker stops the load after evidence of systemic failure. Use all three: timeout + bulkhead + breaker (+ careful retry).

Drill ladder

1. Q: Why must timeout sit under the breaker? A: Breaker counts failures; without timeout, calls hang and never become "failures" until the pool is already empty.
2. Q: Half-open admits 50 probes from 50 pods at once — what goes wrong? A: Recovering dependency sees a thundering herd and fails probes → stuck open longer.
3. Q: Payment returns 400 for bad card data — should that open the breaker? A: No. Client/validation errors are not dependency health.

The pages that follow work the state machine, tuning, and combination with retries and bulkheads in production detail.

I just read a lesson on **Introduction** (System Design) and want to truly understand it. Explain Introduction from first principles using ONE vivid real-world analogy and a visual mental model — draw it as ASCII art or a clear step-by-step diagram — with a concrete example using real numbers. Then ask me one question to check I got the mental picture, and wait for my reply. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Teach me **Introduction** interactively. Ask me ONE guiding question at a time, wait for my answer, and adapt to my confusion — build the idea with me step by step instead of explaining it all at once. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Quiz me on **Introduction** with 5 questions, easy to tricky, ONE at a time. Tell me if each answer is right; at the end, explain clearly what I got wrong and why. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Help me remember **Introduction** for the long term: give the one-sentence intuition, a memorable hook/mnemonic, a tiny worked example, and 3 active-recall flashcards (Q -> A). If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.