Circuit Breaker — The State Machine (Closed → Open → Half-Open)

The problem: one slow service drags down everything

Say your checkout service calls a payment service. Payment gets slow (a DB hiccup). Every checkout request now waits 30 seconds for a timeout; threads pile up waiting; soon checkout runs out of threads and it goes down too — then services calling checkout fail. One slow dependency has cascaded into a system-wide outage. The fix is to stop calling a service that’s clearly broken, and fail fast instead.

The idea: an electrical breaker for service calls

A circuit breaker wraps a remote call and watches it, exactly like the breaker in your home’s fuse box: when something’s wrong it trips, cutting off calls so the failing service can recover and the caller isn’t dragged under. It has three states:

            failures exceed threshold
   ┌──────────┐ ───────────────────────▶ ┌────────┐
   │  CLOSED  │                           │  OPEN  │  ← reject calls instantly ("fail fast")
   │ (normal) │ ◀─────────────────────    └────┬───┘    for a cool-down timeout
   └──────────┘   trial call succeeds          │ after timeout, allow a few trial calls
        ▲                                       ▼
        │                                 ┌────────────┐
        └──────── trial calls pass ────── │ HALF-OPEN  │ ── trial call fails ──▶ back to OPEN
                                          └────────────┘

• CLOSED — normal. Calls pass through; the breaker counts failures.
• OPEN — too many failures. Calls are rejected immediately (no waiting on timeouts) for a cool-down period. The caller can return a fallback (cached value, “try later”).
• HALF-OPEN — after the cool-down, let a few trial calls through. All succeed → close the breaker (recovered). Any fail → re-open (still broken).

Step through the state machine

Trace requests through Closed → Open → Half-Open as a downstream fails and recovers. Predict, on each request, whether the breaker lets it through or fails fast — and watch how many downstream calls it sheds versus no breaker at all.

About the simulator above

The scenario debugger uses a simplified consecutive-fail threshold (trip after 3 failures in a row, cooldown 3 ticks) so the state transitions are easy to step through. The worked example below uses a production-style rolling-window rate (>50% of the last 20 calls). Both are valid breaker configurations; the consecutive-count version is easier to reason about in a tutorial, while the rolling-rate version is less likely to flap on intermittent noise. The state machine behaves the same way in both cases. The debugger also admits exactly one half-open trial call (success closes, failure re-opens), whereas the worked example uses a 2-probe budget — production breakers typically admit 1–5 probes and require consecutive successes, as covered under Operating a breaker in production.

Worked example

Breaker config: trip if >50% of the last 20 calls fail; cool-down 30s.

1. Payment starts failing → failure rate crosses 50% → breaker trips to OPEN.
2. For 30s, checkout’s payment calls return instantly with a fallback (“payment temporarily unavailable”) instead of hanging — checkout stays alive.
3. At 30s the breaker goes HALF-OPEN, lets 2 calls through. Payment has recovered → both succeed → breaker CLOSED, normal service resumes.

Pairing with retry (important nuance)

Retry and circuit breaker are layered: retry handles a transient blip (try again with backoff); the circuit breaker handles a sustained outage (stop trying entirely). Put the breaker outside the retry so that once it’s open, you don’t keep retrying a service that’s down.

Operating a breaker in production

A breaker is not a set-and-forget knob. The dangerous production questions are: How do I know it tripped for the right reason? How do I know it is still open? What happens if the fallback is wrong?

• Expose breaker state as a metric. Every breaker should emit its current state (closed / open / half-open), the count of state transitions, and the number of fail-fast rejections. A breaker that flaps between open and closed is usually telling you the threshold is too tight or the downstream dependency is genuinely unstable.
• Alert on "open too long." A breaker open for minutes means a real outage; a breaker that never opens when the dependency is failing means it is misconfigured or the failure is not being counted (e.g., timeouts are not treated as failures).
• Design the fallback deliberately. "Return null" is not always safe. For a payment call, the fallback might be "try the alternate processor"; for a recommendation call, it might be "return cached defaults"; for a write, there may be no safe fallback and you must fail the user request explicitly.
• Half-open is a probe, not a full reopen. Limit the number of trial calls (e.g., 1–5) and require consecutive successes, not just one lucky success, before closing. A single good response after an outage can be a false positive.
• Test it. Breakers are easy to break in deployment. Use chaos tests or fault injection to verify the breaker opens, returns the fallback, and closes after recovery.

When a breaker is the wrong tool

A breaker earns its keep only against a sustained failure of a remote dependency; three cases call for a different tool. A single transient blip is handled faster by a bare retry with backoff and jitter than by tripping and then waiting out a whole cool-down window. A call that must not silently drop a write is a poor fit: opening the breaker and returning a fallback loses the write, so durable work belongs in an outbox or queue that is drained on recovery, not behind a breaker that fails fast and forgets. And a local, in-process pure function cannot "fail slow" across the network, so there is no cascade to prevent and a breaker only adds state to maintain. One subtlety worth stating explicitly: the reason the machine returns to HALF-OPEN rather than snapping straight back to CLOSED is the half-recovery stampede — throwing the full request volume at a service that has only just come back can knock it over again, so it admits a bounded probe first and closes only once the probes pass.

Takeaways

• A circuit breaker prevents a slow/failing dependency from cascading by failing fast.
• Three states: Closed (normal), Open (reject + cool-down), Half-Open (trial recovery).
• Combine with retry + bulkhead; always provide a fallback for the open state.

Re-authored from-scratch for this guide. Diagrams adapted from Karan Pratap Singh’s System Design (MIT); patterns follow Azure Architecture Center / microservices.io / DDIA conventions.