Circuit Breaker — The State Machine (Closed → Open → Half-Open)

The problem: one slow service drags down everything

Say your checkout service calls a payment service. Payment gets slow (a DB hiccup). Every checkout request now waits 30 seconds for a timeout; threads pile up waiting; soon checkout runs out of threads and it goes down too — then services calling checkout fail. One slow dependency has cascaded into a system-wide outage. The fix is to stop calling a service that’s clearly broken, and fail fast instead.

The idea: an electrical breaker for service calls

A circuit breaker wraps a remote call and watches it, exactly like the breaker in your home’s fuse box: when something’s wrong it trips, cutting off calls so the failing service can recover and the caller isn’t dragged under. It has three states:

            failures exceed threshold
   ┌──────────┐ ───────────────────────▶ ┌────────┐
   │  CLOSED  │                           │  OPEN  │  ← reject calls instantly ("fail fast")
   │ (normal) │ ◀─────────────────────    └────┬───┘    for a cool-down timeout
   └──────────┘   trial call succeeds          │ after timeout, allow a few trial calls
        ▲                                       ▼
        │                                 ┌────────────┐
        └──────── trial calls pass ────── │ HALF-OPEN  │ ── trial call fails ──▶ back to OPEN
                                          └────────────┘

• CLOSED — normal. Calls pass through; the breaker counts failures.
• OPEN — too many failures. Calls are rejected immediately (no waiting on timeouts) for a cool-down period. The caller can return a fallback (cached value, “try later”).
• HALF-OPEN — after the cool-down, let a few trial calls through. All succeed → close the breaker (recovered). Any fail → re-open (still broken).

Step through the state machine

Trace requests through Closed → Open → Half-Open as a downstream fails and recovers. Predict, on each request, whether the breaker lets it through or fails fast — and watch how many downstream calls it sheds versus no breaker at all.

Worked example

Breaker config: trip if >50% of the last 20 calls fail; cool-down 30s.

1. Payment starts failing → failure rate crosses 50% → breaker trips to OPEN.
2. For 30s, checkout’s payment calls return instantly with a fallback (“payment temporarily unavailable”) instead of hanging — checkout stays alive.
3. At 30s the breaker goes HALF-OPEN, lets 2 calls through. Payment has recovered → both succeed → breaker CLOSED, normal service resumes.

Pairing with retry (important nuance)

Retry and circuit breaker are layered: retry handles a transient blip (try again with backoff); the circuit breaker handles a sustained outage (stop trying entirely). Put the breaker outside the retry so that once it’s open, you don’t keep retrying a service that’s down.

Takeaways

• A circuit breaker prevents a slow/failing dependency from cascading by failing fast.
• Three states: Closed (normal), Open (reject + cool-down), Half-Open (trial recovery).
• Combine with retry + bulkhead; always provide a fallback for the open state.

Re-authored from-scratch for this guide. Diagrams adapted from Karan Pratap Singh’s System Design (MIT); patterns follow Azure Architecture Center / microservices.io / DDIA conventions.