Conclusion
Retry Pattern — decision checklist
Retries turn many transient failures into successes. Unbounded, non-idempotent, or multi-layer retries turn outages into stampedes. Use this checklist in design reviews; skip the motivational recap.
When to use
- Failure class is transient (network blip, 503, 429, brief overload).
- Operation is idempotent or protected by an idempotency key / unique constraint.
- You have a deadline and max attempts that still fit UX and dependency SLOs.
- Backoff includes jitter; prefer exponential with cap.
When NOT to use
- Non-idempotent money or inventory mutations without dedupe.
- 4xx validation/auth errors (except rate-limit with Retry-After).
- Breaker is open or dependency is known-down for a maintenance window.
- Another layer already owns retries (avoid r^N amplification).
Policy defaults that are defensible
| Knob | Starting point | Rationale |
|---|---|---|
| Max attempts | 2–3 total (1 original + 1–2 retries) | Enough for blips; bounds latency |
| Timeout per try | ~ dependency p99 × 1.5–2, hard cap | Fail faster than default 30s libraries |
| Backoff | Full jitter, cap 1–5s | Desynchronize recovery |
| Retryable codes | 408, 429, 502, 503, 504, connect errors | Not 400/401/403/404 |
| Idempotency | Key required for POSTs with side effects | Timeout ≠ not-executed |
Why 2–3: with independent transient faults at 90% per-try success, attempts 1/2/3 succeed with 90% / 99% / 99.9% probability (P(success within n) = 1 − (1−p)n = 1 − 0.1n). The 4th attempt buys 0.09 points of success for another full timeout+backoff of tail latency — and against correlated failure (a sustained outage) it buys nothing; that is the breaker's job.
Top 3 production alerts
- Retry rate / attempt histogram — sudden climb means dependency pain or misclassification of errors.
- Amplification ratio (outbound RPS / inbound RPS) — >2–3× sustained is a storm risk.
- Duplicate side-effect indicators (double charge detections, unique constraint conflicts) — idempotency gaps.
Interaction with other patterns
- Timeout: mandatory floor under every retry.
- Circuit breaker: stop retrying when open; half-open probes are not full retry storms.
- Bulkhead: rejected-by-full-pool should usually not retry immediately into the same pool.
- Outbox / queue: durable async retry beats in-request infinite retry for long recovery.
Drill ladder
- Q: Three layers each retry 3 times — worst case multiplier? A: Up to 3³ = 27 at the leaf (if each hop multiplies); design to avoid that.
- Q: User double-clicks Pay; two requests same idempotency key. Correct server behavior? A: One execution; second returns same result (or 409 if in-flight policy says so).
- Q: Prefer in-request retry or queue for a 15-minute dependency outage? A: Queue/outbox with backoff consumers; in-request retry will blow deadlines.
One-line takeaway
Retry only what is safe, few times, with jitter, under a deadline — and own the policy at one layer.
🤖 Don't fully get this? Learn it with Claude
Stuck on Conclusion? Open Claude, copy a block below, and it'll teach you this exact concept — visually and interactively.
Build the mental picture, not memorization.
I just read a lesson on **Conclusion** (System Design) and want to truly understand it. Explain Conclusion from first principles using ONE vivid real-world analogy and a visual mental model — draw it as ASCII art or a clear step-by-step diagram — with a concrete example using real numbers. Then ask me one question to check I got the mental picture, and wait for my reply. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.
Socratic — adapts to where you're stuck.
Teach me **Conclusion** interactively. Ask me ONE guiding question at a time, wait for my answer, and adapt to my confusion — build the idea with me step by step instead of explaining it all at once. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.
Active recall exposes what you missed.
Quiz me on **Conclusion** with 5 questions, easy to tricky, ONE at a time. Tell me if each answer is right; at the end, explain clearly what I got wrong and why. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.
Intuition + hook + flashcards for long-term memory.
Help me remember **Conclusion** for the long term: give the one-sentence intuition, a memorable hook/mnemonic, a tiny worked example, and 3 active-recall flashcards (Q -> A). If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.