Introduction

Retry Pattern: self-heal from blips without creating storms

In distributed systems, transient failures are normal: brief packet loss, a LB target draining, a dependency GC pause, a 503 during deploy. Many of these succeed if tried again moments later. The Retry pattern automatically re-issues a failed operation under policy so users do not see avoidable errors.

The danger is not "retrying" — it is retrying unsafely: non-idempotent operations, no backoff (thundering herd), unbounded attempts (latency and load explosions), or retries stacked at every layer of a call chain (multiplicative amplification).

Mechanism

1. Attempt operation.
2. On retryable failure (timeout, 502/503/429, connection reset), wait backoff(attempt) with jitter.
3. Retry until success or max attempts / deadline.
4. On non-retryable failure (400, 401, 404, business rule), fail immediately.

Exponential backoff + full jitter (common form): sleep random in [0, min(cap, base * 2^attempt)]. Jitter desynchronizes clients so recovery is not a synchronized stampede.

Worked example: charge card timeout

Checkout calls Payment with 200 ms timeout. At t=0 request R1 is sent; Payment charges the card at t=50 ms but the response is lost. Caller sees timeout — uncertain outcome (two generals).

Policy	What happens	Outcome
No retry	User sees error; card may already be charged	Support nightmare
Naive retry, no idempotency key	R2 charges again	Double charge
Retry with idempotency key K	R2 carries K; Payment returns original success	Safe; one charge

Rule: never retry a side-effecting call unless the receiver can dedupe (idempotency key, natural unique constraint, or truly pure read).

Amplification math (why budgets matter)

Edge makes up to 3 attempts, Service A makes up to 3 attempts (3 attempts each, counting the original — not 3 retries on top of it), Service B is the leaf. Worst-case calls to B per user request = 3 × 3 = 9 (and deeper chains get worse). During an outage this is a self-DDoS. Cap with: max attempts, overall deadline, and prefer retry at one layer (usually the edge or the owner of the idempotency key), not every hop.

When NOT to retry

• Non-idempotent writes without a dedupe mechanism.
• 4xx client errors (except 408/429 with respect for Retry-After).
• Circuit breaker is Open — retrying fights the breaker.
• Local logic bugs (retrying will not fix wrong code paths).
• User-aborted requests — do not keep working after cancel.

Failure / operability

• Retry storm: synchronized retries after outage. Detect: RPS to dependency >> user RPS. Fix: jitter, retry budgets, shed load.
• Latency inflation: 3 retries × 1 s timeout = multi-second p99. Fix: tight timeouts + small attempt count + deadline propagation.

  Two clocks, not one: a per-try timeout bounds each attempt; an overall deadline bounds the whole operation. Example: overall deadline 2 s, per-try timeout 500 ms, full-jitter backoff. Attempt 1 fails at 500 ms; sleep 80 ms; attempt 2 fails at 1,080 ms; sleep 150 ms; attempt 3 must finish by 2,000 ms, so its per-try timeout is shrunk to the remaining 770 ms. Rule: before each attempt, set per-try timeout = min(configured per-try, deadline − elapsed); if the remainder is too small to plausibly succeed, abort early instead of firing a doomed call.

• Hidden double side effects: metrics look "successful" after retry while money moved twice. Fix: idempotency + audit.
• Layered retries: SDK + service + gateway all retry. Fix: document single owner of retry policy.

Decision defensibility

Why not infinite retry until success? Unbounded tail latency and infinite load on a sick dependency. Why not only circuit breaker? Breakers handle sustained failure; retries handle short blips without opening. Use: timeout → limited retry with jitter on idempotent ops → breaker when failure budget exceeded.

Drill ladder

1. Q: Timeout then success on server — do you retry? A: Yes only with idempotency; outcome is uncertain without it.
2. Q: Why full jitter vs fixed 1 s sleep? A: Fixed sleep aligns clients into waves; jitter spreads load.
3. Q: HTTP 400 on create user — retry? A: No; fix the request.

I just read a lesson on **Introduction** (System Design) and want to truly understand it. Explain Introduction from first principles using ONE vivid real-world analogy and a visual mental model — draw it as ASCII art or a clear step-by-step diagram — with a concrete example using real numbers. Then ask me one question to check I got the mental picture, and wait for my reply. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Teach me **Introduction** interactively. Ask me ONE guiding question at a time, wait for my answer, and adapt to my confusion — build the idea with me step by step instead of explaining it all at once. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Quiz me on **Introduction** with 5 questions, easy to tricky, ONE at a time. Tell me if each answer is right; at the end, explain clearly what I got wrong and why. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Help me remember **Introduction** for the long term: give the one-sentence intuition, a memorable hook/mnemonic, a tiny worked example, and 3 active-recall flashcards (Q -> A). If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.