Availability

An API gateway reaches high availability by running as a fleet of stateless, interchangeable replicas behind a health-checked load balancer — any replica can serve any request, a failed one is detected by active health checks and pulled from rotation before clients notice, and its share is redistributed to the survivors — while refusing to inherit the downtime of the synchronous dependencies (auth, rate-limit store, service discovery) it touches on every request.

That second clause is the part beginners miss. The gateway is a serial chokepoint: every request to every backend passes through it, so its availability multiplies with everything else on the path. If the gateway is down, healthy backends behind it are unreachable anyway. This makes the gateway the tier that must be the most available in the whole system — and it is uniquely easy to accidentally make it the least available, by hanging its request path off a dependency that is flakier than the gateway itself.

Why redundancy alone is not the whole story: the math

Two rules govern availability of a request path:

• Parallel (redundancy): for N independent replicas where any one can serve the request, availability = 1 − (1 − A)^N. Failure probability shrinks exponentially with N.
• Serial (dependency chain): for components that must all be up, availability = A₁ × A₂ × … Products only shrink — the chain is always less available than its weakest link.

A single gateway instance at A = 99.5% is down ~3.6 h/month. Put 4 behind a load balancer and the chance that all four are down at once is (0.005)⁴ ≈ 6×10⁻¹⁰ — on paper >99.9999999%. Redundancy is cheap and powerful. But the serial rule is what actually caps you: the gateway plus its synchronous dependencies form a chain, and one flaky dependency on the hot path drags the whole gateway below the availability of any single box in it.

Translate targets into budgets before promising anything:

Worked example: designing a gateway tier to 99.95%

Target: the gateway tier (everything the client hits before the business logic runs) must sustain 99.95% — a budget of ~22 min/month. Walk the serial chain. Values below are typical published/measured figures for managed cloud components.

Naive design — validate every token by calling the auth service synchronously, and reject requests (fail-closed) whenever Redis is unreachable. Now all six are in series:

0.99999 × 0.9999 × 0.9999 × 0.9995 × 0.999 × 0.999 ≈ 0.99730 → 99.73%, about 119 min/month of downtime (calendar month of 730.5 h — the same convention as the nines table above). The gateway tier is now less available than the 99.9% Redis it leans on, and blows the 99.95% target — even though every individual box is healthier than that.

Decoupled design — validate tokens locally against a cached JWKS (public keys, refreshed in the background; no per-request network call), and fail-open on the rate limiter (if Redis is unreachable, allow the request rather than 500 it). The two flaky dependencies drop off the availability path entirely:

0.99999 × 0.9999 × 0.9999 × 0.9995 ≈ 0.99929 → 99.93%, about 31 min/month. Add a small backend redundancy improvement and you clear 99.95%. Same hardware, same replicas — the ~88 min/month you bought came entirely from taking two dependencies off the critical path, not from adding more gateways.

The decoupling mechanism, concretely

Three techniques keep a dependency from lending its downtime to the gateway:

• Local validation over remote calls. JWT/JWKS: fetch the identity provider's public keys once, cache them, refresh in the background on a timer. Token signatures are then verified in-process with no per-request hop — the auth service can be down for minutes and requests still authenticate. (This is only safe for stateless checks; token revocation still needs a fresh signal, so cache with a bounded TTL.)
• Fail-open on soft dependencies. A rate limiter exists to shed load, not to gate correctness. If the counter store is unreachable, letting the request through (perhaps with a coarse local fallback limit) is almost always better than returning 5xx. Contrast hard dependencies (does this caller have any valid credential?) where you fail-closed — availability must never override security there.
• Circuit breakers + bounded timeouts. Wrap every downstream call so a slow or dead dependency trips open fast and returns a fast fallback, rather than tying up gateway worker threads until they exhaust and the whole replica stops accepting connections (a slow dependency taking down a fast gateway is the classic cascading failure). When it is the gateway itself that is saturating — a brownout, not a hard failure — degrade by route priority: shed or throttle non-critical routes first (search suggestions, analytics beacons) so the revenue-critical path keeps its latency budget, rather than letting every route degrade equally until nothing meets SLO.

Capacity headroom (N+k). Redundancy only helps if the survivors can carry the load. Size the fleet so that losing an entire AZ still serves peak traffic — if 2 replicas handle peak, run 4 across 2 AZs, not 2 in one AZ. "We had a spare" is worthless if the spare then falls over under the redirected load.

Pitfalls

• The control plane is a hidden single point of failure. Replicas protect against independent hardware failure. They do nothing against a correlated failure — a bad route table, an expired/mis-rotated TLS cert, or a broken config pushed atomically to all N replicas at once. Most large gateway outages are self-inflicted config pushes, not hardware. Roll config out canary-first and keep instant rollback.
• Shallow health checks hide brain-dead replicas. If the LB health check hits /ping and gets 200 while the replica can't reach any backend, traffic keeps flowing into a black hole. The check must exercise real readiness.
• ...but deep health checks cause correlated eviction. If the check calls a downstream and that downstream blips, every replica fails the check simultaneously, the LB drains the entire fleet, and a partial dependency outage becomes a total gateway outage. Health checks must test the replica's own liveness, not its dependencies'.
• Retry storms. A gateway that blindly retries a struggling backend multiplies load on it, turning a recoverable brownout into a full collapse. Use capped retries, jittered backoff, and a circuit breaker.
• No connection draining on deploy. Rolling a deploy without graceful shutdown drops in-flight requests; the terminating replica must deregister from the LB and finish open requests before exiting.
• Single-AZ redundancy. Four replicas in one availability zone give you great numbers on paper and zero protection when that AZ goes dark.

When to use it / when NOT to

The design decision is how to make the gateway redundant. Three options:

• Active-active fleet behind an LB (default). All replicas serve traffic; failover is just "the LB stops sending to the dead one," which is sub-second. Choose this when the gateway is stateless (it should be) and you want the highest availability and full hardware utilisation. Cost: you must externalise all state (rate-limit counters, sessions) and tolerate that any replica may serve any request.
• Active-passive (hot standby). One node serves, a standby waits and takes over on failure via VIP/DNS. Prefer this only when the component genuinely cannot be run active-active — e.g. a legacy stateful gateway or something needing a single writer. Cost: failover is slow (health-check detection + DNS TTL or VIP move = seconds to minutes of hard downtime), and you pay for an idle node. For a stateless gateway this is strictly worse than active-active.
• Managed gateway (AWS API Gateway, managed Envoy/Cloud LB). The provider runs a multi-AZ, auto-scaled fleet and gives you an availability SLA. Choose this when you don't want to own the HA engineering and your traffic/feature needs fit the product. Cost: less control over latency and behaviour, per-request pricing that gets expensive at very high volume, and you inherit the provider's blast radius and quotas.

Decision rule: stateless gateway, high-availability target, cost-sensitive at scale → self-managed active-active across AZs. Small team or spiky/uncertain traffic and you value not being paged → managed gateway. Reach for active-passive only when statefulness forces your hand — and treat its failover time as real downtime in your math.

Takeaways

• The gateway is a serial chokepoint, so its availability multiplies with everything on the path — it must be your most-available tier, and it's dangerously easy to make it the least.
• Redundancy (parallel) crushes independent failure exponentially; it does nothing for correlated failure (bad config, expired cert) — which causes most real gateway outages.
• The biggest availability wins come from taking flaky dependencies off the hot path: local JWKS token checks, fail-open rate limiting, circuit breakers — not from adding more replicas.
• Size for N+k so survivors carry peak after losing an AZ, spread replicas across AZs, and keep health checks liveness-only so a downstream blip can't drain the whole fleet.

Re-authored and deepened for this guide. Sources: Google, Site Reliability Engineering (Beyer et al.) — availability targets, "nines," and embracing risk; Amazon Web Services Well-Architected Framework, Reliability Pillar — multi-AZ redundancy and static stability; Michael T. Nygard, Release It! — circuit breakers, fail-fast, and bulkheads; Martin Kleppmann, Designing Data-Intensive Applications — serial vs. parallel availability and correlated failures; Envoy and NGINX documentation — active health checking, connection draining, and active-active fronting.