medium Design an API Gateway

Design an API Gateway (RESHADED)

An API gateway is the single front door that terminates TLS, authenticates the caller, enforces rate limits, routes to the right upstream, and emits telemetry — so every backend service does not re-implement those cross-cutting concerns. The whole design question is a latency-and-availability accounting: it adds one hop and one shared dependency (the rate-limit store), so you must show the arithmetic that says the hop pays for itself and pick an explicit consistency stance for the counter. This page synthesizes the scattered building blocks (fleet sizing, availability, the Redis limiter, the traced request) into the one end-to-end problem a senior must defend.

R — Requirements

• Functional: authenticate (JWT), authorize, rate-limit per client, route by path/method to an upstream pool, transform request/response, aggregate where needed, emit metrics/traces.
• Non-functional: low added latency (<2 ms in-DC), high availability (it is on every request's critical path → must not be a hard SPOF), horizontal scalability, and graceful degradation when a dependency (the limiter store) is down.

E — Estimation (show the arithmetic; each number drives a decision)

Step	Arithmetic	Decision it forces
Throughput	peak 60,000 RPS; each node sustains ~3,000 RPS → 60,000 ÷ 3,000 = 20 serving nodes (+2 headroom = 22)	stateless nodes behind an L7 LB; scale out horizontally
Limiter load	1 INCR per request → ~60,000 ops/s on the counter store	a single Redis (~100k+ ops/s) fits, but for HA use a two-tier local+global limiter
Bandwidth	60,000 × ~10 KB avg response ≈ 600 MB/s ≈ 4.8 Gbps egress	NIC/LB capacity planning; response streaming, gzip
State size	route table ~10k routes × 1 KB = 10 MB (in-mem per node); counters ~5M clients × ~60 B ≈ 300 MB (Redis)	route table cached in memory; counters in RAM with TTL, not on disk
Latency budget	WAN RTT ~40 ms dominates; gateway work (TLS+auth+INCR+route) ≈ <2 ms in-DC	the extra hop is cheap vs N direct client→service round-trips

S — System interface

Uniform edge API: ANY /{service}/{path} with the caller's JWT in Authorization. Internally the gateway owns: route(method, path) → upstream_pool, allow(client_id) → bool (limiter), and verify(jwt) → claims.

D — Data model

• Route table — (method, path_prefix) → {upstream_pool, timeout_ms, auth_required, rate_limit}. Read-mostly; loaded into each node's memory and refreshed on change (watch/long-poll a config store), so the hot path never hits a DB.
• Rate-limit store (Redis) — key {client_id}:{window}, value = counter, TTL = window. Per request: INCR then compare to the limit; EXPIRE sets the window. Fixed-window is one round-trip; a sliding-window log or token bucket trades more state for smoother limiting.

H/D — High-level & detailed: one traced write path (t=0)

1. t=0 — request hits the L7 LB, which picks a healthy gateway node (least-conn).
2. TLS terminate at the node (keep-alive to upstreams reused).
3. AuthN — verify the JWT signature locally against cached JWKS (~0.1 ms, no network). Bad/expired → 401 immediately.
4. Rate-limit — INCR {client}:{minute} on Redis; if the result > limit → 429. If Redis times out → fail-open (allow) on normal endpoints.
5. Route lookup — in-memory route table → upstream pool; apply timeout.
6. Forward to the upstream (intra-DC ~1–2 ms), await, transform the response.
7. Return to the client; emit metrics/traces asynchronously, off the hot path.

E — Evaluate: the explicit CAP/PACELC stance for the counter store

The rate-limit counter is the one shared, per-request dependency, so its consistency choice is the crux. Choose PA / EL: under a partition, prefer Availability (serve traffic, tolerate a slightly-loose count) over Consistency (blocking requests to agree on an exact count); and with no partition, prefer Latency (a fast local decision) over a strictly-consistent global count. Concretely:

• Fail-open vs fail-closed on a limiter outage — fail-open (allow) for normal endpoints so an infra blip doesn't take the product down; fail-closed (deny) only for abuse-sensitive endpoints (login, payments) where over-admitting is worse than a brief outage.
• Two-tier limiter — a per-node local token bucket (EL: ~0 latency, no network) reconciled to the global Redis counter asynchronously. Approximate at the boundary but fast and partition-tolerant.

Judgment layer — why, why-not, alternatives, trade-off

• Why a gateway: one place for auth, limiting, TLS, routing, and observability — services stop re-implementing them and clients get one stable entry point decoupled from internal topology.
• Why NOT / when not: it is on every request's critical path (a SPOF you must make HA), it adds a hop and a deploy bottleneck (every team's routes funnel through it), and it can grow into a monolith of business logic. For internal east-west traffic it is the wrong tool.
• What else solves this (named alternatives): (1) No gateway — clients call services directly; simplest, but every service re-implements auth/limit and clients couple to topology. (2) Service mesh / sidecar (Envoy per service) — decentralizes the same concerns for east-west traffic with no central hop, at the cost of many moving parts. (3) BFF — a per-client-type gateway that also aggregates; better UX-fit, more surfaces to run.
• The trade-off: gateway = centralization (one policy point, one hop, one thing to make HA) vs mesh = decentralization (no central hop/SPOF, per-service policy, higher operational complexity). Pick the gateway for north-south client traffic and policy uniformity; pick the mesh for east-west service-to-service.

Pitfalls

• Making the limiter store fail-closed by default — one Redis blip 429s all traffic.
• Doing route/config lookups on the hot path (per-request DB reads) instead of an in-memory cache refreshed on change.
• Synchronous metrics/logging on the request path — emit async.
• Letting the gateway accrete business logic until it is a distributed monolith and a deploy bottleneck.

Takeaways

• Size it from arithmetic: 60k RPS ÷ 3k/node = 20 stateless nodes; 1 INCR/request = 60k ops/s on the limiter.
• The hop is cheap (<2 ms in-DC) against the ~40 ms WAN RTT and N saved client round-trips.
• State the counter store's stance out loud: PA/EL, fail-open by default, fail-closed only where abuse > availability.
• Gateway (north-south, centralize policy) vs service mesh (east-west, no central SPOF) is the decision to defend.

---

Re-authored and deepened for this guide, synthesizing standard API-gateway design (Grokking / DesignGurus, Hello Interview), CAP/PACELC (Abadi), and rate-limiting patterns. Re-authored/Deepened for this guide.

I'm working on the problem **Design an API Gateway** (System Design). Give me a HINT LADDER: start with the tiniest nudge, then wait. Only reveal the next, stronger hint when I ask. Do NOT show the full solution unless I type 'show solution'. Keep me doing the thinking. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Explain the optimal approach to **Design an API Gateway** with a VISUAL walkthrough: trace it on a small concrete example using ASCII art / a step-by-step diagram, narrate what changes each step, then give time & space complexity with a one-line derivation. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

I'll paste my solution to **Design an API Gateway**. Review it for correctness, missed edge cases, and time/space complexity, then coach me toward the optimal — don't just rewrite it. Ask me to paste my code now. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Give me 2 problems that use the SAME underlying pattern as **Design an API Gateway**. For each, let me attempt first, then review my answer and name the trigger signal that reveals the pattern. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.