Usage of API gateway

What "usage" means at the edge

An API gateway earns its place at the boundary between untrusted clients and a service fleet by doing five things well: authenticating the caller, enforcing quota, resolving a route, forwarding the call, and shaping the response. Everything else — protocol translation, response aggregation, canary routing — is a variation on those five. The trace below walks one real request through all five in order, with the actual hosts, byte counts, and time budget it consumes.

Traced: one request through the gateway

A mobile client calls GET /orders/8842-919 with a bearer token. The gateway sits at 10.0.2.10:443. Here is what happens, in order, with the clock running:

1. TLS accept + parse (0.10 ms) — the gateway terminates TLS, reads a 512-byte request, and extracts the path and the Authorization header.
2. JWT verification (0.25 ms) — the token's signature is checked against the cached JWKS entry for issuer auth.internal; no network call, so this is pure CPU.
3. Rate-limit check (0.30 ms) — the gateway issues INCR rl:user:8842:min against Redis at 10.0.1.5:6379, gets back 48, compares it to the plan's ceiling of 100/min, and lets the request through. The key carries a 60-second TTL so the bucket self-expires. The INCR and its TTL arm must apply atomically (a single Lua script) — the crash-safety trap and the fixed-window boundary burst are dissected in the Concurrency and Coordination lesson.
4. Routing decision (0.10 ms) — the path pattern /orders/{id} maps to the order-svc upstream pool.
5. Circuit-breaker check (0.05 ms) — the breaker for the order-svc pool, backed by host 10.0.1.4:8090, has recorded 5 consecutive failures in the last 10 seconds and is open. The gateway does not attempt the upstream call.
6. Fallback assembly (0.40 ms) — instead of a 502, the gateway serves a cached last-known-good order summary: 912 bytes of JSON, gzipped to 380 bytes (about 58% smaller) before it goes back over the wire, with a Retry-After: 15 header matching the breaker's half-open timer.

0.10 + 0.25 + 0.30 + 0.10 + 0.05 + 0.40 = 1.20 ms of gateway-side latency, entirely before any (failed) attempt at the upstream — the client sees a fast, deliberate degradation instead of a slow timeout.

Where this pattern breaks

• The gateway becomes a distributed monolith. Once routing, auth, and business-specific response shaping all live in gateway config, every team's deploy queues behind the gateway team's release window.
• Fallback data goes stale silently. The 380-byte cached response in the trace above is only safe because it's marked with an age and a client-visible Retry-After; a fallback with no staleness signal turns a resilience feature into a source of quietly wrong data.
• One gateway, one blast radius. A bad rate-limit config or a bug in the JWT-verification path takes down every service behind the gateway at once, not just one. A route shipped with rl: 10/min instead of 1000/min throttles legitimate traffic to 429s instantly, and because the counter lives in shared Redis the revert still takes seconds to propagate — so gateway config deserves the same canary rollout and 429-spike alerting you would give application code, plus separate limits for the revenue-critical routes so one bad edit cannot floor them all.
• Latency budget creep. 1.2 ms per hop looks free until ten teams each add "just one more" enrichment step and the gateway's own overhead becomes the dominant term in p99.

Versus the named alternatives

The gateway pattern is the right call only for a specific traffic shape: a shared boundary between external clients and many internal services, needing centralized authentication, quota, and routing. Three signals point away from it, toward a different pattern instead.

No gateway — clients call services directly

Signal: one or two services, a single client type, a team that owns both ends. Dropping the gateway removes a hop and its 1.2 ms of overhead — appealing when that hop buys nothing. The cost shows up the moment a second client type or a third service appears: authentication, rate limiting, and routing logic must now be duplicated and kept in sync inside every service, and there are no scoped rate limits shared across services — each one tracks its own counters, so a client can be throttled by service A but not by service B for what should be one unified quota. Choose this only while the surface area is small enough that duplicating cross-cutting logic in each service is genuinely cheaper than operating a gateway.

Service mesh — purely internal service-to-service traffic

Signal: the traffic in question never crosses the trust boundary — it's order-svc calling inventory-svc, not a client calling order-svc. A mesh (sidecar proxies alongside each service, as in Istio or Linkerd) gives every internal call mTLS, retries, and load balancing without funneling east-west traffic through one edge process. That's the opposite trade-off from the gateway: a mesh adds a sidecar hop to every internal call — more moving parts and more per-pod resource overhead — in exchange for no single choke point and no coupling between unrelated services' internal traffic. Applied to the trace above, a mesh would not have helped: the failure sat between an external client and order-svc, exactly the north-south traffic a gateway is built for, not the east-west traffic a mesh is built for. In practice the two are complementary rather than competing — gateway at the edge, mesh internally, each handling the traffic shape it fits.

BFF (Backend-for-Frontend) — multiple client types with divergent needs

Signal: a mobile app and a web app hitting the same services but wanting different aggregations, payload shapes, or field sets. A single shared gateway forces one contract to serve both, which tends to accrete client-specific branches into shared config over time. A BFF per client type moves that divergence into its own small service, owned by the team that owns the client, while the BFFs can still sit behind a thin shared gateway for the concerns that really are common to all clients — TLS termination, coarse rate limiting. The trade-off is more services to run and a second place, BFF then downstream service, where a bug can hide.

Aggregation is where a BFF (or an aggregating gateway) earns or loses its keep, and the deciding factor is fan-out shape. A screen that needs five backends at ~100 ms each costs 5 × 100 = 500 ms if the calls are issued serially, but only about max(legs) + merge ≈ 100 + 20 = 120 ms when they are fanned out concurrently — the aggregator waits on the slowest leg, not the sum. Two rules make that safe: every leg needs its own timeout, or one hung backend freezes the whole screen at the fan-out's mercy; and a failed non-critical leg should return a partial response with a degraded flag rather than failing the entire aggregate. Aggregating reads this way is fine; aggregating transactional writes in the gateway is not — multi-service writes belong in a saga owned by the services, not stitched together at the edge where there is no place to hold the compensation logic.

None of these replace a gateway outright in a system that already has one — they mark where its edge stops paying for itself: push traffic to a mesh once it's internal, to a BFF once it's client-specific, and keep the gateway for the boundary that's genuinely shared.

Sources: Newman, S., Building Microservices (2nd ed., O'Reilly, 2021), ch. 10 — API gateway and Backend-for-Frontend patterns. Richardson, C., Microservices Patterns (Manning, 2019), ch. 8 — external API patterns and BFF; ch. 11 — circuit breaker. Istio and Linkerd project documentation on sidecar-based service-mesh architecture. Netflix Technology Blog, "Fault Tolerance in a High Volume, Distributed System," on circuit-breaker rationale for edge-facing services.