Performance Implications

An API Gateway sits on every request's critical path, so every microsecond it spends terminating TLS, verifying a token, matching a route, and re-opening a backend connection is added to every client call — the whole performance game is keeping that per-hop tax small and bounded while fanning out to backends in parallel so aggregation costs the slowest branch, not the sum.

The original catalog of this page named the right techniques (token bucket, leaky bucket, service mesh load balancing) but never said how any of them work, and answered "latency" with "optimize the code." Below, each mechanism is traced with real numbers so you can reason about the actual cost.

The extra hop, in real milliseconds

The gateway's headline cost is one more network hop. But be honest about which hop: a remote client already pays a wide-area round trip to reach anything in your region, so the gateway does not add that. What it adds is (a) one intra-datacenter hop from gateway to backend (~0.3–0.8 ms) plus (b) the gateway's own CPU work — route match, JWT signature verify, header rewrite (~1–3 ms warm). On the happy path that is a 2–4 ms tax. The danger is the tail: a connection-pool miss forces a fresh TCP+TLS handshake to the backend (adds 2 RTT ≈ 1–2 ms intra-DC, tens of ms if the pool is exhausted and the request queues), and a GC pause or CPU saturation on the gateway itself turns p99 into 50–200 ms.

Latency budget: one aggregated mobile request

A mobile home screen needs profile, recent orders, and recommendations. The gateway fans out to three services. Compare doing it sequentially versus scatter-gather (all three in flight at once):

Same three calls; parallel fan-out nearly halves observed latency because the client waits for the slowest branch (60 ms) instead of the sum (150 ms). "Optimize the aggregation logic" concretely means: issue the backend calls concurrently, set a per-branch deadline (e.g. 80 ms) so one slow service cannot stall the whole page, and return partial results for non-critical branches like recommendations.

Rate limiting: token bucket, mechanically

A token bucket is a counter that refills at a fixed rate and caps at a maximum. The bucket holds up to C tokens and gains r tokens per second; each request removes one token; a request that finds the bucket empty is rejected (HTTP 429). Because a full bucket lets a backlog drain instantly, the pattern permits short bursts up to C while still enforcing a sustained ceiling of r requests/second — which is exactly what real traffic (spiky, then quiet) needs.

Traced example — C = 10, r = 5 tokens/sec (1 token every 200 ms)

The bucket starts full. A client fires a burst of 15 requests at t = 0, then 2 more at t = 0.4 s.

The burst of 10 sailed through (that is the point — occasional bursts are legitimate); the 11th–15th were shed instantly and cheaply; and by 400 ms the steady 5/sec refill had earned back exactly 2 tokens for the next 2 requests. A leaky bucket is the dual: requests enter a fixed-size queue and leave at a constant rate r; it smooths output to a perfectly even stream (good for protecting a fragile downstream) but cannot pass a burst — the 10 simultaneous requests would be spaced 200 ms apart instead of served at once. Choose token bucket to allow bursts, leaky bucket to eliminate them.

Distributed rate limiting and load balancing — the hard parts

Rate limiting across a gateway fleet. With 5 gateway replicas each keeping a local bucket, a global limit of 100 req/s becomes 5×20 — but skewed hashing sends 60% of a user's traffic to one replica and it throttles at 20 while the global count is only 60. The real fix is a shared counter: each replica does an atomic INCR with a TTL against Redis (a sliding-window or a centrally-held token bucket). This costs one ~0.3 ms Redis round trip per request and makes Redis a dependency on the hot path — so production systems use a two-tier scheme: enforce a generous local bucket to shed obvious floods with zero network cost, and reconcile against the shared counter for the true global limit.

Load balancing and the service mesh. A gateway doing central round-robin is blind to backend health between health-check intervals, so it keeps sending requests to a pod that is GC-pausing. A service mesh (e.g. Envoy sidecars) moves the decision to a proxy next to each caller and uses health-aware algorithms — least-request (send to the backend with the fewest in-flight calls) or EWMA latency weighting — plus outlier detection that ejects a slow pod after N consecutive 5xx. "More granular" concretely means the balancing decision is made per-call with live in-flight and latency data, not per-interval from a central table.

Sizing backend connection pools

A gateway keeps a warm pool of keep-alive connections to each backend. If the pool is too small, requests queue for a connection and tail latency spikes; if too large, you waste file descriptors and memory and can overwhelm the backend with idle sockets. A safe size is derived from the maximum concurrent in-flight requests you expect to that backend, not from total gateway traffic.

Worked example: gateway fleet = 5 replicas; target backend = Orders service; peak = 10,000 req/s; Orders mean latency ≈ 30 ms, p99 latency = 60 ms. Little's Law uses the mean sojourn time: λ × W = 10,000 × 0.030 = 300 concurrent requests on average. Sizing a connection pool to the tail is safer, so we use the p99 as a hedge: 10,000 × 0.060 = 600 concurrent requests. With 5 replicas, each gateway needs up to 600 / 5 = 120 concurrent connections. Add headroom for bursts (×1.5) → 180 connections per replica. Set pool max to 180, alarm on pool-wait time, and cap total connections so Orders is not drowned by idle keep-alives.

If the backend is shared by multiple callers, its total connection count is the sum across all gateway replicas and other clients. A backend that accepts at most 1,000 connections can be exhausted by 6 gateways × 180 = 1,080, so the limit must be negotiated with the backend team or enforced by the service mesh.

Source grounding: Little's Law derivation and connection-pool sizing are standard in queueing-theory references (Harchol-Balter, Performance Modeling and Design of Computer Systems) and in the Envoy proxy connection-pool documentation.

Pitfalls

• Connection-pool exhaustion beats CPU to the tail. The gateway keeps a bounded pool of keep-alive connections per backend. Under a spike, requests queue for a free connection; the extra hop that was 2 ms becomes 40 ms of pool-wait. Size pools per backend and alarm on pool-wait time, not just gateway CPU. The sharpest single signal is the gap gateway_p99 − max(upstream_p99): if the gateway's own p99 pulls away from the slowest backend it is fronting, that difference is queueing (pool-wait), not backend slowness.
• Synchronous fan-out with no per-branch deadline. One slow backend (the 60 ms Orders call becomes 5 s) holds the aggregated response — and the gateway thread/connection — hostage. Always set a per-call timeout below the client's timeout and degrade gracefully.
• Per-replica rate limits that don't sum to the global limit. Local buckets under skewed load reject legitimate traffic while the global ceiling is untouched. Use a shared counter for the authoritative limit.
• The gateway as an undeployable monolith. Piling routing + auth + transformation + aggregation into one process makes every service change a gateway change and every gateway deploy a fleet-wide risk. Keep it thin; push per-client shaping into a BFF.
• Double-counting the WAN RTT. Blaming the gateway for latency the client would pay to reach any endpoint leads to "optimizations" that don't move the number. Measure the gateway's marginal cost: intra-DC hop + gateway CPU + queueing.

When to use a gateway — and when not to

Reach for an API Gateway when you have many services behind one public edge and need cross-cutting concerns — TLS termination, auth, rate limiting, request shaping — applied uniformly at the boundary, and clients (mobile, third-party) that benefit from a single stable endpoint and response aggregation. Concrete signals: you are re-implementing JWT verification in every service; a client makes 6 calls to render one screen; you need one place to enforce quotas.

Choose THIS (a shared gateway) when north-south cross-cutting concerns and client simplification dominate. Prefer a service mesh when the pain is service-to-service resilience and load balancing. Prefer direct calls when callers are internal, trusted, and latency-critical. Prefer a BFF when one gateway is being pulled in incompatible directions by different clients. These compose: a gateway for north-south plus a mesh for east-west is the common large-system layout.

Takeaways

• The gateway's honest cost is its marginal latency — intra-DC hop + CPU + queueing (~2–4 ms warm) — not the WAN RTT the client pays anyway; the real risk lives in the tail (pool exhaustion, GC).
• Aggregate with parallel scatter-gather and per-branch deadlines so a client waits for the slowest backend, not the sum — often a 2× win.
• Token bucket allows bursts up to C at a sustained rate r; leaky bucket smooths to a constant rate. Distributed limiting needs a shared counter or it under- or over-throttles.
• A gateway is a deliberate SPOF and bottleneck you accept for uniform edge control; run it HA, keep it thin, and pair it with a mesh for east-west traffic.

Re-authored and deepened for this guide. Draws on Sam Newman, Building Microservices (2nd ed.) on gateways, BFFs, and aggregation; Chris Richardson, Microservices Patterns (API Gateway chapter); the Envoy Proxy docs on least-request/EWMA load balancing and outlier detection; the classic token/leaky bucket definitions from the ATM traffic-shaping literature (Tanenbaum, Computer Networks); and Stripe's and Cloudflare's engineering writeups on distributed rate limiting with shared counters.