Introduction to API Gateway

What is an API gateway?

An API gateway is a server-side component that sits between external clients (web browsers, mobile apps, partner integrations) and a system's backend services, acting as the single entry point for every inbound request. Instead of a client knowing the network address of forty different microservices, it knows one address: the gateway.

On a typical request, the gateway performs some subset of:

• Routing — map an incoming path/method to the backend service that owns it (e.g. GET /orders/42 → orders-service).
• Authentication — validate a token or API key once, at the edge, instead of every service re-implementing it.
• Rate limiting — protect backend services from being overwhelmed by any single client.
• Request/response transformation — translate protocols, reshape payloads, or attach headers before forwarding.
• Observability — a natural place to emit consistent access logs, metrics, and trace headers for every request entering the system.

Centralizing this cross-cutting logic frees individual microservices to focus on their own business logic, and gives consumers of the system one consistent contract instead of many bespoke ones.

Worked trace: one request through the pipeline

The diagram above shows why order matters inside the gateway: AuthN has to run before RateLimit, because the limiter needs to know which client's budget to check. Reverse the two and the only key left is the source IP — which lumps every user behind a shared NAT or CGNAT egress into one bucket, so one noisy tenant exhausts the quota for everyone sharing that address. Walk one request through it.

A mobile client calls GET /orders/42. The gateway rate-limits per client using a token bucket with capacity 100. This client has already spent tokens earlier in the current window, leaving 58 tokens remaining.

The number that changes on every allowed request is tokens remaining — the bucket only ever tracks how much budget is left, and it decrements. Had this client been at 0 tokens remaining, RateLimit would have short-circuited the pipeline with 429 Too Many Requests before Transform or the backend ever saw the request — exactly as the diagram's short-circuit paths show.

Difference between an API gateway and a load balancer

The two are easy to conflate because both sit in front of backend infrastructure and forward traffic — but they make different kinds of decisions.

An API gateway is focused on routing a request to the appropriate microservice based on its URL/contract, while a load balancer is focused on distributing requests evenly across a group of interchangeable replicas of the same service.

Another difference is the type of request each typically handles. An API gateway routes requests based on a URL that identifies a specific API or resource, while a load balancer routes requests sent to a single well-known IP address toward whichever backend server is healthiest and least loaded — it doesn't need to understand what the request is about to make that call.

Common failure modes and pitfalls

• Single point of failure. If the gateway goes down, every client is cut off — even though every backend microservice may be perfectly healthy. Mitigate by running gateway instances behind their own load balancer, across multiple availability zones.
• Added latency. Every request now pays for an extra network hop plus whatever work the gateway does (auth lookups, transformation, logging). Usually a few milliseconds, but it compounds under high fan-out or when the gateway makes synchronous calls to enrich a request.
• Configuration drift and blast radius. Routing rules, rate limits, and auth policy for the whole system now live in one place. A bad deploy to the gateway's config can break every service at once, instead of just the one team that pushed a bug.
• Becoming a dumping ground for business logic. It's tempting to add response aggregation, orchestration, or domain-specific transformation into the gateway because "it already sees every request." Left unchecked, the gateway grows into a shared, hard-to-own monolith that every team is afraid to touch.
• Versioning and backward compatibility. Because the gateway is the contract clients depend on, changing routes or response shapes there is a breaking change for every consumer, not just one service's callers.
• Cascading timeouts under backend failure. If a downstream service slows down, gateway threads/connections can pile up waiting on it, starving capacity for requests bound for unrelated, healthy services. This needs its own timeouts, bulkheads, and circuit breakers at the gateway layer, not just at each service.

When an API gateway is the wrong call

An API gateway is a judgment call, not a default. Before centralizing at the edge, weigh it against these alternatives.

In practice a gateway and a mesh are not mutually exclusive: many systems put a gateway at the edge for north-south traffic (external clients in) and a mesh internally for east-west traffic (service to service) — each solving the problem it's actually good at, instead of one component trying to do both jobs.

Where the route table lives

Everything above assumed the gateway simply knows that /orders/* maps to orders-service. That mapping has to come from somewhere, and there are two answers with different failure modes:

• Static configuration — routes live in a config file or a declarative object (in Kubernetes, a CRD such as an Ingress or Gateway resource) that is versioned and deployed like code. This is simple and auditable, but it is exactly where the "configuration blast radius" pitfall above becomes concrete: one bad config push can 404 every route at once, so route changes should be canaried to a slice of traffic and be one-command rollbackable.
• Service-discovery-backed routing — the gateway watches a registry (Kubernetes Endpoints, Consul) and updates its upstream pool membership without a redeploy: new instances start receiving traffic as they register, removed ones drop out. The failure mode is staleness — a just-killed pod can keep receiving traffic for up to one refresh interval until the gateway's view catches up, which is why discovery-backed routing still needs health checks and retries in front of it.

And what about a request that is in flight when the gateway instance handling it dies? The TCP connection resets and the client sees a connection error — the gateway cannot hand a half-processed request to a peer. Only idempotent requests are safe for the client (or a retry layer) to automatically retry, because the client cannot know whether the backend already applied the work. This is exactly why load balancers do connection draining for planned restarts — finish open requests, stop accepting new ones — and why unplanned instance death is the case idempotency keys exist for.

Check yourself

1. Why must AuthN run before RateLimit? The limiter needs an identity to key the budget on; keyed only by source IP, every user behind a shared NAT/CGNAT egress collapses into one bucket and one noisy tenant exhausts it for all of them.
2. Gateway vs load balancer — what decision does each make? The gateway decides which service owns the request (by URL/contract); the load balancer decides which replica of one service takes it (by health/load).
3. Your gateway config push 404s every route — what limits the blast radius? Canarying the config change to a slice of traffic before full rollout, plus a fast, tested rollback path — per the configuration-drift pitfall above.

Sources: original lesson content from this course's "Introduction to API Gateway" material; the gateway-vs-load-balancer distinction, rate-limiting/token-bucket behavior, and failure-mode guidance cross-checked against standard system-design references (AWS API Gateway and NGINX documentation on reverse proxies and API gateways; Istio and Linkerd documentation on service-mesh sidecar architecture).