Load Balancer vs API Gateway

Load balancing is a function — spread traffic across a pool — and it comes in two builds: an L4 balancer routes on the connection's IP/port 5-tuple without reading the request, while an L7 balancer (ALB, NGINX) is a connection-terminating reverse proxy that can route on the request itself. An API gateway is an L7 application that goes further still: it terminates the connection, parses the HTTP method, path, headers, and token, and runs per-route policy before forwarding. This page contrasts the L4 archetype with the API gateway because that pair marks the two ends of the inspection-depth axis — the L7 balancer sits between them, and the ALB pitfall below shows exactly where it falls short of a gateway. Everything else (auth, rate limiting, aggregation) follows from that one axis: you cannot make a decision on data you never read.

The mechanism: depth of inspection

Picture a request as a sealed envelope inside a shipping box. The address on the box is the TCP/IP 5-tuple: source IP+port, destination IP+port, protocol. The letter inside the envelope is the HTTP request: GET /api/v2/orders/8842, an Authorization header, a body.

• An L4 load balancer reads only the box address. It hashes or round-robins the 5-tuple to a backend, and it never opens the envelope — so it cannot route by URL path, verify a JWT, or rate-limit a specific API. It comes in two mechanically different builds. Packet/flow forwarders (an IPVS director, AWS NLB) never complete the TCP handshake themselves: they forward packets of one end-to-end connection, which is why they preserve the client's source IP on the wire (the NLB does for most target configurations — the exact default depends on target type), can do direct-server-return, and add only ~0.1–0.3 ms while scaling to millions of connections, because they do almost nothing per packet. L4 full proxies (HAProxy in TCP mode) do terminate the client's TCP connection and open a second one to the backend — still blind to HTTP, but the backend now sees the proxy's IP unless you enable PROXY protocol. Both are blind to the request; only the first is a pure relay.
• An API gateway (Kong, Apigee, AWS API Gateway, an Envoy-based mesh edge) opens the envelope. It terminates TLS, parses the full HTTP request, and becomes the request's origin for policy: it authenticates the caller, enforces quotas, rewrites the path, maybe fans out to several services and stitches the responses. That costs a few milliseconds and CPU per request, but it is the only place in the stack that can enforce an API contract.

They are not rivals. In production the standard shape is an L4 LB in front of a horizontally-scaled gateway fleet: the LB gives the fleet a single VIP and spreads connections across gateway instances for availability; each gateway then does the L7 work. The LB scales the thing that does the thinking.

Trace one request through the whole stack

A phone app calls GET https://api.shop.com/api/v2/orders/8842 with header Authorization: Bearer eyJhbGciOiJSUzI1Ni.... Here is exactly what each hop sees and does — note where the request stops being opaque.

The response flows back through the gateway (which adds CORS headers and gzip) over the TLS connection it already terminated. The load balancers touched only the address; every decision that needed the request's contents happened at the gateway.

Where should TLS termination and auth live?

This is the question the two components actually settle. Auth must live at the L7 gateway — it is the shallowest hop that can read the token. Pushing it onto the L4 LB is physically impossible (no visibility into the payload), and pushing it into every microservice duplicates security-critical logic and guarantees drift.

TLS termination is a genuine choice with two common answers:

• Terminate at the gateway (edge). Simplest and most common. The gateway needs plaintext anyway to read the token and path, so terminating there is free. Traffic inside the trusted network can go plaintext or re-encrypt to the mesh.
• Terminate at the L4 LB, re-encrypt to the gateway (TLS passthrough is the third option). Note that a TLS-terminating LB is necessarily operating in the full-proxy build — it holds the keys and re-originates the connection, even though its routing decision still uses only the 5-tuple. Used when a security policy forbids plaintext anywhere, or when the LB offloads TLS to dedicated hardware. Costs a second handshake and, critically, hides the client IP unless you propagate it (PROXY protocol at L4, or X-Forwarded-For at L7). Forget that and every per-IP rate limit and audit log records the LB's address instead of the real caller.

Pitfalls

• Trying to route by URL on an L4 LB. A classic misconfiguration: someone puts orders and payments behind one NLB and expects /payments/* to reach a different pool. L4 never sees the path — every request lands on whatever the 5-tuple hash chose. Path routing needs L7 (an ALB or a gateway). Why the naive version is wrong: the LB is asked to branch on data that is still inside an encrypted, unopened envelope.
• Mistaking an L7 ALB for an API gateway. An Application Load Balancer does TLS termination and path/host routing, so it looks gateway-shaped. But it has no JWT verification, no per-consumer API keys, no quota buckets, no request aggregation, no schema validation. If you need those, the ALB is a router, not a gateway.
• Lost client IP behind SNAT. An L4 LB doing source NAT makes every request appear to come from the LB. This is the full-proxy/NAT mode from the mechanism section — packet/flow forwarders preserve the source address; it is the L4 full proxies (and any NAT hop) that erase it. Rate-limit or geo-block by source IP at the gateway and you throttle all users together, or none. Enable PROXY protocol / trust X-Forwarded-For.
• TCP-only health checks lying to you. An L4 LB health-checks by opening a socket. A gateway instance whose upstream is down but whose port is still listening looks "healthy," so the LB keeps sending it traffic that all returns 500. Use L7 health checks (GET /healthz expecting 200) for anything past the L4 layer.
• The gateway as a distributed monolith. Every team piles routing, transforms, and business rules into the shared gateway config. It becomes a deploy bottleneck and a single blast radius: one bad plugin takes down every API. Keep the gateway to cross-cutting concerns; push business logic into services.

When to use which — and the trade-offs

Reach for a plain load balancer (no gateway) when: traffic goes to one homogeneous service or a monolith, there is no per-consumer policy, and you just need to spread load and fail over dead instances. You gain the lowest possible latency (sub-millisecond at L4) and dead-simple operations; you give up any ability to enforce an API contract at the edge.

Reach for an API gateway when: many backend services must present one public surface, and you need centralized authn/authz, per-key rate limiting, request/response transformation, versioning, or response aggregation. You gain a single enforcement point and a stable client contract; you pay a few milliseconds per request, add a component that must itself be scaled and made highly available, and risk over-centralizing.

Versus the nearest alternative, an L7 reverse proxy (NGINX / AWS ALB): the proxy gives you TLS termination and path/host routing cheaply and with tiny latency. It does not give you first-class API management — no key issuance, no quotas, no token introspection, no composition. Choosing the proxy trades away policy features for simplicity and speed; choosing the gateway trades latency and a heavier operational surface for governance.

Crisp rule: choose an L4 load balancer when raw throughput, connection scale, and simple fan-out dominate; choose an API gateway when policy and a unified contract dominate; choose an L7 proxy when you only need routing plus TLS, not API management. And remember they compose — an L4 LB in front of a gateway fleet, with internal LBs behind it, is the default production topology, not an either/or.

Takeaways

• The dividing line is depth of inspection — and load balancers exist at both depths: L4 LBs route on the 5-tuple and never open the request; L7 LBs (ALB, NGINX) are connection-terminating reverse proxies that route on it; API gateways terminate it and run per-route policy on its contents.
• Any concern that needs the request's content — auth, path routing, per-consumer rate limits, aggregation — must live at the L7 gateway, because it is the shallowest hop that can read it.
• They are complementary layers: the standard shape is an L4 LB spreading connections across a scalable gateway fleet, which then fans out to services through internal LBs.
• An L7 ALB looks gateway-shaped but is only a router; if you need API keys, quotas, or composition, that gap is exactly what an API gateway fills.

Re-authored and deepened for this guide, drawing on the NGINX documentation (L4 vs L7 proxying), the AWS Elastic Load Balancing docs (Network Load Balancer vs Application Load Balancer), the Kong and AWS API Gateway feature references, Envoy's listener/filter model, and Sam Newman's "Building Microservices" (2nd ed.) on edge gateways and where cross-cutting concerns belong.