Authorization — PKCE, Multi-Tenancy, Zanzibar & Revocation (Deep Dive)

Authorization — PKCE, Multi-Tenancy, Zanzibar & Revocation (Deep Dive)

Every gap this page closes has the same shape: a check trusts the wrong input, or trusts a cache that is older than the fact it needs, or assumes a dependency is always up. PKCE fixes a check that used to trust possession of a secret; algorithm pinning fixes a verifier that trusts a header the attacker controls; tenant scoping fixes a check that trusts a client-supplied ID; fail-closed design fixes the assumption that the policy engine is always reachable; Zanzibar's zookie fixes a check that trusts a cache that hasn't caught up yet; refresh-token rotation fixes the assumption that a stolen token stays quiet. This page is the "what a senior engineer actually watches for" layer on top of the OAuth/JWT/RBAC-ABAC-ReBAC mechanics covered elsewhere in this guide.

1. PKCE — binding the code to whoever started the flow

A confidential client (a backend with a secret) authenticates its token exchange with a client_secret. A public client — a single-page app or a mobile app — ships its code to the user's device, so anything baked into that bundle is not a secret; any string in it can be extracted with dev tools or a decompiler. PKCE (RFC 7636) replaces "prove you know the secret" with "prove you are the same party that started this specific flow":

1. Client generates a random code_verifier (43–128 characters) and computes code_challenge = BASE64URL(SHA256(code_verifier)).
2. /authorize request carries only the challenge: code_challenge=E9Melhoa2OwvFrEMTJguCHaoeK1t8URWbuGJSstw-cM&code_challenge_method=S256. The verifier itself never leaves the device yet.
3. The authorization server stores the challenge alongside the issued authorization code.
4. On the token exchange, the client finally sends the raw code_verifier. The server recomputes SHA256(code_verifier) and checks it equals the stored challenge before minting tokens.

Trace why interception fails: an attacker who captures the redirect (a malicious app registered on the same custom URI scheme, a leaky referrer header, a compromised network) obtains the authorization code — but not the code_verifier, which never appeared in that URL. Redeeming the stolen code at /token without the matching verifier fails the SHA256 check. The proof is bound to the device that generated the verifier, not to whoever holds the code.

Why it now applies to every client, not just public ones: a confidential client's secret protects the token exchange only if nothing else leaks the code first — PKCE is a second, independent binding that survives even if a secret ever leaks or is misconfigured, so current guidance (OAuth 2.0 Security BCP) is PKCE for all clients, confidential or not. The flow it replaces — the implicit flow, which returned tokens directly in the redirect fragment with no code-exchange step at all — is deprecated precisely because it has no equivalent binding: a token in a URL fragment can leak via browser history, referrer headers, or a malicious redirect, and there is no secret or verifier to check at the end.

2. RS256 verification via JWKS — and the attack that same setup enables

A real Google id_token is signed RS256 (RSA signature over SHA-256): Google's authorization server holds a private key and signs; any verifier fetches Google's public keys from a JWKS endpoint (https://www.googleapis.com/oauth2/v3/certs), matches the token header's kid (key id) to the right public key in that JSON Web Key Set, and checks the signature with it. This is asymmetric on purpose: the public key can be published and cached everywhere without letting anyone forge a token, because forging RS256 requires the private key.

The exact fact that makes this safe — the verification key is public — is what enables the classic RS256→HS256 algorithm-confusion attack. HS256 is symmetric: the same secret both signs and verifies. Many older JWT libraries read the algorithm to use for verification from the token's own alg header and passed a single "key" variable into a generic verify(token, key) call. If a verifier is configured to accept RS256 tokens with "the public key" as that key variable, and it does not also pin the algorithm, an attacker can:

1. Take the RSA public key (deliberately public — anyone can fetch it from the JWKS endpoint).
2. Forge a token with header {"alg":"HS256","typ":"JWT"} and any payload they want (e.g. {"sub":"attacker","role":"admin"}).
3. Compute signature = HMAC-SHA256(header.payload, public_key_bytes) — using the public key's raw text as the HMAC secret. This is legal HMAC math; there is nothing to "break."
4. Send the forged token. The verifier reads alg: HS256 from the header, runs HMAC-SHA256 with "the key" (the same public key variable it uses for RS256), gets a match, and accepts a token the attacker fully controls.

The bug is not in RSA or HMAC — both are doing exactly what they are asked. The bug is a verifier that lets the attacker's own header choose which algorithm — and therefore which trust model — to apply. Mitigation: pin the expected algorithm outside the token. The verifier call should look like jwt.verify(token, publicKey, { algorithms: ["RS256"] }) — an explicit allow-list the library enforces itself, never derived from the token under test. Modern JOSE libraries require this list; treat any verifier without one as broken.

3. Multi-tenant authorization — scope every check by a token-derived tenant_id

In a multi-tenant system, "is this user allowed to read this invoice" is incomplete without "and is this invoice actually in this user's tenant." The tenant_id used for that second check must be derived from the verified token — a claim the authorization server put there when it authenticated the user — never taken from anything the client supplies on the request itself (a URL path segment, a query parameter, a request body field, or a header the client controls).

The failure, concretely: a service exposes GET /api/tenants/{tenantId}/invoices/{id} and authorizes with something like if (token.isValid()) { return db.query("SELECT * FROM invoices WHERE tenant_id = ? AND id = ?", request.tenantId, request.id) }. The token proves the caller is a real, logged-in user of some tenant — but the query trusts request.tenantId from the URL. A user authenticated for tenant 42 simply requests /api/tenants/17/invoices/9001; the query scopes by tenant 17 (attacker-supplied), not by the tenant the token actually proves, and returns another company's invoice. This is Broken Object-Level Authorization (BOLA/IDOR) at the tenant boundary, and it is exploitable purely by editing an ID — no token forgery required.

The fix is structural, not a better check: read tenant_id only from the validated token's claims, and either ignore the URL's tenantId entirely (scope every query by token.tenant_id) or, if it must appear in the URL for routing/readability, treat a mismatch between token.tenant_id and the URL's value as a hard failure — never let the URL value substitute for or override the token's value in the actual data access.

Claims that commonly drive authorization decisions: sub (who), roles/groups (RBAC), scope (OAuth-granted actions, e.g. invoices:read), tenant_id (multi-tenant scoping), and permissions (fine-grained ABAC/PBAC). The API gateway or service parses these from the verified token and passes them to the PDP as the subject's attributes. A claim the client could modify without server verification is not an authorization input — it is an attack surface.

4. Fail-closed vs availability — the PDP becomes a hard dependency

The secure default when a Policy Decision Point cannot be reached is fail closed: deny, return 403, rather than permit an unverified request. That default is correct — but it has a direct, often under-priced consequence: it makes the PDP's availability a multiplicative factor in the product's availability. If every protected request calls the PDP synchronously and the PDP is down, degraded, or unreachable across a network partition, every one of those requests now fails — the app servers and database can be perfectly healthy and the product is still down, because the authorization layer, whose job is to prevent a narrow class of misuse, has become a single point of failure for the whole system.

Mitigations that keep fail-closed without making the PDP a global outage switch:

• Embedded / replicated PDP (sidecar). Run the policy engine (e.g. OPA) as a sidecar next to each service, with policy bundles pushed or pulled locally, so the "call" is a local in-process evaluation, not a network hop to one central service. A regional or host-level failure now affects one instance, not the whole fleet.
• Bounded last-known-good caching. If the PDP is briefly unreachable, serve the last decision the service actually received for that principal/resource, but only within a bounded staleness window (seconds to low tens of seconds) — trading a small, deliberate consistency gap for not failing every request the instant the network hiccups.
• Degrade to read-only. Under PDP outage, deny anything destructive or state-changing (writes, deletes, privilege changes) but allow reads served from a recent cached decision — asymmetric risk: a wrong read is usually recoverable, a wrong write often is not.

5. A distributed PDP under partition — Zanzibar's CAP stance

Google's Zanzibar is exactly the "PDP at global scale" case: a single authorization service backing Drive, Calendar, and other properties, answering permission checks from anywhere in the world. Rather than requiring every check to synchronize with the single freshest global write before answering — which would mean failing or stalling checks during any partition or replication lag — Zanzibar leans toward availability with bounded, explicit staleness: a replica keeps answering checks from the snapshot it currently has, and the caller states, via a token attached to the check, exactly how fresh that snapshot needs to be for this specific decision. That token is the zookie (§7), and it is the mechanism that lets Zanzibar avoid the two bad extremes — always-stale (permit things that were just revoked) and always-synchronous (403 the world every time a link between regions is slow). The stance is not hypothetical caution — the paper (Pang et al., USENIX ATC 2019) reports serving roughly 10 million authorization QPS at <10 ms p95 globally, with >99.999% availability over three years of production use. A latency budget that tight is only reachable by answering from local replicas, which is exactly why the design accepts bounded staleness rather than global synchrony.

6. The new-enemy problem — the canonical bug the zookie prevents

Name the bug precisely: an ACL revocation and a later content write are two independent writes to two different stores (a relationship/ACL store and a content store), each replicating on its own schedule. If a reader's check lands on a replica that has absorbed the content write but not yet the earlier revoke, the reader — someone who was just removed — can see content created after their removal. They have become a "new enemy" who was never supposed to see it, and the system lets them in purely because the two writes raced across replicas in the wrong order relative to each other.

Trace it with illustrative timestamps (the numbers are for teaching the mechanism, not measured figures): Alice removes Bob from folder F at t=10.000; that write commits at the authoritative replica R1 immediately but takes until t=10.200 to reach a second region, R2. Fifty milliseconds later, at t=10.050, Alice creates a new document D inside F (inheriting F's membership); this write, on a different, faster-replicating content store, reaches R2 by t=10.060. Bob's read of D is routed to R2 at t=10.100. At that instant, R2 already has D (10.060 < 10.100) but has not yet received the revoke (10.200 > 10.100) — so the check "is Bob a member of F?" still evaluates true on that replica, and Bob wrongly reads a document that was created strictly after his access was pulled.

7. Zookie derivation — a snapshot watermark that lower-bounds freshness

A zookie is not a secret and not a signature over the data — it is an opaque token that encodes a snapshot point in the underlying globally-ordered store (in Zanzibar's implementation, a timestamp from the store's consistency mechanism). Every write returns one; a check can be handed one and required to honor it. Concretely, the fix to the trace above: the write that creates D is itself causally dependent on the current membership of F — Alice could only create it under the assumption that membership already reflects her revoke — so that write is chained to a zookie ≥ t=10.000. When Bob's read of D is required to honor that zookie, R2 is no longer allowed to answer from whatever snapshot it happens to have; it must answer from a snapshot at least as new as the zookie's watermark. Since R2 only reaches that watermark at t=10.200, the read either blocks or is served by a replica that has already caught up — and only then does it correctly see the revoke and return DENY. The zookie turns "how fresh must this one check be" from an unanswerable global question into a specific, checkable lower bound tied to the exact write the check depends on.

8. Practical revocation architecture — short JWT + opaque refresh + rotation

The standard shape that resolves "JWTs can't really be revoked" without giving up JWT's scaling story is a two-tier token pair:

• Access token: a short-lived (minutes), stateless JWT. Verified by signature only — no database round-trip — so it scales the same way any self-contained token does. Its blast radius if stolen is bounded by its own short exp.
• Refresh token: long-lived, opaque (a random string, not a JWT), stored server-side (hashed, like a password) and actually checked against server state on every use. This is the real revocation point: delete or mark the row and the refresh token is dead immediately, unlike a JWT that stays valid until it expires no matter what the server does.

That alone still leaves a gap: a stolen refresh token is valid for as long as its own long lifetime, unless something notices the theft. Refresh-token rotation with reuse detection closes it: every time a refresh token is used, the server issues a brand-new refresh token and immediately invalidates ("marks used") the one just presented, chaining both to the same token family. If an already-invalidated refresh token is ever presented again, there are only two explanations — a client retried after losing a response (rare, recoverable), or someone is replaying a token they stole earlier while the legitimate holder has since rotated past it (a compromise in progress). The safe policy treats any reuse as the latter and revokes the entire family — every token descended from that original login — forcing both the attacker and the legitimate user to authenticate again. This bounds a stolen refresh token's usable window to "before the legitimate client's next refresh," rather than "until its long expiry."

9. The boundary case with no per-request boundary: long-lived connections and batch jobs

"Authorize every request" assumes there is a request to hang the check on. A websocket held open for hours, a streaming subscription, or a long-running batch job that processes millions of records under one identity's permissions has no such boundary — the permission that was valid when the connection opened or the job started may be revoked minutes into a session that has no natural next "request" to re-check at. Four complementary answers, chosen by how much revocation lag is tolerable:

• Short token TTL forcing reconnect. The token used to establish the channel expires like any other; the channel must present a fresh one to continue, either by fully reconnecting or by renegotiating a new token over the still-open socket at the TTL boundary. Revocation takes effect within one TTL window.
• Periodic re-check. The server independently re-asks the PDP "is this principal still permitted?" on a fixed interval (e.g. every 30–60 seconds) regardless of whether the connection or job is otherwise idle, bounding staleness to that interval without requiring a reconnect.
• Revocation push. Instead of polling, the PDP (or a pub/sub layer next to it) actively notifies the services holding long-lived state when a relevant permission changes, so the specific connections tied to that principal or resource are invalidated the moment the change happens rather than at the next poll.
• Batch jobs — snapshot vs per-record re-check. A job that reads a permission once at start and runs for hours accepts staleness for its whole duration; a job that re-checks per object it touches pays a per-record cost for correctness. This is the same "how often do you re-authorize" question as the streaming case, just amortized over records instead of time.

Judgment layer — when to use which, and the named alternative

• Central Zanzibar-style ReBAC vs embedded per-service authorization. Reach for a Zanzibar-style central relationship store when access genuinely follows a graph of relationships and sharing — folders, ownership, "friends of friends" — and you can afford to operate a globally replicated system as infrastructure in its own right. Prefer embedded, per-service authorization (local RBAC/ABAC checks, no external graph) when your rules are a handful of stable roles or simple attribute conditions: you avoid running a second distributed system, at the cost of duplicating simple logic per service instead of centralizing it.
• Fail-closed vs fail-open. Fail-closed is the default for anything destructive or sensitive — a wrong permit is often unrecoverable. Fail-open (permit during an outage, with everything permitted logged for after-the-fact audit and reconciliation) is a deliberate, narrow exception reserved for low-risk, read-mostly paths where availability clearly outweighs the risk of a brief, logged over-permission — never for actions that move money or destroy data.
• Refresh-token rotation vs a single long-lived refresh token. Rotation costs real state — you must persist a "used" marker and a family chain, and handle the retried-request edge case without false-positive revocations. The payoff is bounding a stolen refresh token's usable window to one rotation cycle instead of its full lifetime; for anything beyond a low-value, low-risk client, that trade is worth the extra state.
• Opaque-token introspection vs self-contained JWT for the access token itself is the general version of the trade-off this page's revocation architecture already resolves for the common case (JWT access token, opaque refresh token) — reach further toward opaque + introspection everywhere only when instant revocation of every token, not just the refresh token, is a hard requirement (e.g. a banking session), accepting the per-request lookup cost that comes with it.

Pitfalls

• Trusting a client-supplied tenant_id (or any client-supplied identity claim) instead of the one on the verified token — the tenant-isolation bug in §3 and the algorithm-confusion bug in §2 are the same shape: letting the caller pick which trust boundary to check against.
• Verifying a JWT's signature without pinning the algorithm — "signature valid" only means "signed by someone with a key the verifier accepted for whatever algorithm the token claims," which is meaningless if the token itself gets to choose that algorithm.
• Making the PDP a synchronous, unreplicated hard dependency with no cache or embedded fallback — the correct fail-closed default then quietly becomes "our authorization layer is the single point of failure for the whole product."
• Assuming a globally distributed authorization graph is simply "consistent" — Zanzibar-scale systems buy availability with bounded staleness, not by making staleness disappear; treating it as free consistency is how the new-enemy bug slips through code review.
• Letting two causally related writes (a revoke and a later write that depends on it) replicate through independent, unchained paths — without a shared freshness token between them, their relative order is whatever each store's replication schedule happens to produce.
• Rotating refresh tokens without reuse detection — rotation alone just relabels the token each time; the security property comes from treating a replayed, already-used token as a compromise signal, not from rotation by itself.
• Never re-checking a long-lived connection — a revoked user keeps acting for the entire remaining life of an open socket or job if nothing forces a re-authorization point into it.

Takeaways

• PKCE binds an authorization code to whoever started the flow via a hash proof, closing the gap the implicit flow could never close for clients that cannot hold a secret — which is why it is now recommended for every client, not only public ones.
• The same public-key setup that makes RS256/JWKS verification scale (the key is meant to be public) is exactly what an unpinned verifier turns into an RS256→HS256 forgery path; pin the algorithm outside the token, never read it from the token's own header.
• Multi-tenancy, fail-closed availability, and the new-enemy problem are one family of bug: a check trusted something — a client-supplied ID, an always-up dependency, a cache — that was not actually guaranteed to be correct at the moment of the check. The zookie is the general fix pattern: attach a freshness lower bound to the specific write a check depends on, instead of assuming either "always fresh" or "always stale."
• Revocation in practice is short-lived stateless access tokens plus a long-lived opaque refresh token that is genuinely checked against server state, with rotation-plus-reuse-detection turning a stolen refresh token into a tripwire instead of a standing backdoor.

Sources: RFC 7636 (PKCE) and the OAuth 2.0 Security Best Current Practice (RFC 9700, PKCE for all clients, refresh token rotation and reuse detection); RFC 7519 (JWT) and the OWASP JSON Web Token Cheat Sheet (algorithm-confusion attacks); Google's Zanzibar paper (Pang et al., USENIX ATC 2019) for the new-enemy problem and zookies; OWASP API Security Top 10 (BOLA/IDOR at the tenant boundary); Google SRE Book (fail-closed vs availability trade-offs). Re-authored/Deepened for this guide.

Refresh-token family state

Storing a family lineage lets the server detect token replay and contain the blast radius of a stolen refresh token.

When NOT to choose each piece

• When NOT long-lived JWTs alone: need immediate revoke → short access TTL + refresh rotation (or opaque tokens + server session store).
• When NOT a remote PDP on every request without cache: PDP outage becomes total outage; pair with fail-closed policy + short-lived decision cache and stampede protection.
• When NOT Zanzibar-scale graph authz: simple RBAC on a single tenant admin app — a relation graph is overkill and ops-heavy.

Interviewer follow-ups & drills

1. Why not put tenant_id only in the path and trust the client? Client can swap IDs; scope from token claims and enforce server-side.
2. Ops signals: refresh-family reuse events, JWKS fetch failures, PDP latency/error rate, cross-tenant deny rate spikes.
3. Drill: Access JWT TTL 15m, stolen refresh used after rotation — what must revoke? Entire refresh family; force re-login.