hard Rate-Limit Hot Keys

Rate-Limit Hot Keys — sharding & local pre-aggregation

A distributed rate limiter usually treats its counter store (Redis) as uniform — but a limit is keyed per client/tenant, and one celebrity tenant or a single hot API key sends all its INCRs to one key, which lives on one shard/slot. That slot becomes a throughput and tail-latency hotspot that no amount of cluster scaling fixes: adding Redis nodes re-distributes different keys, but a single key cannot be split across slots by the cluster. The mechanism to internalize: sharding helps many keys, never one hot key. The fix has to change how that one key is counted.

Traced example (reuse the 60k-RPS gateway)

Peak 60,000 RPS across 20 gateway nodes. Suppose one celebrity tenant is 1/3 of traffic → ~20,000 RPS on a single key. Every request does INCR {celeb}:{window}, so that key's slot takes 20,000 ops/s while the other slots sit near-idle. That one slot's CPU and its p99 now gate the whole limiter.

Mitigation 1 — sub-counter sharding (split the key)

Replace the one key with K sub-counters {celeb}:{window}:0 … :K-1. Each request INCRs a random (or round-robin) sub-shard, so the 20,000 ops/s spread across K slots — at K=8, 2,500 ops/s per slot, which the cluster can now place on different nodes. The limit check reads all K and sums them.

• Cost: every check is a K-read fan-out (a pipeline/MGET of K keys) and K× the key memory. The per-shard view is partial, so an exact atomic "is this the Nth request" is lost — you compare the summed total.
• When: moderate hotspots where you still want a near-exact global count and can afford the read fan-out.

Mitigation 2 — local pre-aggregation + periodic flush

Each gateway node counts the hot key in local memory and flushes the accumulated delta to Redis on an interval (say every 100 ms). Redis now sees 20 nodes × 10 flushes/s = ~200 ops/s for that key instead of 20,000 — a ~100× reduction — and the hot slot stops being hot.

• Cost: it is approximate. Between flushes, nodes decide locally, so the global limit can overshoot by up to (number of nodes × each node's per-interval local allowance) at the flush boundary, and up to ~100 ms of staleness. You are trading exactness for hotspot relief.
• When: extreme hotspots where a slightly-loose limit is acceptable and Redis-load / blip-tolerance matters more than precision.

Selection & trade-off

Sharding keeps the limiter centralized and near-exact but taxes every check with a K-read fan-out; local pre-aggregation slashes store load ~100× and survives store blips but is approximate at the boundary. Rule of thumb: reach for sharding when accuracy is the priority and the hotspot is moderate; reach for local pre-aggregation when the hotspot is severe and approximate limiting is fine. They compose — shard the global counter and pre-aggregate locally — for the worst celebrity-tenant cases.

Pitfalls

• Assuming a bigger/again-sharded Redis cluster fixes a single hot key — it cannot; the key is the unit of placement.
• Sharding but forgetting the read is now a fan-out — a naive per-check MGET of K keys on the hot path adds latency.
• Local pre-aggregation with too long a flush interval → large overshoot; too short → you lose the load-reduction benefit. Tune the interval to the acceptable overshoot.
• Not detecting which keys are hot — apply these mitigations selectively (only to keys above a traffic threshold), not to every key.

Takeaways

• Cluster scaling distributes many keys; it never splits one hot key — the counting scheme must change.
• Sub-counter sharding: INCR a random of K sub-keys, sum on read — near-exact, costs a K-read fan-out.
• Local pre-aggregation: count in memory, flush deltas periodically — ~100× less store load, approximate at the boundary.
• Detect hot keys and apply selectively; combine both for celebrity tenants.

---

Re-authored and deepened for this guide, drawing on standard hot-key / hot-shard mitigations (sub-counter sharding, client-side pre-aggregation) from large-scale rate-limiting practice. Re-authored/Deepened for this guide.

I just read a lesson on **Rate-Limit Hot Keys** (System Design) and want to truly understand it. Explain Rate-Limit Hot Keys from first principles using ONE vivid real-world analogy and a visual mental model — draw it as ASCII art or a clear step-by-step diagram — with a concrete example using real numbers. Then ask me one question to check I got the mental picture, and wait for my reply. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Teach me **Rate-Limit Hot Keys** interactively. Ask me ONE guiding question at a time, wait for my answer, and adapt to my confusion — build the idea with me step by step instead of explaining it all at once. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Quiz me on **Rate-Limit Hot Keys** with 5 questions, easy to tricky, ONE at a time. Tell me if each answer is right; at the end, explain clearly what I got wrong and why. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Help me remember **Rate-Limit Hot Keys** for the long term: give the one-sentence intuition, a memorable hook/mnemonic, a tiny worked example, and 3 active-recall flashcards (Q -> A). If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.