Introduction to Caching

A cache works because a small pool of fast storage placed in front of a slow source can answer most requests from a working set that is far smaller than the whole dataset: you pay the slow source's latency once to populate an entry, then serve every subsequent request for that key from memory until it is evicted or expires. The entire payoff rests on one physical fact — reading from RAM is roughly five orders of magnitude faster than a cross-datacenter round trip — so if even a modest fraction of requests hit the fast layer, the average latency collapses toward the fast number.

The latency hierarchy that motivates caching

Caching is not a clever trick; it is arbitrage against a steep cost gradient. These are the canonical order-of-magnitude numbers (Jeff Dean's "Latency Numbers Every Programmer Should Know," updated to modern hardware):

A cache moves the answer up this table: an in-memory cache turns a 0.5 ms in-datacenter DB round trip (or a 150 ms cross-continent call) into a ~100 ns memory read. That is the whole game — and it is why "just add RAM in front of it" is the single highest-leverage latency optimization in systems work.

Worked example: what a 90% hit ratio actually buys

Suppose a read is served either from a Redis cache (1 ms end-to-end, including the network hop to the cache node) or, on a miss, from Postgres (50 ms including query + round trip). Let h be the hit ratio. The blended average latency is:

avg = h × t_hit + (1 − h) × t_miss

Note the subtlety: a miss usually costs more than a bare DB read, because you pay the cache lookup (1 ms), then the DB (50 ms), then a write-back to the cache. So a truthful miss cost is ~51 ms. Walking the ratio:

Two lessons fall out. First, the curve is dominated by the tail: going from 90% → 99% (a mere 9 points) roughly quarters the average latency, because each remaining miss is 51× more expensive than a hit. Second, a cold or badly-keyed cache with a low hit ratio can be net negative — at h=0 you added 1 ms of pure overhead to every request and got nothing. Hit ratio is the metric that decides whether the cache is an asset or a liability.

Key terminology, tied to the mechanism

• Cache hit — the requested key is present (and not expired); served from the fast layer. Hits are what create the payoff.
• Cache miss — the key is absent; you fall through to the source, then typically populate the cache so the next request hits. A miss costs more than a plain source read (lookup + source + write-back).
• Eviction — the cache is deliberately small (that is why it is fast and cheap), so when it fills, an eviction policy (LRU, LFU, etc.) removes entries to make room. Eviction is capacity-driven and involuntary.
• Expiration / TTL — a time-to-live that forces a refresh regardless of space. This is the main lever against staleness.
• Staleness — the cached copy no longer matches the source because the source changed after the copy was made. Every cache trades some freshness for speed; the discipline of caching is bounding that staleness.

Where caches live

The same mechanism recurs at every layer, differing only in what "fast" and "source" mean: CPU L1/L2/L3 (source = RAM), OS page cache and browser cache (source = disk/network), an in-memory cache like Redis or Memcached (source = database), the database's own buffer pool (source = disk pages), and a CDN edge cache (source = origin server, replacing a cross-continent trip with a nearby one). "Database caching" specifically means the DB keeping hot pages/results in its buffer pool — not a separate product.

Pitfalls

• Low hit ratio makes the cache a tax. As the worked table shows, at h≈0 you have only added latency. This happens with high-cardinality keys (e.g. caching per-request search URLs almost nobody repeats) or a cache too small for the working set — it thrashes, evicting entries before they are re-read.
• Stale reads after writes. If you update the DB but forget to invalidate or update the cache, clients keep reading the old value until the TTL expires. "There are only two hard things in Computer Science: cache invalidation and naming things" (Phil Karlton) is about exactly this.
• Thundering herd / cache stampede. When a hot key expires, thousands of concurrent misses hit the source simultaneously, all trying to recompute the same value — the DB can fall over precisely because the cache was doing its job until that instant. Try it: 10k rps hot key, TTL 60s — stampede risk at expiry; name one mitigation (single-flight).
• Unbounded cache = memory leak. A cache with no eviction policy and no TTL grows until the process is OOM-killed. A cache must have a bound.
• Caching non-idempotent or user-specific data under a shared key. Serving one user's authenticated response to another because the cache key omitted the user/tenant is a classic and dangerous bug.

When to reach for a cache — and when not to

A cache is the right tool when the access pattern is read-heavy with temporal locality: the same keys are requested repeatedly within a short window, and the data can tolerate being slightly stale. Concrete signals: read:write ratio well above ~10:1, a skewed key distribution (a small "hot set" serves most traffic), an expensive-to-produce value (a heavy join, an aggregation, a remote API call), and a business tolerance for bounded staleness (a product page can be seconds stale; a bank balance at a teller cannot). The inverse signals disqualify just as fast: write-heavy keys needing strong freshness (balances, live inventory) and per-user data with almost no reuse pay the cache's costs without its payoff.

Trade-offs vs. named alternatives

Choose a cache when the working set is much smaller than the dataset and reads dominate. You gain latency (the ~8× above) and offload the source. It costs you: a second system to operate, a coherence problem (invalidation) you did not have before, and the possibility of serving stale data.

• vs. a read replica of the database. A replica gives you more read throughput with the same query model and full consistency semantics, and no invalidation logic — but each read still costs a full DB round trip, so it does not fix latency. Prefer a replica when you need to scale reads while keeping strong-ish consistency and rich queries; prefer a cache when you specifically need per-request latency to drop and can tolerate staleness.
• vs. just a bigger / faster database (or more indexes). Tuning the source removes the need for a separate layer and its coherence bugs — but you eventually hit the floor of what a durable, consistent store can do, and it is far more expensive per QPS than RAM. Prefer optimizing the DB first when hit ratios would be low or writes dominate; add a cache once a genuine hot read set emerges.
• vs. precomputing / materialized views. Materialization gives predictable, always-warm reads with no miss penalty, but the write side pays to keep it current and it is inflexible to new query shapes. Prefer it for a small, known set of expensive aggregates; prefer a cache for a large, unpredictable key space with locality.

The senior instinct: do not add a cache reflexively. Add it when you can point to a specific hot read path, estimate the hit ratio, and accept a defined staleness bound. If any of those three is missing, a cache adds complexity and stale-data risk without a guaranteed win.

Fast disqualifiers

• Working set ≫ RAM and hit rate stays low

Design-review follow-ups: Why not cache everything? Fill cost + staleness + stampede risk must beat DB cost. What do you watch in operation? Hit rate, miss latency, and stampede DB QPS on TTL boundaries.

Takeaways

• Caching is latency arbitrage: it moves answers up a hierarchy where RAM (~100 ns) beats an in-DC round trip (~0.5 ms) by ~5,000× and a cross-continent trip (~150 ms) by ~a million×.
• The blended latency h·t_hit + (1−h)·t_miss is dominated by the tail — a high hit ratio is everything, and a low one makes the cache net-negative because misses cost more than an uncached read.
• Every cache trades freshness for speed; the engineering is in bounding staleness (TTL + invalidation) and bounding memory (eviction), not in the lookup itself.
• Reach for a cache only on read-heavy paths with locality and a tolerated staleness bound; otherwise a read replica, a tuned DB, or materialized views may fit better.

Re-authored and deepened for this guide. Latency figures adapted from Jeff Dean & Peter Norvig's "Latency Numbers Every Programmer Should Know" (with Colin Scott's updated interactive tables) and Brendan Gregg's Systems Performance. Caching patterns and failure modes draw on Alex Xu, System Design Interview (Vol. 1), Martin Kleppmann, Designing Data-Intensive Applications (Ch. 1, 5), and the Redis and AWS ElastiCache documentation. The invalidation aphorism is Phil Karlton's.