Why is Caching Important

Why is Caching Important

A cache is a small, fast store that sits in front of a slower, more expensive source of truth and keeps copies of the data you are most likely to ask for again. The whole idea rests on one stubborn fact of computing: the fastest work is the work you never do. If you computed an answer a moment ago, recomputing it or fetching it from disk over the network is pure waste. Caching turns "do the expensive thing every time" into "do it once, then reuse."

Two forces make caching almost unavoidable in real systems. First, access patterns are skewed — a tiny fraction of items (hot keys, trending posts, the logged-in user's own profile) account for most requests. Second, the latency and cost of storage layers differ by orders of magnitude: a CPU cache hit is nanoseconds, RAM is ~100 ns, a local SSD is ~100 µs, and a cross-service database query can be 1–10 ms. Caching lets the common case ride the fast layer while the rare case pays full price.

How it works, precisely

A cache is fundamentally a key-value map with an eviction policy (because it is bounded) and an invalidation strategy (because the source of truth changes). On each request you perform a lookup: a cache hit returns the stored value immediately; a cache miss falls through to the origin, and you typically populate the cache with the result on the way back. The fraction of requests served from cache is the hit ratio, and it is the single number that governs how much a cache helps.

Because memory is finite, when the cache fills you must evict something. Common policies are LRU (evict the least-recently-used entry, betting recent use predicts future use), LFU (evict the least-frequently-used), and TTL expiry (each entry lives for a fixed time). Writes need a policy too: write-through updates cache and origin together (consistent, slower writes); write-back updates the cache first and flushes later (fast, but risks loss); cache-aside (a.k.a. lazy loading) leaves the application to read the cache, miss, load from the DB, and write back — the most common pattern for web backends.

A worked scenario

Suppose a social feed service handles 10,000 QPS of profile reads. Each read hits Postgres directly at 5 ms latency, and the DB can sustain roughly 12,000 QPS before it saturates — so you are already at 83% capacity with no headroom. You add a Redis cache in front with cache-aside and a 60-second TTL.

Social-graph read traffic is heavily skewed — measured production workloads (e.g. Facebook's memcache fleet; Nishtala et al., Scaling Memcache at Facebook, NSDI 2013) see a small hot set serve the vast majority of reads — so assume the top ~50,000 profiles capture a 95% hit ratio. (That 95% is a measured/assumed workload property, not a consequence of "Zipfian" alone: pure Zipf s=1 over millions of keys gives a flatter curve; real feeds are steeper because of trending amplification.) Now only 5% of traffic — 500 QPS — reaches Postgres. The DB load drops from 10,000 to 500 QPS (a 20× reduction), instantly restoring headroom. Average read latency becomes 0.95 × 0.2 ms + 0.05 × 5 ms = 0.44 ms, an ~11× speedup. Strictly, a miss also pays the 0.2 ms failed cache lookup before falling through, so the honest average is 0.95 × 0.2 + 0.05 × 5.2 ≈ 0.45 ms — negligible here, but that lookup tax is exactly why a low-hit-ratio cache is net-negative: at h = 0 you add 0.2 ms to every request for nothing (see Cache Performance Metrics for the break-even hit rate). And crucially, the cache absorbs traffic spikes: a celebrity post that 10× the reads on one key barely touches the origin because that key is already hot in cache.

Notice the leverage: raising the hit ratio from 95% to 99% cuts DB traffic from 500 to 100 QPS — another 5× — which is why interviewers care far more about hit ratio than raw cache speed.

Trade-offs, and when NOT to cache

Caching is not free. You trade consistency for latency and load reduction: a cached value can be stale relative to the source of truth for up to its TTL. You add an extra moving part (the cache tier) that can fail, and you take on the hard problem of invalidation. Weigh caching against the named alternatives before reaching for it:

• Read replicas — scale reads without staleness beyond replication lag and without invalidation logic. Prefer these when you need strong-ish consistency and your bottleneck is raw read throughput on relational queries, not per-key latency.
• Denormalization / precomputation — bake the answer into the write path (e.g. a materialized view or a fan-out-on-write feed). Better than caching when the read is a heavy aggregation that rarely changes.
• Vertical scaling / better indexes — often a missing index, not a missing cache, is the real fix. Cache last, after the query itself is tuned.

Use a cache when: reads dominate writes, access is skewed toward hot keys, the origin is expensive or capacity-constrained, and some staleness is tolerable. Avoid it when: data must be strictly fresh (balances, inventory, auth tokens), access is uniform with no hot set (hit ratio will be low and you gain little), or write-heavy churn constantly invalidates entries so you pay to populate values nobody rereads.

Pitfalls an interviewer probes

• Cache stampede (thundering herd) — a hot key expires and thousands of concurrent misses hammer the DB at once. Mitigate with request coalescing / single-flight locks, staggered TTL jitter, or serving-stale-while-revalidate.
• Invalidation is the hard problem — be ready to discuss write-through vs. TTL vs. explicit purge, and the risk of serving stale data. "There are only two hard things in CS..." is the expected nod.
• Hot-key / cache penetration — queries for keys that don't exist bypass the cache every time; cache negative results (short TTL) or use a Bloom filter. A single ultra-hot key can also overload one shard.
• Wrong metric focus — candidates brag about Redis being fast; the interviewer wants the hit ratio, the eviction policy fit, and what happens on a total cache failure (can the origin survive the full unbuffered load? often not — that's a cascading outage).

Key takeaways

• Caching exploits skewed access patterns and the huge latency gap between storage layers to make the common case cheap; hit ratio, not cache speed, is the metric that matters.
• The core mechanism is a bounded key-value store with an eviction policy (LRU/LFU/TTL) and a write/invalidation strategy (cache-aside, write-through, write-back), each trading freshness against speed.
• Reach for a cache only when reads dominate, keys are hot, the origin is costly, and staleness is tolerable — otherwise prefer read replicas, precomputation, or better indexes.
• Interviewers probe the failure modes: stampedes, invalidation correctness, cache penetration, and whether your origin survives a full cache outage.