Storage Estimation Playbook

The formula

Storage = writes/day × bytes/write × 365 × years × replication factor

Then add indexes/metadata overhead (~10–30%) and don’t forget replication (×3 is the common default).

Steps

1. Writes/day from QPS: write_QPS × 86,400.
2. Bytes/write: use the “bytes per thing” anchors (text ~300 B, row ~1 KB, photo ~1 MB).
3. Multiply by retention in days, then by replication.

Worked example — a photo service

10M photos/day, ~1.5 MB each (compressed + a thumbnail), kept 5 years, ×3 replication:

• Per day: 10⁷ × 1.5 MB = 15 TB/day.
• 5 years: 15 TB × 365 × 5 ≈ ~27 PB, ×3 replication ≈ ~80 PB.
• Gate: petabytes of immutable blobs ⇒ object storage (S3), not a database.

Common mistakes

• Forgetting replication (off by 3×) and indexes/overhead.
• Confusing storage (bytes) with bandwidth (bits) — see the Gbps vs GB/s trap.

Second worked example — chat metadata

• 1M users send 50 messages/day; message ~200 B + 50 B metadata = 250 B.
• Daily writes = 1M × 50 × 250 B = 12.5 GB/day.
• 5-year retention: 12.5 GB × 365 × 5 = ~22.8 TB; ×3 replication = ~68 TB.
• Add indexes (~20%); keep media attachments in a separate object store → chat metadata fits in sharded SQL, media in S3. Compress text (250 B often → ~80 B) and set a per-class replication factor — not everything needs ×3.

When the formula misleads

Trap	What the naive formula does	Correct move
Peak vs average writes	Uses daily mean → under-provisions for Black Friday 10× spikes if you only size disks for average	Size durable storage for peak daily ingest × retention; size *bandwidth* separately for peak QPS
Hot vs cold data	Keeps 5 years all on SSD ×3	Tier: 30–90d hot, rest cold/archive; replication factor can drop on cold
Dedup / compression	Assumes raw bytes forever	Measure post-compress and content-hash dedup for media before quoting PB
Secondary indexes	Only multiplies row size	Each secondary index can be 20–100% of primary; full-text can exceed primary
Multi-region	×3 local only	Active-active can mean ×3 per region or async copies; state the topology

Choosing RF per data class

RF=2 is defensible when the data is re-derivable or has an authoritative copy elsewhere — thumbnails regenerable from originals in S3, caches, search indexes rebuildable from the primary store. You only need to survive one node loss long enough to re-replicate, and the second copy exists to keep serving during that repair window. RF=3 is the floor when this IS the only copy: with RF=2, the repair window after the first loss is a single-copy exposure window, and disk fleets at PB scale lose disks daily — 27 PB ÷ 10 TB = 2,700+ disks at ~1–2% AFR is 27–54 failures/yr, so single-copy windows recur weekly.

• Chat metadata (only copy, serves reads during repair) → RF=3 = ~68 TB.
• Thumbnails (regenerable from S3 originals) → RF=2 = tens of TB saved at zero durability cost.
• Cold archive behind erasure coding drops below RF=2-equivalent overhead — see Replication in Capacity Estimation.

Boundary example A: 1 TB/day raw media. Formula says 1×365×5×3 = ~5.5 PB. With 2:1 compress + 30% dedup + 90-day hot only: hot ≈ 1×0.5×0.7×90×3 ≈ 95 TB hot; cold archive ~ years on cheaper storage. Same “5 years of photos,” totally different bill.

Boundary example B (growth compounds): flat 15 TB/day × 5 yr ≈ 27 PB raw. At +20%/year write growth: sum = 15×365×(1.2⁰+…+1.2⁴) ≈ 15×365×7.44 ≈ ~40.7 PB (~49% higher) before RF. Opposite trap: 30-day TTL on a 12.5 GB/day chat log → steady-state ~375 GB, not 5×365×12.5 GB.

See also: Senior corrections — fan-out, peak, utilization & durability.

Drills

1. Recompute the photo service if thumbnails are 50 KB and only originals need 5-year retention.
2. When would you choose RF=2 instead of RF=3 for a data class? (Answered in “Choosing RF per data class” above — try it from memory first.)

---

Formulas are standard/public-domain engineering math. Approach and reference-table format adapted from the System Design Primer (CC BY 4.0), Jeff Dean’s latency numbers, the DesignGurus capacity-estimation guide, and Little’s Law.

I just read a lesson on **Storage Estimation Playbook** (System Design) and want to truly understand it. Explain Storage Estimation Playbook from first principles using ONE vivid real-world analogy and a visual mental model — draw it as ASCII art or a clear step-by-step diagram — with a concrete example using real numbers. Then ask me one question to check I got the mental picture, and wait for my reply. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Teach me **Storage Estimation Playbook** interactively. Ask me ONE guiding question at a time, wait for my answer, and adapt to my confusion — build the idea with me step by step instead of explaining it all at once. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Quiz me on **Storage Estimation Playbook** with 5 questions, easy to tricky, ONE at a time. Tell me if each answer is right; at the end, explain clearly what I got wrong and why. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Help me remember **Storage Estimation Playbook** for the long term: give the one-sentence intuition, a memorable hook/mnemonic, a tiny worked example, and 3 active-recall flashcards (Q -> A). If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.