Read Heavy vs Write Heavy System

A workload is read-heavy or write-heavy because of which operation you can cheaply make more copies of: reads can be fanned out to any number of identical stale-tolerant replicas and caches, so you scale reads by duplication; writes must converge to a single authoritative order (one row, one log, one committed sequence), so you scale writes by reducing per-write cost — batching, appending sequentially, and deferring work. That asymmetry is why the two are tuned in opposite directions, and why bolting read tactics onto a write problem (or vice versa) fails.

The signal is the read:write ratio measured in operations per second, not row counts. Content and social feeds typically run 10:1 to 1000:1 reads-to-writes (a tweet is written once, read millions of times); metrics/telemetry, event logs, and IoT ingestion invert it — near 1:50 or worse. Above roughly 95% reads you optimize the read path first; below roughly 80% reads (a sustained write flood) the write path is the bottleneck and no amount of caching helps.

Worked trace 1 — a read-heavy feed (200:1)

Take a news/social service at peak: 50,000 read QPS and 250 write QPS (a 200:1 ratio). A single Postgres primary handles maybe a few thousand simple queries/sec before CPU and buffer contention saturate it, so 50,250 QPS on one node is impossible. Layer the read tactics and watch the load collapse:

The whole point: the primary went from an impossible 50,250 QPS to a comfortable 250. Cache did the heavy lifting (95% absorbed in RAM); replicas mopped up the misses; the primary is reserved for the one thing that cannot be duplicated — the authoritative write.

The trap nobody sees in the diagram: read-your-writes vs replica lag

Replicas are asynchronous: the primary commits, returns success, then ships the WAL. Under normal load a replica trails by tens of milliseconds; but most engines apply replication on a single thread, so a write burst or one long transaction can push lag to seconds or minutes. Now combine that with a load balancer that sends the user's next read to a random replica:

This is a read-your-writes violation, and it is the single most common production bug of the replica pattern. Fixes, cheapest first: (1) after a user's write, pin their reads to the primary for a few seconds (session stickiness); (2) return the commit's log position (LSN) to the client and require the serving replica to have applied ≥ that LSN before answering (monotonic/session consistency); (3) write the new value into the cache on the write path so the user reads their own write from RAM. Budget the lag explicitly — alert at >1 s, page at >5 s — because lag is invisible until a user complains.

Worked trace 2 — write-heavy ingestion (1:50)

Now flip it: a metrics/telemetry service ingesting 200,000 writes/sec with only a trickle of dashboard reads. Caching and replicas are useless here — every write must be durably recorded, and the bottleneck is fsync. An SSD fsync costs ~1 ms; a naive "commit each write with its own fsync" loop is therefore capped near 1,000 writes/sec, 200× short. Two mechanisms rescue it:

• Group commit / batching. Buffer writes arriving within a small window and flush them behind a single fsync. Batch 1,000 writes per flush and the disk does one fsync for the group — but an fsync of a large batch is no longer ~1 ms: flushing ~1,000 records means writing hundreds of KB plus the flush-window wait, so a realistic cycle is on the order of 5–10 ms. At ~100–200 flushes/sec × 1,000 writes/flush the single-disk ceiling lands at approximately 100,000–200,000 writes/sec — not the 1,000,000/sec the naive 1 ms figure would suggest. The cost is latency: a write now waits up to the flush window (say 5 ms) before it is durable, and anything buffered but not yet flushed is lost on a crash.
• Append-only / LSM storage. A B-tree (Postgres, MySQL/InnoDB) updates rows in place: a 200-byte update can dirty a full 8 KB page plus a WAL record plus, right after a checkpoint, a full-page image — heavy random I/O and large write amplification under load. An LSM engine (Cassandra, RocksDB, ScyllaDB) instead appends every write to an in-memory memtable and flushes it as one sequential SSTable, converting random writes into streaming ones. That is what lets a Cassandra node sustain tens of thousands of writes/sec where a B-tree primary stalls.

Combine both plus sharding across, say, 8 nodes and 200,000 writes/sec becomes ~25,000/node — well within LSM range. The reads that do happen are the price you pay (see pitfalls).

Pitfalls

• Cache stampede (thundering herd). A hot key expires; thousands of concurrent requests all miss simultaneously and hammer the primary in the same millisecond — the DB you protected gets a spike worse than no cache. Fix with per-key locks / request coalescing, or probabilistic early recomputation before expiry.
• Stale reads from replica lag. The read-your-writes bug above, plus non-monotonic reads: two consecutive reads hit replicas at different lag and the value appears to go backwards. Route related reads to one replica or enforce session consistency.
• Single-threaded replication apply. A write burst on the primary makes lag balloon on replicas because the apply thread can't keep up — exactly when you most need them, they serve the stalest data. Watch for engines with parallel apply, and cap long transactions.
• Batching enlarges the data-loss window. Group commit and async writes trade durability for throughput: a crash loses everything buffered since the last flush. Never batch writes that must survive a crash without an explicit, bounded window and acknowledgment strategy.
• LSM read amplification & compaction stalls. A point read may probe the memtable plus several SSTables (bloom filters help but don't eliminate it), so reads are slower than a B-tree. And if ingest outruns background compaction, the engine issues write stalls — throughput cliffs to zero until compaction catches up.
• Hot shards (celebrity problem). Sharding by user ID spreads load only if it's uniform; one viral account or one hot time-bucket concentrates all traffic on a single shard, recreating the bottleneck you sharded to remove.

When to use it / when NOT — and the trade-offs

Reach for the read-heavy stack (cache + replicas) when the ratio is ≥95% reads, the same keys are hit repeatedly (high cache locality), and users tolerate seconds of staleness. Do not when reads are unique/uncacheable (analytical scans over arbitrary ranges — a cache never warms) or when correctness demands the freshest value on every read (bank balances before a transfer).

Cache (Redis) vs read replica — both offload reads, but differently. A cache gives sub-millisecond RAM reads and huge fan-out relief, but you own invalidation, stampede risk, and it only serves what you thought to cache. A replica speaks full SQL (arbitrary queries, joins), needs no invalidation logic, but costs a real query per read and carries lag. Choose the cache when a small hot set dominates and staleness is fine; prefer replicas when queries are diverse and you can't enumerate cache keys — and in practice, layer both.

Reach for the write-heavy stack (LSM + batching + async + sharding) when writes are the sustained bottleneck, writes are append-like (events, metrics, logs), and reads are few or tolerate higher latency. Do not when you need fast point/range reads on the freshest data with strong consistency, or when write volume actually fits a single tuned B-tree primary — the LSM's read penalty and operational weight aren't worth it.

LSM vs B-tree — the core write decision. LSM turns random writes into sequential appends (far higher ingest, lower write latency) but pays with read amplification and constant background compaction (CPU, I/O, occasional stalls). A B-tree gives fast in-place reads and simple operations but does random-write I/O with high write amplification that collapses under a write flood. Choose LSM when ingest dominates and reads are point-lookups or scans that tolerate a few SSTable probes; prefer a B-tree when reads are the hot path and writes fit the primary.

CQRS vs one shared model. Splitting writes (command) from reads (query) lets each side scale and be modeled independently — but you inherit eventual consistency between them, a projection/sync pipeline, and roughly double the moving parts. Choose CQRS when read and write shapes genuinely diverge at scale; prefer one model when a single well-indexed store still keeps up — most systems never need CQRS.

The senior instinct: measure the ratio and the tail latency budget first, then add the cheapest tactic that clears the bottleneck (usually a cache for reads, batching for writes) before reaching for replicas, sharding, or CQRS — each of which buys scale by spending consistency and operational simplicity.

Takeaways

• Reads scale by duplication (caches, replicas, CDN); writes scale by cheapening each write (batching, sequential appends, deferring) — they pull in opposite directions.
• The dominant read-path failure is replica lag breaking read-your-writes; budget lag explicitly and pin post-write reads to the primary or enforce session consistency.
• The dominant write-path lever is amortizing fsync via group commit and choosing LSM over B-tree for ingest — both trade durability window and read speed for throughput.
• Every tactic here buys scale by spending consistency or operational complexity; add the cheapest one that clears the measured bottleneck, not the fanciest.

Re-authored/Deepened for this guide. Draws on Martin Kleppmann, Designing Data-Intensive Applications (replication lag, read-your-writes and monotonic-read consistency, B-tree vs LSM storage engines); the PostgreSQL documentation on WAL and group commit; Apache Cassandra and RocksDB design notes on LSM-tree write/read amplification and compaction; and Redis documentation on caching patterns and stampede mitigation.