System Design Trade-offs

Functional = WHAT it does (post tweet, follow user, load timeline). Non-functional = HOW WELL (latency, availability, scale).

• Functional defines the API surface → you can't size anything without it.
• Non-functional drives every trade-off → p99 < 200ms + 99.99% uptime + 500M reads/day forces caching, replication, and CDN.

Cause→effect: state 200M DAU, 100:1 read:write → that single number tells you it's read-heavy → you justify caching + read replicas later. Cost: time spent here is time NOT drawing; under-spending here means you over-engineer the wrong axis (e.g. sharding writes when the pain is reads).

Decide by access pattern + relational need, not popularity.

• SQL (Postgres/MySQL): strong schema, multi-row ACID transactions, ad-hoc joins. Single primary caps writes at roughly 5k–20k writes/sec.
• NoSQL wide-column (Cassandra/DynamoDB): denormalized, partition-key access, horizontal writes to 100k+/sec, but joins/transactions are weak.

Cause→effect: timeline = "give me last 50 tweets for a key" → single-partition lookup → NoSQL is a clean fit. Billing/payments = needs ACID → SQL. Cost of NoSQL: you denormalize, so one logical update fans out to many rows you must keep in sync in app code — no foreign keys to save you.

They're orthogonal. Latency = time for ONE request. Throughput = requests/sec across all.

• Batching 100 writes into one disk flush: each waits longer (+250ms latency) but you do 10k instead of 1k ops/sec (throughput up).
• The SLA word decides: if it says p99 latency < 200ms, the batch breaks it. If it says handle 10k req/s, the batch is the fix.

Cause→effect: pipelining/batching amortizes fixed per-op cost over many ops → throughput rises → but in-flight queuing → tail latency rises. Trade-off: on fixed hardware you generally trade one for the other; raising both at once means buying capacity, not free lunch. Name which one the user actually feels.

Because consistency trades against availability + latency (CAP under partition / PACELC when there's no partition).

• Strong: the write must be acknowledged by a quorum/primary before reads see it → a cross-region read may pay +80ms coordination, and if the primary is partitioned, writes BLOCK.
• Eventual: any replica answers immediately (~5ms local) and converges in ~10–500ms → stays available during partitions.

Cause→effect: a profile photo being 200ms stale harms nobody → choose eventual for speed + uptime. A bank balance must be exact → choose strong, accept the latency and reduced availability. Cost of eventual: your app must tolerate reading its own stale writes ("read-your-writes" anomalies) unless you add sticky routing.

Mechanism: cache serves hot keys from RAM, skipping disk + query planning.

• Mean read ≈ 0.9×1ms + 0.1×40ms = 4.9ms. But the p99 isn't this average — it's dominated by the misses, so it still lands near the DB's own tail (~40ms+) unless the DB tail itself shrinks.
• Why the tail also improves: 90% of traffic never reaches the DB → offered load drops → the DB's queue drains → even the 10% misses wait less. So p99 falls, but because the DB got faster, not because 4.9ms is the p99.

Three costs you must name:

• Staleness/consistency: cache and DB disagree until TTL expires or you invalidate. Pick TTL by tolerance: 60s for a feed, 0 (write-through) for a price.
• Eviction: RAM is finite → LRU/LFU drops cold keys → a scan can blow the working set out and tank the hit rate.
• Stampede: a hot key expires → 10k requests miss simultaneously → they all hammer the DB at once. Fix with request coalescing / a short lock / probabilistic early refresh.

• Read-through: on a miss, the cache loads from DB, stores it, returns it. First read of a key is slow (40ms); all subsequent are fast (1ms). Cache only holds what's actually requested → small footprint.
• Write-through: every write goes to cache AND DB synchronously → cache is never stale, but every write pays both latencies (~5ms + ~40ms) and you cache data that may never be read.

Cause→effect: a feed is read 100:1 and tolerates seconds of staleness → read-through, let cold data stay out. A checkout price must never be wrong and is read right after write → write-through guarantees freshness. Cost: read-through risks serving stale until next load (and eats a cold-miss on first access); write-through wastes writes + RAM on rarely-read keys. (Write-back is faster still — ack the write from cache, flush to DB later — but risks data loss if the cache dies before flushing.)

Mechanism (primary–replica): writes go to the primary, which streams its change log to replicas; reads fan out across replicas.

• Fixes reads: 1 primary + 3 replicas ≈ 4x read capacity → CPU drops, read p99 falls.
• Does NOT fix writes: every write still funnels through the one primary → write ceiling unchanged. For that you need sharding or peer-to-peer (multi-primary).

The bug: replication lag (10–500ms) → a user posts a comment (write→primary) then refreshes (read→replica that hasn't caught up) → "my comment vanished!" Fix with read-your-writes routing (send that user's reads to the primary briefly). Trade-off vs peer-to-peer: P2P/multi-primary lets any node take writes (higher write availability) but introduces write–write conflicts you must resolve (last-writer-wins, vector clocks) — primary–replica avoids conflicts but has a single write bottleneck + a failover gap when the primary dies.

They operate at different layers and do different jobs:

• Reverse proxy (nginx): the base primitive — terminates TLS, forwards to a backend, can cache static assets. "Hide my servers behind one address."
• Load balancer: a reverse proxy specialized for distributing traffic across N identical instances (round-robin/least-connections) + health checks. Job: spread load, survive an instance dying. Operates L4 (transport) or L7 (HTTP).
• API gateway: L7, per-API logic — auth, rate-limiting (1000 req/min/key), request routing to different microservices, response aggregation, versioning. Job: one front door with policy for many services.

Cause→effect: the LB answers "which of my 10 identical app servers?"; the gateway answers "is this caller allowed, throttled, and which service owns /payments?". You run both because they solve different problems: a common topology is LB → gateway → services, where the LB spreads raw connections across a fleet of gateway instances and the gateway then applies policy and fans out to the right microservice. Cost: two hops, two more components to operate, monitor, and fail over — extra latency and operational surface in exchange for clean separation of "distribute load" from "enforce API policy."