Knowledge Guide
Home › Hands-On Builds › SD Design Labs

Design Instagram / News Feed

Design Instagram / News Feed

Here's the answer that feels obviously right and is a trap: "the feed is just the posts from everyone you follow, newest first — so when someone opens the app, query all their followees' recent posts, merge-sort by timestamp, return the top page." It sounds like one SQL query. It is one SQL query. And it collapses the instant you put a real user behind it. A person who follows 1,000 accounts now triggers a 1,000-way scatter-gather + an in-memory sort on every single feed open, and they open the feed dozens of times a day. Multiply that by hundreds of millions of daily users (we compute the exact query rate below) and your posts database is being asked for tens of millions of point-lookups per second whose results you then throw away and recompute five minutes later when they refresh. That's the first wall. The second wall has a name and a face: the celebrity. The naive fix for the read cost is "precompute each user's feed when a post is made" — but a celebrity with 100 million followers who posts one photo now generates 100 million feed writes from a single tap. This lab derives the design that survives both walls at once, and the punchline is that you don't pick one strategy — you pick different strategies for different users, on purpose.

1. The Trap — the "obvious" feed breaks two different ways

Trap A — build the feed on read (pull). The naive read-time feed:

// NAIVE getFeed(user)
followees = follows.where(follower = user)        // e.g. 1,000 accounts
posts = postsDB.where(author IN followees)         // scatter-gather across shards
         .orderBy(created_at DESC).limit(50)        // merge-sort every time
return posts

Trace the cost. A user following 1,000 accounts makes the DB fetch recent posts from 1,000 authors (spread across many shards), then sort them, every time they pull to refresh. Nothing is reused between two refreshes 30 seconds apart — the same 1,000-way query runs again. In the BOTE below we compute this at ~29 million posts-store queries/second system-wide, to serve feed opens whose results are almost identical to the last open. You are recomputing an answer that barely changed, at the busiest point in the system, on the path the user is literally staring at. Feed opens vastly outnumber posts (people read far more than they post), so you've put your heaviest amplification on your highest-traffic path. That's backwards.

Trap B — precompute the feed on write (push), naively, for everyone. The obvious fix for Trap A: when someone posts, immediately write that post's ID into every follower's precomputed feed, so a feed open is just "read my ready-made list" — O(1). Beautiful for the reader. Now a normal user with 500 followers posts once → 500 tiny feed writes, cheap and async, nobody notices. Then a celebrity with 100 million followers posts one photo. That single POST becomes 100,000,000 feed writes. At a sustained fan-out capacity of 50,000 writes/sec that one post takes 100,000,000 ÷ 50,000 = 2,000s ≈ 33 minutes to fully propagate — during which the queue is jammed and everyone else's posts are delayed behind it. Push turns a celebrity's single tap into a write storm sized to their follower count, and it storms exactly where push was supposed to be cheap. Feel both failures before reading on: pull dies on the read path for high-following users, push dies on the write path for high-follower users. The design has to answer both.

2. Scope it like a senior (ask before you design)

Don't draw a box yet. Each of these answers changes the architecture, not just a constant:

Assume for this lab: 500M DAU; each user posts 2 photos/day and opens the feed 10×/day; average user has ~500 followers and follows ~500 accounts; follower distribution is power-law with celebrities up to 100M followers; feed may lag a few seconds (eventual consistency OK); default feed is reverse-chronological with a ranked-feed variant discussed; each photo stored in ~3 resolutions averaging ~300 KB total delivered; media is far more read than written. Peak factor 3× over average.

3. Reason to the design (derive it, don't recite it)

Attempt 1 — fan-out-on-read (pull). Store each post once; build the feed at read time by querying followees and merging. Writes are trivial (one row per post) and there's zero write amplification — great for someone who posts to millions of followers, because a celebrity's post is just one insert. But the read path pays for it: every feed open is an N-way scatter-gather + sort where N = accounts-followed, and feed opens are your highest-volume operation. This is Trap A. Pull is cheap for high-follower accounts, brutal for high-following readers and high read volume.

Attempt 2 — fan-out-on-write (push). Invert it: when a post is made, precompute by appending its ID into every follower's materialized feed (a per-user list in Redis). A feed open becomes an O(1) read of a ready list — the read path, your hottest path, is now dirt cheap and trivially cacheable. Write cost is proportional to follower count, but for a normal user (~500 followers) that's 500 cheap async list-appends nobody waits on. Since reads ≫ writes and most accounts have bounded followers, push amortizes beautifully: pay a little at write time so every one of the many reads is free. This is the correct default. Then the celebrity walks in (Trap B): 100M followers → 100M writes per post → a fan-out storm that jams the queue for everyone. Push is cheap for readers and bounded-follower authors, catastrophic for high-follower authors.

The key insight — the two strategies fail on opposite axes, so use each where it wins: the HYBRID. Pull is bad exactly where push is good (high read volume, bounded followers) and good exactly where push is bad (extreme follower counts). So classify accounts by a follower threshold T: normal users (below T) use push — their posts fan out into follower feeds, keeping the common read O(1); celebrities (above T) are exempt from fan-out entirely — their posts are stored once and pulled + merged at read time. A reader's feed is then two things unioned at read: their precomputed push-feed (from all the normal accounts they follow) merged with a live pull of recent posts from the handful of celebrities they follow. Because a typical user follows only a few celebrities, that read-time merge is tiny — you've kept push's cheap reads while defusing the one case that blows push up. This is the whole design; everything else is plumbing.

Media is a separate problem from metadata — don't conflate them. A photo is a large immutable blob; a post is a small mutable-ish metadata record (caption, author, timestamp, likes). Storing multi-hundred-KB blobs in your posts DB bloats every row and every backup and makes sharding miserable. Instead: the blob store (S3-style object storage) holds the bytes, fronted by a CDN that caches images at edge PoPs near users; the posts DB holds only metadata plus the media URL. On upload the client sends bytes to the blob store (often via a pre-signed URL, so the bytes never transit your app servers) and the app writes only metadata. On read, the feed returns metadata + CDN URLs and the client fetches images straight from the nearest edge — so ~99% of the enormous media read bandwidth (computed below) never touches your origin. Metadata is small, cacheable, and fans out; media is huge, immutable, and belongs on a CDN. Keep them apart.

4. Build it — the design worksheet

This is the artifact you'd produce on a whiteboard in a 45-minute round. Work it in order; every number is traced, not asserted — recompute it yourself before an interview.

Functional requirements

Non-functional requirements

1. Back-of-envelope estimation (BOTE)

2. API sketch

POST /post
     body: { media (uploaded to blob via pre-signed URL), caption }
     -> 201 { post_id, media_url, created_at }

GET  /feed?cursor=<opaque>&limit=50
     -> 200 { items: [ {post_id, author, media_url, caption, created_at} ],
              next_cursor }        // push-feed  +  merged celebrity pull

POST /follow    body: { target_user_id }   -> 204
DELETE /follow  body: { target_user_id }   -> 204

3. Data model & partitioning

4. High-level diagram — write fan-out, read path, and CDN

The write path (blue) stores media in blob+CDN, writes metadata to the Posts DB, and — only for normal users — enqueues an async fan-out that pushes the post ID into each follower's Redis feed. Celebrity posts (red) skip fan-out and sit once in storage. The read path (green) reads the reader's materialized feed O(1) from Redis and merges in a live pull of recent posts from the few celebrities they follow; images (amber) come straight from the CDN edge, so origin bandwidth stays tiny.

Instagram/News Feed hybrid architecture: write path (blue) from Client to Post Service, which stores media in Blob store + CDN, writes metadata to the Posts DB sharded by user_id, and enqueues async fan-out workers that push a post ID into each follower's per-user Redis feed cache for normal users only; celebrity posts (red) are not fanned out and are stored once to be pulled at read time; the read path (green) reads a user's materialized feed O(1) from Redis and merges in a live pull of recent posts from the few celebrities the user follows, while images are served from the CDN edge with roughly 99 percent origin offload.
Instagram/News Feed hybrid architecture: write path (blue) from Client to Post Service, which stores media in Blob store + CDN, writes metadata to the Posts DB sharded by user_id, and enqueues async fan-out workers that push a post ID into each follower's per-user Redis feed cache for normal users only; celebrity posts (red) are not fanned out and are stored once to be pulled at read time; the read path (green) reads a user's materialized feed O(1) from Redis and merges in a live pull of recent posts from the few celebrities the user follows, while images are served from the CDN edge with roughly 99 percent origin offload.

5. Rubric — what a strong answer hits

5. Break it — three traced failures of the naive designs

1. The celebrity fan-out storm (why pure push is unshippable). A celebrity has 100M followers and taps "post." Under pure fan-out-on-write, that one post must append a post-ref to 100,000,000 feed lists. Trace the time on the fan-out tier: at a sustained 50,000 writes/s, 100,000,000 ÷ 50,000 = 2,000 seconds ≈ 33 minutes to fully propagate a single post. Two compounding harms: (a) followers see the post minutes apart — "stale fan-out"; (b) far worse, that job monopolizes the shared queue, so every normal user's post is stuck behind the celebrity's 100M writes. If three celebrities post around one live event, the queue backs up for everyone. This is the failure the hybrid exists to kill: exempt above-T accounts from fan-out entirely, store their post once, and merge it at read time → the 100M writes become zero.

2. Feed cache staleness & the un-fanned edit/delete. The materialized feed is a denormalized copy of post IDs. Now a user deletes a post, or you fan out a post and then its author gets blocked/unfollowed. The push-feeds already contain the stale ref. If getFeed blindly trusts the cached list, followers see a "ghost" post that 404s when the client fetches its media, or content that should be gone. Fix: the feed list stores only post IDs; on read, hydrate them through the Posts DB (or a posts cache) which is the source of truth for existence/visibility, and drop refs that no longer resolve. Lesson: the materialized feed is an index, not the data — never let denormalized copies be authoritative for existence.

3. Thundering herd on a viral post (hot key + cache stampede). A post goes viral; its media object and metadata are requested millions of times a second. Two distinct hot-key problems: (a) media — solved by the CDN, whose whole job is to absorb read concentration at the edge, so a viral image is served from thousands of PoPs, not your origin; (b) metadata — if that one post's metadata is cached and the entry expires, thousands of concurrent feed hydrations all miss simultaneously and stampede the Posts DB for the same row (a cache stampede / thundering herd). Fix: single-flight / request-coalescing on cache miss (one rebuild, others wait) plus a soft-TTL so the entry is refreshed before it hard-expires. Lesson: a pure read with no write contention still bites you on cache expiry, not on the read itself — see Cache Stampede & Invalidation.

6. Optimise — with trade-offs vs named alternatives

Every row is "buys X, costs Y, use when Z" against a specific named alternative — that's the judgment an interviewer is scoring.

DecisionOption AOption BPick when
Feed materializationFan-out-on-write (push)Fan-out-on-read (pull)Push when reads ≫ writes and follower counts are bounded (the common user) — pay ~500 cheap writes once so the many feed opens are O(1). Pull when the author has extreme followers (a post is 1 write vs 100M) or when a user posts far more than they're read. The point of the next row is you don't have to choose globally.
Who gets push vs pullHybrid: push below threshold T, pull-and-merge above TOne strategy for everyoneHybrid essentially always at social-network scale with a power-law follower distribution — it takes push's cheap reads for the 99.9% and pull's cheap writes for the celebrity 0.1%. One-strategy-for-all only when there are provably no celebrities (bounded followers) — then pure push is simpler and wins.
Feed rankingReverse-chronologicalML-ranked (engagement score)Chrono when simplicity/predictability matters and "newest" is the product promise — it's a pure merge by timestamp, no scoring service. Ranked when engagement is the goal (most consumer feeds): you add a scoring service and must decide when to score (see Defend) — buys engagement, costs a whole ranking subsystem + explainability/debuggability.
Feed storage modelPre-materialized per-user feed (Redis list)On-demand merge at read (compute every open)Pre-materialized when read latency and read volume dominate — feed open is one list read. On-demand merge only for the celebrity slice (you can't afford to materialize their 100M copies) or a low-traffic product where the memory cost of materializing isn't worth it. The hybrid uses BOTH: materialized for the push part, on-demand for the celebrity part.
Fan-out targetsPush to ALL followers eagerlyPush to active users only (lazy for dormant)Push-active-only when a large fraction of followers are dormant (never open the app) — materializing feeds they'll never read wastes writes and memory; compute their feed lazily if they return. Push-all only when nearly all followers are active or you need instant availability for everyone. Cuts fan-out writes proportionally to your dormant fraction.
Media deliveryServe images from origin/app serversBlob store + CDN edgeCDN essentially always for user media — the ~350 GB/s egress makes origin serving a non-starter, and images are immutable so edge caching is trivially correct. Origin-serving only for tiny/private/short-lived assets where edge caching buys nothing.

7. Defend under drilling

Q1 — "How do you handle a celebrity with 100M followers?"
I don't fan their posts out at all. Accounts above a follower threshold T are exempt from fan-out-on-write; their post is stored once in the Posts DB and nothing is pushed. On the read side, when any user opens their feed, I read their materialized push-feed (all the normal accounts they follow) O(1) from Redis and separately do a live pull of recent posts from the small set of celebrities they personally follow — typically a handful — then merge the two by rank/time. So the celebrity's 100M-write storm becomes zero writes, and the cost is a tiny bounded read-time merge per feed open. That's the whole reason the hybrid exists: it moves the celebrity's cost off the write path (where it's 100M) onto the read path (where it's ~a few extra lookups per reader).

Q2 — "Chronological vs ranked — how does ranking change the architecture?"
Chrono is just a merge by created_at — the materialized feed can literally be a time-sorted list and you're done. Ranking injects a scoring service and forces a "when do you score?" decision. Three options: (a) score at fan-out/write time and store a score with each feed entry — cheap reads, but the score is stale by read time and you re-score on signal changes; (b) score at read time over a candidate set — freshest but adds latency and compute to your hottest path; (c) the common production answer — a two-stage pipeline: fan-out builds a candidate feed (retrieval), then a lightweight ranker scores the top few hundred candidates at read time using features (recency, affinity, engagement, media type) from a feature store. So ranking doesn't replace push/pull — it sits on top: fan-out becomes candidate generation, and you add ranking + a feature store + model serving. Buys engagement; costs a whole subsystem plus the ability to explain "why am I seeing this."

Q3 — "Where does the threshold T come from, and what if an account crosses it?"
T comes from the write-amplification budget: it's the follower count above which fanning out a post costs more than the read-time merge would. Concretely, pick T so the fan-out tier's sustained write budget isn't dominated by a few big accounts — tens of thousands to ~a million followers is the usual band. Crossing it must trigger reclassification: a growing account promoted past T stops being fanned out (new posts pulled instead); if you forget to reclassify, that account keeps storming the queue as it grows. Demotion matters too — a shrunk account can rejoin push. The bug to avoid is a static flag set once at signup.

Q4 — "A user's feed is missing a post someone swears they made. Debug it."
Walk the two paths. If the author is a normal user, the post-ref should have been pushed — check the fan-out queue: is it backed up (a celebrity storm ahead of it, Break-it #1)? Did the worker fail after the DB write but before enqueue (so the post exists but was never fanned out)? Did the reader's feed list get trimmed past it? If the author is a celebrity, nothing was pushed by design — the post should appear via the read-time pull-merge; check that the reader's celebrity-follow list includes them and the merge ran. And check hydration (Break-it #2): the ref may be present but dropped because the Posts DB says it's deleted/hidden. The freshness SLA I stated in scoping (a few seconds' lag is OK) tells me whether a 10-second delay is a bug or expected.

Q5 — "Follow/unfollow at scale — when I follow someone, do I backfill their old posts into my feed?"
Generally no eager backfill of the full history — that's an unbounded write on a follow action and follows can be bursty. For a normal account you might push their most recent K posts into the new follower's feed (bounded), or simplest: let the next natural feed rebuild/merge pick them up. For a celebrity you follow, you get their recent posts for free via the read-time pull — no backfill needed at all, which is a nice side-benefit of the hybrid. Unfollow is lazy: don't scrub their refs out of the materialized feed synchronously; filter at hydration time and let trimming age them out. This keeps follow/unfollow cheap and bounded.

Q6 — the 100× escalation: "Now it's 50B DAU-equivalent / a global live event where 200M people open the feed in the same 10 seconds."
Two pressures: read spike and correlated fan-out. For the read spike, the feed-cache read path scales horizontally — shard the Redis feed cluster by user_id (consistent hashing so adding nodes doesn't reshuffle everything) and add replicas; the CDN already absorbs the media spike at edge. The danger is cache stampede on a few hot posts (Break-it #3) — single-flight + soft-TTL, and pre-warm the obvious hot content (the event's posts) into cache. For correlated fan-out (many big accounts posting about the event at once), the hybrid already keeps celebrities off the write path, and the async queue absorbs bursts by design — but I'd cap per-account fan-out concurrency so no single account (even sub-T) can monopolize workers, and prioritize the queue so small-account posts aren't stuck behind large ones. The architecture doesn't change shape at 100× — I add shards/replicas and rely on the async queue + CDN as shock absorbers; what I watch is tail latency (p99) on feed open and queue depth on fan-out.

8. You can now defend

Re-authored/Deepened for this guide. Model-answer reading: Designing Instagram, Designing Facebook Newsfeed, and News Feed: Fan-out-on-Write vs Fan-out-on-Read (System Design Problems). Concept cross-refs: What is CDN, Cache Read Strategies, Cache Stampede & Invalidation, and Consistent Hashing. Escalation ideas pair with the Designing Twitter's Timeline walkthrough and the Twitter timeline sizing drill.

🤖 Don't fully get this? Learn it with Claude

Stuck on Design Instagram / News Feed? Open Claude, copy a block below, and it'll teach you this exact concept — visually and interactively.

🎨 Explain it visually

Build the mental picture, not memorization.

I just read a lesson on **Design Instagram / News Feed** (Hands-On Builds) and want to truly understand it. Explain Design Instagram / News Feed from first principles using ONE vivid real-world analogy and a visual mental model — draw it as ASCII art or a clear step-by-step diagram — with a concrete example using real numbers. Then ask me one question to check I got the mental picture, and wait for my reply. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.
🤔 Walk me through it (interactive)

Socratic — adapts to where you're stuck.

Teach me **Design Instagram / News Feed** interactively. Ask me ONE guiding question at a time, wait for my answer, and adapt to my confusion — build the idea with me step by step instead of explaining it all at once. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.
🧪 Quiz me & fix my gaps

Active recall exposes what you missed.

Quiz me on **Design Instagram / News Feed** with 5 questions, easy to tricky, ONE at a time. Tell me if each answer is right; at the end, explain clearly what I got wrong and why. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.
🧠 Make it stick

Intuition + hook + flashcards for long-term memory.

Help me remember **Design Instagram / News Feed** for the long term: give the one-sentence intuition, a memorable hook/mnemonic, a tiny worked example, and 3 active-recall flashcards (Q -> A). If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

📝 My notes

← Design Typeahead / Search AutocompDesign a Web Crawler →