Knowledge Guide
HomeHands-On BuildsSD Design Labs

Design Dropbox / File Sync

Design Dropbox / File Sync

Here's the answer that gets nodded-along for thirty seconds and then falls apart: "store the file in S3, keep a row in a database pointing at it, and when the file changes, upload the new version." It sounds complete. Now watch it break the instant you attach real user behavior to it. A user has a 1 GB video-project file open in their editor; every few seconds the app autosaves, changing maybe 10 bytes of metadata deep inside it. Your naive design re-uploads the entire 1 GB on every autosave. Ten autosaves = 10 GB of upload traffic to move 100 bytes of actual change — that's ~99.99999% of the bytes on the wire being pure waste, on one file, from one user, in a couple of minutes. Multiply by millions of active users and you haven't built Dropbox, you've built a bandwidth incinerator. And that's before the second failure: two users (or the same user's phone and laptop) both have the same 4 GB OS installer or the same viral PDF, and you store every copy in full — petabytes of the identical bytes, paid for over and over. File sync looks like "put files somewhere"; the defining problem is moving and storing as few bytes as physically possible. This lab derives the design — chunking, content-addressed dedup, delta sync — that survives the autosave question.

Scope it like a senior (ask before you design)

Do not draw a single box yet. The design changes completely depending on the answers here — each question below is load-bearing:

Assume for this lab: 500M registered users, 100M daily-active, ~3 devices per active user; average 50 GB stored per user; a mix that includes large media (so chunking matters); in-place edits to large files are common (so delta sync matters); the same file can be edited on two devices, at least one possibly offline (so conflict resolution is in-scope); files can be shared with per-file ACLs; sync latency of a few seconds is acceptable (eventual consistency for file bytes, strong consistency for the metadata that tells a client which version is current).

Reason to the design (derive it, don't recite it)

Attempt 1 — store each file as one blob, re-upload on change. PUT /files/{id} writes the whole file to object storage; a metadata row records file_id → blob_url, version. This is the naive design from The Trap, and it has two independent fatal flaws. First, no delta: a 10-byte edit to a 1 GB file re-ships the full gigabyte, because the unit of change is the whole file. Second, no dedup: the identical 4 GB installer stored by a million users is a million full copies — you pay for 4 PB to hold 4 GB of distinct bytes. Both flaws share one root cause: the file is the unit. Fix the unit and both flaws dissolve.

Attempt 2 — split each file into chunks; make the chunk the unit. Cut every file into blocks of ~4 MB. Now an edit only needs to re-upload the chunks that actually changed, not the whole file — a 10-byte edit touches one 4 MB chunk, a 256× reduction on our 1 GB file (traced in Build It). This kills the delta problem. But how do we know which chunks changed, and how do we avoid storing two identical chunks twice? We need a name for each chunk that depends only on its contents.

Attempt 3 — name each chunk by the hash of its contents (content-addressable storage). Hash every chunk with SHA-256; the hash is the chunk's key in the block store. Two chunks with identical bytes — same user, different users, doesn't matter — produce the same hash and therefore map to the same key, so they're stored exactly once globally. That's dedup, for free, as a side effect of the naming scheme. And delta sync becomes trivial: to sync an edited file, the client re-chunks it, hashes each chunk, and asks the server "which of these hashes do you already have?" — then uploads only the hashes that come back unknown. The content-hash is doing double duty: it's the dedup key and the change-detector. This is the crux of the entire design.

Attempt 4 — but fixed-size chunk boundaries defeat dedup under insertions. Here's the subtle killer. If chunk boundaries are at fixed byte offsets (0–4MB, 4–8MB, ...), then inserting a single byte near the front of a file shifts every subsequent byte by one position. Every chunk boundary now falls on different content, so every chunk downstream of the insertion gets a new hash — and dedup/delta collapse: you re-upload almost the whole file for a one-byte insert. The fix is content-defined chunking (CDC): instead of cutting at fixed offsets, slide a small rolling hash (e.g. a Rabin fingerprint) over the bytes and place a boundary wherever the rolling hash hits a target pattern (say, low N bits all zero). Now boundaries are anchored to content, not position. Insert a byte and only the one chunk containing the insertion changes; the boundary-defining patterns downstream are the same bytes as before, so the same cut points reappear and every downstream chunk keeps its old hash. CDC is what makes delta sync survive insertions, not just in-place overwrites. (Trade-off vs fixed-size in Optimise.)

Attempt 5 — separate strongly-consistent metadata from eventually-consistent blocks, and sync by notification. We now have two very different kinds of data. The chunk bytes are large, immutable (a hash names exactly one byte-sequence forever), and can live in cheap, eventually-consistent object storage. The metadata — the ordered list of chunk-hashes that is a file at a given version, plus the version counter — is tiny but must be strongly consistent and transactional: a client must never see version 5's chunk-list while another client is mid-commit, or it'll reconstruct a corrupt file. So: a Metadata Service (small, transactional, sharded by user/file) owns file→chunk-list+version; a Block Store (object storage, keyed by chunk hash) owns the bytes. Sync is then: client commits new metadata → metadata version bumps → a Notification Service tells the user's other devices "file X is now v(n+1)" → each peer pulls the new chunk-list, diffs it against what it has, and fetches only the unknown chunks. Push notification (not polling) because 100M devices polling every few seconds would be a self-inflicted DDoS (see Optimise).

Build it — the design worksheet

This is the artifact you'd produce on the whiteboard in a 45-minute round. Work it top to bottom; every number below is traced from the scoping assumptions, not asserted — recompute them yourself before an interview.

1. Requirements

Functional: upload/download files; sync changes across a user's devices within seconds; edit the same file offline on multiple devices and reconcile on reconnect (with a defined conflict rule); share files with per-file ACLs; keep version history (at least enough to resolve conflicts). Non-functional: minimize bytes on the wire (delta sync) and bytes at rest (dedup); durability of stored bytes is paramount (losing a user's file is unacceptable — target 11 nines via erasure coding/replication); the metadata that names the current version must be strongly consistent; the block bytes may be eventually consistent; high read availability.

2. Back-of-envelope estimation (BOTE)

3. API sketch

// --- metadata / negotiation ---
POST /files/{fileId}/commit
     body: { chunks: [hash, ...ordered], baseVersion }
     -> 200 { version, missingChunks: [hash, ...] }   // which chunks server still needs
     -> 409 CONFLICT { serverVersion }                // baseVersion is stale (see Break It #1)

GET  /files/{fileId}/metadata
     -> 200 { version, chunks: [hash, ...ordered], size, mtime }

POST /chunks/has            // negotiate BEFORE uploading bytes
     body: { hashes: [hash, ...] }
     -> 200 { have: [hash, ...], need: [hash, ...] }  // dedup check

// --- block store (bytes) ---
PUT  /chunks/{hash}         // upload one chunk; server verifies SHA-256(body)==hash
     -> 201 | 200 (already had it, no-op)
GET  /chunks/{hash}         // download one chunk (peer reconstruction)

// --- sync ---
GET  /sync/subscribe        // long-poll / WebSocket; server pushes {fileId, version} on change

Note the shape: bytes and metadata are different endpoints on different stores, and /chunks/has lets the client dedup before sending a single byte over the wire — the negotiation is what makes delta sync cheap.

4. Data model & partitioning

5. High-level diagram — the sync round-trip

Device A edits a file: the watcher fires, the chunker re-cuts it (content-defined), each chunk is hashed, and A asks the Metadata Service which hashes are already known (a). A uploads only the unknown chunks to the Block Store (b) and commits the new ordered chunk-list + version to the Metadata Service (c). The version bump fans out through the Notification Service (d, e) to Device B, which pulls the new chunk-list (f) and fetches only the changed chunks from the Block Store (g) to reconstruct the file. Notice the byte-path (blue) and the metadata-path (amber) never merge into one store.

Dropbox file-sync mechanism: Device A's watcher detects a change, the chunker cuts the file into content-defined ~4MB chunks, each is hashed with SHA-256; A asks the Metadata Service which hashes it already has, uploads only the new chunks to the Block Store, and commits the new ordered chunk-list plus version to the Metadata Service. The version bump fans out through the Notification Service to Device B, which pulls the new chunk-list and fetches only the changed chunks from the Block Store to reconstruct the file.
Dropbox file-sync mechanism: Device A's watcher detects a change, the chunker cuts the file into content-defined ~4MB chunks, each is hashed with SHA-256; A asks the Metadata Service which hashes it already has, uploads only the new chunks to the Block Store, and commits the new ordered chunk-list plus version to the Metadata Service. The version bump fans out through the Notification Service to Device B, which pulls the new chunk-list and fetches only the changed chunks from the Block Store to reconstruct the file.

6. Rubric — what a strong answer hits

Break it — three concrete failures the naive design walks into

1. Concurrent edits from two devices — the lost update, traced. Device A and Device B both have file F at version 5. A edits it offline on a plane; B edits it on the couch. B reconnects first and commits → server is now at v6 (B's chunk-list). A's plane lands and A commits, still believing the base is v5. If the server blindly accepts, A's commit overwrites v6 — B's edit is silently gone, a classic lost update. The fix is optimistic concurrency on the metadata commit: the client sends baseVersion; the server does a compare-and-set (UPDATE ... WHERE current_version = baseVersion). A's commit arrives with baseVersion=5 but the server is at 6 → 409 CONFLICT. Now A knows it forked. The product decision (not a bug fix): keep both — the client materializes A's version as F (conflicted copy from A's-laptop 2026-07-11) and both files sync to everyone. This is exactly what real Dropbox does, and it's why the metadata store must be strongly consistent — a compare-and-set is meaningless on an eventually-consistent store where two nodes both think they're at v5. (Deeper: conflict-resolution strategies — LWW, vector clocks, logical clocks; for auto-merge instead of conflicted copies, CRDTs.)

2. A chunk upload fails mid-sync — the partial-file trap. A file has new chunks [b1, b2, b3]. The client uploads b1 and b2, then the network drops before b3. If the client had committed the new metadata first (chunk-list = [b1,b2,b3]), any peer that pulls that metadata will try to fetch b3, get a 404, and reconstruct a corrupt file — the metadata promises a chunk the block store doesn't have. The fix is ordering: upload all chunks to the block store first, then commit the metadata last, as the atomic "publish" step. Metadata at version n+1 is only ever written after every chunk it references is durably in the block store; a peer that sees v(n+1) is guaranteed every referenced chunk exists. If b3's upload fails, the commit never happens, the file stays at v(n) for everyone, and the client retries — and because chunk uploads are content-addressed and idempotent (PUT /chunks/{hash} of an already-present chunk is a no-op), retrying re-sends only b3, not b1/b2. Metadata-last is the transactional boundary that keeps the block store's eventual consistency from ever showing a torn file.

3. Hash collision — the dedup-corruption nightmare, and why it's a non-issue. Dedup's correctness rests on "same hash → same bytes." If two different chunks hashed to the same key, the second would be dropped as a "duplicate" and every file referencing it would silently get the wrong bytes — undetectable, catastrophic data corruption. So: is this a real risk? With SHA-256 (256-bit space) and our 2.5×10¹² chunks, the birthday-bound probability of any accidental collision is ~n²/(2×2²⁵⁶) ≈ (2.5×10¹²)² / (2×1.16×10⁷⁷) ≈ 2.7×10⁻₅₃ — astronomically smaller than the probability of a cosmic ray flipping the bit anyway. So you do not add a collision-check for accidental collisions — the cryptographic hash width is the guarantee, and that's why you pick SHA-256, not a truncated/fast hash. The one caveat worth stating: SHA-256 is chosen partly because it's collision-resistant against a malicious actor deliberately crafting two colliding chunks (a broken hash like MD5/SHA-1 can be attacked this way). A defense-in-depth option for the truly paranoid or security-sensitive: on a hash match, do a cheap length check, or in the rare high-value case a full byte-compare before dropping the "duplicate." But for the default design, the collision probability is the answer — you compute it, show it's negligible, and move on.

Optimise — with trade-offs

Bottlenecks worth naming out loud:

DecisionOption AOption BPick when
Chunk boundariesFixed-size (cut every 4 MB by offset)Content-defined (CDC, rolling hash)CDC almost always for a sync product — it's the only one that survives insertions (a 1-byte insert re-uploads 1 chunk, not the whole file). Fixed-size only when edits are guaranteed to be pure in-place overwrites or append-only (e.g. log files), where boundaries never shift — then fixed-size is simpler and cheaper to compute.
Where dedup happensClient-side (ask /chunks/has before upload)Server-side (upload everything, dedup at write)Client-side to save bandwidth (the scarce resource for a sync product) — the delta never leaves the device. Server-side only when you can't trust the client to hash correctly, or bandwidth is free but you still want storage dedup — it saves disk but not the upload. Most sync products do both: client-side to save the wire, server-side refcounting as the source of truth.
Sync unitFull-file transfer on any changeDelta sync (changed chunks only)Delta sync for large files with small edits (the whole point). Full-file only for tiny files where chunking overhead exceeds the savings (see small-file optimization) or files that are always fully rewritten anyway (delta buys nothing).
Cross-device consistencyStrong (a commit blocks until all devices ack)Eventual (commit returns immediately; peers catch up via notify)Eventual for the file bytes — nobody expects a laptop edit to appear on a powered-off phone instantly, and blocking on offline devices would make every save hang. Strong only for the metadata version counter (so conflict detection's compare-and-set is correct). This split — strong metadata, eventual blocks — is the core consistency decision.
Conflict resolutionLast-writer-wins (drop the loser)Keep both as conflicted copies (or CRDT auto-merge)Conflicted-copies for opaque binary files (docs, images) where you can't safely merge and silently dropping a user's edit is unacceptable. LWW only when losing an edit is genuinely fine (ephemeral caches). CRDT auto-merge when the file format is structured and mergeable (collaborative text) and you're willing to build format-aware merge logic.
Notification deliveryClient pollingServer push (long-poll / WebSocket)Push at this device scale — polling 100M devices generates ~20M req/s of near-empty traffic vs ~116K real deliveries/s. Polling only acceptable for tiny device counts or when a minutes-stale sync is fine and you want dead-simple stateless servers.

Defend under drilling

Q1 — "Why chunk at all? Why not just diff the two file versions and send the byte-diff?"
Byte-diffing (like rsync's rolling-checksum algorithm) works for a single sender→receiver pair, but it doesn't give you the two things chunking does: global dedup (a byte-diff between my v5 and v6 tells you nothing about the identical chunk sitting in another user's file) and a content-addressed store where the same bytes are stored once across the entire system. Chunk-hashes make dedup a property of the storage layer, not a per-transfer computation. rsync-style delta is actually complementary — you can use a rolling hash inside the chunker (that's what CDC is), but the chunk-hash is what makes storage and cross-user dedup work.

Q2 — "How big should a chunk be? Defend 4 MB."
It's a direct trade-off. Smaller chunks → finer-grained delta (a small edit re-uploads less) and better dedup (more chance two files share a chunk), but more chunks means more metadata: halving chunk size doubles the chunk-index (our 160 TB → 320 TB) and doubles the per-file chunk-list and the number of hash negotiations. Larger chunks → less metadata and fewer round-trips, but a tiny edit now re-uploads a bigger block and dedup gets coarser. 4 MB sits where the metadata stays manageable (2.5T chunks, ~400 TB total) while a typical edit still only re-ships a few MB. If the workload were tiny files with tiny edits, I'd argue it down toward 1 MB; if it were huge media rarely edited in place, I'd argue it up. State the workload, then pick.

Q3 — "How do you resolve conflicting edits from two offline devices?"
Metadata commits carry a baseVersion and the server does compare-and-set on the file's current_version. Whichever device reconnects and commits first wins the version bump; the second device's commit arrives with a stale baseVersion and gets a 409. At that point it's a product decision, not a data-loss event: for opaque files (the default), the loser is materialized as a "conflicted copy" — a new file that syncs to everyone, so no edit is ever silently dropped. For a structured, mergeable format you could instead run a CRDT/three-way merge and auto-reconcile. The thing I will not do is last-writer-wins that silently discards an edit — that's the one outcome users never forgive. This is precisely why metadata is strongly consistent: compare-and-set is only correct if there's a single authoritative version counter.

Q4 — "How do you sync a 50 GB file efficiently — and what about a poor connection that keeps dropping?"
50 GB ÷ 4 MB = ~12,800 chunks. First sync is unavoidably ~50 GB of bytes, but every subsequent edit is delta: change 100 MB in the middle → ~25 chunks → ~100 MB on the wire, not 50 GB. For the flaky connection, chunking is the resumability mechanism: each chunk is an independent idempotent PUT /chunks/{hash}, so a dropped connection resumes at chunk boundaries — you re-send only the in-flight chunk, never restart the 50 GB. The client tracks which hashes are confirmed-uploaded and only commits metadata once all 12,800 are durable (Break It #2). Contrast the naive full-file design: a drop at 49 GB means restarting from zero.

Q5 — "What happens to dedup and sharing when a chunk is shared across users — and how do you ever delete anything?"
A chunk in the block store is keyed by content hash and may be referenced by many files across many users, so you can't delete it when one user deletes their file. Each chunk-index entry carries a reference count (or a reference set); deleting a file version decrements the refcounts of its chunks, and a chunk is only garbage-collected when its count hits zero. Sharing is orthogonal to dedup: dedup is "these bytes exist once"; access is "user U is allowed to read file F" — enforced at the metadata/ACL layer, never by chunk possession. Critically, a user must not be able to probe GET /chunks/{hash} for a hash they guessed to read someone else's data; chunk reads are authorized through file ACLs, not open by hash. GC on a 2.5-trillion-chunk store is itself a hard background job (mark-and-sweep with refcounts, done incrementally).

Q6 — the 100× escalation: "Storage per user grows 100×, to 5 TB average. What breaks first?"
Logical storage goes 25 EB → 2.5 ZB, physical (at 2.5× dedup) → ~1 ZB, and unique chunks 2.5T → 250 trillion. The first thing to break is not the block store (object storage scales horizontally — it's just more machines and cost). It's the chunk index: 250T entries × 64 B = 16 PB of index that every single upload must query ("do you have this hash?") on the hot path. That index becomes the bottleneck — you'd shard it far more aggressively (still by hash, consistent-hashing), push a bloom-filter in front of each shard so a "definitely-new" chunk skips the lookup, and cache hot hashes at the client. The dedup ratio also matters more at this scale: a small improvement from 2.5× to 3× saves ~160 EB of physical storage — so you'd invest in better CDC tuning and cross-user dedup. The design's shape holds (chunk+hash+delta+notify); what changes is that the metadata/index tier, not the byte tier, is where the engineering goes.

You can now defend


Re-authored/Deepened for this guide. Model-answer reading: Designing Dropbox — Chunking, Dedup & Delta Sync, Traced (System Design Problems); crux deepened in Data Compression vs Data Deduplication. Chunk sharding via Consistent Hashing; durability/object-store layer in Distributed File System — Durability Arithmetic, Erasure Coding and Cloud Storage & Data Movement. Conflict handling cross-referenced with Conflict Resolution Strategies and CRDTs.

🤖 Don't fully get this? Learn it with Claude

Stuck on Design Dropbox / File Sync? Open Claude, copy a block below, and it'll teach you this exact concept — visually and interactively.

🎨 Explain it visually

Build the mental picture, not memorization.

I just read a lesson on **Design Dropbox / File Sync** (Hands-On Builds) and want to truly understand it. Explain Design Dropbox / File Sync from first principles using ONE vivid real-world analogy and a visual mental model — draw it as ASCII art or a clear step-by-step diagram — with a concrete example using real numbers. Then ask me one question to check I got the mental picture, and wait for my reply. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.
🤔 Walk me through it (interactive)

Socratic — adapts to where you're stuck.

Teach me **Design Dropbox / File Sync** interactively. Ask me ONE guiding question at a time, wait for my answer, and adapt to my confusion — build the idea with me step by step instead of explaining it all at once. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.
🧪 Quiz me & fix my gaps

Active recall exposes what you missed.

Quiz me on **Design Dropbox / File Sync** with 5 questions, easy to tricky, ONE at a time. Tell me if each answer is right; at the end, explain clearly what I got wrong and why. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.
🧠 Make it stick

Intuition + hook + flashcards for long-term memory.

Help me remember **Design Dropbox / File Sync** for the long term: give the one-sentence intuition, a memorable hook/mnemonic, a tiny worked example, and 3 active-recall flashcards (Q -> A). If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

📝 My notes