Designing Pastebin

Standalone note — this page is sections 9–12 only. Pastebin’s full design (requirements, API, high-level architecture, capacity, DB schema, key generation, read/write path) lives with the classic problem write-up and intentionally reuses much of the URL shortener skeleton. Treat this page as the delta: what changes when the service stores multi-MB bodies instead of only redirect metadata.

• Depends on: Designing a URL Shortening Service (key gen, tiny metadata rows, non-guessable IDs, basic purge ideas).
• Expect from sections 1–8 elsewhere: create/read paste API; short code → metadata; capacity (writes, storage for large bodies); choose object store for content + SQL/NoSQL for metadata; rate limits.
• This page owns: content-addressed dedup + refcount purge (§9), dual partitioning/replication (§10), CDN vs app cache (§11), content-hosting security (§12).

9. Purging or DB Cleanup

The original chapter's answer here was the URL shortener's lazy-deletion-vs-active-scan story: check expiry lazily on read, and also run a periodic cleanup job over expired rows. That much still applies unchanged to Pastebin's metadata table. What does not carry over is what "deleting a paste" means, because a metadata row here is not the whole record — it is a pointer to a separately stored, potentially multi-megabyte content blob. Deleting the row is not the same as deleting the data.

If two different pastes happen to contain byte-identical text (a shared license header, a stack trace ten different users hit the same bug and pasted, a snippet copied from a tutorial), storing that text twice wastes space. Content-addressable storage — keying the object store by a hash of its content instead of an arbitrary ID — lets Pastebin store it once and have every matching paste's metadata row point at the same blob. That single change is exactly what breaks the URL shortener's simple "delete the row, delete the data" purge story: a blob can now be referenced by more than one row, so purging one row must not delete a blob other live pastes still depend on. Reference counting is the fix — track how many metadata rows point at each content hash, and garbage-collect the blob only once that count reaches zero.

Concurrency hazard: the check-then-act race

The trace below assumes one paste at a time. In production, two clients can paste identical bytes simultaneously, and both app servers may reach step 3, see a miss, and upload the same blob twice. Worse, one server may increment a reference count while another is garbage-collecting the blob because its last metadata row was just deleted. The fix is to make the content-index update atomic: use a conditional write so only the first writer creates the blob (e.g., S3 If-None-Match or a transactional upsert in the metadata DB), and make reference-count increments/decrements atomic transactions. Without that, deduplication leaks storage and garbage collection can delete a blob that another paste still references. Atomic counters alone are not enough to close the GC side of this race, though: a collector that read count=0 can still physically delete the blob just after a concurrent writer's dedup hit incremented it 0→1. The collector must either tombstone the blob and wait out a grace window before physical deletion, or CAS it into a "deleting" state that a concurrent dedup lookup treats as a miss (forcing a fresh upload) — the delete decision and the delete marker have to land in the same atomic step.

Traced: one paste's life from write to purge

1. Client submits paste text (up to the 10MB limit).
2. The app server hashes the body (e.g. SHA-256) to get a content key.
3. It checks the content index for that hash. Hit: increment the existing blob's reference count and skip the upload entirely. Miss: upload the blob to the object store keyed by that hash and initialize its reference count to 1.
4. Either way, insert a new metadata row {urlHash, contentHash, expire_date, owner} — this row is what's unique per paste; the blob underneath it may not be.
5. Time passes. The paste's expire_date arrives, or the user deletes it early.
6. The purge job (or a lazy check on read) removes that metadata row.
7. Removing the row decrements the reference count on its contentHash.
8. Only when that count hits zero does a separate garbage-collection pass delete the actual blob from the object store — until then, some other live paste's metadata row is still pointing at it.

Crash window between steps 3 and 4: if the app server dies after uploading the blob (reference count 1) but before inserting the metadata row, the blob is orphaned with a nonzero count that no row will ever decrement. A background reconciler that periodically compares reference counts against live metadata rows — or creating blobs in a pending state with a TTL that only the metadata commit clears — reclaims these mid-write orphans.

When this is not worth doing: hashing every incoming paste costs CPU on the write path, and the content-index lookup is an extra hop before the upload can even start. If most pastes in the traffic mix are unique human-typed snippets — the common case for a general-purpose Pastebin, as opposed to, say, a CI system re-pasting the same build log a thousand times — the dedup hit rate can be too low to pay back the added write latency and index-maintenance cost in storage savings. Reference-counted dedup is a design to reach for once you have evidence of a duplicate-heavy workload (shared snippets, bot-generated repeats, popular gists), not a default to bolt onto every write path.

10. Data Partitioning and Replication

The URL shortener partitions one small table: a short code and its destination URL both live in a single tiny row, so "shard key" and "storage system" are the same decision — hash the short code, route to a shard. Pastebin split its storage into two systems (Section 7): a metadata database and an object store. That split means partitioning is now two separate decisions, not one, and they don't share an answer.

Metadata DB: shard by urlHash

This half doesn't change from the URL shortener's reasoning. Rows are small (<1KB), keyed by urlHash, and a uniform hash of a short code spreads keys evenly across shards with no natural hot spot — the same consistent-hashing argument applies unchanged.

Object store: shard by contentHash, not urlHash

This half is new, and it follows directly from the dedup design in Section 9. If the object store were sharded by urlHash, a single content blob shared by pastes with unrelated urlHashes could need to live on multiple shards at once, defeating the point of storing it once — or, worse, dedup gets silently abandoned and the same bytes are stored per shard. Sharding the object store by contentHash instead keeps every distinct blob on exactly one shard regardless of how many urlHashes point at it, with a useful side effect: a cryptographic hash is already uniformly distributed, so contentHash-based sharding gives even load distribution for free, and the shard key doubles as an integrity check (recompute the hash on read and compare).

Replication also diverges

The URL shortener replicates its one small table with straightforward 3x full-copy replication (a leader plus two followers) — cheap, because each row is a few hundred bytes. Applying that same 3x-full-copy scheme to multi-megabyte paste blobs would triple the object storage bill for every paste. Object stores at this scale typically use erasure coding instead: split each blob into k data fragments plus m parity fragments spread across independent failure domains, so any k of the (k+m) fragments reconstruct the original. This gives comparable durability to 3x replication at a fraction of the storage overhead for large objects — e.g., Reed–Solomon (10,4) stores 1.4 bytes per byte versus 3.0 for triplication, about 53% less, while tolerating the loss of any 4 fragments — and it is not a trade worth making for the metadata row, where erasure coding's fixed per-object overhead would dominate a payload already smaller than the coding metadata itself.

11. Cache and Load Balancer

The URL shortener's caching story is a single decision: keep the hot {urlHash → longURL} mappings in an LRU app-level cache (e.g. Memcached), because the whole working set of popular short links is a set of tiny strings that comfortably fits in RAM, and a cache hit fully answers the request. Pastebin's read path returns two different kinds of object, and that split forces two caching strategies layered on top of each other rather than one shared cache.

Metadata: same small-row app cache as the URL shortener

The {urlHash → contentHash, expire_date, owner} row is still small and still benefits from exactly the same LRU app-cache treatment, for exactly the same reason — no delta here.

Content: CDN, not app cache

The paste body is a different shape of problem: it can be several megabytes, and unlike a URL redirect target (read once by the browser to issue a redirect and then discarded) a popular paste's body may be fetched repeatedly by many readers. Two properties make it a good fit for a CDN specifically, rather than a bigger app-level cache: it is immutable once written (a paste is never edited, only expires — nothing to invalidate mid-life), and readers are geographically distributed, so the value is in serving bytes from a point of presence near the reader rather than from a single app-tier cache near the database. Put content behind a CDN, addressed by the paste's URL or contentHash, with the object store as the CDN's origin on a cold-edge miss. Holding multi-megabyte blobs in the same process-local cache used for tiny metadata rows would either blow the cache's memory budget or evict the hot metadata rows that actually need to stay in RAM — which is why this is a split, not just a bigger version of the URL shortener's single cache.

Load balancer: unchanged

Application servers remain stateless in both designs, so the load balancer itself — round-robin or least-connections across a fleet of app servers — carries no Pastebin-specific delta. The interesting difference here lives entirely on the caching axis, not the load-balancing axis.

12. Security and Permissions

For the URL shortener, the only user-supplied input is a destination URL that the service never stores meaningfully beyond a string to redirect to, so its security section is short: make short codes non-guessable/non-sequential so one user can't enumerate another's links, and rate-limit the API key. Pastebin inherits that same non-guessable-key requirement, but storing the payload itself — not just a pointer to someone else's page — changes what "security" has to cover.

The non-guessable key now doubles as access control

For unlisted pastes, knowledge of the urlHash is often the only check standing between a reader and the paste's contents — a capability-URL pattern. That's a heavier burden than it was for the shortener: guessing a short code just reveals a redirect target the attacker could likely find another way; guessing a paste key can directly expose whatever was pasted, and people habitually paste exactly the kind of content that shouldn't leak — API keys, credentials, internal stack traces, config files. The same "non-guessable" requirement from the shortener is doing strictly more security work here.

Content scanning — a concern that doesn't exist for a URL shortener at all

A URL shortener never looks at the destination page's contents, so it has no content-moderation surface. Pastebin hosts the content directly, which makes it directly responsible for what's stored: scanning uploads for malware signatures, for patterns that look like leaked secrets (high-entropy strings, known credential formats), and for abuse content (spam, phishing kits, doxxing) is a first-class part of the write path here, not an afterthought.

Hosting abuse, not just redirect abuse

A malicious short URL is an abuse-of-redirection problem — it points somewhere bad. A malicious paste is an abuse-of-hosting problem — the bad content lives on the service's own infrastructure and can itself be linked to from elsewhere as though the service were the origin (phishing instructions, malware droppers fetched by URL, spam pages). Rate-limiting API keys, the shortener's main defense, doesn't touch this; it requires the content-scanning story above plus takedown and reporting tooling the shortener never needed.

Read-side access control, not just write-side key generation

Because the payload can be sensitive, pastes commonly need visibility tiers a URL shortener has no equivalent for: public, unlisted (guessable-but-unlisted, per above), and private/password-protected. Password-protected pastes add a credential check to the read path — something the shortener's redirect handler never needed, since a public URL was never treated as something to protect in the first place. Encryption at rest for the object store and TLS in transit matter proportionally more here too, because it is the blob itself that may hold the sensitive material, not just a database column pointing at someone else's page.

Source

Base structure and requirements from the "Designing Pastebin" chapter (this page), whose Sections 9–12 originally deferred entirely to Designing a URL Shortening Service like TinyURL for Purging, Data Partitioning and Replication, Cache and Load Balancer, and Security and Permissions. This revision keeps that page as background for the mechanics that genuinely carry over unchanged (lazy-vs-active TTL sweeps, urlHash-based metadata sharding, 3x replication of small rows, non-guessable keys, stateless load balancing) and replaces the four stub cross-references with reasoning specific to Pastebin: content-addressable storage with reference-counted purging (Section 9), a second content-hash-based partitioning scheme for the object store with erasure-coded replication (Section 10), a CDN layer for large immutable blobs on top of the same small-row app cache (Section 11), and content-scanning plus read-side access control driven by the fact that Pastebin stores the sensitive payload itself rather than a pointer to it (Section 12).

🪜 Drill ladder: Designing Pastebin

1. Content expiration: deleting a metadata row does not delete the blob — walk the refcount decrement and the garbage-collect-at-zero step, and say why lazy TTL check on read is not enough on its own.
2. Dedup race: two clients paste identical bytes at once — what breaks without an atomic conditional create (S3 If-None-Match / transactional upsert), and how can a refcount race delete a blob a live paste still points at?
3. Partitioning: why shard the metadata DB by urlHash but the object store by contentHash? What goes wrong if you shard blobs by urlHash instead?
4. Replication cost: why erasure-code multi-MB blobs (k data + m parity) instead of 3× full copies, yet keep 3× copies for the tiny metadata row?
5. CDN vs app cache: why does the immutable multi-MB body belong behind a CDN while the sub-KB metadata row stays in the LRU app cache — what breaks if you put the blob in the same LRU?
6. Capability-URL security: why is a guessable paste key a strictly bigger risk than a guessable short code, and when is content-hash dedup NOT worth its write-path CPU + index cost?