System Design Problems — The Recurring Resolution Mechanisms (Reconciliation, Leases, Hot-SKU FCFS, DST & Celebrity Fan-out) (Deep Dive)

The System Design Problems slice names where a payment call, an upload, a booking, a flash sale, a short link, a feed, or a calendar reminder can go wrong — but most of those pages stop at naming the failure point and defer the actual fix ("handle idempotency," "prevent double-booking," "resolve DST") to the reader's imagination. This page is the missing layer: seven resolution mechanisms that keep reappearing across unrelated problems, each traced with real values so the mechanism, not just its name, is transferable to a problem you haven't seen before. It assumes the building blocks covered elsewhere in this guide — idempotency keys, fencing tokens, fan-out-on-write, CRDTs — and shows exactly where each one gets reached for and why nothing weaker survives contact with the failure mode.

Mechanism: a payment call has three possible outcomes — success, failure, and the one that actually causes incidents: no response at all. A network blip, a load balancer timeout, or a provider-side stall means your service genuinely does not know whether the card was charged. Retrying blindly risks a double charge; giving up risks losing a paid order. The fix is two mechanisms working together, not one: an idempotency key makes it safe to retry (the provider deduplicates on the key and returns the original result instead of charging twice), and a reconciliation job resolves the case a retry can't reach — total silence, where you have nothing to retry against yet because you don't know if the first attempt is still in flight or already decided.

Traced, with real states:

The key insight interviewers probe for: idempotency keys and reconciliation solve different halves of the same problem. The key makes an action safe to repeat once you've decided to repeat it; it does nothing to tell you whether you should, because the caller that timed out has no signal at all. Reconciliation is what converts "I don't know" into a decision, by asking the party that actually knows — the provider's ledger, not your own logs. Skipping reconciliation and just leaving records PENDING forever is the failure mode that shows up as silently vanished orders in production.

Mechanism: a Google-News-style ranking feed, a view counter, or any "recent stats" surface commonly runs two layers at once: a fast, approximate real-time layer (stream aggregation over the last few minutes) and a slower, exact batch layer (a nightly or hourly job that recomputes the true totals from the full event log). At read time the two layers are merged into one number the user sees. The naive merge — real_time_count + batch_count — double-counts every event that both layers have already seen, because the batch job's cutoff and the stream's cutoff don't line up to the millisecond.

The merge algorithm: the batch layer must expose the exact timestamp (or offset) through which it is complete — call it the batch watermark, W. The stream layer then only contributes events with timestamp > W; anything at or before W is assumed already folded into the batch total and is dropped from the real-time side of the merge. So the read-time formula is not addition of two whole counts, it's batch_total(≤W) + stream_total(>W) — the watermark is the seam, and the merge is only correct if both layers agree on what it means for an event to be "before" it.

The watermark rule that makes this hold under out-of-order arrival: a watermark of value W is a stream engine's claim that no more events with event-time ≤ W will arrive, generally derived from max(event-time seen) − allowed lateness. Set the lateness bound too tight and genuinely late events (a mobile client that was offline) get silently dropped from the stream side and never re-merged — a permanent undercount at the boundary. Set it too loose and the real-time number lags further behind wall-clock time than the product can tolerate. This exact trade-off, plus the late-data-arrival failure modes, is covered in depth in Event-Time, Watermarks & Late Data elsewhere in this guide; the point specific to this page is that the watermark is what makes read-time merge safe — without it, a lambda-style split into "fast approximate" and "slow exact" layers has no principled way to combine the two without either gaps or double-counts at the seam.

Mechanism: chunked upload (Dropbox-style file sync) and segment ownership (a reminder/alert worker claiming a shard of due reminders) share the same shape — exactly one worker should be writing a given segment at a time, but the worker can pause (GC, a slow disk, a network partition) for an unbounded, unknowable length of time. A distributed lock by itself only proves who currently holds the lease according to the lock service; it says nothing about whether a paused holder is about to wake up and write with stale authority. The fix is a lease (a time-bounded grant: "you own segment S until t+30s, renew or lose it") paired with a fencing token — a monotonically increasing number issued with every lease grant that the segment's storage layer itself checks, rejecting any write whose token is lower than the highest one it has already accepted.

Concretely for chunked upload: worker A leases segment 4 of a file, gets token 71, then stalls past its lease TTL. The lease expires; worker B leases segment 4, gets token 72, and writes its chunk, which the storage layer records as "segment 4, highest token = 72." Worker A wakes up unaware anything happened and tries to write segment 4 presenting token 71 — the storage layer compares 71 against the 72 it has already seen and rejects the write. Without the token check, A's stale write would silently corrupt or overwrite B's already-durable chunk. This is the identical mechanism this guide's Split-Brain, Fencing & Safe Failover page uses for leader promotion, and the same one CAP in Practice traces for a generic lock-plus-resource pair — the lesson that generalizes here is that the token must be enforced at the last hop (the storage write), never only at the lock service, because the lock service has no way to reach into a paused worker and stop it from acting on outdated information.

Mechanism (seat/room booking): booking a seat looks like it needs a distributed transaction — check availability, then reserve, atomically, possibly across shards if seats for one venue are spread across partitions. In practice you can usually avoid the cross-shard transaction entirely by (a) partitioning so that all seats for one resource (one showtime, one room) live on a single shard, keyed by resource id, and (b) reserving with a single-row conditional update — a compare-and-set on that row's status column, e.g. UPDATE seats SET status='HELD', holder=:user WHERE seat_id=:id AND status='FREE'. The database's own row-level atomicity is the only coordination primitive needed: if two requests race for the same seat, exactly one UPDATE matches the WHERE status='FREE' predicate and affects a row; the loser's statement affects zero rows and the caller treats that as "already taken," no lock service, no 2PC, no cross-shard commit protocol required. This is the same pattern the guide's Designing Ticketmaster — Seat Reservation Traced page walks end to end, and it's worth naming explicitly here because it's the general answer to "how do I book X without a distributed transaction": shard so contention is single-row, then let the storage engine's native atomic compare-and-set do the work a transaction manager would otherwise need to coordinate.

Mechanism (reaction / like counter): a "likes" counter looks like a simple increment, but a naive counter += 1 on every click is wrong the moment a user can toggle their reaction (like → unlike → like) or the client retries a request. The fix is a read-modify-write guard keyed by the user's own reaction row, not a blind increment on the aggregate: store one row per (user_id, item_id) recording the user's current reaction state, and derive the counter delta from the actual state transition — no-row → liked is +1; liked → liked (a retried request) is +0; liked → unliked is −1. The per-user row is updated with an atomic compare-and-set on its own previous state (so two racing requests from the same user can't both apply +1), and only a successful, real transition emits a delta to the aggregate counter, which itself is incremented atomically. This is the same shape as the seat CAS — push the contention down to one narrow, single-row atomic operation, and let a derived value (the delta) come from a verified state change rather than from trusting the request to be idempotent on its own.

Mechanism: a flash sale's hottest SKU is a single row of inventory that thousands of requests contend for in the same second. A single-row compare-and-set (as in booking, above) is correct but throughput-bounded — every request pays a full lock/CAS round-trip against the same row, and CPU-bound serialization on one row caps you far below the request rate a flash sale actually sees. Two different mechanisms trade off differently here, and the difference matters for whether FCFS survives:

• Single ordered queue per SKU, batched decrement. Route all requests for a hot SKU into one strictly-ordered queue (a single partition in Kafka or SQS FIFO, keyed by SKU), so arrival order is fixed the instant a request is appended — nothing can jump ahead of an earlier request once it's in the log. A single consumer drains the queue in small batches on a fast tick (e.g. every 20–50ms) and applies one atomic decrement per batch, marking requests WON in strict arrival order up to whatever inventory remains, and SOLD OUT beyond it. This exactly preserves global FCFS — the traced example below shows why — because ordering happens once, at ingestion, and the batching only parallelizes the decrement's throughput, not the sequence.
• Pre-allocate into sub-buckets (sharded counters). Split the SKU's inventory into K sub-buckets up front (1,000 units → 10 buckets of 100) and hash incoming requests across buckets so each bucket's CAS runs independently and in parallel — much higher raw throughput than one row. The catch: this only approximates FCFS. A request that lands in a lightly-loaded bucket can win against an objectively earlier request that happened to land in an already-exhausted bucket — the global arrival order was never actually recorded anywhere, only per-bucket order was. Rebalancing spare units from under-drained buckets into exhausted ones near depletion narrows the gap but never fully closes it. Sub-bucketing is the right call when "approximately fair, much higher throughput" is an acceptable trade for the business; it is the wrong call when a support ticket saying "I clicked first and lost" must be provably false.

Traced with real numbers, the ordered-queue approach:

Inventory starts at 5. Tick 1 drains {r1,r2,r3} in the order they arrived and one atomic decrement takes inventory from 5 to 2, all three WON. Tick 2 drains {r4,r5,r6}: r4 and r5 consume the remaining 2 units and WIN; r6, though it arrived before any hypothetical later request, exhausts the row and is SOLD OUT — but critically, r6 lost to r4 and r5 only because they arrived earlier in the same queue, never because of which bucket or shard it happened to land in. Tick 3's r7 finds inventory at 0. No request anywhere in this trace could have jumped the queue; the only lever the system has is the batch cadence, and shrinking that cadence (or reacting eagerly instead of on a fixed tick) is how you buy back the latency the batching costs.

Mechanism: the redirect status code a URL shortener returns for GET /abc123 is not a cosmetic choice — it decides whether your own server sees the click at all. A 301 (Moved Permanently) tells the browser this mapping will never change; compliant browsers and intermediate caches (CDNs, corporate proxies) cache the redirect and, on every subsequent click, resolve straight to the long URL without ever hitting your server again. That's minimum latency and minimum server load for the shortener — and it means your analytics pipeline sees only the first click per client, because every repeat click is served from cache. A 302 (Found / Temporary) tells the browser the mapping might change, so it must not cache it — every single click, first or hundredth, round-trips to your server, giving you a complete per-click event stream (referrer, timestamp, geography, device) at the cost of the server having to handle every click's traffic.

There's no universally correct answer — it's a direct trade of latency/cost against analytics completeness, and the interview signal is naming that trade-off explicitly rather than picking a code by default. A shortener whose product is "make links load fast" (e.g. an internal tool with no monetization on click data) should default to 301. A shortener whose product is the click analytics (marketing campaign links, affiliate tracking, ad redirects) must use 302, or a 301 with a short-lived cache-control override, because losing repeat-click visibility defeats the product's purpose. Because the single hottest short link in a system is itself a hot key under either scheme, this page's own cache-stampede mechanism (§7) and the guide's dedicated Cache Stampede: single-flight, soft TTL & dedup page both apply directly to protecting that link's lookup path regardless of which redirect code you choose.

Mechanism (fan-out threshold): a Twitter/Instagram-style timeline can push a new post into every follower's precomputed feed at write time (fan-out-on-write) — cheap to read, but the write cost is O(followers) per post, which is fine at a few hundred followers and catastrophic at fifty million. The alternative, fan-out-on-read, does nothing at write time and instead merges a user's followed accounts' posts at read time — flat write cost regardless of follower count, but every feed load now does real work. The standard resolution is a hybrid keyed by a follower-count threshold: fan-out-on-write for the overwhelming majority of accounts under the threshold (say, 10k–100k followers depending on write capacity), and fan-out-on-read for accounts above it — a celebrity's post is not pushed to millions of precomputed feeds; instead, any follower's feed read merges in the celebrity's recent posts live, and normal accounts' posts get merged in from their own precomputed feed segment. The cost reasoning is explicit and worth stating in an interview: fan-out-on-write cost is paid once per post times follower count, so it scales with the celebrity's audience; fan-out-on-read cost is paid once per feed load times however many celebrities that reader follows, which is bounded and small for almost everyone. This exact threshold and hybrid is traced end to end, with real QPS numbers, in this guide's Designing Twitter's Timeline — Fan-out on Write, Traced and News Feed: Fan-out-on-Write vs Fan-out-on-Read pages.

Thundering herd on the celebrity read path: pushing celebrities onto fan-out-on-read concentrates enormous read traffic onto one hot cache key — that celebrity's recent-posts cache entry. If that single key expires (TTL eviction) or the cache node restarts, thousands of concurrent feed reads can miss simultaneously and all hit the database at once to recompute the same value — a classic cache stampede. The standard guard is single-flight / request coalescing: the first miss acquires a short-lived rebuild lock and populates the cache; every other concurrent request for the same key waits on that in-flight rebuild instead of independently querying the database, so N simultaneous misses cost one database read, not N. Pairing this with a soft TTL (serve the slightly-stale cached value while one request rebuilds in the background) avoids a latency spike entirely for everyone but the unlucky first request. This is worked in full, including the soft-TTL variant, in Cache Stampede: single-flight, soft TTL & dedup and Cache Stampede & Invalidation — Thundering Herd and the Hard Half in this guide.

DST policy (reminder/calendar systems): a recurring reminder or calendar event stored as a local wall-clock time ("9:00 AM every weekday, America/New_York") hits two edge cases twice a year that have no universally correct resolution — only an explicit, documented one. In spring-forward, clocks jump from 1:59:59 AM directly to 3:00:00 AM; a reminder scheduled for 2:30 AM does not exist that day. In fall-back, 1:00:00–1:59:59 AM occurs twice; a reminder scheduled for 1:30 AM is ambiguous between two distinct real instants an hour apart. The only defensible engineering answer is to pick and document one policy per case rather than let the underlying date library's default behavior decide silently: for the non-existent spring-forward gap, either shift the reminder forward to the first valid instant after the gap (2:30 AM → 3:00 AM) or skip that single occurrence — shifting forward is the more common product choice, since a missed reminder is usually worse than a slightly-late one. For the doubled fall-back hour, deterministically pick the first occurrence (the pre-transition, still-daylight-saving instant) unless the product has a specific reason to prefer the second; consistently picking one and storing the resolved UTC instant (recomputed at each DST transition boundary, since the wall-clock-to-UTC offset for future occurrences changes with the transition date) avoids the alternative failure mode of firing a recurring reminder twice or not at all around every DST boundary.

• Reconciliation without a terminal timeout on PENDING. If the reconciliation job itself can't reach the provider (an extended provider-side outage), records must not stay PENDING forever — define a maximum reconciliation window after which a human-reviewed manual resolution path kicks in; an infinite retry loop is not a design, it's a deferred incident.
• Watermark set from a single client's clock. Basing the watermark purely on the fastest-arriving producer's clock (rather than a bound like max_seen − allowed_lateness across all producers) silently drops every slower producer's legitimate late data forever — it never gets a second chance to be merged in.
• A fencing token issued but never checked at the resource. Generating monotonic tokens is worthless if the storage layer that receives writes doesn't compare and reject — the token has to be enforced at the last hop, not just handed out by the lock service.
• Sub-bucketing a hot SKU and calling it FCFS. If the interviewer or the product spec explicitly requires provable first-come-first-served, sharded counters silently violate it under load; say so rather than presenting sub-bucketing as a free lunch.
• Treating 301 as strictly "better" than 302. A 301 that later needs to change (a shortener rotating destinations, or an incorrectly minted link) will keep resolving to the old destination in every browser and CDN that already cached it, for as long as that cache entry lives — sometimes effectively forever for infrequent visitors.
• A fixed follower-count fan-out threshold that never gets revisited. An account crossing the threshold needs a migration path (draining its precomputed fan-out entries and switching reads to the pull path) — treating the threshold as a one-time classification instead of a state transition leaves accounts stuck on the wrong side as they grow.
• Resolving DST from the server's local timezone instead of the event's stored IANA timezone. A reminder is ambiguous or missing only in the timezone the user actually meant (e.g. America/New_York); resolving the gap/overlap using the server's own timezone (often UTC, which has no DST) silently produces the wrong local behavior for the user.

Both decisions share a shape: there is no context-free right answer, only a right answer given which cost the product can least afford — server load and staleness on one side, per-event visibility and write amplification on the other. Naming the trade-off, not just the mechanism, is what separates "I know fan-out-on-write" from "I know when fan-out-on-write is wrong."

1. An unknown outcome (a timeout, not a failure) needs a different mechanism than a known failure — idempotency keys make retries safe, but only a reconciliation job that asks the true source of state can resolve silence into a decision.
2. Merging a fast approximate layer with a slow exact layer is only safe with an explicit watermark defining the seam; without one, "lambda-style" merges either double-count or drop data at the boundary.
3. Contention on a single hot row (a lock, a seat, a SKU) is resolved by narrowing the atomic operation to that one row (CAS, lease+fencing, ordered-queue+batch) rather than reaching for a heavier distributed-transaction primitive — but check whether the narrowing preserves the guarantee you actually need (true FCFS) or only an approximation of it.
4. Every "which mechanism" question in this page resolves to a stated trade-off, not a universal winner — 301 vs 302, fan-out-on-write vs on-read, and sub-bucketed vs strictly-ordered inventory all trade a cost the interviewer expects you to name against a cost you're accepting.

Related pages

• System Design Problems II — Crawler, Ride-Hailing, Chat, File-Sync & Search: Resolution Mechanisms (Deep Dive) — the companion deep dive covering the rest of the series
• Designing Dropbox — Chunking, Dedup & Delta Sync, Traced — the chunked-upload lease/fencing example named in §3
• Quorum in Practice — The Two Inequalities, Conflict Resolution, Read-Repair, Partitions & Tail Latency (Deep Dive) — related single-row contention resolution mechanisms
• Idempotency & “Exactly-Once Is a Myth” — the foundational idempotency-key mechanism used throughout §1
• Designing Reminder Alert System — the reminder-worker and DST-scheduling problems traced in §3 and §7