Distributed IDs & Transaction Protocols — Snowflake Failure Modes, 2PC Blocking, TCC's Three Problems & Saga Isolation (Deep Dive)

Four mechanisms show up over and over in distributed-systems interviews, and each one has a specific failure mode that the textbook description skips: a Snowflake ID generator that assumes the clock only moves forward, a Two-Phase Commit coordinator whose crash leaves a participant unable to move at all, a TCC (Try-Confirm-Cancel) protocol whose Confirm/Cancel messages can arrive out of order or twice, and a Saga whose steps commit independently and so are never truly isolated from each other. This page traces each failure mode with real values — not just names it.

Split-brain detection, fencing tokens, and CRDT conflict resolution are covered in depth elsewhere in this guide (see CAP in Practice — Fencing Tokens, Sloppy Quorums, BFT & Consistency Models, Split-Brain, Fencing & Safe Failover, and the CRDT pages); this page only recaps them briefly at the point they intersect with saga multi-master conflicts, and instead spends its depth on the four mechanisms above. A higher-level comparison of 2PC vs Saga vs TCC already exists in this guide (2PC vs Saga vs TCC — Distributed Transactions); this page is the depth layer underneath that comparison — the specific bugs each protocol produces in production and why.

1. Snowflake IDs: a clock and a counter, not a random number

A Snowflake ID is a single 64-bit integer built by concatenating three fields, most-significant bit first: an unused sign bit, a millisecond timestamp, a worker id, and a per-millisecond sequence counter. IDs are therefore roughly time-ordered by construction — sort them numerically and you get approximately the insertion order — and unique because no two generators ever emit the same (timestamp, worker id, sequence) triple, as long as three invariants hold. Every real production incident with Snowflake IDs is one of those three invariants quietly breaking.

(a) Clock rewind — the reused-timestamp duplicate

The generator's uniqueness guarantee depends on one assumption: the clock it reads only moves forward. NTP does not guarantee that. If a host's clock is running fast, an NTP correction can step the clock backward by tens or hundreds of milliseconds in a single adjustment (rather than the gradual slew NTP prefers) — and the generator, having no memory of physical time, happily reuses a millisecond it has already issued IDs for.

Trace it with real values on a single worker (worker id = 7), sequence counter reset to 0 at the start of each new millisecond:

The duplicate is silent at mint time — both calls return successfully, no exception, no retry signal — and only surfaces later as a primary-key collision, a cache-key collision, or two unrelated business objects that compare equal. The fix is a guard the naive walkthrough above skips: before emitting, compare the current clock reading to last_seen_ts. If now < last_seen_ts, the generator must not mint — it either blocks and retries until the clock catches back up past last_seen_ts, or raises an explicit ClockMovedBackwardsException so the caller can fail loudly rather than silently mint a collision. Twitter's reference Snowflake implementation takes exactly this stance: refuse to generate during a detected backward jump rather than degrade uniqueness.

(b) Worker-id coordination — the 10-bit ceiling

Uniqueness across workers (not just across time on one worker) depends on no two live generators ever holding the same worker id at the same time. With 10 bits allotted to the field, the id space is exactly 2¹⁰ = 1024 values (commonly split further into a datacenter-id and a node-id sub-field, e.g. 5 bits each). Two failure shapes come from this field, and they are opposite problems:

Collision from static/manual assignment. If worker ids are hardcoded in config or assigned by a human spreadsheet, a deploy mistake (two instances launched with the same configured id, or a stale id left behind after a host is repurposed) silently produces two generators minting from the same id space — and the sequence-counter uniqueness guarantee, which only holds per worker id per millisecond, is gone. The standard fix is to make worker-id assignment a real coordination problem: each generator acquires an ephemeral lease on a free slot from ZooKeeper or etcd at startup (e.g. a sequential ephemeral znode, or an etcd lease with a TTL), holds it only while its session is alive, and the coordination service releases the slot automatically if the process dies without a clean handoff — so a crashed worker's old id becomes available again instead of staying "reserved" forever or, worse, double-issued.

Ceiling from scale. The other side of the same field is a hard capacity limit: at most 1024 concurrently live generators, full stop, no matter how the bits are sub-divided. A fleet that outgrows the id-space ceiling has to widen the field (fewer sequence bits, or a longer ID beyond 64 bits) — a breaking schema change for every existing ID's meaning — which is why the bit widths are usually chosen with real headroom (1024 workers × 4096 ids/ms/worker = ~4.19M ids/ms fleet-wide, per millisecond, is normally far beyond one service's actual write rate) rather than tuned tight to current scale.

(c) The enumeration/security trade-off

The same property that makes Snowflake IDs useful for sorting and sharding — they are roughly time-ordered — is a direct information leak when the ID is exposed to an untrusted client. Two consecutive orders, invoices, or user-signup rows get IDs that differ by a small, predictable delta, which lets an outside observer (a) estimate how many rows of that type exist and how fast they're being created — your daily order volume, deducible from two timestamps a competitor can see in two receipts — and (b) simply increment or decrement an ID they legitimately hold and probe for neighboring resources (IDOR — Insecure Direct Object Reference), especially if authorization checks are weak on the read path.

The fix is a split between internal and external identifiers, not abandoning Snowflake: keep the Snowflake ID as the primary key and sort/shard key inside the system, where its ordering property is valuable for index locality and range scans, but never hand it to an untrusted client directly. Expose a second, opaque identifier at the API boundary instead — a random UUIDv4, a Hashids/Sqids-style reversible obfuscation of the internal id, or a per-resource random token stored alongside the row — so the public-facing value carries no ordering information and is not sequentially guessable. This is exactly the same internal/external split used for database auto-increment primary keys, generalized to Snowflake's ordered-but-not-random IDs.

2. Two-Phase Commit: why the block is fundamental, not a bug

2PC makes a distributed write atomic across N participants by splitting the decision from the execution into two round trips. Phase 1 — prepare/vote: the coordinator sends PREPARE(tx) to every participant; each participant does everything needed to guarantee it can commit if told to — acquire the necessary locks, write the change to its durable redo log — and replies VOTE YES only once that state is safely persisted, or VOTE NO and immediately releases its own locks if it can't proceed. Phase 2 — commit/abort: only after collecting votes from every participant does the coordinator decide — COMMIT if all voted yes, ABORT if any voted no — and broadcasts that single decision to everyone.

The blocking problem lives entirely in the gap between those two phases. A participant that has replied VOTE YES has made a promise it cannot take back: it might be told COMMIT a moment later, so it must keep holding its locks and keep the prepared state durable. If the coordinator crashes in exactly that gap — after collecting all votes but before the decision is broadcast — the participant is stuck. It cannot guess: unilaterally aborting is unsafe because the coordinator might already have told a different participant to commit (now these two participants disagree about the outcome of the same transaction); unilaterally committing is equally unsafe because some other participant might have voted NO, which the crashed coordinator never got to broadcast. The only safe move is to wait — holding every lock it acquired in phase 1 — until the coordinator recovers and tells it the real answer, or until an operator intervenes.

This is why the block is fundamental to the protocol, not an implementation bug to be patched away: 2PC buys atomicity by having every participant agree to give up its own ability to decide unilaterally the moment it votes yes, in exchange for a promise that a single coordinator will tell everyone the same answer. That promise has no timeout that is always safe — any timeout a participant picks risks guessing wrong exactly in the crash scenario above. The protocol trades liveness (the guarantee that progress eventually happens) for safety (the guarantee that no two participants ever disagree about the outcome), and a coordinator crash at the worst possible moment is precisely when that trade is paid.

The standard mitigations attack this exact gap rather than removing it: Three-Phase Commit (3PC) inserts a PRE-COMMIT acknowledgement between vote-collection and the final commit, so a timed-out participant that reached pre-commit can safely assume commit was decided (non-blocking under a bounded-delay, no-network-partition assumption — an assumption that fails under real partitions, which is why 3PC is rarely used as-is in practice). Consensus-based commit coordination — replacing the single coordinator with a Raft- or Paxos-backed group that logs the PREPARE/COMMIT decision to a majority before acting — removes the single point of failure instead: the decision survives any one coordinator crash because a replacement can read the same durable log and finish the broadcast, so participants are only ever blocked as long as it takes a new leader to be elected, not indefinitely.

3. TCC (Try-Confirm-Cancel): explicit reservations, and their three notorious bugs

TCC avoids 2PC's cross-service lock-holding by replacing generic prepare/commit with three business-level calls each participant implements itself: Try reserves a resource without finalizing it (e.g. move funds into a "frozen/pending" bucket, not out of the account); Confirm finalizes a Try that already succeeded (release the frozen funds as a completed debit); Cancel releases a reservation a Try made, restoring the prior state (unfreeze the funds). Because Try already did the risky, resource-holding part, Confirm and Cancel are meant to be simple and non-blocking — but that split across independent services, retried over an unreliable network, is exactly what produces the three problems Seata (the widely used Java TCC framework) documents as the recurring production failures of the pattern.

(a) Empty rollback

A Cancel call arrives at a participant for a transaction whose Try never ran there at all — or ran but was lost before any state was recorded. This happens routinely: if the orchestrator times out waiting for Try's response and unilaterally decides to roll back the whole transaction, Cancel is dispatched to every participant regardless of whether their Try actually landed. A participant implementation that assumes "Cancel always undoes a Try that happened" will either throw (looking for state that doesn't exist) or, worse, silently corrupt unrelated state by cancelling against a default/zero record it shouldn't have touched. Fix: Cancel must first check whether a transaction-status record exists for that tx id; if none exists, it inserts a record marking the transaction as cancelled-without-a-try and returns success as a no-op — both so the Cancel call is safely idempotent and so a Try that shows up later for the same tx id (see (b)) can see that a Cancel already happened for it.

(b) Hang / suspension — the out-of-order delivery

A Try call is delayed on the network (retried, queued behind a slow link, or simply reordered) and arrives at the participant after the Cancel for the same transaction has already been processed. Without a guard, the late Try executes as if nothing happened: it reserves the resource (freezes the funds), permanently — because the orchestrator has already moved on believing the transaction was fully rolled back, no Confirm or second Cancel is ever coming to release it. The resource is now stuck ("hung"/suspended) in a frozen state forever, invisible to the orchestrator's bookkeeping. Traced:

Fix: the same transaction-status record used to make Cancel idempotent doubles as the guard here — before a Try reserves anything, it must check whether a Cancel (or Confirm) has already been recorded for that tx id, and if so, refuse to act rather than executing a Try nobody will ever finalize or release. Both (a) and (b) are solved by the same piece of state: a durable, per-transaction status record checked before every phase acts.

(c) Idempotency of Confirm/Cancel under retry

Network calls get retried — a client that doesn't receive an ack for Confirm has no way to tell whether the first call was lost in flight or the response was lost on the way back, so it retries the same Confirm (or Cancel). If Confirm's implementation is "release the frozen funds" expressed as a naive decrement/increment rather than a state transition, a duplicate Confirm double-releases — crediting the same amount twice. Fix: every Confirm and Cancel must be a state-transition guarded by the same transaction-status record, not a bare arithmetic operation: Confirm checks the record is in the TRIED state before transitioning it to CONFIRMED and performing the side effect exactly once; if the record is already CONFIRMED, the call is a no-op that returns success without repeating the side effect. Cancel is symmetric against CANCELLED.

All three problems reduce to one missing piece of infrastructure: a durable, queryable transaction-status table (tx id → phase: TRIED / CONFIRMED / CANCELLED) that every phase consults before acting and updates atomically with its own side effect. TCC frameworks like Seata provide this as a first-class piece of the runtime precisely because hand-rolled TCC participants forget to build it and hit exactly these three bugs in production.

4. Saga isolation anomalies and the Garcia-Molina countermeasures

A saga achieves atomicity-like all-or-nothing behavior across services by chaining local transactions with compensations, but it buys that with a cost 2PC and TCC don't pay in the same way: each step's local transaction commits — and becomes visible to every other reader — the moment it finishes, long before the whole saga finishes or is known to succeed. There is no cross-step isolation. Three classic anomalies fall directly out of that:

Lost update: saga A reads a value, saga B reads the same value and writes an update based on it, then saga A writes its own update based on the stale value it read — overwriting B's write as if it never happened. Dirty read: a second saga reads a value written by step 2 of saga A and acts on it, but saga A's later step fails and compensates step 2 back — the second saga acted on data that, in the final consistent state, never should have existed. Fuzzy/non-repeatable read: a saga reads the same value twice at two different steps and gets two different answers because another saga committed a change in between — violating the assumption that a single saga sees a consistent snapshot across its own steps.

Hector Garcia-Molina and Kenneth Salem, who introduced sagas (1987), also catalogued the standard countermeasures — techniques applied at the application level, since there is no database engine enforcing isolation across services:

• Semantic lock: mark a row touched by an in-flight saga step with an explicit PENDING flag (application-level, not a DB row lock). Other sagas that see PENDING either wait, refuse, or apply their own compensating logic instead of reading it as a normal committed value — directly blocking the dirty-read and fuzzy-read paths above.
• Commutative updates: structure updates so order doesn't matter — balance += 10 and balance += 5 commute regardless of which saga's step runs first, so interleaving two sagas' updates on the same row can never lose one, sidestepping the lost-update anomaly by construction rather than by locking.
• Reread-value ("re-read before act"): re-fetch the value immediately before the step that depends on it and verify it hasn't changed since the earlier read in the same saga, aborting/retrying the step if it has — a targeted, saga-local fix for fuzzy reads without locking the row for everyone else.
• Pessimistic view: reorder a saga's steps so that operations most likely to need compensation run first, before steps that would be expensive or risky to see in a dirty-read intermediate state — reduces the window in which an anomaly can be observed rather than eliminating it outright.
• By-value: choose which countermeasure to apply per-transaction based on the business risk of the value involved — a $5 line-item gets a cheap, lock-free strategy; a $50,000 wire transfer gets semantic locking — because uniformly applying the strongest countermeasure everywhere is needlessly expensive for low-risk operations.

Multi-master write conflicts: LWW vs CRDT vs home-region

A related but distinct problem shows up when the same logical record can be written in more than one region/replica at once (multi-master), which sagas spanning regions can trigger: two writes to the same key race, and something has to decide the outcome. Three strategies, in order of how much they quietly discard versus formally reconcile:

Last-Write-Wins (LWW): attach a timestamp (or hybrid logical clock) to every write and keep whichever has the higher timestamp on conflict. Simple and requires no coordination, but it silently drops the losing write's data — there is no merge, no notification, the loser simply vanishes, and clock skew across regions can pick the "wrong" winner by wall-clock time even though it was causally earlier from the client's point of view. CRDTs (Conflict-free Replicated Data Types): design the data type itself (a counter, a set, a map) so that concurrent updates merge deterministically via a mathematical join — every replica converges to the same state with no data silently lost and no central coordinator, at the cost of a constrained type system (not every business object maps cleanly onto a CRDT) and, for some CRDT variants, growing metadata overhead. See the CRDT pages in this guide for the state-based vs operation-based mechanics. Home-region ownership: assign each record a single authoritative region (by user's signup geography, tenant, or shard key) and route all writes for that record to its home region, so concurrent writes to the same record from two regions structurally cannot happen — conflicts are prevented rather than resolved after the fact, at the cost of higher write latency for users who aren't near their record's home region and a real failure mode if the home region itself is unreachable (does the write block, or fail over and risk exactly the conflict this was meant to prevent?).

Multi-master conflicts and network partitions are close cousins of split-brain: two writers that each believe they are the sole authority for a record are the same shape of problem as two nodes that each believe they are the leader. The fencing-token and quorum defenses covered in depth in CAP in Practice and Split-Brain, Fencing & Safe Failover elsewhere in this guide apply directly here: a fencing token attached to writes lets a home region (or a quorum) reject a stale writer's update even if that writer hasn't yet realized it lost ownership.

The CAP/PACELC stance: RPO=0 forces CP

Underneath all three conflict strategies is a single availability-vs-consistency decision that CAP/PACELC makes explicit. If the business requirement is RPO = 0 (zero acceptable data loss on failover — every acknowledged write must survive any single failure), the only honest implementation is synchronous replication: the primary must wait for acknowledgement from every replica (or a durable quorum) before acknowledging the client, so that no acknowledged write can ever exist on the primary alone. That is a CP choice under CAP — the system must refuse to accept writes (sacrifice availability) during a partition rather than accept a write that might not survive a failover. LWW and CRDTs are, not coincidentally, the tools reached for when a system instead chooses AP and accepts a nonzero RPO in exchange for staying available through a partition — there is something to reconcile only because both sides kept accepting writes.

Judgment layer: choosing among 2PC, Saga, and TCC

2PC — use when participants are few, co-located (same datacenter, low-latency network), short-lived, and you need real ACID atomicity with a generic (non-business-aware) protocol, e.g. a classic XA transaction across two databases in one cluster. Avoid it across services owned by different teams or across WAN links: the blocking window in a coordinator crash is proportional to how long recovery takes, and holding row locks across a slow cross-region hop for that duration is a throughput and availability cost most services can't absorb.

TCC — use when you need something close to 2PC's assurance (an explicit reservation phase that can still be reversed) but across services that refuse to hold generic database locks for another team's coordinator, and you're willing to write three business-aware methods per participant (Try/Confirm/Cancel) instead of relying on a generic protocol. Common in payments and inventory holds, where "freeze then finalize or release" maps naturally onto the resource. Cost: it is the most implementation-heavy of the three — every participant author must correctly handle empty rollback, hang, and idempotency, or the framework must (Seata does).

Saga — use for long-running workflows spanning many services (order → payment → inventory → shipping) where holding any kind of cross-service lock for the whole duration is unacceptable, and the business can tolerate a window where intermediate, partially-completed state is visible to other readers (mitigated with the countermeasures above) in exchange for compensating actions instead of true rollback. Avoid it where a partial, briefly-visible intermediate state is unacceptable to the business (e.g. a regulatory reporting event that must never be seen before it's final) — that calls for TCC's reservation phase or 2PC's true atomicity instead.

LWW vs CRDT vs home-region for multi-master conflicts follows the same shape of trade-off: LWW when losing data occasionally is acceptable and simplicity matters most; CRDTs when the data type fits the constrained CRDT model and losing no update is a hard requirement; home-region ownership when the data model can be sharded by a natural owner and you'd rather prevent the conflict than resolve it, accepting the latency and failover cost for users away from their home region.

Pitfalls

• "NTP only slews, it never steps." Slewing is the preferred correction, but a sufficiently large drift, a misconfigured NTP client, or a manual clock fix can still step the clock backward. A generator that doesn't guard against now < last_seen_ts is one bad NTP event away from minting a duplicate ID.
• "A 2PC timeout makes the participant safe to unblock." No timeout is universally safe: guessing abort risks disagreeing with a coordinator that actually decided commit elsewhere, and vice versa. A timeout only bounds how long you wait before escalating to an operator or a recovery protocol — it doesn't remove the fundamental uncertainty.
• "Cancel/Confirm are simple, so they don't need their own state." All three Seata problems — empty rollback, hang, double-execution under retry — come from treating Confirm/Cancel as bare side effects instead of guarded state transitions against a durable transaction-status record.
• "Sagas are eventually consistent, so isolation doesn't matter." Eventual consistency is about the data converging; saga isolation anomalies are about other readers seeing intermediate state that may later be compensated away entirely — a different problem that eventual consistency alone does not fix.
• "Exposing the Snowflake ID directly is fine, we don't have a public API." Internal dashboards, support tools, and even URL query parameters logged by a CDN can leak volume information to anyone who can see them; the internal/external ID split is cheap enough to apply by default rather than only after an incident.

Key takeaways

• Snowflake IDs are only unique if three invariants hold: the clock never rewinds past the last issued timestamp (guard with now < last_seen_ts → refuse/wait), worker ids are leased uniquely (ZooKeeper/etcd, not hardcoded, within the 10-bit/1024-worker ceiling), and time-ordered IDs are never handed to untrusted clients directly (use an opaque external id).
• 2PC's coordinator-crash block is a fundamental safety-over-liveness trade, not a fixable bug: a participant that voted YES cannot unilaterally decide without risking disagreement with whatever the coordinator told (or didn't tell) another participant. 3PC and consensus-backed coordinators bound the block; they don't eliminate the trade.
• TCC's three production bugs — empty rollback, hang/suspension, and double-execution under retry — all stem from treating Confirm/Cancel as bare side effects; the fix in every case is the same durable, per-transaction status record consulted before every phase acts.
• Sagas trade cross-service atomicity for local-commit visibility, producing lost update/dirty read/fuzzy read; Garcia-Molina's countermeasures (semantic lock, commutative updates, reread-value, pessimistic view, by-value) manage the anomaly rather than eliminate it, and an RPO=0 requirement forces a CP (synchronous replication) choice regardless of which countermeasure is used.

Sources: X. (Twitter) Snowflake project documentation, 2010 (the 64-bit id layout and the clock-rewind refusal stance). H. Garcia-Molina & K. Salem, "Sagas," ACM SIGMOD, 1987 (the saga model and its isolation countermeasures). Seata (Alibaba) documentation and design notes on TCC mode (the empty-rollback, hang/suspension, and idempotency problem framing). J. Gray, "Notes on Data Base Operating Systems," 1978, and the classic Two-/Three-Phase Commit literature (D. Skeen, "Nonblocking Commit Protocols," ACM SIGMOD, 1981) for the 2PC blocking proof and the 3PC non-blocking attempt. E. Brewer, CAP theorem, and the PACELC extension (D. Abadi, 2012) for the RPO/CP framing. Re-authored/Deepened for this guide.