Designing a Digital Wallet — TC/C, Event Sourcing & Sharded Raft, Traced

The only design where "eventually consistent" is not an option

A digital wallet moves money between accounts. The requirement that shapes everything: money must never be created or destroyed. A lost transfer is a customer's missing funds; a duplicated one is money invented from nothing. Both are unacceptable in a way that a lost like or a stale feed is not, and that is what makes this design different from the rest of this guide.

Scope

• Transfer between two wallets; check balance; view transaction history.
• ~1 million transfers/second at peak in the demanding version of the problem.
• Correctness and auditability above availability — refusing a transfer is far better than mis-executing one.
• Every state change must be reproducible — you must be able to prove how a balance came to be.

Start with the ledger, not the balance

The naive schema is wallets(user_id, balance) and UPDATE balance = balance - 100. It is wrong for a reason deeper than concurrency: it destroys history. After the update you know the balance and nothing about how it got there, so you cannot audit, cannot answer a dispute, and cannot detect that a bug corrupted it.

Instead use double-entry bookkeeping: every transfer writes two immutable ledger entries — a debit and a credit — sharing a transaction ID, and summing to exactly zero. A balance is derived by summing entries (in practice, a cached snapshot plus entries since). The consequence is the most valuable property in the design: if the sum of all entries is not zero, you have a bug, and you can detect it mechanically. Money cannot leak silently, because leaking would violate an invariant you can check continuously across the whole system.

The distributed problem: two wallets, two shards

At scale wallets are sharded by user, so a transfer usually spans two shards — two databases, no shared transaction. Three approaches, in increasing practicality:

Two-phase commit (2PC)

A coordinator asks both shards to prepare, then tells both to commit. It provides atomicity and has two well-known problems: it is blocking (if the coordinator dies after prepare, participants hold locks indefinitely, unsure whether to commit or abort), and it holds database locks for the duration of a network round trip. At a million transfers/second, lock-holding across the network is not viable.

TC/C — Try, Confirm, Cancel

The practical version, and the one traced in the diagram. It is 2PC moved into the application, where the intermediate state is a real business concept rather than a database lock:

1. Try — reserve resources. Wallet A's balance drops and 100 moves into a held bucket; wallet B records 100 as pending in. Each shard commits its own local transaction immediately, so no locks are held across the network.
2. Confirm — finalize. A's hold clears; B's pending becomes real balance.
3. Cancel — compensate. A's hold is released back to available; B discards the pending entry.

Why this is genuinely better: the failure that matters — insufficient funds — is discovered in Try, before anything is committed anywhere, so the cancel path is a release rather than a reversal of applied money. And at no point is the 100 both spendable in A and available in B, because held funds are not spendable. The reservation is what buys atomicity without distributed locks.

The cost, stated honestly: an intermediate state is visible. A user can see money leave their available balance before it arrives at the destination, so the product must show "pending" states, and the phase transitions must be persisted in a phase-status table so a crashed coordinator can be resumed rather than guessed at. TC/C does not remove the coordinator problem; it makes the stuck state recoverable and non-blocking.

Saga

A sequence of local transactions, each with a compensating transaction, executed forward and unwound backward on failure. More flexible than TC/C and appropriate for long multi-step workflows — but for a two-party transfer it is weaker, because compensation happens after money has already moved, so there is a window in which the system is observably imbalanced. Prefer TC/C for transfers; prefer saga for orchestration across many services where reservation is not possible.

Idempotency, non-negotiably

Every mutating request carries a client-supplied idempotency key, stored with the result. A retry with the same key returns the original outcome rather than executing again. Without this, any timeout is unresolvable: the client cannot tell whether the transfer happened, and retrying might double-pay. With it, retrying is always safe, which is what allows aggressive retries everywhere else in the system.

The subtlety: the idempotency record and the ledger entries must be written in the same local transaction. If the key is recorded first and the write then fails, the retry is suppressed and the transfer is silently lost; if the write succeeds and the key is not recorded, the retry double-applies. This is one of the few places where "it's just a cache" reasoning causes financial loss.

Reproducibility: event sourcing and CQRS

To prove how a balance arose, store the commands and the events they produced, in order, as the system of record. Balances become a projection of that log. Two properties follow that are hard to get any other way:

• Determinism / replay. Re-run the event log through the same logic and you must arrive at the same balances. This makes auditing mechanical, and makes recovery from a logic bug a matter of fixing the code and rebuilding the projection.
• Historical state. "What was this balance last Tuesday?" is answerable by replaying to that point, rather than by hoping someone kept a backup.

Replaying from the beginning gets slow, so take periodic snapshots and replay only the events after the most recent one — the same checkpoint pattern as everywhere else. CQRS separates the write model (append commands/events, optimized for correctness and throughput) from read models (balances, statements, analytics projections), each shaped for its query. The honest cost: read models are eventually consistent with the event log, so a balance display can lag a transfer by milliseconds — which is why authorization decisions must read the write model or a strongly-consistent projection, never the analytics one.

Determinism is a real engineering constraint, not a free property. Replay only reproduces the same result if the logic contains nothing non-deterministic — no wall-clock reads, no random values, no unordered map iteration, no calls to external services whose answers may differ. Timestamps and any external decision must be captured in the event so the replay uses the recorded value. Systems that neglect this discover during their first audit that replay produces different numbers, which destroys the entire value of the design.

Scaling: sharded Raft groups

Correctness needs replication with a strong guarantee, so each shard is a Raft group: a leader plus followers, with writes committed once a majority has them. Raft gives a single, agreed order of events per shard and survives a minority of failures with no data loss — exactly the guarantee a ledger needs.

One Raft group cannot handle a million transfers/second, so run many groups, each owning a slice of the wallet keyspace. Total throughput scales with group count; each group independently orders its own events.

What this costs is the thing to say out loud: a transfer spanning two groups spans two consensus domains, so there is no single log that orders both sides — which is precisely why TC/C exists above. Sharded Raft gives you scalable strong consistency within a shard and hands you the cross-shard problem to solve at the application layer. For per-node throughput, high-performance implementations lean on a local embedded store (RocksDB-style log-structured storage) and avoid copying between kernel and user space, since at this rate the bottleneck becomes the write path itself rather than consensus.

Which mechanism, when

Decision	Option	Choose when	Breaks when
Cross-shard atomicity	TC/C	Two-party transfers at high rate	Intermediate "pending" state must be modelled and shown
Cross-shard atomicity	2PC	Low rate, same datacenter, existing XA support	Coordinator failure blocks participants holding locks
Cross-shard atomicity	Saga	Long multi-service workflows	Money moves before compensation — observable imbalance
State model	Event-sourced ledger	Auditability and replay are requirements	Requires strict determinism; more storage; steeper to build
State model	Mutable balance column	Prototypes, non-financial counters	Any dispute, audit, or bug investigation — history is gone
Replication	Sharded Raft groups	Strong consistency at scale	Cross-shard transactions still need TC/C on top
Replication	Async primary/replica	Read scaling for non-authoritative views	Authoritative balances — failover can lose committed writes

Pitfalls

• Mutable balances with no ledger. Unauditable, and imbalance becomes undetectable.
• Floating-point money. Use integer minor units (cents) or a decimal type; binary floating point cannot represent 0.10 exactly, and the error accumulates into real discrepancies.
• Idempotency key written outside the transaction that writes the ledger — loses or duplicates transfers on retry.
• Non-determinism in replayed logic (clock reads, randomness, external calls), which silently breaks reproducibility until an audit exposes it.
• Authorizing spend from an eventually-consistent read model, permitting a double-spend within the projection lag.
• Held funds with no expiry. A crashed coordinator leaves money reserved forever; holds need timeouts and a reaper, and the reaper must be idempotent against a late confirm.
• No reconciliation. Even with a perfect design, reconcile against external systems (bank, PSP) — the ledger can be internally consistent and still disagree with reality.

Cost model — what dominates the bill

A wallet's cost is consensus writes and retained history — and unlike most systems here, the data can never be deleted, so storage grows monotonically forever.

Rough BOTE at 1 million transfers/second. Each transfer is 2 ledger entries plus command and event records — call it ~500 bytes of durable data per transfer, so 500 MB/s of ledger writes, ~43 TB/day. With Raft replication factor 3, that is ~130 TB/day provisioned, and because financial records are typically retained for 7 years, the trajectory is measured in hundreds of petabytes. At even $10/TB-month for cold tiers, one year of that history is on the order of $500,000/month and rising every month.

Compute is dominated by consensus: every write requires a majority round trip, so per-transfer latency has an irreducible network component and each shard's throughput is bounded by its slowest majority. Reaching a million/second means on the order of hundreds to thousands of Raft groups, each with 3 replicas — thousands of nodes whose sizing is set by fsync and consensus throughput rather than by CPU.

Dominant line items: replicated, permanently-retained ledger storage; then the consensus node fleet; then reconciliation and audit batch jobs, which re-read enormous history ranges.

Levers: snapshot aggressively and tier old events to cheap object storage (they must be retained, not fast — a 7-year-old event needs to be readable for an audit, not in microseconds); compress the event log, which is highly repetitive and compresses very well; and batch multiple events into one Raft append, which amortizes the consensus round trip across many transfers and is the single biggest throughput-per-dollar win. What you must not do is reduce the replication factor or shorten retention to save money — those are the guarantees you are being paid to provide.

Operability: the fingerprints of a broken wallet

The signal that matters most is one no other system in this guide has: the ledger sum drifting from zero. It should be computed continuously and treated as a sev-1, because a non-zero sum means money has been created or destroyed and every downstream number is suspect. It also localizes well — the imbalanced transaction ID tells you exactly which code path is broken.

Held funds with growing age is the stuck-coordinator fingerprint: transfers that passed Try and never reached Confirm or Cancel. Customers experience it as money that has left their balance and not arrived, so hold-age is a customer-impact metric, not a technical one. Idempotency-store hit rate spiking means clients are retrying heavily — usually a timeout somewhere upstream — and it is a healthy sign that the guard is working and an unhealthy sign about latency.

Replay producing different balances than the live projection is the determinism failure, and it is the one that invalidates the design's core promise; it should be tested continuously by replaying a sample window and diffing, not discovered during an audit. Raft leader elections clustering on particular groups points at a slow disk or a saturated node, and because writes stall during an election, it shows up as periodic transfer latency spikes on a subset of users — which looks random until you group by shard.

Watch also for reconciliation diffs against external systems trending non-zero, which is the only way to catch errors where your ledger is internally perfect and disagrees with the bank. Signals worth having: ledger-sum invariant (continuous), hold count and age distribution, phase-status table entries stuck per phase, idempotency hit rate, replay-versus-projection diff, per-group Raft election rate and commit latency, and external reconciliation diff by counterparty.

---

Authored for this guide to cover the digital wallet design (Alex Xu Vol. 2, ch. 27 — not present in the Vol. 1 PDF); TC/C phase and double-entry-ledger diagram hand-authored as SVG. Complements the existing "Designing Payment System" page and this guide's Saga pattern, Raft/consensus, and idempotency material.

I just read a lesson on **Designing a Digital Wallet — TC/C, Event Sourcing & Sharded Raft, Traced** (System Design) and want to truly understand it. Explain Designing a Digital Wallet — TC/C, Event Sourcing & Sharded Raft, Traced from first principles using ONE vivid real-world analogy and a visual mental model — draw it as ASCII art or a clear step-by-step diagram — with a concrete example using real numbers. Then ask me one question to check I got the mental picture, and wait for my reply. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Teach me **Designing a Digital Wallet — TC/C, Event Sourcing & Sharded Raft, Traced** interactively. Ask me ONE guiding question at a time, wait for my answer, and adapt to my confusion — build the idea with me step by step instead of explaining it all at once. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Quiz me on **Designing a Digital Wallet — TC/C, Event Sourcing & Sharded Raft, Traced** with 5 questions, easy to tricky, ONE at a time. Tell me if each answer is right; at the end, explain clearly what I got wrong and why. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Help me remember **Designing a Digital Wallet — TC/C, Event Sourcing & Sharded Raft, Traced** for the long term: give the one-sentence intuition, a memorable hook/mnemonic, a tiny worked example, and 3 active-recall flashcards (Q -> A). If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.