Time, Clocks & Ordering — Why You Can't Trust the Wall Clock

You cannot trust the clock on the wall

"Which write happened last?" feels obvious — just compare timestamps. But across machines, wall clocks drift, and NTP corrections can make a clock jump backwards. Order two events on two servers by their timestamps and you can get the order wrong — which, with last-write-wins, silently loses data. Distributed systems need a different notion of time: logical time, built on causality, not seconds.

Happens-before: the only ordering you can trust

Event a happens-before b if a could have caused b: same process and earlier, or a is a send and b its receive. (Exactly the happens-before from the concurrency memory model — same idea, now across machines.) Events with no happens-before path either way are concurrent — there is no "true" order, and pretending otherwise is the bug.

Logical clocks

• Lamport clock (a single counter): increment on each event; on send, attach it; on receive, set max(local, received) + 1. Guarantees a → b ⇒ L(a) < L(b). But the converse fails — equal/ordered counters can't distinguish causal from concurrent.
• Vector clock (one counter per node): can detect concurrency — if neither vector dominates the other, the events are concurrent (a true conflict). This is how Dynamo-style stores detect conflicting writes instead of silently dropping one.
  Micro-trace: replica A stamps a write [1,0]; replica B, not having seen it, stamps its own [0,1]. Component-wise, neither is ≤ the other → provably concurrent → the store keeps both as siblings for the app to merge. Had B seen A's write first, B's would carry [1,1] ≥ [1,0] — a clean overwrite. That ≤-test is the entire conflict detector.

Between vector clocks and TrueTime sits the Hybrid Logical Clock (HLC) — a single ~64-bit timestamp that is causal like a Lamport clock yet stays within the skew bound of wall time; it is the pragmatic default in CockroachDB and MongoDB. See the deep Time & Clocks page for its update rules.

So how does Spanner order globally?

Google TrueTime faces the clock problem head-on: it exposes time as an interval [earliest, latest] with bounded uncertainty (atomic clocks + GPS), and on commit it waits out the uncertainty so two transactions can't overlap. Buying real ordering costs a deliberate wait — a vivid reminder that distributed time isn't free.

Pitfalls

• Last-write-wins by wall-clock timestamp — clock skew silently discards the "earlier" (by real causality) write. Use logical clocks or version vectors.
• Assuming monotonic time — NTP can step backwards; use a monotonic clock for durations.
• Ignoring concurrency — concurrent writes are a conflict to resolve, not an ordering to guess.

Takeaways

• Wall clocks drift and jump — never order distributed events by timestamp alone.
• Happens-before is the trustworthy order; Lamport gives a consistent total order, vector clocks detect concurrency/conflicts.
• Real global ordering (Spanner TrueTime) costs a commit-wait — time isn't free.

---

Re-authored for this guide; Lamport-clock diagram hand-authored as SVG. Follows Lamport (1978) and DDIA ch. 8–9. See also: (Concurrency) Memory Model & happens-before, Replication (LWW conflicts), MVCC. Go deeper: Time, Clocks & Ordering — Lamport, Vector Clocks, HLC & TrueTime (the full clock ladder, with the vector-clock trace and commit-wait mechanics).

Which clock for which problem — and how LWW silently loses data

Decision table — which clock for which problem?

Problem	Use	Never use
Order events within one process	Monotonic clock	Wall clock
Detect causality across nodes	Vector clock / version vector	Lamport clock alone
Total order across replicas	Consensus epoch + log offset	Wall-clock timestamp
Bound commit wait globally	TrueTime-style interval clock	NTP alone

Failure trace — last-write-wins silent data loss

1. A client sets x=1; the write is stamped by Node A, whose clock runs 20 ms fast → timestamp 10:00:00.030 (real time ≈ .010).
2. 5 ms later (real time ≈ .015) the same client sets x=2; this write is stamped by Node B, whose clock is accurate → timestamp 10:00:00.015.
3. In real time x=2 happened after x=1, so x=2 is the value the user expects to survive.
4. Last-write-wins keeps the larger timestamp: .030 (x=1) beats .015 (x=2) — so the store keeps the older value x=1.
5. The genuinely-later write x=2 is silently discarded: a fast clock on Node A quietly ate the user's most recent update.

Moral: with skewed wall clocks, "largest timestamp" ≠ "happened last." Detect the conflict with a version vector instead of guessing an order — which is exactly what Dynamo-style stores do.

Drill ladder

• L1: Why can NTP make a clock run backwards?
• L2: Two Lamport clocks: L(a)<L(b). Can a and b be concurrent?
• L3: How does Spanner's TrueTime prevent two transactions from overlapping?

Ops signal: watch NTP/chrony offset and clock-step events (a backward step is the moment LWW quietly loses data), the version-vector conflict rate (rising conflicts mean concurrent writes you are silently resolving), and — on Spanner-style systems — the TrueTime commit-wait histogram (that wait is the price of global order).

I just read a lesson on **Time, Clocks & Ordering — Why You Can't Trust the Wall Clock** (System Design) and want to truly understand it. Explain Time, Clocks & Ordering — Why You Can't Trust the Wall Clock from first principles using ONE vivid real-world analogy and a visual mental model — draw it as ASCII art or a clear step-by-step diagram — with a concrete example using real numbers. Then ask me one question to check I got the mental picture, and wait for my reply. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Teach me **Time, Clocks & Ordering — Why You Can't Trust the Wall Clock** interactively. Ask me ONE guiding question at a time, wait for my answer, and adapt to my confusion — build the idea with me step by step instead of explaining it all at once. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Quiz me on **Time, Clocks & Ordering — Why You Can't Trust the Wall Clock** with 5 questions, easy to tricky, ONE at a time. Tell me if each answer is right; at the end, explain clearly what I got wrong and why. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Help me remember **Time, Clocks & Ordering — Why You Can't Trust the Wall Clock** for the long term: give the one-sentence intuition, a memorable hook/mnemonic, a tiny worked example, and 3 active-recall flashcards (Q -> A). If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.