Time, Clocks & Ordering — Why You Can't Trust the Wall Clock

You cannot trust the clock on the wall

"Which write happened last?" feels obvious — just compare timestamps. But across machines, wall clocks drift, and NTP corrections can make a clock jump backwards. Order two events on two servers by their timestamps and you can get the order wrong — which, with last-write-wins, silently loses data. Distributed systems need a different notion of time: logical time, built on causality, not seconds.

Happens-before: the only ordering you can trust

Event a happens-before b if a could have caused b: same process and earlier, or a is a send and b its receive. (Exactly the happens-before from the concurrency memory model — same idea, now across machines.) Events with no happens-before path either way are concurrent — there is no "true" order, and pretending otherwise is the bug.

Logical clocks

• Lamport clock (a single counter): increment on each event; on send, attach it; on receive, set max(local, received) + 1. Guarantees a → b ⇒ L(a) < L(b). But the converse fails — equal/ordered counters can't distinguish causal from concurrent.
• Vector clock (one counter per node): can detect concurrency — if neither vector dominates the other, the events are concurrent (a true conflict). This is how Dynamo-style stores detect conflicting writes instead of silently dropping one.

So how does Spanner order globally?

Google TrueTime faces the clock problem head-on: it exposes time as an interval [earliest, latest] with bounded uncertainty (atomic clocks + GPS), and on commit it waits out the uncertainty so two transactions can't overlap. Buying real ordering costs a deliberate wait — a vivid reminder that distributed time isn't free.

Pitfalls

• Last-write-wins by wall-clock timestamp — clock skew silently discards the "earlier" (by real causality) write. Use logical clocks or version vectors.
• Assuming monotonic time — NTP can step backwards; use a monotonic clock for durations.
• Ignoring concurrency — concurrent writes are a conflict to resolve, not an ordering to guess.

Takeaways

• Wall clocks drift and jump — never order distributed events by timestamp alone.
• Happens-before is the trustworthy order; Lamport gives a consistent total order, vector clocks detect concurrency/conflicts.
• Real global ordering (Spanner TrueTime) costs a commit-wait — time isn't free.

---

Re-authored for this guide; Lamport-clock diagram hand-authored as SVG. Follows Lamport (1978) and DDIA ch. 8–9. See also: (Concurrency) Memory Model & happens-before, Replication (LWW conflicts), MVCC.

I just read a lesson on **Time, Clocks & Ordering — Why You Can't Trust the Wall Clock** (System Design) and want to truly understand it. Explain Time, Clocks & Ordering — Why You Can't Trust the Wall Clock from first principles using ONE vivid real-world analogy and a visual mental model — draw it as ASCII art or a clear step-by-step diagram — with a concrete example using real numbers. Then ask me one question to check I got the mental picture, and wait for my reply. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Teach me **Time, Clocks & Ordering — Why You Can't Trust the Wall Clock** interactively. Ask me ONE guiding question at a time, wait for my answer, and adapt to my confusion — build the idea with me step by step instead of explaining it all at once. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Quiz me on **Time, Clocks & Ordering — Why You Can't Trust the Wall Clock** with 5 questions, easy to tricky, ONE at a time. Tell me if each answer is right; at the end, explain clearly what I got wrong and why. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Help me remember **Time, Clocks & Ordering — Why You Can't Trust the Wall Clock** for the long term: give the one-sentence intuition, a memorable hook/mnemonic, a tiny worked example, and 3 active-recall flashcards (Q -> A). If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.