hard Event-Time, Watermarks & Late Data

A watermark is a heuristic assertion, not a fact — and it is what triggers a window to close

In a stream, every event carries two clocks. Event time is when the thing actually happened (a click, a sensor reading, a payment). Processing time is when your system happens to observe it. These are never the same: a mobile client buffers offline and uploads later, a network hop stalls, a retry redelivers — so events arrive late and out of order relative to when they occurred. If you group events into windows ("total clicks per 10-second bucket"), you must decide which clock defines the bucket boundaries. Event time is the only one that gives a reproducible, physically-meaningful answer — but it creates a hard question: when can you be sure you've seen everything that belongs in a window, so you can close it and emit a result? You can never be certain — a straggler event could always still be in flight. A watermark is the stream processor's answer anyway: a running, self-declared claim of the form "I assert I have now seen every event with event-time ≤ T." It is a heuristic, not a proof, and that is precisely why it can be wrong.

How the watermark actually advances

A common, simple generator — "bounded out-of-orderness" — tracks the maximum event-time seen so far and subtracts a fixed slack that represents the worst lateness you're willing to tolerate:

// per source partition, updated on every incoming event
maxEventTimeSeen = max(maxEventTimeSeen, event.eventTime)
watermark        = maxEventTimeSeen - allowedLatenessBound

// a window fires the instant: watermark >= window.endTime

When the stream has multiple partitions (Kafka topic-partitions, for example), the operator's effective watermark is the minimum across all of its input partitions — the whole point of a watermark is a claim that everything up to T has arrived, so it can only be as fresh as the slowest input. This single design choice is also where the "idle source" pitfall below comes from.

Traced: window [0,10), a straggler arrives after the watermark has already passed it

Allowed-lateness bound = 2. Events arrive in this order (arrival order ≠ event-time order):

Arrival #	Event event-time	max-seen so far	watermark = max-seen − 2	window [0,10) state
1	2	2	0	open, holds {2}
2	5	5	3	open, holds {2,5}
3	9	9	7	open, holds {2,5,9}
4	14	14	12	watermark(12) ≥ window_end(10) → CLOSES, emits sum({2,5,9}) = 16
5	8 (straggler)	14	12	window already fired; event-time 8 ≤ watermark 12 → too late — dropped, or routed to a side output as a late correction

Notice the straggler's event-time (8) genuinely belongs inside [0,10) — it is not late in any real sense, it was simply delayed in transit. The watermark's assertion "I've seen everything ≤ 12" was wrong: it had not actually seen event #5 yet. That is not a bug in the implementation — it is the fundamental trade-off of using a heuristic to decide "have I seen enough?" over an unbounded, unreliable stream.

Why the watermark can be wrong

• A straggler after the watermark passed. As traced above — either silently dropped (data loss in the aggregate) or, if the engine supports it, emitted as a late correction via a side output that a downstream consumer must explicitly merge in.
• An idle or stalled source stalls the whole watermark. Because the global watermark is the minimum across partitions, one partition that stops producing (a quiet Kafka partition, a paused producer) freezes the merged watermark at its last value forever — every window downstream stays open indefinitely, memory grows, and results never emit, even though every other partition is flowing fine. Engines mitigate this with explicit idleness detection (mark a source "idle" so it's excluded from the min until it produces again).
• Too-aggressive a bound drops real data. A small allowedLatenessBound makes the watermark race ahead quickly (low latency), but if real-world jitter is larger than the bound assumes, genuinely on-time-but-slightly-delayed events keep landing after their window already closed — the same failure as the straggler above, just happening routinely instead of as an edge case.

The judgment layer

Tight bound vs loose bound. A tight allowedLatenessBound (e.g. a few hundred milliseconds) closes windows and emits results fast — good for dashboards and alerting where freshness matters — but drops or corrects more late data, because real network/client jitter regularly exceeds a small bound. A loose bound (minutes) waits longer before closing a window, so far fewer events are dropped, at the direct cost of end-to-end latency and more window state held in memory while it waits. There is no bound that is simply "correct" — it is a dial you set based on how much you value freshness versus completeness for that specific pipeline.

The mitigation, not a cure: watermark + allowed lateness + side output. Engines like Flink let you additionally keep a window's state alive for a further grace period after the watermark passes its end, so late-but-not-too-late events still trigger a recomputed emission, and only events later than that get routed to a side output for manual handling instead of silently vanishing. This narrows the failure window but does not eliminate it — a late-enough event can always still slip past any finite bound.

Named alternative: processing-time windows. Instead of grouping by when an event happened, group by when the server observed it — no watermark machinery needed at all, a window always closes exactly on schedule (e.g. every 10 wall-clock seconds), and there is nothing to be "late." The cost is correctness: the result depends on network conditions and processing delay at the time, not on reality, so it is not reproducible — replaying the same events through a slower or backed-up pipeline produces a different answer, and you cannot correctly reprocess historical data at all. Processing-time windows are the right choice only when approximate, best-effort freshness matters more than a correct answer (e.g. a rough live "requests per second" gauge); anything that must be auditable or reproducible (billing, analytics, ML feature pipelines) needs event-time + watermarks despite the extra complexity and latency.

Takeaways

• A watermark is a heuristic claim — "seen all event-time ≤ T" — not a guarantee; it is generated as max-seen-event-time − allowed-lateness-bound and is, by construction, sometimes wrong.
• It is what triggers windows to close and emit; a straggler that arrives after the watermark has passed its event-time is either dropped or handled as a late correction via a side output.
• Bound size is a direct latency-vs-completeness trade-off; because the merged watermark is a minimum across partitions, one idle source can stall every downstream window indefinitely.
• Processing-time windows sidestep the whole problem but give up reproducibility and correctness under lateness — use event-time + watermarks whenever the answer must be correct, not just prompt.

---

Synthesized from the Dataflow Model paper (Akidau et al., VLDB 2015), Streaming Systems (Akidau, Chernyak & Lax, O'Reilly), Apache Flink's watermark & allowed-lateness documentation, and Kleppmann's Designing Data-Intensive Applications (ch. 11, Stream Processing). Re-authored/Deepened for this guide. See also: The 8 Fallacies of Distributed Computing, Idempotency & Exactly-Once, Stateful Stream Processing: Checkpoints, Restart & TTL.

I just read a lesson on **Event-Time, Watermarks & Late Data** (System Design) and want to truly understand it. Explain Event-Time, Watermarks & Late Data from first principles using ONE vivid real-world analogy and a visual mental model — draw it as ASCII art or a clear step-by-step diagram — with a concrete example using real numbers. Then ask me one question to check I got the mental picture, and wait for my reply. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Teach me **Event-Time, Watermarks & Late Data** interactively. Ask me ONE guiding question at a time, wait for my answer, and adapt to my confusion — build the idea with me step by step instead of explaining it all at once. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Quiz me on **Event-Time, Watermarks & Late Data** with 5 questions, easy to tricky, ONE at a time. Tell me if each answer is right; at the end, explain clearly what I got wrong and why. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Help me remember **Event-Time, Watermarks & Late Data** for the long term: give the one-sentence intuition, a memorable hook/mnemonic, a tiny worked example, and 3 active-recall flashcards (Q -> A). If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.