hard Web Crawler Freshness: re-crawl scheduling & change detection

Freshness is a scheduling problem, not a crawling problem

A crawler that re-fetches every page on the same clock is wrong twice over: too slow for the pages that change by the hour (the index goes stale, users see dead deals and yesterday's headlines) and too fast for the pages that never change (wasted bandwidth, wasted CPU, and a politeness budget burned on fetches that can only ever return the same bytes). The fix is to treat each page's change rate as something you estimate from its own history, schedule its next re-crawl proportional to that rate, and — before paying for a full re-fetch — ask the cheapest possible question first: “did anything actually change?”

Estimating change rate: a Poisson process per URL

Model each page's edits as a Poisson process with rate λ (expected changes per day). If Δt has passed since the last crawl, the probability the page has changed by now is P(changed) = 1 − e^(−λ·Δt). You never know λ up front, so you estimate it online with an exponential backoff-and-advance rule (the approach behind Cho & Garcia-Molina's freshness studies and production crawlers like Googlebot):

• Last check found the page changed → your interval was too long for this page — shrink it (e.g. halve it), floored at a minimum (e.g. 1 hour, so no host gets hammered).
• Last check found the page unchanged → your interval was too short — extend it (e.g. double it), capped at a maximum (e.g. 90 days, so nothing is abandoned forever).

This converges without ever needing to know the “true” λ: a fast-changing page keeps getting its interval squeezed until it's checked about as often as it actually changes; a static page's interval keeps stretching until it's checked rarely. The schedule chases the evidence.

Traced: four pages, four schedules

Page	Interval before	Check result	New interval
news.example.com/home	1 hour	conditional GET → 200, changed	stays 1 hour (already at the floor)
blog.example.com/post-42	7 days	conditional GET → 304 (3rd unchanged check in a row)	extends to 14 days
docs.example.com/faq	30 days	conditional GET → 304	extends to 60 days
shop.example.com/deal-19	30 days	conditional GET can't be trusted (host omits ETag) → full fetch, hash changed (price update)	shrinks to 15 days; if the very next check also shows a real change, shrinks again to 7 days

Notice the news page never leaves the floor, the docs page keeps drifting toward the cap, and the deal page oscillates downward while it's actively being edited — three very different schedules, each earned by that page's own history, with no global constant anywhere.

Cheap change detection before paying for a full re-fetch

• Conditional GET — the crawler stores the ETag and/or Last-Modified from the previous fetch and sends them back as If-None-Match / If-Modified-Since. If nothing changed, the server replies 304 Not Modified with headers only — no body. The cost of a “still fresh” check drops from a full page transfer to a couple of round-trip headers.
• Content hash fallback — not every server implements conditionals correctly (some never set an ETag; some send a Last-Modified that never moves). When you can't trust the headers, do the full fetch anyway but compare a hash of the normalized body against the stored hash. This also catches the opposite failure: a page that *does* return a fresh Last-Modified or a byte-different response purely from volatile boilerplate — an ad slot, a “rendered at …” timestamp, a CSRF token — without any change to the content that actually matters.
• Push signals — a sitemap's <lastmod>, an RSS/Atom feed, or WebSub (PubSubHubbub) lets a site tell you what changed instead of you guessing by polling. Cheapest option when offered, but coverage is partial and some sites publish stale lastmod values, so it supplements polling — it doesn't replace the floor/cap schedule above.

Priority: freshness value × change probability, bounded by politeness and budget

Two pages with the same λ don't deserve the same crawl slot: a heavily-linked homepage costs more to leave stale than an obscure archive page with identical change frequency. So the frontier doesn't just sort by “interval elapsed” — it ranks by freshness value × P(changed since last crawl), and only the top of that ranking gets a fetch slot once the crawl budget (finite fetches/day) and per-host politeness (robots.txt crawl-delay, rate limits) are accounted for. A page whose “ideal” interval says “check me now” still waits if checking it would blow the host's rate limit or the day's total fetch budget.

Pitfalls

• One interval for everything — either starves fast-changing pages (a stale index is the classic complaint about a slow search engine) or burns the crawl budget re-fetching pages that can't have changed.
• Ignoring ETag/Last-Modified — doing a full GET every time a conditional GET would have returned 304 wastes bandwidth and CPU on both ends for zero freshness gain, and burns politeness budget that could have gone toward a page that actually changed.
• Trusting a naive full-page hash — hashing raw HTML flags volatile boilerplate (ads, timestamps, session tokens) as a “change” on every visit, so the interval never grows and the page gets checked forever at the floor for no real gain. Normalize or diff the meaningful content region first.
• Hammering a host on a mass shrink — when many URLs on one domain shrink their interval at once (a site relaunch, a flash sale), the per-host politeness limit still applies. A shrunk interval is a request for priority, not permission to ignore crawl-delay.

Selection & trade-offs

Fixed interval vs adaptive re-crawl. A single global interval is simple — one config value, nothing to estimate or store per URL — and is fine for a small, homogeneous corpus where every page really does change at about the same rate. It stops working the moment the corpus is heterogeneous (which real crawls always are): adaptive scheduling costs extra state per URL (λ estimate, last ETag/hash, current interval) but spends a fixed crawl budget where staleness actually costs something, instead of spreading it evenly across pages that don't need it.

Conditional GET vs full-fetch+hash vs push. Conditional GET is the cheapest per-check cost when the server implements it honestly — use it as the default. Full fetch + content hash is the fallback for hosts that don't support (or lie about) conditionals; it costs a full transfer but still avoids treating boilerplate churn as a real change. Push (sitemap lastmod, RSS, WebSub) is cheapest of all when a site offers it — you're told, you don't have to ask — but coverage is partial and some feeds are stale, so it augments the polling floor rather than replacing it. A production crawler runs all three: push where available, conditional GET as the default poll, full-fetch+hash as the fallback for uncooperative hosts.

Takeaways

• Re-crawl interval should track a per-page estimate of change rate (λ), not a single global constant — shrink on a detected change, extend on none, floored and capped.
• A conditional GET turns “check if this changed” into a near-free operation (304, no body) — use it before ever paying for a full re-fetch, with content-hash as the fallback for uncooperative hosts.
• Scheduling priority is freshness value × P(changed), always bounded by per-host politeness and total crawl budget — an urgent page still waits its turn if fetching it now would violate either.
• Naive change detection (full-page hash, trusting every Last-Modified) breaks the whole scheme by flagging boilerplate churn as real change, which prevents the interval from ever growing.

---

Synthesized for this guide from the freshness/re-crawl literature — Cho & Garcia-Molina, “Estimating Frequency of Change” (the online λ-estimation and backoff-and-advance scheduling model), HTTP conditional-request semantics (RFC 7232 ETag/If-None-Match), and production crawler practice (Googlebot's crawl-rate/crawl-demand model, sitemap lastmod). Diagram hand-authored as SVG. Complements the “Designing a Web Crawler — Frontier, Dedup & Politeness, Traced” page, which flags freshness as a hard part without covering the scheduling mechanism — this page is that mechanism.

I just read a lesson on **Web Crawler Freshness: re-crawl scheduling & change detection** (System Design) and want to truly understand it. Explain Web Crawler Freshness: re-crawl scheduling & change detection from first principles using ONE vivid real-world analogy and a visual mental model — draw it as ASCII art or a clear step-by-step diagram — with a concrete example using real numbers. Then ask me one question to check I got the mental picture, and wait for my reply. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Teach me **Web Crawler Freshness: re-crawl scheduling & change detection** interactively. Ask me ONE guiding question at a time, wait for my answer, and adapt to my confusion — build the idea with me step by step instead of explaining it all at once. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Quiz me on **Web Crawler Freshness: re-crawl scheduling & change detection** with 5 questions, easy to tricky, ONE at a time. Tell me if each answer is right; at the end, explain clearly what I got wrong and why. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Help me remember **Web Crawler Freshness: re-crawl scheduling & change detection** for the long term: give the one-sentence intuition, a memorable hook/mnemonic, a tiny worked example, and 3 active-recall flashcards (Q -> A). If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.