hard DNS Failover & Anycast

Two ways to move traffic off a dead site — at two different layers

Both DNS failover and anycast answer “a site just died; how do I steer clients to a healthy one?” They differ in which layer does the steering, and that single fact drives everything else. DNS failover changes the answer to a name lookup — you point the record at a new IP. It's simple and works anywhere, but the old answer is cached in resolvers, browsers, and OSes across the whole internet, so it clears slowly. Anycast changes the route to a single IP — the same address is announced from many sites and BGP delivers each client to the nearest. Failover happens in the routing fabric in seconds, and clients never re-resolve. The catch is that BGP doesn't know about your TCP connections.

DNS failover: the TTL lies to you

The naive mental model is “set TTL=60s, so failover takes 60s.” The real recovery time is the sum of every cache between you and the client, most of which you don't control:

1. Detection. Your health checker must first notice site A is down and flip the record — call it ~20s.
2. Recursive resolver cache. A resolver that fetched the record at t=5s serves the old IP until its copy expires at t=65s. Worse, many resolvers clamp the TTL to a floor (they won't honor a value below, say, 30–60s) and some ISP resolvers simply ignore low TTLs and hold the record for minutes — pushing this out toward 300s.
3. Client / browser pinning. Browsers keep their own DNS cache (Chrome pins for roughly a minute) and the OS stub resolver caches too — another ~60s on top, regardless of what the record says.
4. Open connections. A DNS change does nothing to an already-established connection. A browser with an HTTP keep-alive socket to the dead IP keeps using it until the connection idles out or errors.

Traced example

Record www → 1.1.1.1, TTL 60. At t=0 site A (1.1.1.1) dies; health check flips it to 2.2.2.2 at t=20s.

Client situation	Recovers at
Resolver cached at t=5s, honors TTL	~t=65s (fresh lookup returns 2.2.2.2)
Resolver clamps min-TTL to 300s	~t=305s — 5 minutes
Browser tab pinned + keep-alive socket open	only after the socket drops, then a fresh lookup

So the same failover event resolves for some users in about a minute and for the tail in five-plus minutes. The TTL sets a floor on how fresh an answer can be — it does not bound how long stale answers keep flowing.

Anycast: failover in the routing layer

Anycast announces the same IP prefix (say 203.0.113.0/24) via BGP from every point-of-presence. A client just dials 203.0.113.10; the internet's routers, following BGP's best-path selection, deliver its packets to whichever announcing site is nearest. There is no per-client “pick a site” step and no DNS round-trip to a specific instance — the address itself is everywhere.

Failover is a route withdrawal. When site A dies (or is drained), it stops announcing the prefix. BGP reconverges — typically in seconds — and every router that used to send that client to A now sends it to the next-best site, B. The client never re-resolved DNS and never changed the IP it's talking to. This is why anycast failover is fast and TTL-independent.

The long-lived-TCP nuance

Because anycast pins a client to nothing — it merely follows whatever route is best right now — a path change mid-connection is dangerous. If BGP reconverges, a route flaps, or ECMP rehashes the flow, packets for an already-open TCP connection can suddenly arrive at a different site that has no state for that connection. It replies with a RST and the connection dies. For a stateless, one-shot exchange over UDP — a DNS query — this is invisible: the next packet just goes to a site that can answer. For a long-lived TCP flow — a large download, a WebSocket, a database connection — it's a hard reset the client must detect and retry. That's exactly why anycast is the default for DNS and CDN edge entry (short, stateless, often UDP) and why long-lived TCP over anycast needs care: keep the anycast frontend stateless, or use anycast only to steer the client to a site and then pin the session to that site's unicast address.

Judgment: DNS failover vs anycast vs a load-balancer VIP

Mechanism	Layer / speed	Gain	Cost / when NOT to
DNS failover	naming layer; minutes (TTL + pin bound)	dead simple, works everywhere, no special network kit	slow and non-deterministic; useless for sub-minute RTO; you don't control downstream caches
Anycast	routing layer; seconds	fast, no re-resolution, naturally routes to nearest PoP, absorbs DDoS	needs BGP (own IP space, ASN, peering) — operationally heavy; mid-connection reset risk for long-lived TCP
LB / VIP	within one site; sub-second	instant failover between backends behind one virtual IP; health-checked	the VIP is local — it does not survive the whole site/region going away

How a senior decides. Inside a datacenter or AZ, a load-balancer VIP gives the fastest, cleanest failover between backends — but it can't save you when the whole site is gone. Across regions where a few minutes of RTO is acceptable and you can't (or won't) run BGP, DNS failover is the pragmatic active/passive tool. When you need fast, global, latency-aware entry — or you're serving DNS itself, or absorbing a DDoS — anycast wins, provided the frontend is stateless enough to tolerate the reset nuance. Real systems layer all three: anycast steers each user to the nearest PoP, DNS handles coarse region-level steering and disaster failover, and a VIP/LB does the fast in-site work behind it.

Pitfalls

• Trusting the TTL as your failover time — downstream resolver clamps, ISP overrides, browser pinning, and open keep-alive sockets routinely stretch it to minutes.
• Setting TTL to 1s to “fix” it — many resolvers ignore sub-minute TTLs anyway, and you've just multiplied query load on your DNS for little gain.
• Running long-lived TCP (databases, WebSockets, big transfers) directly over a churny anycast prefix and being surprised by resets on route changes.

Takeaways

• DNS failover steers at the naming layer; its real speed is detection + resolver cache + client pin + open-connection lifetime — minutes, not the TTL.
• Anycast steers at the routing layer via BGP; failover is a route withdrawal that reconverges in seconds with no client re-resolution.
• Anycast is ideal for stateless / short / UDP traffic (DNS, CDN edge); a mid-connection route change can RST a long-lived TCP flow, so keep anycast frontends stateless or use it only to steer.
• Pick by scope and RTO: VIP inside a site, DNS across regions when minutes are fine, anycast for fast global latency-aware entry — and layer them.

---

Re-authored for this guide; DNS-delay and anycast diagrams hand-authored as SVG. Follows Cloudflare's and the wider CDN literature on anycast and BGP, Google SRE practice on DNS-based load balancing, and standard resolver/TTL behavior (RFC 1035, browser DNS caching). See also: Load Balancing, DNS, TCP Deep, Designing for Failure. Deepened for this guide.

I just read a lesson on **DNS Failover & Anycast** (System Design) and want to truly understand it. Explain DNS Failover & Anycast from first principles using ONE vivid real-world analogy and a visual mental model — draw it as ASCII art or a clear step-by-step diagram — with a concrete example using real numbers. Then ask me one question to check I got the mental picture, and wait for my reply. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Teach me **DNS Failover & Anycast** interactively. Ask me ONE guiding question at a time, wait for my answer, and adapt to my confusion — build the idea with me step by step instead of explaining it all at once. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Quiz me on **DNS Failover & Anycast** with 5 questions, easy to tricky, ONE at a time. Tell me if each answer is right; at the end, explain clearly what I got wrong and why. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Help me remember **DNS Failover & Anycast** for the long term: give the one-sentence intuition, a memorable hook/mnemonic, a tiny worked example, and 3 active-recall flashcards (Q -> A). If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.