hard Scale-In Safety & Connection Affinity

Scaling in is the dangerous direction

Scaling out adds a fresh instance that starts empty and takes new work — nothing can break. Scaling in deletes an instance that is right now holding live TCP connections and half-finished requests, so unless you first stop new traffic and let the in-flight work finish, those requests are severed mid-flight and the client gets a 5xx. The whole discipline of safe scale-in is one rule: deregister, drain, then terminate — never terminate first.

The drain mechanism, layer by layer

“Draining” means: remove the instance from the load balancer’s rotation so it receives no new requests, but keep its existing connections alive until they complete (or a timeout fires), and only then kill the process. Every layer implements the same idea:

• Load balancer — connection draining / deregistration delay. On AWS ALB/NLB a deregistered target enters the draining state: the LB routes no new requests to it but lets in-flight ones finish, up to the deregistration delay (default 300s). Only after that does the target leave the pool.
• Kubernetes — endpoint removal + preStop + SIGTERM + grace period. Deleting a pod does two things at once: it is removed from the Service’s Endpoints (so kube-proxy / the ingress stop sending it new traffic), and its preStop hook fires, then the container gets SIGTERM. The app should catch SIGTERM, stop accepting connections, and finish in-flight work before exiting. If it is still alive after terminationGracePeriodSeconds (default 30s), the kubelet sends SIGKILL — a hard, un-drainable kill.
• The app itself. A graceful shutdown means: on SIGTERM, close the listening socket (refuse new connections), let handlers in progress return, flush and close, then exit 0.

The infamous race: endpoint removal is asynchronous. For a brief window after SIGTERM, the LB/kube-proxy may not have learned the pod is gone and can still send it new requests — which now hit a socket that is closing, producing 502s. The standard fix is a preStop hook that sleeps a few seconds: keep serving normally while deregistration propagates, then begin shutdown.

Traced scale-in timeline (drain vs hard kill)

Autoscaler removes instance B (3 → 2). B is serving 12 in-flight requests. Compare the two paths step by step:

t	Graceful drain	Hard kill (terminate first)
0.0s	B deregistered from LB → no new requests routed to B	B receives SIGKILL immediately
0.0s	preStop sleep 5s lets deregistration propagate; B keeps serving its 12	12 open connections severed → clients get TCP RST
0–18s	SIGTERM; B finishes in-flight, in-flight count 12 → 0	12 requests fail → 502s; clients retry
~18s	process exits 0; B terminated cleanly → 0 errors	retries pile onto A & C; if they were already near capacity, they shed load → 503s (retry storm)

The drain path never has a moment where a request is routed to a gone instance, so it avoids the 502/503 window entirely. The hard-kill path converts a routine scale-in into a burst of user-visible errors — worse, the retry traffic can tip the surviving instances over, so a capacity reduction becomes an outage.

Affinity decides what a scale-in costs

If every instance is stateless, draining is the whole story. But the moment a client is pinned to a specific backend — “affinity” — removing that backend can destroy state the client depended on. There are two families, and they behave very differently under scale-in.

LB-issued (cookie / sticky) affinity

The load balancer pins the client: on the first response the LB sets its own cookie (e.g. ALB’s AWSALB, or an app cookie like JSESSIONID) that encodes which target to use, and thereafter routes that client to the same backend. This is how “keep the user on the node that holds their in-memory session” is implemented. On scale-in, if the pinned target is the one removed, the LB simply re-pins the client to a survivor — but the in-memory session on the dead node is gone. The user is logged out, their half-built cart evaporates, their WebSocket drops. Sticky sessions turn a stateless-looking service into a stateful one whose state disappears with the instance.

Client / consistent-hash affinity

Here the client (or a hash ring) picks the backend from a key: node = ring.lookup(hash(user_id)). With consistent hashing, removing one node reassigns only that node’s share of the keyspace (~1/N of keys) to its clockwise neighbour; every other key stays on the same node. Contrast naive hash(key) % N: dropping from N to N−1 servers changes the modulus, so almost every key maps somewhere new — a mass cache miss / re-shuffle. That remap-only-1/N property is exactly why consistent hashing is the affinity of choice for caches and sharded stores that scale in and out routinely.

Pitfalls engineers actually hit

• Grace period shorter than the longest request. A 30s grace with 60s report-generation requests means the slow ones get SIGKILLed anyway. Size the grace period (and LB deregistration delay) to the p99 request duration, or those requests still drop.
• App ignores SIGTERM. If your process doesn’t trap SIGTERM and close the listener, it runs until SIGKILL — draining does nothing. Frameworks don’t all do this for you.
• The deregistration race. Skipping the preStop sleep leaves the window where new traffic still arrives at a closing pod → sporadic 502s during every scale-in that look like a flaky app.
• Long-lived connections never drain. WebSockets / gRPC streams / SSE can stay open indefinitely; connection draining waits forever (or hits the timeout and cuts them). You need app-level shutdown signalling (e.g. gRPC GOAWAY) to ask clients to reconnect elsewhere.
• Sticky sessions hide statefulness. Everything looks fine until an autoscale-in (or a spot reclaim, or a deploy) removes the pinned node and a slice of users is logged out — an intermittent bug that never reproduces in staging.

Judgment: how a senior engineer decides

Connection draining vs hard kill

Drain whenever requests are user-facing or non-idempotent — the cost is a bounded shutdown delay (seconds to minutes) and slightly slower scale-in, and you gain zero dropped requests. Hard kill is acceptable only when work is fully idempotent and cheaply retriable and shutdown speed matters more than a few reset connections (e.g. a stateless batch worker pulling from a queue that redelivers on no-ack). For anything serving HTTP to users, hard kill is the wrong default — it trades a few seconds of patience for user-visible 5xx and a possible retry storm.

Sticky (cookie) sessions vs stateless+external store vs consistent-hash affinity

• Sticky cookie sessions — cheapest to bolt on (flip a flag on the LB), no external dependency, and locality is great while the node lives. Cost: you are now stateful and scale-in / instance loss = lost sessions. Choose only for low-stakes state you can afford to lose, or as a stopgap.
• Stateless + external session store (Redis/DB) — the default for correctness. Any instance can serve any request, so scale-in is free and drains cleanly; the cost is a network hop per request to fetch session and an external store to run and keep available. Choose when losing a session is unacceptable (auth, checkout).
• Consistent-hash affinity — the pick when you want locality for performance (in-process cache, sharded state, sticky routing to the node that owns a key) but must survive membership changes. You keep warm caches through scale events because only ~1/N of keys move; the cost is more moving parts (a ring, virtual nodes, rebalancing) and that the ~1/N of keys whose owner left still miss/rebuild. Choose for caches, sharded stores, and stateful stream processors.

Rule of thumb: reach for stateless + external store first (it makes scale-in a non-event); use consistent-hash affinity when locality is a real performance win; treat sticky cookies as a last resort for disposable state.

Takeaways

• Scale-in is riskier than scale-out because you are deleting live in-flight work — always deregister → drain → terminate, never terminate first.
• Draining is the same idea at every layer: LB deregistration delay, K8s endpoint-removal + preStop + SIGTERM + grace period, and an app that shuts down gracefully. Mind the async-deregistration race (the preStop sleep).
• Cookie/sticky affinity makes in-memory sessions die with the instance; consistent-hash affinity remaps only ~1/N of keys, so caches stay warm across scale events.
• Prefer stateless + external session store; use consistent-hash affinity for performance locality; keep sticky sessions for state you can afford to lose.

---

Synthesized from AWS ELB/ALB target-deregistration & connection-draining docs, the Kubernetes pod-termination lifecycle (preStop / SIGTERM / terminationGracePeriodSeconds), Karpenter / cluster-autoscaler graceful-shutdown guidance, and consistent-hashing treatments in Designing Data-Intensive Applications (Kleppmann) and the original Amazon Dynamo paper. Re-authored/Deepened for this guide.

I just read a lesson on **Scale-In Safety & Connection Affinity** (System Design) and want to truly understand it. Explain Scale-In Safety & Connection Affinity from first principles using ONE vivid real-world analogy and a visual mental model — draw it as ASCII art or a clear step-by-step diagram — with a concrete example using real numbers. Then ask me one question to check I got the mental picture, and wait for my reply. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Teach me **Scale-In Safety & Connection Affinity** interactively. Ask me ONE guiding question at a time, wait for my answer, and adapt to my confusion — build the idea with me step by step instead of explaining it all at once. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Quiz me on **Scale-In Safety & Connection Affinity** with 5 questions, easy to tricky, ONE at a time. Tell me if each answer is right; at the end, explain clearly what I got wrong and why. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Help me remember **Scale-In Safety & Connection Affinity** for the long term: give the one-sentence intuition, a memorable hook/mnemonic, a tiny worked example, and 3 active-recall flashcards (Q -> A). If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.