Distributed & Network Gotchas — Schema Evolution, WebSocket Reconnect, Exactly-Once Sink, Serverless Cost, Session-Loss & DoS Mechanics (Deep Dive)

These are seven senior-level “gotcha” questions that show up across system-design and backend interviews because each one hides a race, an asymmetry, or a false assumption behind an innocent-sounding scenario. They span topics — serialization, networking, streaming, concurrency, cost, caching, security — but they share a shape: something looks safe (“the schema just gained a field,” “the write got an ack,” “the idempotency key stopped duplicates”) until you trace the exact bytes or the exact interleaving and find the gap. Where relevant this page cross-references the guide’s existing DDoS Attacks page (SYN backlog economics), the Exactly-Once is a Myth page (the dedup-race pattern, reused here at the sink boundary), and the Distributed Cache Cluster page (replication lag and failover). The goal is the same every time: name the mechanism, not just the symptom.

1. Schema evolution: adding a field without breaking old clients

The mechanism that makes a schema change safe is always the same question: when an old reader meets new data, or a new reader meets old data, what happens to the field that only one side knows about? The three common formats answer that question completely differently, because each anchors its compatibility contract on a different thing — field names, field numbers, or a pair of schemas.

The one rule that generalizes across all three: never let “the same identifier” mean two different things over time — don’t reuse a Protobuf tag, don’t repurpose a JSON key for a different type, don’t drop an Avro default. That single discipline is what makes “just add a field” actually safe.

The compatibility modes (Confluent Schema Registry framing, but the concept generalizes)

• Backward compatible: a consumer running the new schema can read data produced under the previous schema. This is what lets you upgrade consumers before producers — the common, safest rollout order.
• Forward compatible: a consumer still running the previous schema can read data produced under the new schema. This is what lets you upgrade producers before every consumer has caught up.
• Full compatible: both directions hold — either side can be upgraded first without breaking the other. This is the safest but most restrictive mode (e.g. new fields must have defaults, nothing gets removed without one).
• Transitive variants (backward-transitive, etc.) check the new schema against every previous version on file, not just the immediately prior one — important once a topic has had several revisions and old data is still being replayed.

A schema registry (Confluent-style, used with Avro or Protobuf) exists to enforce one of these modes automatically: it rejects a schema registration that would violate the configured compatibility mode, instead of discovering the break in production when an old consumer chokes on a replayed message.

2. The WebSocket reconnect gap: where did the 10 seconds of messages go?

A raw WebSocket is just a pipe — it has no memory of what passed through it before the current connection existed. If the pipe breaks and reopens, the new connection starts from nothing; whatever the server pushed while the pipe was down is gone unless something outside the socket remembers it. That something is a per-connection (or per-session) sequence number paired with a server-side buffer the client can replay from.

Traced: client drops for 10 seconds

The two moving parts are inseparable: the sequence number is what lets the client prove precisely what it last saw (so the server replays the exact gap, no more, no less — no duplicate replay, no silent gap), and the durable buffer is what lets the server answer “what happened between 104 and now” even though the socket that would have carried it is gone. If the client was offline longer than the buffer’s retention window, the server can’t answer precisely — the correct behavior is to tell the client explicitly (“gap exceeds retention, resync from source of truth”) rather than silently resuming and leaving an undetected hole. An equivalent pattern: instead of a ring buffer, give the client a durable inbox (e.g. a per-user outbox table or queue) it re-reads from its own last cursor on every reconnect — same idea, framed as pull instead of push-replay.

3. Stream-to-sink exactly-once: the sink has to cooperate

A stream processor can guarantee it will not lose or skip an input record — that’s just checkpointing the consumer offset after (or atomically with) processing, then replaying from the last checkpoint on restart. What it cannot unilaterally guarantee is that the corresponding output write to an external sink happens exactly once, because a crash between “wrote to the sink” and “committed the checkpoint” forces a replay that writes to the sink again. There are exactly two ways to close that gap, and both require the sink’s help:

• Atomic co-commit (two-phase commit style). The checkpoint update and the sink write are committed as a single atomic unit — the sink participates in the same transaction as the stream’s offset commit (this is how Kafka-to-Kafka exactly-once works: the transactional producer atomically writes output records and advances the consumer group’s offset). This requires the sink to support participating in a distributed/atomic transaction with the stream’s checkpoint store — true for another Kafka topic, rare for an arbitrary HTTP API or a database with no XA/2PC support.
• Idempotent (offset-keyed) sink — the far more common answer. Let the checkpoint and the write be non-atomic, accept that a crash can cause the write to replay, and make the replay a no-op: the sink keys the write by the source offset (or an id derived from it) and applies it exactly the same atomic-CAS discipline as §4 below — “insert this offset’s result if it hasn’t been applied yet, else skip.” Concretely: an upsert keyed by a stable event id instead of an append, or a processed_offsets table with a unique constraint checked in the same write transaction as the actual sink mutation.

Either way, the underlying insight is identical to the idempotency-key problem below — “exactly-once” is never really achieved by making delivery exactly-once (impossible, per the guide’s Exactly-Once is a Myth page); it’s achieved by making a duplicate delivery harmless at the point where it would otherwise cause damage. Here that point is the sink’s write path, not the consumer’s read path.

4. Idempotency-key concurrency: how the second worker is actually stopped

Job #8842 gets picked up twice — a queue visibility timeout expired while worker A was still mid-processing, so worker B also claimed it, or a duplicate enqueue happened upstream. Both workers reach the “send confirmation email” step at roughly the same time. An idempotency key only prevents the double-send if the check that “have I sent this already?” is atomic — a plain read-then-write check is a race, not a guard.

Traced: the naive race vs. the atomic fix

Both reads ran before either write — the classic read-modify-write hazard, the same shape as a lost counter++ update. The fix moves the “have I claimed this?” check into the storage layer itself, as a single atomic operation that only one caller can win:

-- Redis: only one caller's SETNX returns 1
SETNX key:8842 "claimed"   -- worker A: returns 1 (won the claim) -> proceed to send
SETNX key:8842 "claimed"   -- worker B: returns 0 (key already exists) -> skip, do not send

-- equivalently, at a SQL store:
INSERT INTO idempotency_keys(job_id) VALUES (8842)
  ON CONFLICT (job_id) DO NOTHING;  -- unique constraint decides the winner
-- rows_affected = 1 -> you won, send the email (same transaction)
-- rows_affected = 0 -> someone already claimed it, skip

The database (or Redis, atomically) serializes the two contenders on the key itself, so exactly one SETNX/INSERT can succeed — the decision is made by the storage engine, never by a prior read the caller performed. This is the identical CAS (compare-and-swap) discipline used at the sink boundary in §3.

5. Serverless cost reality: the bill is rarely dominated by compute

The pitch for serverless is “pay only for the compute you use,” so teams model cost as invocations × duration × memory and are then surprised when the actual AWS bill looks nothing like that estimate. The mechanism: a request almost never touches only the function’s compute meter — it also pays for everything that gets it to the function and everything the function reaches out for.

• API Gateway, priced per request (roughly $1–$3.50 per million requests depending on API type and tier) — a pure per-request tax that scales with traffic regardless of how little work each Lambda invocation does. At high request volumes with cheap, fast functions, this line item can dwarf the Lambda compute line entirely.
• Data transfer and NAT gateway hops. A function placed inside a VPC (common once it needs to reach a private RDS instance or internal service) that makes outbound calls routes through a NAT gateway, which bills for data processed — a cost that has nothing to do with CPU time and everything to do with how chatty the function is.
• The externalized-state tax. Because a function instance is stateless between invocations (and may be a fresh cold instance on top of that), any state it needs — config, feature flags, session data, a cache lookup — requires a network round trip on every single invocation to fetch what a long-lived server process would just be holding in memory. Each of those round trips is itself a billed request against whatever datastore holds the state, multiplying the request-based cost model rather than the compute-based one.

The mental model to carry into a cost estimate: serverless cost is dominated by the number of network hops a request causes — API Gateway’s per-request fee, NAT/data-transfer for anything crossing a VPC boundary, and one round trip per piece of externalized state — not by how many CPU-milliseconds the handler itself burns. A function that does 5ms of real work but makes three external calls to reconstitute its state will often cost more, per invocation, than a function that spins the CPU for 200ms with no external calls at all.

6. Session-loss on Redis failover: the acked write that never replicated

Sequence: the client writes a new session (say, right after login), Redis’s primary applies it and returns an ack, and then the primary dies before that write has propagated to any replica. A replica is promoted (Sentinel or Cluster failover) — but it never received that write, so the session simply does not exist on the new primary. The user, who was just told “logged in successfully,” is logged out on the very next request.

The mechanism is that Redis replication is asynchronous by default: the primary acknowledges a write to the client as soon as it applies the write locally, and streams the command to replicas afterward, off the critical path, for latency reasons. That ordering — ack-then-replicate rather than replicate-then-ack — is exactly what makes the write’s durability conditional on the primary surviving long enough to finish propagating it. (The guide’s Distributed Cache Cluster page covers the mirror-image symptom of the same asynchrony: a read routed to a replica right after a write to the primary can return the old value, because the replica hasn’t caught up yet.)

Mitigations — none of them make it free

• WAIT (synchronous-ish acknowledgment). The client issues WAIT numreplicas timeout after the write, blocking until at least numreplicas replicas confirm they applied it (or the timeout elapses) before treating the write as durable. This narrows the loss window a great deal but is not a true guarantee — a promoted replica can still fail to be one of the replicas that acknowledged, and the timeout path re-opens the race entirely if hit.
• Put the session in a durable, non-cache store (a primary database with real replication/durability guarantees, or a managed store with synchronous replication) and use Redis purely as a read-through cache in front of it, not as the source of truth. This removes the async-replication risk entirely for the data that matters, at the cost of the latency Redis existed to avoid.
• Tolerate and design for occasional re-login. If the product can absorb an infrequent forced re-auth (rare failover events, short session lifetimes anyway), treat this as an accepted trade-off rather than an engineering problem to eliminate — often the pragmatic answer once you’ve sized how rare primary failover actually is.

7. DoS mechanics: three distinct classes, three distinct defenses

The guide’s DDoS Attacks page covers volumetric floods (SYN flood backlog exhaustion, UDP reflection/amplification) in depth. There is a third class worth naming explicitly because it exhausts neither bandwidth nor a fixed-size table by brute force: slow-connection exhaustion, plus the specific mechanics of the SYN-cookie defense and a fourth-ish variant, TLS-handshake CPU asymmetry.

SYN cookies: encoding state into the sequence number so there is no state to hold

A SYN flood works because the server allocates a backlog slot the instant it sees a SYN, before the handshake is proven genuine. SYN cookies remove the thing being exhausted: instead of allocating a slot and remembering “I sent a SYN-ACK to X, waiting for its ACK,” the server encodes everything it would have remembered directly into the 32-bit sequence number of the SYN-ACK itself — a keyed hash (MAC) of the connection’s 4-tuple (source/dest IP and port), a secret known only to the server, a coarse time window (so cookies expire), and a few bits for the negotiated MSS. It sends that crafted sequence number and then forgets the connection attempt entirely — zero backlog entry, zero memory held.

When the real client answers with its final ACK, the ack number it must send back is exactly ISN+1 — which only a client that actually received the SYN-ACK (i.e. actually owns the claimed source IP) can produce. The server, on receiving that ACK, recomputes the same hash from the packet’s own 4-tuple and the current time window and checks it matches the returned value; only at that point, having verified the round trip really happened, does it allocate the real connection state (recovering the negotiated MSS from the encoded bits). A spoofed SYN flood never produces this ACK — the attacker doesn’t control the spoofed IP’s inbox, so it never sees the SYN-ACK to answer — meaning the server verifies nothing and stores nothing for every spoofed packet, no matter how many arrive. The trade-off: because only a limited number of bits are available in a sequence number, SYN cookies can’t faithfully encode every TCP option (large window scaling in particular can get dropped or reduced while cookies are active), which is the same limitation noted on the DDoS page.

Slowloris: exhausting the connection pool with almost no bandwidth

Slowloris is a third DoS class, distinct from both volumetric (saturate the pipe) and classic application-layer (flood with complete, valid-looking requests) attacks: it exhausts the server’s concurrent-connection pool using barely any bandwidth at all. The attacker opens many HTTP connections and sends a valid but incomplete request — a partial header, or headers with no terminating blank line — then trickles a few more bytes every so often, just often enough to stay under the server’s read/idle timeout, so the connection is never considered dead and never completes either. Each such half-open request occupies one worker thread, process, or connection slot for as long as the attacker keeps trickling bytes. A thread-per-connection server with a modest worker pool (historically the failure mode Apache’s prefork MPM hit) can be fully saturated by a few hundred to a few thousand such slow connections — trivial for a single attacking machine, no botnet or amplification required, because the attack is priced in held connections, not packets per second.

TLS-handshake CPU asymmetry

A TLS handshake requires the server to perform genuinely expensive asymmetric-crypto operations (key exchange, certificate signing/verification) for every incoming attempt, while triggering one costs the client comparatively little. A flood of handshake initiations (or repeated renegotiation requests on already-open connections) forces the server to burn real CPU cycles on cryptographic math per attempt — exhausting compute rather than memory (SYN flood) or connection slots (Slowloris). This is why TLS termination at scale is commonly offloaded to dedicated hardware or a CDN/load balancer designed to absorb exactly this asymmetry.

Mitigations, matched to the mechanism

• SYN flood → SYN cookies (above) — solves backlog exhaustion specifically; does nothing for the other two classes.
• Slowloris → aggressive header/read timeouts, a cap on concurrent connections per IP, and an event-driven reverse proxy (nginx-style, handling many idle-ish connections cheaply) placed in front of a thread-per-connection app server, so the proxy fully buffers a request before ever handing it to a worker thread.
• TLS CPU exhaustion → rate-limit handshakes per IP, enable session resumption/tickets (skips the full asymmetric handshake cost on reconnects), disable client-initiated renegotiation, and offload termination to hardware or a CDN built to absorb the asymmetry at scale.

Pitfalls

• Schema evolution: reusing a Protobuf field number after “removing” a field, or repurposing a JSON key for a different type — both silently reinterpret old bytes as the wrong value instead of failing loudly.
• WebSocket reconnect: resuming with “give me anything new” instead of an explicit last-seen sequence number — without the exact cursor, the server cannot tell the client precisely what it missed, so it either re-sends duplicates or silently drops the gap.
• Exactly-once sinks: committing the checkpoint before confirming the sink write succeeded — a crash in between causes the write to be silently skipped on replay, the opposite failure mode from the duplicate-write case.
• Idempotency keys: recording the claim in a store that isn’t transactional with the effect (e.g. Redis key, Postgres side-effect) — the two can drift if one commits and the other rolls back.
• Serverless cost: estimating spend from Lambda’s pricing page alone and ignoring API Gateway, NAT, and per-invocation state round trips — the estimate can be off by an order of magnitude at scale.
• Redis failover: assuming an ack from Redis means the write is durable across a failover — async replication means acked and durable are not the same claim.
• DoS defenses: deploying only SYN cookies and declaring the service “DoS-hardened” — they do nothing against Slowloris or TLS-handshake exhaustion, which need entirely different mitigations.

The judgment layer

Protobuf vs Avro vs JSON, when to pick which. Reach for Protobuf when you control both ends of an RPC boundary and want compact, fast, statically-typed wire messages with strong compile-time contracts (gRPC services). Reach for Avro when you’re writing to a shared, long-lived log (Kafka topics with many independent consumer teams) where a central schema registry enforcing compatibility modes matters more than per-message size, and where the writer/reader schema-resolution model handles evolution across many historical versions gracefully. Reach for JSON when human-readability and tooling ubiquity (browsers, curl, ad-hoc debugging) outweigh compactness and you’re willing to enforce compatibility by convention and code review rather than by the format itself.

2PC-style atomic sink commit vs idempotent offset-keyed sink. Prefer the atomic co-commit only when the sink is capable of participating in one transaction with the stream’s own checkpoint store (same-system cases like Kafka-to-Kafka) — it’s strictly stronger but narrowly available. For everything else (an arbitrary external API, a database without cross-system transaction support), the idempotent/offset-keyed sink is the practical default: it costs a bit of extra sink-side bookkeeping (a dedup table or an upsert key) but works with any sink that can do a conditional write.

Redis WAIT vs a durable store vs tolerating re-login. WAIT is the right dial to reach for first because it’s a one-line change with no architectural cost — use it when the data is important but a total redesign is not justified. Move to a genuinely durable store when the data (payment state, order state) truly cannot be allowed to vanish, accepting the latency Redis was there to avoid. Tolerating occasional re-login is the right call only after you’ve actually measured how rare primary failover is in your environment — treating a once-a-year event as demanding a full durable-store migration is over-engineering; treating a frequent one that way is under-engineering.

Takeaways

• Every “evolve this safely” question reduces to: what happens when one side knows about a field/element the other doesn’t — Protobuf answers it with never-reused tag numbers, Avro with schema resolution + defaults, JSON by convention only.
• A socket with no memory needs an external one: WebSocket gap recovery is a sequence number plus a durable buffer/inbox the client can resume from precisely, not a vague “send me anything new.”
• “Exactly-once” is never delivery becoming exactly-once (impossible); it’s a duplicate being made harmless at the point of impact — the sink’s write path here, the same atomic-claim mechanism as the idempotency-key race.
• Read-then-write is a race whenever two workers can interleave; the fix is always to push the “have I already done this?” check into an atomic operation the storage layer serializes (SETNX / ON CONFLICT), never a prior SELECT/GET.
• An acked write is not a durable write once asynchronous replication is in the picture (Redis failover, or any primary-replica system with async replication) — durability claims and ack claims must be evaluated separately.
• DoS is not one problem: memory exhaustion (SYN flood), connection-pool exhaustion (Slowloris), and CPU exhaustion (TLS handshake asymmetry) each need a defense matched to the resource actually being drained.

Related pages

• Kafka — Delivery Semantics, Exactly-Once & Producer Back-Pressure (Deep Dive) — Kafka's own exactly-once sink mechanics in depth
• Difference Between LongPolling, WebSockets, and ServerSent Events — compares WebSockets against alternative transports
• Session & Cookie Security — session durability and security concerns
• Serverless Architecture vs Traditional Serverbased — broader serverless cost/architecture trade-offs
• DDL Locking & Online Schema Change — schema evolution in relational databases, a parallel mechanism