Cloud Storage & Data Movement — S3 ETag/Cost/Lifecycle, CDC Replication-Slot Back-pressure & Read-Your-Writes (Deep Dive)

The common thread: guarantees that either cost money or that someone downstream must honor

Object storage, change-data-capture, and read-your-writes look unrelated, but each one is the same trade dressed differently: you can make a guarantee cheap, fast, or safe to ignore — pick at most two. Cheap storage defers cost onto the read you didn't plan to make; a low-overhead CDC pipeline defers back-pressure onto the primary database it claims not to touch; and a scaled-out read path defers "did my own write happen" onto the client, unless something explicit routes around the lag. This page works the arithmetic and traces the race/failure in each case — the stampede, idempotency-key, retry-amplification, and dual-write/outbox mechanics that recur here already have their own deep pages elsewhere in this guide; this page adds the storage- and data-movement-specific angle on top of them.

1. Storage-tier cost arithmetic: hot vs. cold, and the retrieval-fee blowup

A "cold" tier (Glacier / Archive-class storage) is cheap for one mechanical reason: the provider stores your bytes on denser, slower media with fewer standby replicas ready to serve a read instantly — retrieving them means re-hydrating onto fast media first, which costs a per-GB retrieval fee and takes minutes to hours, not milliseconds. You are not paying less for the same service; you are pre-paying with latency and later paying with a retrieval fee for the same bytes.

Worked example (illustrative, rounded list-price figures — the arithmetic pattern is the lesson, not today's exact price sheet). Take a 100 TB (100,000 GB) dataset kept for one year:

Moving it to Deep Archive saves $27,600 − $1,188 = $26,412/year — a real win, but only if you rarely need the data back. A full restore at the "standard" retrieval speed (≈12h) runs ≈$0.02/GB, so one full-dataset restore costs 100,000 × 0.02 = $2,000. Now ask the question tiering guides never ask: how many full restores per year before Deep Archive stops being the cheaper option? Set the two annual costs equal:

The break-even is ≈13 full restores/year — roughly once a month. That is the retrieval-fee blowup pitfall stated as arithmetic: archived data is cheap to hold and expensive+slow to read, so any workload that reheats the same cold data more than about once a month should simply not have been archived, and one panicked mass-restore (a legal hold, an ML re-training run, a mis-scoped audit) can wipe out a year of "savings" in a single event — before even counting the 12–48h wait.

A second, separate surprise is the minimum-storage-duration charge: Deep Archive bills a minimum retention window (illustratively ≈180 days) — delete or transition an object before that window elapses and you are still billed for the remaining days at the archive rate, as if you'd kept it. Delete a 100,000 GB Deep-Archive dataset at day 30 of a 180-day minimum: you owe the remaining 150 days regardless — at ≈$0.000033/GB-day that's 100,000 × 150 × 0.000033 ≈ $495 for data you no longer have, on top of the $99 you already paid for the 30 days it actually sat there. Trivial for one object; real money when a lifecycle rule is mis-tuned across a whole fleet of buckets.

When time-based lifecycle is the wrong tool. Rule-based lifecycle transitions (e.g. "move to Glacier after 90 days") assume access temperature falls monotonically with age. That assumption breaks on bimodal / re-heating access: a three-year-old video goes viral, a closed legal case is re-opened, an ML dataset gets pulled back for a retraining run. Age-based rules will have already archived that object, so the very moment it goes hot again is the moment you eat the retrieval fee and the multi-hour wait. S3 Intelligent-Tiering exists for exactly this: it monitors real access per object and auto-promotes/demotes tiers, for a small per-object monitoring fee, so temperature — not age — decides the tier. Use time-based rules only when cool-down is genuinely monotonic (logs, backups, completed-order archives); use access-pattern tiering whenever the distribution is unpredictable or known to be bimodal.

2. The S3 ETag gotcha: why md5sum file stops matching the ETag

For a single-part upload, S3's ETag genuinely is the MD5 digest of the object's bytes (for objects without SSE-C/SSE-KMS encryption applied), e.g. "9b2cf535a4b1e6d0f2a8c3d7e5f10abc" — so md5sum myfile equals it, and a lot of tooling grew up assuming that always holds.

It stops holding for a multipart upload (mandatory above 5GB, and used by default by most SDKs/CLIs well below that). S3 does not hash the whole object for a multipart ETag — it cannot, because it never sees the object as one contiguous stream. Instead:

1. Each part (you supply, up to 10,000 of them) gets its own MD5 digest, computed independently as the part arrives.
2. S3 concatenates the raw 16-byte binary digests of every part, in part order.
3. It MD5-hashes that concatenation, and appends -N, where N is the number of parts.

So a 4-part upload might produce an ETag like "a1b2c3d4e5f60718293a4b5c6d7e8f90-4" — a hash of hashes, not a hash of bytes. This is not a bug or an inconsistency to work around; it is a categorically different computation, so md5sum myfile will never equal a multipart ETag, no matter how many times you recompute it. A pipeline that asserts local_md5 == etag as an integrity check will report every large object as "corrupted" and be wrong every time.

A second, independent exception: server-side encryption with SSE-C or SSE-KMS also breaks the ETag=MD5 assumption, even for a single-part upload with no chunking involved at all — the ETag is simply not an MD5 digest of the object data once that encryption is applied. This holds regardless of part count, so a pipeline that special-cases only "large/multipart objects" and assumes small single-part objects are always safe will still be wrong for any encrypted bucket.

The fix is to stop overloading the ETag as an integrity primitive at all: request one of S3's explicit additional checksums at upload time — x-amz-checksum-sha256 (or -crc32/-crc32c/-sha1) — which S3 stores and returns independent of how the object was chunked or encrypted, and which your client can recompute and compare directly. This is the fix for encrypted objects too, not just multipart ones. Where you must stay ETag-based (matching against an existing store, no re-upload), the alternative is to replicate S3's exact multipart algorithm client-side (same part boundaries, same MD5-of-MD5s, same -N suffix) — workable, but brittle to any change in part size, and does not help at all for encrypted objects.

3. The conditional-write race: compare-and-swap over HTTP

A plain PUT has no notion of "based on what version" — S3 just overwrites whatever bytes are currently there. Trace the race two clients hit with no conditional check:

Nothing failed, nothing errored — B's PUT simply overwrote A's, and neither client nor any log entry records that a write vanished. This is the classic lost update.

Conditional PUT (If-Match) fixes it by turning the write into optimistic-locking compare-and-swap: the client sends If-Match: "v7" alongside the PUT, and S3 checks the object's current ETag against that header before writing. Re-trace the same race with conditional writes on:

The second writer now gets a loud, detectable conflict instead of a silent overwrite — the same shape as a database's optimistic-concurrency check, just expressed as an HTTP precondition.

Three "but why"s worth being able to answer cold:

• Why does plain PUT allow this at all? Because PUT's contract is "replace whatever is there," full stop — there is no implicit version check unless the client asks for one via If-Match/If-None-Match. Safety here is opt-in, not a default.
• Weak vs. strong ETags. A weak ETag (prefixed W/) only promises semantic equivalence ("close enough to serve a cached copy"), not byte-for-byte identity — the spec forbids using a weak ETag with If-Match or Range, because a CAS or a byte-range stitch needs a guarantee weak ETags don't make. Always use strong (content-hash) ETags for a compare-and-swap.
• ETag derivation across replicas. S3 itself derives ETags from content, so this specific failure doesn't happen inside S3 — but the moment you put a CDN or reverse proxy in front of heterogeneous origins that derive ETags from mtime/inode instead of a content hash, two backends serving byte-identical content can hand out different ETags. That silently breaks both 304 conditional GETs (needless re-fetches) and any If-Match CAS logic layered on top (spurious 412s). Derive ETags from content, not filesystem metadata, if you're building this yourself.

4. CDC replication-slot back-pressure: when a stalled consumer takes down the primary

A log-based CDC connector (Debezium, Postgres logical replication, MySQL binlog readers) holds a replication slot on the source database — a bookmark saying "don't discard any WAL/binlog after this point, I might still need it to resume." That is what makes CDC resumable and gap-free after a connector restart. It is also exactly the mechanism that turns a downstream problem into a primary-database outage.

If the consumer stalls — the sink is down, a poison event wedges the pipeline, Kafka is unreachable — the slot's confirmed position stops advancing. The primary keeps taking writes and keeps generating new WAL, and it cannot recycle WAL segments the stuck slot might still need, by design (that's the durability guarantee the slot exists to provide). WAL accumulates without bound on the primary's disk until the disk fills, at which point Postgres can no longer fsync new writes and effectively stops accepting them — or crashes. A consumer that was supposed to be a read-only, "non-intrusive" downstream tap has taken down the source of truth.

Mitigations, in the order they should actually be built: (1) monitor slot lag directly — bytes/LSN behind, not just "is the connector process alive"; a connector can be up and still not confirming. (2) Bound the retention — Postgres 13+'s max_slot_wal_keep_size caps how much WAL a slot can hold hostage; past that cap Postgres invalidates the slot itself rather than filling the disk, converting an unbounded outage into a bounded, recoverable one. (3) A drop-slot circuit breaker: an explicit, tested policy that proactively drops a slot once lag crosses a threshold, accepting a costly-but-bounded re-snapshot over an open-ended primary outage. (4) Treat connector liveness/lag as a tier-1 dependency of the source database's own health, not a downstream team's problem — because under this failure mode, it is the source database's problem.

This is the same territory as the dual-write problem that the Transactional Outbox / CQRS pages in this guide already cover in depth: outbox trades a bounded, pollable table (no slot, no back-pressure risk, slightly higher latency) for CDC's lower latency and lower source-write overhead — but CDC pays for that with exactly this back-pressure exposure on the source's disk. Reach for outbox when a stalled consumer must never be able to threaten the primary; reach for log-based CDC when latency matters more and you're willing to operate slot-lag monitoring as a first-class alert.

5. Read-your-writes: routing, tokens, and recovering ordering

Scale a read path across replicas and you buy throughput at the cost of a lag window — the replica a user's next read lands on may not have applied the write they just made. Nothing is wrong under the hood, but "I just posted this and it's gone" is a real, common, avoidable failure of user-visible correctness. Three concrete strategies, cheapest-and-narrowest to most general:

1. Route the writer's own reads to the primary for a short window after their write (or simply return the resulting state directly in the write response, so the client never has to re-read at all). Simple, but only covers "the same user, right after their own write" — it does nothing for a second user reading data the first user just changed.
2. Version/timestamp token. The write returns a monotonically increasing version or LSN; the client carries it forward (e.g. ?since=v42) on subsequent reads, and the read path routes to (or blocks briefly on) a replica whose applied position is ≥ that token — a "bounded staleness" read. This is the most general option: it composes across services and survives a client moving between requests/sessions, because the guarantee travels with the token, not with a sticky connection.
3. Sticky sessions — pin a user's whole session to one node/replica. Cheapest to reason about, but it couples correctness to node affinity: a failover or a load-balancer reshuffle silently breaks the guarantee, and it does nothing once the user's request legitimately needs to fan out to multiple services.

Recovering ordering where it matters follows the same shape as CQRS/CDC event streams: you almost never need global order, you need per-entity order. Partition the event/CDC stream by the entity's own key (e.g. orderId) so every event for one entity lands on one partition and is delivered in order, while events for different entities may freely interleave — that's harmless, because nothing downstream depends on cross-entity ordering. Reserve a single partition (and its throughput ceiling) only for the rare case that genuinely needs cross-entity order, like a double-entry ledger.

The idempotency gate on timeout retries is the sharpest edge in this whole area: a timeout is ambiguous — the request may have already succeeded and only the response was lost in transit. Blindly retrying a non-idempotent write on timeout (`POST /charge`) risks a duplicate side effect (double charge, duplicate order) precisely because you can't tell "it never ran" from "it ran and I never heard back." Safe retry on timeout requires either a naturally idempotent operation (PUT/DELETE/GET) or a client-supplied idempotency key the server dedupes against, returning the original result on replay rather than re-executing. GET/read paths can retry freely; write paths must not, unless gated this way — full mechanism and the double-charge trace live on this guide's idempotency/exactly-once page, and the layered-retry math that turns retries into a metastable overload (retry budgets, single-layer retry, jitter) is covered on the Circuit Breaker/Retry/Rate-Limiting page — reach for those for the derivation; the point here is that this same timeout ambiguity is exactly what a version-token read-your-writes retry must also respect (retry the read, not the write, on ambiguity). Likewise, a cache sitting in front of a read-your-writes path has its own coincident failure mode — many clients missing the same just-invalidated key at once (cache stampede) — with its own single-flight/soft-TTL fix covered on this guide's caching pages; the two problems compound if a version-gated read miss also fans out as an uncoalesced stampede of DB reads.

Pitfalls

• Asserting md5sum file == ETag as an integrity check on any object that might have been multipart-uploaded — it will falsely flag correct large objects as corrupted.
• Time-based lifecycle rules on data with bimodal/re-heating access — the archived object is exactly the one that goes viral, and you pay the retrieval fee at the worst possible moment.
• Deleting or re-transitioning an object before its minimum storage duration elapses — you're still billed for the remainder, for data you no longer have.
• Writing with a plain, unconditional PUT on any object more than one writer can touch — silent lost update, with no error and no log trail.
• Using a weak (W/) ETag with If-Match or Range — the spec forbids it because weak ETags don't promise byte-identity, which a CAS or range-stitch requires.
• Monitoring "is the CDC connector process alive" instead of its actual slot lag — a wedged-but-running connector still fills the primary's disk.
• Retrying a non-idempotent write after a timeout with no idempotency key — the timeout doesn't tell you whether the first attempt already succeeded.
• Reaching for a single global ordering guarantee (one partition, one queue) when only per-entity order is actually required — a needless throughput ceiling.

Judgment layer

Lifecycle rules vs. Intelligent-Tiering: use fixed time-based transitions when the cool-down is genuinely monotonic and predictable — logs, backups, completed-order archives — because they're deterministic and carry no per-object monitoring fee. Reach for Intelligent-Tiering the moment access is unpredictable or known to be bimodal (user-generated content, datasets that can be reopened for audits or retraining) — the small monitoring fee buys automatic promotion instead of a reactive, expensive mass-restore discovered the hard way.

Choosing a read-your-writes strategy: reach for a primary-read window (or "return state in the write response") first — it's the cheapest fix and covers the dominant case of a user re-reading their own recent write. Reach for a version/timestamp token when the guarantee has to survive across services or session boundaries, or when you need "read at least as fresh as write X" rather than just "read my own write" — it's more plumbing but composes correctly under failover and multi-service fan-out, which sticky sessions do not. Reserve sticky sessions for the narrowest, single-service, sessionless-alternative-is-worse case, and know going in that a failover event will quietly break the guarantee it provides.

Takeaways

• Cold storage isn't just "cheaper" — model it as storage-cost saved minus (retrieval-cost × expected reheats); past a computable break-even, archiving loses.
• A multipart S3 ETag is a hash of part-hashes with a -N suffix, not md5(file) — verify integrity of large objects with an explicit additional checksum (x-amz-checksum-sha256), not the ETag.
• Conditional PUT (If-Match) is compare-and-swap over HTTP: it turns a silent lost update into a loud, retryable 412 — but only with strong, content-derived ETags.
• A replication slot trades resumability for a back-pressure risk the source pays for if the consumer stalls; monitor lag directly and cap retention rather than trusting a liveness check alone.

Related pages

• Transactional Outbox — Solving the Dual-Write Problem — the alternative to CDC that trades latency for removing the replication-slot back-pressure risk in §4.
• Idempotency & “Exactly-Once Is a Myth” — the full double-charge trace behind the timeout-retry gate in §5.
• What Are the Differences Between a Circuit Breaker, Retry With Backoff, and Rate Limiting — the layered-retry math that turns naive retries into a metastable overload.
• Caching — Deep Stampede (XFetch), LRU Lock Contention & Redis Ops (Deep Dive) — the single-flight/soft-TTL fix for the cache-stampede compounding failure noted in §5.

Re-authored/Deepened for this guide.