Data Partitioning — Fan-out Stragglers, Salting Read Cost & Vertical-Split Hazards

The guide already covers local-vs-global secondary indexes, request routing, rebalancing, and failover consistency (see the Data Partitioning pages on those). This page adds four cross-cutting judgments that a senior probes once the basic scheme is on the board: how fan-out kills tail latency, what salting costs on the read side, why a vertical split introduces a write-atomicity hazard, and the direction-dependent availability of sharding.

1. The straggler problem: fan-out and p99

Any query that isn't answerable from a single partition becomes a scatter-gather: fan out to all N shards, wait for all, merge. The trap is that the response time is the maximum of N latencies, so the whole query's p99 is driven by the p99 of any one shard — and it gets worse as N grows. If each shard independently returns slow 1% of the time, a 100-shard fan-out is slow (≥ one straggler) about 1 − 0.99^100 ≈ 63% of the time. So tail latency is amplified precisely by the parallelism that was supposed to help.

Mitigations: hedged/backup requests (re-issue to a replica after a short delay, take the first to return), tighter per-shard timeouts with partial results, and — best — designing the access pattern so the common query hits one partition (partition by the key you query by).

2. Salting a hot key: the read-side bill

Salting spreads a hot key (a celebrity user, a viral post) by appending a random bucket suffix — user123#0 … user123#9 — so writes spread across 10 partitions instead of hammering one. The immediate senior counter is "now how do you READ that user?" You've converted a single-partition read into a fan-out over all 10 buckets plus a merge — you re-introduced §1's straggler problem for that key. So salting is a write-hotspot fix that you pay for on every read. Scope it deliberately: salt only the keys that are actually hot (not the whole space), and keep the salt factor as small as the write load requires, because it multiplies read cost 1:1.

3. Vertical split: a hidden dual-write hazard

Splitting a row vertically — hot columns (presence, last-seen) in a fast store, cold columns (profile, bio) in another — is a sound performance move, but it silently turns one atomic row update into two writes to two stores. A crash between them leaves the record half-updated with no transaction to roll it back: the presence card says online, the profile disagrees. You now own a consistency problem you didn't have. Options and their cost: accept eventual consistency and reconcile (cheapest, if the two halves can diverge briefly), a saga/outbox with compensation (more machinery), or keep them co-located and solve the hotspot another way (caching the hot columns) if atomicity really matters. Name the trade-off out loud — interviewers look for whether you notice the atomicity you gave up.

4. Sharding's availability is direction-dependent

A subtle result worth stating precisely: sharding contains the blast radius for a single-key operation (one shard down takes out only 1/N of keys, the rest are fine) — but it multiplies failure probability for an all-shard query (a scatter-gather fails if any shard is down, so availability is p^N, dropping fast as N grows). More shards = better single-key isolation and worse whole-dataset-query availability. Replication is what reconciles the two: replicate each shard so a single-key op survives a node loss, and a fan-out can route each sub-query to a live replica. Sharding partitions load; replication buys availability — you almost always need both, and it's replication, not sharding, that makes the fan-out survivable.

Judgment layer & takeaways

• Fan-out: avoid it for hot paths by partitioning on the query key; where unavoidable, use hedged requests + partial results and watch that p99 = slowest-shard p99.
• Salting: a targeted write-hotspot tool, not a default — it taxes every read of the salted key. Prefer a cache or a single dedicated partition for a known celebrity if reads dominate.
• Vertical split: great for hot/cold separation, but you inherit dual-write atomicity — pick eventual-reconcile vs saga vs stay-co-located explicitly.
• Availability: sharding isolates single-key failures but compounds all-shard-query failures; replication is the counterweight — design them together.

Re-authored/Deepened for this guide.