Designing for Change You've Already Shipped — Versioning Contracts, APIs & Persisted Data

Designing for Change You've Already Shipped — Versioning Contracts, APIs & Persisted Data

The Open/Closed Principle teaches a comfortable move: add new behavior without editing existing source. But it quietly assumes you control all the code. This page is about the harder, real-world case — the one that actually shows up in a staff interview and in production: v1 is already live. Old consumers are running. Old data is sitting in Kafka, in S3, in a Postgres column, serialized inside a cache. You cannot recompile the world at midnight. You must ship v2 while v1 keeps breathing.

The two prompts we will fully equip you for: "add a required field to an event that 40 services consume, with zero downtime" and "migrate a persisted/serialized object across versions." Both are the same underlying problem wearing different clothes.

1. Mechanism first — why in-place change is off the table

The moment a contract has live consumers, it stops being your data structure and becomes a shared one. A contract is any of three things: a wire format (the event/message on the bus), an API (the request/response shape), or a persisted blob (a row, a serialized object, a document). Once other code — that you do not deploy atomically with yourself — reads or writes that contract, old and new versions must coexist in time. There is no instant where every producer and every consumer flips together. Deploys are rolling; queues hold in-flight messages written by yesterday's code; a blob written two years ago is read for the first time today.

So the whole discipline reduces to preserving two properties across a version change:

• Backward compatibility — new code can read old data. The v2 consumer must still make sense of a v1 message. This is what protects you against the backlog and the slow producer.
• Forward compatibility — old code can tolerate new data. The v1 consumer, not yet upgraded, must not crash when it sees a v2 message carrying fields it has never heard of. This is the harder, more-often-forgotten direction.

Two techniques buy you both. Additive-only change: you only add optional things; you never remove, rename, or repurpose. And tolerant readers (Postel's Law — "be conservative in what you send, liberal in what you accept"): a reader ignores fields it does not recognize instead of rejecting the whole message. Get these two habits into your bones and most versioning questions answer themselves.

2. The concrete rules (the ones you recite in the interview)

• Add only OPTIONAL fields. A new field must have a defined behavior when absent (a default, or "treat as unknown"). If new data is meaningless without it, that is a required field, and required fields are never added in one step (see §4).
• Never reuse or renumber a field tag. In protobuf a field is identified on the wire by its number, not its name. If you delete field 3 and later add a different field as 3, old bytes for the old field 3 will be silently misread as the new one — a data-corruption bug with no exception. Mark deleted numbers reserved. In Avro, the equivalent safety valve is aliases so a rename still resolves.
• Never repurpose the meaning of an existing field. Overloading status to also mean "priority for legacy rows" is a silent contract break — the field's semantics are part of the contract just as much as its type.
• A required field arrives only through expand–contract. add optional → backfill / dual-write → flip to required once every producer emits it. Three deploys, never one.
• Renaming is removing + adding. Treat any rename as delete-then-add and it will scare you into doing it safely (keep both, migrate, then retire).

3. Serialization formats — what each does for compat, and how each breaks

The format you pick decides how much of this the tooling enforces for you.

Protocol Buffers
  Identity on the wire = field NUMBER. Names are cosmetic.
  message OrderPlaced {
    string order_id     = 1;
    int64  amount_cents = 2;
    string customer_tier = 3;   // added later — safe, new number
    reserved 7;                  // a number we deleted, fenced off forever
    reserved "old_flag";
  }
  Forward compat: an old parser PRESERVES unknown fields (it keeps the
    raw bytes and re-emits them) — so a v1 service can round-trip a v2
    message without losing tier. This is the killer feature.
  Backward compat: missing scalar fields read back as the type default
    (0, "", false) — so absence is indistinguishable from "explicitly 0".
    Use 'optional' (proto3) if you must tell those apart.
  How it BREAKS: reusing/renumbering a tag; changing a field's type in a
    non-wire-compatible way (e.g. int32 -> string).

Avro
  No field numbers. Compat comes from a WRITER schema and a READER schema,
  resolved field-by-field by NAME at decode time (schema resolution).
  - Reader has a field the writer lacks  -> reader's DEFAULT is used (backward).
  - Writer has a field the reader lacks  -> it is IGNORED (forward).
  - Field renamed                        -> use an ALIAS so it still resolves.
  Crucial operational fact: the reader must OBTAIN the writer's schema.
  In practice a Schema Registry stores schemas and enforces a compatibility
  mode (BACKWARD / FORWARD / FULL) at register time — it REJECTS an
  incompatible schema before it can ever be produced. That is the real win:
  the break is caught at CI/registration, not at 3am in a consumer.
  How it BREAKS: adding a field WITHOUT a default (now neither side can
  fill it); no access to the writer schema at read time.

JSON (schemaless)
  No enforcement at all — the discipline lives entirely in your code.
  Forward compat: your parser must IGNORE unknown keys, not throw
    (Jackson: @JsonIgnoreProperties(ignoreUnknown = true), or
     FAIL_ON_UNKNOWN_PROPERTIES = false). This is the "tolerant reader".
  Backward compat: a missing key must map to a sensible default, never NPE.
  How it BREAKS: a strict deserializer that rejects unknown fields (the
    default in many libraries!) turns every additive change into an outage
    for un-upgraded consumers. Changing a field's JSON type is unchecked
    and blows up only at the reader.

The trade is legibility vs. enforcement. JSON is human-readable and needs no registry, but every rule above is a code review away from being violated. Protobuf/Avro give you machine-checked evolution — at the cost of a schema registry, code generation, and opaque bytes on the wire.

4. Expand–contract, traced: "40 services consume OrderPlaced; add a required customerTier"

You cannot add a required field in one move, because in the seconds-to-hours of a rolling deploy there exist simultaneously: producers still on v1 (emitting no tier), consumers on v1 (that would choke on it), messages already in the queue written last week, and consumers on v2. "Required in one deploy" breaks all four. The fix is to stretch the change across three phases and never let any phase require both sides to be new at once.

Phase 1 — EXPAND the readers.  Field is OPTIONAL, nullable, defaulted.
  Deploy ALL 40 consumers first. Each tolerates tier being absent
  (default to e.g. STANDARD). Nobody produces it yet. Nothing breaks
  because nothing changed on the wire.

Phase 2 — Producers emit it.  Now that every reader tolerates presence
  AND absence, turn on the producer(s). New messages carry tier; old
  in-flight messages still don't — both are fine by Phase 1.

Phase 3 — Backfill / replay (only if history must have it).  Reprocess
  or dual-write the old records so historical data also carries tier.
  For an event log: replay with an upcaster (see §5). For a table:
  a backfill job populates the column in batches.

Phase 4 — CONTRACT.  Once (a) every producer emits tier and (b) no
  un-backfilled data remains in any reader's window, flip it to
  required / non-null and delete the default-handling branch. Only
  now is it truly "required" — and it cost you zero downtime.

Time	Producers	Consumers	Wire state	Safe?
T0 (start)	v1 (no tier)	v1	no tier	baseline
T1 (Phase 1)	v1 (no tier)	v2 tolerant	no tier	yes — reader defaults
T2 (Phase 2)	v2 emits tier	v2 tolerant	mixed old+new	yes — reader handles both
T3 (Phase 3)	v2	v2	backfilled	yes — history repaired
T4 (Phase 4)	v2	v2 requires tier	all new	yes — default branch removed

The load-bearing insight: consumers are deployed before producers. You always widen the set of things a reader will accept before anything starts emitting the new thing. Do it the other way and un-upgraded consumers meet data they can't parse.

5. Migrating a persisted / serialized object across versions

Persisted data is the brutal case: the "old producer" is the past, and you can't upgrade it. A blob written by v1 code will still be read by v5 code years later. Three tools:

(a) Version every record. Stamp each stored payload with its shape — a schema_version column, or a leading version byte in the serialized blob. Without it, a reader has to guess which shape it's holding, and guessing is how corruption starts.

(b) Upcasters. An upcaster is a pure function that transforms an old payload into the next shape on read. You chain them: a v1 blob is upcast v1→v2→v3 before your domain code ever sees it, so the domain only ever knows the latest shape. This is exactly how event-sourcing systems (Axon, EventStore) handle decade-old events.

(c) Avoid raw Java Serialization for durable data. java.io.Serializable ties the on-disk form to your class's private fields and its serialVersionUID. Change the class and either the UID auto-changes (old blobs become unreadable — InvalidClassException) or you pin the UID and silently mis-map fields. It's opaque, JVM-only, a well-known deserialization-gadget security risk, and offers no story for optional fields or defaults. For anything long-lived, serialize through an explicit schema (JSON/protobuf/Avro) you control and can evolve.

// A versioned, upcaster-based read path in Java.
// The stored form is explicit JSON with a "schemaVersion" tag —
// NOT Java Serializable. The domain object is always the latest shape.

record CustomerV3(String id, String name, Tier tier, Region region) {}

interface Upcaster {
    int fromVersion();                       // the version this handles
    ObjectNode apply(ObjectNode node);       // returns the NEXT version's shape
}

// v1 -> v2 : split legacy "fullName" into name; introduce tier=STANDARD
final class V1toV2 implements Upcaster {
    public int fromVersion() { return 1; }
    public ObjectNode apply(ObjectNode n) {
        n.put("name", n.path("fullName").asText(""));   // rename via copy
        n.remove("fullName");
        n.put("tier", "STANDARD");                       // default for old data
        n.put("schemaVersion", 2);
        return n;
    }
}

// v2 -> v3 : add region, defaulted; derive from a legacy field if present
final class V2toV3 implements Upcaster {
    public int fromVersion() { return 2; }
    public ObjectNode apply(ObjectNode n) {
        String legacyZip = n.path("zip").asText("");
        n.put("region", legacyZip.startsWith("9") ? "WEST" : "UNKNOWN");
        n.remove("zip");              // consume the legacy field it replaced
        n.put("schemaVersion", 3);
        return n;
    }
}

final class CustomerReader {
    private final Map<Integer, Upcaster> chain;   // fromVersion -> upcaster
    private final ObjectMapper mapper;
    static final int CURRENT = 3;

    CustomerV3 read(String storedJson) throws IOException {
        ObjectNode node = (ObjectNode) mapper.readTree(storedJson);
        int v = node.path("schemaVersion").asInt(1);  // absent == oldest
        while (v < CURRENT) {                          // upcast step by step
            Upcaster up = chain.get(v);
            if (up == null) throw new IllegalStateException("no upcaster from v" + v);
            node = up.apply(node);
            v = node.get("schemaVersion").asInt();
        }
        node.remove("schemaVersion");                 // internal migration metadata, not a domain field
        // mapper is a tolerant reader (FAIL_ON_UNKNOWN_PROPERTIES disabled), so any
        // stray legacy key an upcaster forgot to drop is ignored rather than throwing.
        return mapper.treeToValue(node, CustomerV3.class);
    }
}

Note what this buys you: writes always produce the current shape; reads lazily migrate old shapes forward; and you never have to run a big-bang migration over the whole datastore before shipping v3. (You may still backfill lazily-on-write or in a background job to eventually retire the oldest upcasters.)

6. API versioning — the same problem at the HTTP edge

Public APIs face identical constraints (clients you don't deploy), plus a naming decision:

• URI versioning (/v1/orders, /v2/orders): explicit, cacheable, trivially routable, obvious in logs. Cost: the version leaks into every URL and clients hard-code it; a "version" technically identifies a resource, which purists dislike.
• Header / content negotiation (Accept: application/vnd.acme.order.v2+json): keeps URLs stable and clean, lets you version representations independently. Cost: invisible in a browser, easy to get wrong in caches/proxies, harder to test by hand.

Whichever you pick, for additive changes you should not need a new version at all — a tolerant reader on the client makes v2 fields free. Reserve a whole new version number for breaking changes you cannot make additively.

Consumer-driven contract tests (Pact). The structural safety net: each consumer publishes the exact requests/responses it depends on; the provider replays those "pacts" in CI and fails the build if a change would break any real consumer. This turns "did I break someone downstream?" from a prod incident into a red pipeline — the same shift Avro's registry gives you for messages.

Deprecation windows. When you must retire v1, announce it, emit the Sunset HTTP header (RFC 8594) and the Deprecation header (RFC 9745) with the cutoff date, monitor who's still calling it, and only remove after traffic drains. Never silently 410 a version clients still use.

7. Pitfalls interviewers push on

• Reusing a deleted field number/tag. Old bytes get reinterpreted as the new field — silent corruption, no exception. Always reserved the number (protobuf) or keep the name via alias (Avro).
• Making a field required in a single deploy. Breaks in-flight messages and un-upgraded producers instantly. Required is a destination reached by expand–contract, never a starting move.
• A dual-write that clobbers. During migration, writing both old and new locations without a clear source of truth lets a lagging writer overwrite fresh data. Make one side authoritative and reconcile; prefer write-new-read-both, then read-new.
• Assuming all clients upgrade at once. They don't. There is always a long tail on the old version; design for coexistence, not for a flag day.
• Java Serializable for durable data. Brittle across class changes, JVM-only, a deserialization security hole. Use an explicit, evolvable schema for anything that outlives one deploy.
• Forgetting forward compat. Teams reflexively make new code read old data (backward) and forget old code must survive new data (forward). The un-upgraded consumer is the one that pages you.

8. Selection & trade-offs — which strategy, when

Strategy	Gains	Costs	Wins when
Schema registry + protobuf/Avro	Machine-enforced compat at register/CI time; compact fast bytes; explicit evolution rules; forward compat via unknown-field preservation / resolution	Registry to run & secure; codegen & build coupling; opaque non-human-readable payloads; a learning curve	High-throughput internal event buses with many independent teams/services (your 40-consumer case). The enforcement is worth the infra.
Schemaless JSON + tolerant reader	Zero infra; human-readable; trivial to start; polyglot; additive changes are free if readers are tolerant	No enforcement — every rule is a code-review away from violation; type changes blow up only at the reader; larger payloads	Small surface, few consumers, fast iteration, or public webhooks where readability & no-tooling matter more than guarantees.
Hard API versioning (v1/v2 endpoints)	Clean break for genuinely incompatible changes; old & new served side by side; simple mental model for external clients	Duplicated code paths to maintain; migration burden pushed onto clients; version sprawl if overused	Public/partner APIs facing a breaking change that cannot be made additive. Use sparingly — additive-first, version only when forced.

The meta-rule: prefer additive evolution over versioning, and prefer tooling-enforced compat over convention as the blast radius (number of independent consumers) grows. You escalate to a hard new version only for a break you cannot express additively.

Key takeaways

• Once a contract has live consumers it is shared and time-spanning; you can't change it in place — old and new must coexist, so you engineer for backward (new reads old) and forward (old tolerates new) compatibility via additive-only change + tolerant readers.
• A required field is reached, never added: expand (deploy tolerant consumers first) → producers emit → backfill → contract to required. Consumers before producers, always.
• For durable data, version every record and upcast on read; ditch raw Java Serializable for an explicit, evolvable schema.
• Escalate from additive + tolerant JSON → schema-registry protobuf/Avro → hard API versions as consumer count and the cost of a silent break rise; add Pact and Sunset headers so breaks surface in CI, not in prod.

Sources: Kleppmann, Designing Data-Intensive Applications, ch. 4 "Encoding and Evolution"; Protocol Buffers & Apache Avro schema-evolution docs; Pact (consumer-driven contract testing); Martin Fowler, "TolerantReader"; RFC 8594 (the Sunset header).

I just read a lesson on **Designing for Change You've Already Shipped — Versioning Contracts, APIs & Persisted Data** (OO & Low-Level Design) and want to truly understand it. Explain Designing for Change You've Already Shipped — Versioning Contracts, APIs & Persisted Data from first principles using ONE vivid real-world analogy and a visual mental model — draw it as ASCII art or a clear step-by-step diagram — with a concrete example using real numbers. Then ask me one question to check I got the mental picture, and wait for my reply. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Teach me **Designing for Change You've Already Shipped — Versioning Contracts, APIs & Persisted Data** interactively. Ask me ONE guiding question at a time, wait for my answer, and adapt to my confusion — build the idea with me step by step instead of explaining it all at once. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Quiz me on **Designing for Change You've Already Shipped — Versioning Contracts, APIs & Persisted Data** with 5 questions, easy to tricky, ONE at a time. Tell me if each answer is right; at the end, explain clearly what I got wrong and why. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Help me remember **Designing for Change You've Already Shipped — Versioning Contracts, APIs & Persisted Data** for the long term: give the one-sentence intuition, a memorable hook/mnemonic, a tiny worked example, and 3 active-recall flashcards (Q -> A). If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.