Evolving APIs & Event Schemas — Changing a Live Contract Without Breaking Consumers

Evolving APIs & Event Schemas — Changing a Live Contract Without Breaking Consumers

The drill: "An event that 40 services consume needs a new required field. Ship it with zero downtime — how?" The trap answer is "add the field and make it required." That single deploy breaks every consumer that hasn't been updated, and breaks the ones that have the moment they read an old event still in the topic. A live contract has data in flight (in-topic, in-flight requests, replay logs) written by every version of every peer that is currently running. You are never editing one schema; you are editing a population of readers and writers that overlap in time.

The mechanism that makes zero-downtime evolution possible is schema resolution: a consumer reads bytes written under a writer schema using its own reader schema, and the deserializer reconciles the two. Whether that reconciliation succeeds defines two properties you must reason about explicitly:

• Backward compatible — a new reader can read data written by an old writer. (Consumers upgrade first; safe to add optional fields and delete fields.)
• Forward compatible — an old reader can read data written by a new writer. (Producers upgrade first; safe to add fields and delete optional fields — the old reader ignores what it doesn't know.)

"Required field, 40 consumers, zero downtime" is impossible in one step precisely because a brand-new required field is neither backward- nor forward-compatible: old data lacks it (new reader chokes), and old readers can't know it exists (they'd be fine, but you can't require producers to emit it until they're all upgraded). The answer is to never let a single deploy require a field that isn't universally present yet — you stage it. That staging is expand-contract.

Expand-contract: staging a required field across 40 consumers

Expand-contract (a.k.a. parallel-change) splits one breaking change into a sequence of individually-safe deploys. The invariant you hold at every step: no reader ever encounters data it cannot parse, and no field is required until it is universally present. Four phases:

1. Expand — add the field to the schema as optional with a default. Nothing emits it yet; every existing reader keeps working (forward compat: they ignore it).
2. Migrate producers — roll all producers to emit the field. Still optional in the schema, so in-flight old events (without it) remain legal.
3. Verify + migrate consumers — monitor that 100% of new events carry the field and that no old events remain in any topic/replay window; then roll consumers to rely on it.
4. Contract — only now flip the field to required (drop the default), and delete any old code paths that tolerated its absence.

Traced timeline for the 40-consumer event (OrderPlaced gaining a required currency field):

Phase	Schema state	Producers	40 Consumers	Data in topic	Invariant held?
0 — before	no currency	none emit	none read it	all lack field	baseline
1 — Expand	currency optional, default "USD"	none emit yet	upgrade freely, no rush	all lack field	Yes — readers ignore unknown / see default
2 — Producers	optional	every producer emits it (producer count is independent of the 40 consumers)	mix of old/new	new events have it, old ones don't	Yes — field still optional
3 — Verify	optional	all emit	roll all 40 to require it in their logic	100% carry field; old ones aged out of retention/replay	Yes — verified by metric before relying
4 — Contract	currency required, no default	all emit	all rely	all carry field	Yes — safe to enforce

The subtle step is 3: the default in phase 1 is what lets a phase-2 consumer read a phase-0 event without a null-pointer — but you must not flip to required until replay/retention windows can no longer surface a field-less event. If your topic retains 7 days and you can replay from offset 0, "old data" isn't gone when producers upgrade; it's gone when the retention/replay horizon passes. Enforce required only after that.

Schema registry compatibility modes

A schema registry (Confluent Schema Registry, Apicurio, AWS Glue) turns these compatibility properties into a publish-time gate: when a producer tries to register a new schema version for a subject, the registry rejects it unless it satisfies the configured mode. This moves "did I break a consumer?" from a 3am incident to a CI failure. The modes:

Mode	Checks new schema can...	Lets you safely	Who upgrades first
BACKWARD (default)	read data written by the previous schema	delete fields; add optional fields	consumers
FORWARD	have its data read by the previous schema	add fields; delete optional fields	producers
FULL	both of the above vs the previous schema	add/remove only optional fields	either order
*_TRANSITIVE	same check vs all prior versions, not just the last	evolve safely even against old replayed data	as above

TRANSITIVE matters exactly for the replay case above: BACKWARD only guarantees the new reader handles the immediately previous writer, so a v5 consumer replaying v1 events from the start of a Kafka topic can still break. BACKWARD_TRANSITIVE checks against v1..v4, which is what you want whenever old data is re-readable.

How the format encodes it — Avro vs Protobuf

Avro uses full writer/reader schema resolution: the writer schema travels with (or is referenced by) the data, and the reader reconciles field-by-field by name. Adding a field with a default is both backward- and forward-compatible — a new reader fills the default for old data; an old reader never sees the field. No default = not backward compatible.

{ "type": "record", "name": "OrderPlaced", "fields": [
    { "name": "orderId",  "type": "string" },
    { "name": "amount",   "type": "long" },
    { "name": "currency", "type": "string", "default": "USD" }   // Phase-1 addition: safe
] }

Protobuf reconciles by numeric field tag, not by name. Unknown fields are preserved and skipped by old readers (forward compat is built in), and proto3 has no required — every field is effectively optional on the wire, so absence is just the default value. You get expand-contract "for free" at the wire level, but you must enforce "present" in application logic during phase 3.

message OrderPlaced {
  string order_id = 1;
  int64  amount   = 2;
  string currency = 3;   // new tag; old readers skip tag 3, new readers default ""
}

The never-reuse-a-field-tag rule

Because Protobuf keys bytes by tag number, reusing a deleted field's tag number for a different field silently corrupts data: a new reader interprets old bytes stored under tag 3 as the new field's type, and an old reader interprets the new field as the deleted one. This is a data-integrity bug, not a parse error — the worst kind. Same discipline for Avro aliases and any positional format. Mark removed tags reserved so the compiler refuses reuse:

message OrderPlaced {
  reserved 3;                 // 'currency' was here; never reuse
  reserved "currency";        // and never reuse the name
  string order_id = 1;
  int64  amount   = 2;
  string currency_code = 4;   // the replacement gets a fresh tag
}

Rule of thumb: tags are append-only and immutable for the life of the contract. You add tags, you retire tags, you never recycle tags.

Consumer-driven contract tests (Pact)

A schema registry checks structural compatibility — can the bytes be parsed. It cannot tell you that Consumer #17 depends on status only ever being one of three values, or that it reads amount in cents. Consumer-driven contract testing (Pact) closes that gap by inverting who owns the expectation: each consumer publishes a pact — a set of concrete request/response (or message) expectations it actually relies on — and the provider's CI verifies against every consumer's pact before it can merge.

1. Consumer test runs against a Pact mock, recording "when I send X, I expect a response shaped like Y" → publishes the pact to a broker.
2. Provider pipeline replays every registered consumer's pact against the real provider. If a change would break any consumer's recorded expectation, the provider build fails.
3. can-i-deploy gates the actual release: it asks the broker whether this provider version is compatible with the consumer versions currently in each environment.

The payoff for the 40-consumer event: a producer physically cannot ship a change that violates a consumer's stated expectation, because that consumer's pact is a required check on the producer's pipeline. Registry catches "unparseable"; Pact catches "parseable but semantically wrong for someone." Use both — they cover different failure classes.

API versioning & deprecation for synchronous endpoints

Events lean on schema evolution; request/response APIs add an explicit version selector when a change genuinely can't be made compatibly. Two placements:

• URI versioning — /v1/orders, /v2/orders. Blunt but obvious: cacheable, trivially routable, greppable in logs, easy for clients. Cost: the version leaks into every URL and tends to version the whole surface at once.
• Header / content negotiation — Accept: application/vnd.myapp.v2+json (or a custom Api-Version header). Keeps URIs stable and lets you version a representation rather than a path. Cost: invisible in a browser, easy to forget, and intermediary caches must vary on the header.

Prefer compatible evolution (add fields, never repurpose or remove) over minting a new version; reserve v2 for a real break. When you must deprecate, run a deprecation window and announce it in-band with standard headers so clients discover it programmatically:

Deprecation: @1735689600                       // RFC 9745: Structured-Fields Date (Unix ts) = Wed, 01 Jan 2025 00:00:00 GMT
Sunset: Sat, 31 Jan 2026 23:59:59 GMT          // RFC 8594: endpoint stops working on/after this date
Link: <https://api.example.com/docs/v2>; rel="successor-version"

Deprecation (RFC 9745) says "still works, but stop using it" — note its value is a Structured-Fields Date (an @-prefixed Unix timestamp), while Sunset (RFC 8594) is a classic HTTP-date; the two headers were standardized six years apart and do not share a syntax. Sunset commits to a date the endpoint will stop working. Emit both during the window, track callers still hitting the old version via metrics, and only remove the endpoint after usage drops to zero (or the sunset date, whichever the contract promised).

Trade-off: how to manage evolution

Approach	How it works	Strengths	Weaknesses / when not to
Schema registry + Protobuf/Avro	Binary encoding, compatibility enforced at publish/CI time	Compact wire size; evolution rules automated & enforced; TRANSITIVE covers replay; strong typing	Registry is a runtime/build dependency; binary is harder to eyeball; tooling & codegen overhead. Overkill for a tiny internal surface.
Schemaless JSON + tolerant reader	Postel's Law: emit strictly, ignore unknown fields, tolerate missing ones with defaults	Zero infra; human-readable; trivial to start; language-agnostic	No enforcement — drift is caught only at runtime (or by a customer); "ignore unknown" is hand-discipline that erodes; no single source of truth. Fine for a small, high-trust team; dangerous across 40 teams.
Hard v1/v2 endpoints or topics	Publish/serve each version in parallel, migrate consumers, retire the old one	Clean isolation; the breaking change is explicit and reversible; no clever compat reasoning	N× fan-out and dual-publish cost; you must migrate all 40 consumers anyway; combinatorial version explosion if used routinely. Reserve for genuinely incompatible breaks, not everyday field additions.

The mainstream answer for a many-consumer event bus is schema registry + Avro/Protobuf with BACKWARD_TRANSITIVE, plus Pact for semantic expectations. Tolerant-reader JSON is the pragmatic default for small or public-facing surfaces; hard versioning is the escape hatch for the rare change that no amount of expand-contract can make compatible.

Takeaways

• You are never editing one schema — you are editing a live population of overlapping readers and writers. Reason in backward (new reader / old data) and forward (old reader / new data) compatibility, not "the new version."
• A new required field is neither; add it via expand-contract — optional-with-default → all producers emit → verify universal presence past the replay horizon → flip to required.
• Let a schema registry gate structure at publish time (use *_TRANSITIVE when old data is replayable) and Pact gate per-consumer semantics in CI; they catch different failures.
• Never reuse a field tag/number — mark it reserved. Prefer compatible evolution over new versions; when you must break, run a deprecation window with Deprecation (RFC 9745) and Sunset (RFC 8594) headers and retire only at zero usage.
• Reason in absolute events, not percentages, at fleet scale. A 0.1% deserialization-failure rate on a 1M msg/s bus is 1,000 poisoned messages every second; and a hard v1→v2 cutover dual-publishes both versions, doubling produce bandwidth for the whole migration window — budget both.

---

Sources: Martin Fowler, "ParallelChange" (expand-contract) and "Consumer-Driven Contracts"; Confluent Schema Registry compatibility documentation (BACKWARD/FORWARD/FULL/TRANSITIVE); Apache Avro specification (schema resolution & defaults); Protocol Buffers language guide (field numbers, reserved, proto3 optionality); Pact documentation (consumer-driven contracts, can-i-deploy); RFC 8594 (the Sunset HTTP header) and RFC 9745 (the Deprecation HTTP header). Re-authored from-scratch for this guide; diagram and traced timeline hand-authored.

I just read a lesson on **Evolving APIs & Event Schemas — Changing a Live Contract Without Breaking Consumers** (System Design) and want to truly understand it. Explain Evolving APIs & Event Schemas — Changing a Live Contract Without Breaking Consumers from first principles using ONE vivid real-world analogy and a visual mental model — draw it as ASCII art or a clear step-by-step diagram — with a concrete example using real numbers. Then ask me one question to check I got the mental picture, and wait for my reply. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Teach me **Evolving APIs & Event Schemas — Changing a Live Contract Without Breaking Consumers** interactively. Ask me ONE guiding question at a time, wait for my answer, and adapt to my confusion — build the idea with me step by step instead of explaining it all at once. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Quiz me on **Evolving APIs & Event Schemas — Changing a Live Contract Without Breaking Consumers** with 5 questions, easy to tricky, ONE at a time. Tell me if each answer is right; at the end, explain clearly what I got wrong and why. If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.

Help me remember **Evolving APIs & Event Schemas — Changing a Live Contract Without Breaking Consumers** for the long term: give the one-sentence intuition, a memorable hook/mnemonic, a tiny worked example, and 3 active-recall flashcards (Q -> A). If you're unsure or a claim isn't standard, say so and reason from first principles instead of guessing.