Avro schema evolution when the consumer is a different company

Schema Registry compatibility modes tell you whether a schema change is legal. They do not tell you whether it is safe to deploy. Those are different questions, and the gap between them widens sharply once the consumer belongs to a different company.

What the compatibility modes actually check

The modes are precise, and worth restating without the marketing.

The non-transitive modes are the trap. BACKWARD compares your new schema against exactly one predecessor. A consumer that has been offline for a while, or a topic with long retention, is reading messages written by schemas from several versions ago — and nothing checked those.

If the data outlives two deployments, use the transitive mode. With Kafka retention measured in days and consumers that can lag, it usually does.

Across a boundary, the default is FULL_TRANSITIVE

Inside one team, BACKWARD_TRANSITIVE is defensible: you control the consumers, you can deploy them first, you can coordinate.

Across an organizational boundary you cannot. You do not know which version they are on, you cannot make them deploy, and you will find out about a break from a support ticket rather than a pipeline. FULL_TRANSITIVE costs you the ability to remove a required field in one step — which is a good thing, because that is precisely the change that breaks people.

Set it on the subject, not globally, and treat any request to loosen it as a design conversation rather than a config change.

Defaults matter more than types

This is the rule that decides whether an addition is genuinely additive.

{
  "type": "record",
  "name": "CardOperation",
  "fields": [
    { "name": "operationId", "type": "string" },
    { "name": "cardId",      "type": "string" },
    { "name": "amountMinor", "type": "long" },
 
    // Safe: optional with a default. Old readers ignore it,
    // new readers get null when reading old data.
    { "name": "channel", "type": ["null", "string"], "default": null }
  ]
}

Without "default": null, the field is required, and a reader using the new schema against old data has nothing to put there. The registry will reject it under a backward-compatible mode — which is the system working, but people routinely respond by loosening the mode instead of adding the default.

The subtler failure is on the consumer side. A defaulted null is indistinguishable from an explicit null. If channel = null means "written before we had channels" in one case and "the operation had no channel" in another, the consumer cannot tell them apart, and someone will eventually write business logic on that ambiguity. Where the distinction matters, encode it — an UNKNOWN enum member says something a null cannot.

The consumer that rejects unknown fields

Compatibility modes assume the consumer is a well-behaved Avro reader that ignores fields it does not know. Plenty of consumers are not.

I learned this on my own product, in JSON rather than Avro, and the lesson transfers exactly: a client deserializing into a strict type — Jackson with FAIL_ON_UNKNOWN_PROPERTIES, a generated model with an exhaustive constructor, a strongly-typed language with no unknown-key escape hatch — breaks when you add a field. The registry says the change is backward compatible. The registry is right. The consumer still breaks.

Which produces the operating rule I now apply by default:

Adding a field to a contract consumed by someone else is a breaking change until proven otherwise.

Proving otherwise means knowing how their deserializer behaves, and the only reliable way to know that is to ask them and write the answer down next to the schema.

Removing a field takes quarters

Under FULL_TRANSITIVE you cannot delete a field that lacks a default, and you should not delete one that has a default either — not in a single release.

The sequence:

1. Deprecate in the schema. Mark it in the doc string with a date. This is documentation, not enforcement, and that is fine — it is the artifact consumers actually read.
2. Stop writing meaningful values. Producers emit the default. The field is still present on the wire.
3. Wait. Long enough that every consumer has deployed at least once and any retained data written with real values has aged out. Across companies, a quarter is a realistic unit.
4. Confirm consumers no longer read it. Ask. If your registry or platform exposes per-consumer schema versions, use them, but the conversation is the part that matters.
5. Remove it.

Steps 3 and 4 are what people skip when a deadline is close, and it is the skipping rather than the removal that causes the incident.

Version the topic when you cannot evolve

Sometimes the change is genuinely incompatible: a field's type must change, a record must be restructured, a unit was wrong.

Do not fight the registry. Publish to a new subject — card-operations.v2 — and run both for the length of the migration. Producers write to both, consumers migrate on their own schedule, and the old topic is retired when it has no consumers left.

This costs double writes and a period of duplication. It is dramatically cheaper than a coordinated same-day cutover across two companies, which is the alternative, and which fails.

What to enforce in CI

Compatibility should fail a build, not a deployment.

# Fails the build if the schema is incompatible with the registered history
mvn io.confluent:kafka-schema-registry-maven-plugin:test-compatibility

Plus one thing the plugin cannot check, which is worth a review checklist item: every added field has a default, and the meaning of that default is written in the doc string. A schema that passes compatibility and leaves the semantics of null undefined has moved the problem into somebody's business logic instead of solving it.

Related

Schema evolution is the contract half of event-driven work; the delivery half is idempotency keys are not enough when money moves. If the events are produced by change data capture rather than by an application, Debezium against a database you do not own covers what that adds operationally.