🔀 Encoding & Schema Evolution

Prerequisites: Data Models, API Design | You'll be able to: define backward and forward compatibility precisely and say who reads and who writes in each direction; choose between JSON, Protocol Buffers, and Avro with reasons that survive scrutiny; and answer the senior follow-up "how do you roll this schema change out?" for databases, services, and event pipelines.

🧨 The problem (why this exists)

You add a field — a single column, deleted_at, so rows can be soft-deleted instead of vanishing. You test it, you ship it. Mid-deploy, a support ticket lands: a record the customer clearly deleted has come back from the dead.

This isn't a bug in your code so much as a fact about the universe it runs in.

Your deploy is a rolling upgrade — the new version goes out a few nodes at a time, monitored, with no downtime, because you cannot replace every process at once [i]. For a stretch of minutes, some nodes know about deleted_at and some have never heard of it. A new node writes a record carrying the field; an old node reads it, does its job, and writes it back — and because its record has no deleted_at, the field evaporates on the way back to disk [i]. The delete is undone by a peer that meant no harm.

Scale the timeline up and the collision reappears in a second dimension.

A schema-on-read store enforces nothing and holds a mixture of formats written at different moments; even a relational database, migrated only via ALTER, keeps old rows in place [i]. So a row you wrote five years ago still sits there in its original encoding, waiting for code you wrote this morning. Old code meets new data across a rolling upgrade; new code meets old data across the passage of time — both permanent conditions of any system that stays up and keeps changing.

This is evolvability — building systems easy to change [i] — reduced to its mechanical core: how do you encode data so two versions of your software, and two eras of your data, coexist without corrupting each other? Get it right and "your application's evolution [is] rapid and your deployments [are] frequent" [i] — you ship additively, all day, without fear.

🔄 Intuition first

Everything here rests on two words people constantly swap by accident. Nail them once.

Backward compatibility: new code can read old data. You look backward in time at data written by an earlier version — normally the easy direction, since the author of the new code knows the old format and handles it deliberately [i].

Forward compatibility: old code can read new data. You ask code written before to cope with data from the future — the tricky direction, because old code must gracefully ignore additions it doesn't understand rather than choke or discard them [i].

A picture that sticks: backward is reading your ancestors' letters; forward is your ancestors reading yours. Which direction you need depends entirely on who writes and who reads, and that flips with the arrangement.

• In one database under a rolling upgrade: new node writes, old node reads → forward; old node writes, new node reads → backward — both at once, so a database needs both [i].
• For a service API: an older client calling a newer server needs backward compatibility on the request and forward compatibility on the response; a newer client flips both [i].

Say "who reads, who writes" every time and you'll never reverse the direction in an interview — which is exactly where candidates trip.

The second intuition is about schemas.

The second intuition is about schemas — a contract stating what fields exist and their types. Some formats carry it inside every message (JSON writes "userName" next to the value); others keep it outside, agreed in advance (Protobuf ships only the number 1; Avro ships nothing but the bare value). Where the contract lives determines how compactly you encode and how gracefully you change — the ladder next.

⚙️ How it works

In memory, data is objects, lists, and trees threaded with pointers; on disk or a network, pointers are meaningless, so it must become a self-contained sequence of bytes [i]. Turning the in-memory shape into bytes is encoding (serialization); the reverse is decoding (parsing, deserialization) [i]. Every claim about compatibility is a claim about a specific format, so we climb the ladder worst to best, using one running record: {userName, favoriteNumber, interests}.

🚫 Rung 1 — Language-native serialization (the trap)

Java's Serializable, Python's pickle, Ruby's Marshal save and restore an object graph in a couple of lines [i]. It's a trap, for four compounding reasons:

• language lock-in (bytes tied to one language's object model);
• security holes (decoding instantiates arbitrary classes, a recurring path to remote code execution);
• versioning as an afterthought;
• and bad efficiency (Java's is infamously slow and bloated).

The verdict: don't use it beyond a transient, same-process purpose [i].

📄 Rung 2 — JSON, XML, CSV (the lingua franca, and what's lossy)

Standardized, cross-language, human-readable text gives you JSON, XML, and CSV — good enough that they dominate interchange between organizations, where agreeing on the format beats being pretty or fast [i]. But "human-readable" hides real losses [i]:

• Numbers are ambiguous. XML and CSV can't tell a number from a digit-string without a schema; JSON separates strings from numbers but not integers from floats. Worse, an IEEE 754 double can't represent integers above 2^53 exactly, so a language parsing JSON numbers as floats (JavaScript) mangles them — X (Twitter) returns 64-bit tweet IDs twice, once as a number and once as a decimal string [i].
• No binary strings. JSON/XML carry Unicode text but not raw bytes, so people Base64-encode binary data — hacky, and about 33% larger [i].
• CSV has no schema and is vague about commas, newlines, and escaping, with rules not every parser implements correctly [i].

Schema languages exist, but evolving them is its own problem.

Schema languages exist — JSON Schema is widely adopted (OpenAPI, registries, databases), with validation like "this port is 1 to 65,535" [i] — but content models, conditional rules, and remote references make evolving them compatibly genuinely hard [i].

Text's size cost prompted binary JSON variants (MessagePack, CBOR, BSON) [i]. But they keep JSON's model and still prescribe no schema, so they must embed every field name in the bytes [i]. MessagePack shrinks our record from 81 bytes to 66 — the field names still ride along, so you've lost human-readability for a small win [i].

🏷️ Rung 3 — Protocol Buffers (field tags earn their keep)

Protocol Buffers (Google) — and its cousin Thrift (Facebook) — require a schema in an IDL, from which a code generator emits encode/decode classes in many languages; the schema is deliberately simple, just fields and types, no value validation [i].

The move that changes everything: instead of field names, the bytes carry field tags — the numbers 1, 2, 3 you assign in the schema, acting as compact aliases [i]. Type and tag pack into one byte, and integers use variable-length encoding (a varint), where the top bit of each byte says whether more follow, so small numbers cost one byte and large ones grow only as needed [i]. Our record drops to 33 bytes; there's no array type, so a repeated field is just the same tag appearing several times [i].

Now watch the evolution rules fall straight out of that one choice — tags, not names, identify fields [i]:

• Rename a field freely — names aren't in the bytes, so a rename is invisible to encoded data.
• Never change a tag — it would orphan every record ever written.
• Add a field with a new tag. Old code hitting an unknown tag reads the datatype annotation, learns how many bytes to skip, and skips it — preserving the unknown field. Forward compatibility, for free.
• New code reading old data finds the added field missing and fills a type default (empty string, 0) — backward compatibility.
• Remove a field like adding one, in reverse — and reserve the dead tag forever.
• Changing a type risks truncation. Widening int32→int64 lets new code read old values (missing bits become 0), but old code reading a new 64-bit value into a 32-bit variable truncates on overflow.

🪶 Rung 4 — Avro (no tags, two schemas)

Avro (2009, a Hadoop subproject Protobuf didn't fit) carries, remarkably, no field tags and no type markers at all [i]. Our record is 32 bytes, the most compact of the lot; a string is just a length prefix plus UTF-8 bytes, with nothing marking it as a string [i].

If nothing in the bytes says what anything is, how does decoding work?

You walk fields in schema order and let the schema supply each type — so the reader must use the exact schema the writer used, or the bytes decode to garbage [i]. Avro's central idea: the writer's schema (whatever encoded the data) and the reader's schema (what the consumer expects) may differ [i]. To decode, Avro holds both and resolves them — matching fields by name so order can differ; a writer field the reader doesn't want is ignored; a field the reader expects but the writer lacks is filled from a default in the reader's schema [i].

That resolution is the evolution story, and it restates our two words exactly [i]:

• Forward = writer on a newer schema than reader; backward = writer on an older schema.
• You may add or remove only a field that has a default value [i]. No default: adding breaks backward compatibility, removing breaks forward.
• Nullability is explicit: null is never a valid default alone, so to allow null you use a union type like union { null, long }, with null first — verbose, but it stamps out a class of bugs [i].
• Renaming works via aliases in the reader's schema (backward but not forward compatible), and so does adding a union branch [i].

So how does the reader get the writer's schema?

The reader must still obtain the writer's schema, and shipping it in every record would erase the savings. So the solution depends on the setting [i]: a big file (a Hadoop dump) writes the schema once at the front — Avro's object container files; a database of individual records tags each with a version number and keeps a table of schema versions to look up (Confluent's Kafka registry and LinkedIn's Espresso work this way); a network connection negotiates the version at handshake [i].

That version table is worth having anyway — it documents history and lets you check compatibility before deploying [i]. And because Avro matches by name with no tags to hand-assign, it's uniquely friendly to dynamically generated schemas: dump a relational database, machine-generate a schema one field per column, and on a column add/drop just regenerate — by-name matching still lines up old readers with new writers [i]. Protobuf can't: it needs hand-assigned tags babysat on every change [i].

🎯 The genuine merits of schemas

The idea isn't new — it echoes ASN.1 (1984, still encoding SSL's X.509 certificates), too complex to pick today [i]. What modern schema-based binary formats buy is concrete [i]: more compact than binary JSON (no field names); a schema that doubles as always-current documentation (you can't decode without it); a registry to check compatibility before deploying; and, in typed languages, codegen with compile-time checks. You get much of schema-on-read JSON's flexibility with far better guarantees — if you keep live schema versions few [i].

🌊 The dataflow modes — where compatibility actually bites

Compatibility is a relationship between an encoding process and a decoding process — it exists anywhere you send data to something that doesn't share your memory [i]. Four modes matter.

Through databases. Storing a value is "sending a message to your future self" — backward compatibility [i] — but concurrent old and new processes during a rolling upgrade add the forward direction too [i]. And a database holds values written five milliseconds and five years ago; deploying new code leaves old data in its original encoding — "data outlives code" [i]. Rewriting every row is expensive, so databases avoid it: a nullable-column add needs no rewrite (missing values read as null), and LSM engines rewrite to the latest format during compaction [i]. Schema evolution thus makes the database look like one schema though storage is a museum of versions; immutable snapshots are the moment to re-encode consistently, into Avro container files or Parquet [i].

Through services. Clients call servers exposing an application-specific API — a service — meant to be independently deployable, so old and new clients and servers run at once and must stay compatible across API versions [i]. REST versus RPC, IDLs, and why a network call can't be made to look local are owned by the API Design lesson. The one fact to carry here is the compatibility direction: a simplifying assumption databases don't get is that servers upgrade first and clients second, so you need only backward compatibility on requests and forward compatibility on responses [i]. Across organizational boundaries it's harder — a provider can't force clients to upgrade, so compatibility may hold indefinitely, and breaking changes force multiple API versions side by side [i].

Through workflows and durable execution. Split an app across services and one operation — a payment — becomes a workflow: a graph of tasks (fraud check → debit card → deposit funds) run by a workflow engine [i]. Durable execution engines (Temporal, Restate) give exactly-once semantics: on a mid-workflow failure the framework re-runs the code but skips the RPCs and state changes already completed, replaying their logged results from a write-ahead log [i]. That replay imposes a hard determinism constraint: because it expects a re-execution to make the same calls in the same order, reordering function calls yields undefined behavior — so you deploy a new version separately and avoid nondeterministic ingredients like random numbers or the clock (frameworks supply deterministic substitutes and analyzers to catch violations) [i]. (We revisit determinism under faults in Distributed Data.)

Event-driven, through a broker. A request is an event sent through a message broker that stores it temporarily; the sender doesn't wait [i]. The encoding rules carry straight over — a message is bytes plus metadata, and the common pattern is Protobuf/Avro/JSON with a schema registry beside the broker checking compatibility [i]. The trap recurs: a consumer republishing to another topic must preserve unknown fields, or it re-creates the field-loss corruption on the stream [i]. The Patterns module's event-driven lesson goes deeper.

The opening database collision, drawn as the rolling upgrade that makes it permanent:

Step 2 demands forward compatibility (old code reading new data), step 4 backward (new code reading old data) — which is why a database, alone among the modes, needs both.

⚖️ Trade-offs

The cross-cutting axis: text is readable and needs no schema sharing but is bulky and loosely typed; schema-driven binary roughly halves the payload and turns the schema into live documentation and a compatibility gate — at the cost of readability and sharing the schema [i].

🔢 Numbers that matter

DDIA's measured or stated values for the running record and its formats:

The interview headline: binary schema formats roughly halve payload versus JSON (33 B vs 81 B), and that win compounds in the internal service fan-out where one request becomes dozens of calls you control; at a public mobile edge one round-trip dwarfs the delta, so JSON's debuggability usually wins. Rule of thumb, not from source: keep concurrently-live schema versions in the low single digits; DDIA says only "keep the number minimal" [i].

🏭 In production

Kafka pipelines run on a schema registry. Confluent's Schema Registry (and Red Hat's Apicurio) sit beside the broker, store every schema version, and reject a producer whose new schema would break the configured compatibility mode before a bad message ships [i]. Each message carries a schema-ID; consumers fetch the writer's schema by ID and resolve it against their reader's [i]. This is DDIA's "database of schema versions" made standard infrastructure — the reference architecture behind "how would you evolve events in a Kafka system?" AsyncAPI plays the OpenAPI role for messages [i].

gRPC is Protobuf on the wire, inheriting its evolution rules — add a tag, never reuse one, reserve the dead ones [i]. Teams enforce this with schema-linters (Buf) in CI that fail the build on a wire-breaking change, turning "don't reuse tag 2" into a gate.

"Data outlives code" is a daily reality. LSM engines (Cassandra, RocksDB) rewrite records to the latest format during compaction, so old encodings age out on their own [i]; relational shops lean on nullable-column-add-without-rewrite to ship changes on live multi-terabyte tables [i]. The genuinely hard migrations — splitting a column into a table, a value into a list — fall to application-level backfills, which DDIA flags as an open research problem [i].

Durable execution is spreading for exactly-once workflows. Temporal and Restate back payment flows where a mid-sequence crash must never charge a card without delivering the goods — impossible to wrap in one database transaction once a third-party gateway is involved [i]. The catch is the determinism constraint: the engine replays logged history, so you can't reorder workflow code and must deploy new versions separately; sneaking a now() or rand() into it is the classic outage [i].

🛠️ Hands-on: evolve a schema

A runnable, from-first-principles demo lives at _proof-of-concepts/02-data-foundations/05-encoding-and-evolution/ in the repo root — a tiny tag/type/value binary codec (the essence of Protobuf/Thrift), no external libraries.

cd _proof-of-concepts/02-data-foundations/05-encoding-and-evolution
./run            # the three evolution scenarios
./run test       # mypy --strict + unit tests + demo

It writes a record with schema v1 and reads it back with an evolved v2 (and vice versa): a new reader reads old data (the added field falls back to its default; a field renamed on the same tag still resolves), an old reader reads new data (the unknown tag is skipped), and reusing a tag for a different type raises EvolutionError — the one change compatibility can't survive. The rule that drops out: add fields on new tags, give them defaults, never reuse a tag.

🪤 Pitfalls & interview traps

Other traps interviewers probe:

• Reversing the direction. Anchor on who reads, who writes: new code reading old data is backward; old code reading new data is forward [i].
• Reusing a Protobuf tag. Deleting email = 2 then shipping phone = 2 makes old records decode their email bytes as phone — silent corruption. Reserve dead tags forever [i].
• Assuming int32→int64 is always safe. New code reads old data fine, but old code reading a large new value into a 32-bit variable truncates it [i].
• Forgetting Avro needs the writer's schema. "Just use Avro, it's compact" is half an answer — without an object container file or registry to carry the writer's schema, the bytes are undecodable [i].
• Avro defaults and null. Add/remove is compatible only if the field has a default, and null isn't a default unless the field is a union with null first [i].
• The senior follow-up: "how do you roll this schema change out?" The expected shape: additive change only; for a database, ship the compatible schema and backfill asynchronously, relying on both directions during the rollout [i]; for a service, exploit servers-first ordering (backward requests, forward responses) and version only when a break is unavoidable, running versions side by side [i]; for events, register the new schema and let the registry enforce compatibility [i]. For anything genuinely breaking, expand-then-contract: add the new, migrate readers, remove the old in a later deploy — and always preserve unknown fields.

✅ Check yourself

Prose questions — answer out loud before expanding.

🔬 PoC — Proof of concepts

Run it yourself. Encoding & schema evolution — three evolution scenarios (add a field, drop a field, change a type) run through a real encoder to show exactly when old and new readers/writers stay compatible and when they break. From _proof-of-concepts/02-data-foundations/05-encoding-and-evolution/, run ./run.

Study real implementations.

• Protocol Buffers — proto3 guide — field-number rules, reserved, and the forward/backward-compatibility guarantees this lesson turns on.
• Apache Avro — the writer-schema/reader-schema model, the other major approach to evolution and the one that leans hardest on a schema registry.
• Confluent Schema Registry — how compatibility is actually enforced in a streaming pipeline: register a schema, and incompatible changes are rejected at publish time.

📚 Sources

• DDIA2 ch. 5 pp. 161–163 (compatibility & rolling upgrades) — rolling upgrades and client-update lag; schema-on-read holding mixed formats; backward vs forward definitions and request/response directions; the read-modify-write field-loss hazard (Fig. 5-1).
• DDIA2 ch. 5 pp. 163–169 (encoding, language-native, JSON/XML, binary JSON) — encode/decode terms; the four failings of language-native serialization; JSON/XML/CSV losses (numbers, 2^53, Base64 +33%, CSV); JSON Schema; MessagePack 66 B vs 81 B.
• DDIA2 ch. 5 pp. 169–172 (Protocol Buffers) — IDL + codegen; field tags; packed type+tag byte and varints; 33 B; repeated; tag-based evolution (rename, add, remove/reserve, type-widening truncation).
• DDIA2 ch. 5 pp. 172–178 (Avro & merits of schemas) — no tags, 32 B; writer's vs reader's schema resolution by name; add/remove-with-default rule; union types for null; aliases; object container files, version tables, negotiation; dynamic schemas; ASN.1 lineage; the merits of schema-based encodings.
• DDIA2 ch. 5 pp. 178–181 (dataflow: databases & services intro) — "message to your future self"; concurrent access needing both directions; "data outlives code"; nullable-column adds and LSM compaction; archival snapshots and Parquet; independent deployability.
• DDIA2 ch. 5 pp. 186–191 (RPC, durable execution, event-driven) — servers-first direction (backward requests / forward responses); cross-org indefinite compatibility and side-by-side versions; workflows, durable execution's replay and determinism constraint; brokers, schema registries, AsyncAPI, preserving unknown fields on republish.
• REST/RPC framing and the network-call reasoning are developed in the sibling API Design lesson; service-transport detail is deferred there.
• Flags: the Data Models prerequisite link targets a module-02 lesson authored alongside this one; it resolves once that ships. "Low single digits of live schema versions" is a rule of thumb, not sourced. Buf, Confluent/Apicurio, and Cassandra/RocksDB are cited as industry illustrations of DDIA's mechanisms, not as DDIA claims.