Data formats: JSON, Protobuf, Avro and friends

Why it matters

Every byte on the wire is paid for in bandwidth, parsing time and storage, and every field name is a promise to whoever reads it. The format you pick decides whether a message is readable in a log, how much smaller it can get, how fast a service can parse ten thousand of them a second, and what happens when you add a field next month. The choice is rarely dramatic, but the wrong one in the wrong place (text in a hot internal path, an untyped blob in a long-lived event stream) shows up as cost and incidents later.

The idea

A serialization format turns in-memory structures into bytes and back. The axes that matter are human readability, size, parse speed, whether a schema is required, and how the format handles evolution.

	JSON	Protocol Buffers	Avro	MessagePack / CBOR
Readable	Yes	No	No	No
Schema	Optional (JSON Schema, OpenAPI)	Required (.proto), compiled	Required, usually shipped with the data or via a registry	None (schemaless binary JSON)
Size	Largest: keys repeated in every message	Small: field numbers, varints	Smallest: no field tags, schema known to both sides	Smaller than JSON, keys still present
Speed	Slowest	Fast, generated code	Fast	Fast
Evolution	Add fields freely; no enforcement	Field numbers; add optional, never reuse numbers	Reader and writer schemas resolved; defaults required	Same as JSON, unenforced
Home turf	Public REST APIs, browsers, configs	gRPC, service-to-service	Kafka events, data lakes	Drop-in JSON replacement where size matters

Schema evolution rules

Whatever the format, the same discipline applies. Add fields as optional with sensible defaults. Never remove a field that any reader still expects; mark it deprecated and stop writing it. Never change a field's type or meaning; add a new field instead. In Protobuf, never reuse a field number, because old data encoded with it will be misread. In Avro, readers use their own schema plus the writer's, so every new field needs a default. A schema registry (common with Kafka) stores versions, checks compatibility on publish, and lets consumers fetch the schema by id instead of shipping it with every message.

Where each lives in a real system

JSON at the public edge, because browsers and partners parse it with no toolchain and humans read it in logs. Protobuf between services, because gRPC uses it and typed generated clients catch mismatches at compile time. Avro or Protobuf on event streams and in data lakes, where billions of records make per-byte savings matter and schemas must be enforceable over years. Compress JSON (gzip or brotli) before replacing it; compression recovers much of the size gap, and replacing the format mostly buys parse speed and typing.

Trade-offs

Text formats are debuggable and universal and cost size and speed. Binary formats are compact and fast and need schemas, codegen and tooling to inspect. Schemaless binary gets size without typing and without evolution guarantees. The hidden cost of any typed format is organisational: someone must own the schemas and the compatibility checks, or they become a new source of breakage.

In an interview

Say the format per layer with a reason: "JSON over REST at the edge for compatibility; protobuf over gRPC internally for speed and typed contracts; events on Kafka in Avro with a schema registry enforcing backward compatibility." Then the evolution rule: "Add optional fields with defaults, never remove or retype, never reuse a field number."