Structured output and schemas

Why it matters

A model that answers in prose is a chat feature; a model that answers in a shape your code can read is a component. Most production calls are the second kind: classify this ticket, pull the fields from this invoice, pick a tool and its arguments. Free text fails those jobs in small, constant ways: a stray sentence before the JSON, a renamed field, a number quoted as a string, a markdown fence around everything. At one failure in thirty, a pipeline of 200,000 documents a day produces 6,000 exceptions. Interviewers ask how you get reliable structure from a probabilistic generator because the answer shows whether you have shipped one.

The idea

Four ways give structure; they differ in who enforces it.

From prompting to constraint

Prompt and parse asks for JSON in the instructions and parses whatever comes back. Nothing is enforced; your parser catches failures after the fact. Fine for prototypes.

JSON mode makes the provider guarantee syntactically valid JSON. It will parse; it may still have wrong keys, a missing field or an invented enum value.

Schema-constrained output passes a JSON schema with the request and the provider guarantees conformance: the strongest guarantee the API offers and the default for extraction and classification.

A tool call declares the shape as a tool's input_schema and asks the model to call it; the arguments object is the answer, validated against the schema. On most APIs this is the same constraint machinery under a different name, and the natural choice when the structured result is an action rather than a record.

How constrained decoding works

The provider compiles the schema into a grammar. At each step the sampler works out which tokens could extend the partial output while keeping it valid under that grammar, and masks every other token to zero probability before sampling. The model never emits an unquoted key, an unclosed bracket or a value outside an enum, because those tokens were never on offer. The guarantee is syntactic: the output is a valid instance of the schema. Whether the values are true is a question constraint cannot answer.

Designing the schema

The model fills the schema one token at a time, so make the shape easy to fill correctly.

• Flat beats nested. One level of objects; arrays of scalars where possible. Deep nesting invites misplaced fields.
• Few optional fields. Each optional field is a decision the model must make; require what you can and use an explicit "unknown" rather than absence.
• Enums for closed sets. A category, a status, a yes-or-no: list the values, and the sampler cannot invent a fifth.
• Descriptions on fields. The description is the only per-field instruction the model sees; say what goes there, in what unit, and what to do when it is missing.
• Order fields as a person would fill them. Generation is sequential; evidence before conclusion makes the conclusion better.

{
  "type": "object",
  "properties": {
    "intent": { "type": "string", "enum": ["refund", "shipping", "account", "other"] },
    "order_id": { "type": "string", "description": "Order number as written, or empty if none" },
    "urgency": { "type": "integer", "minimum": 1, "maximum": 5 }
  },
  "required": ["intent", "order_id", "urgency"],
  "additionalProperties": false
}

Validate and retry

Constraint guarantees shape, not sense. After the output parses, run semantic validation: the order id exists, the dates are in order, the line items sum to the total. On failure, retry once with the validator's message appended ("urgency must be between 1 and 5; you returned 7") at temperature zero; most failures clear on the first retry. Cap retries at two, then route the item to a bigger model or a human queue and count it. The retry rate deserves a dashboard: it shows when a schema, prompt or model change has moved the failure floor.

The canonical uses

Extraction turns a document into a record: invoice fields, contract clauses, CV facts. Classification turns text into one of a known set: intent, routing target, risk tier. Both want a flat schema, enums, temperature zero, and an evaluation set with known answers so you measure field-level accuracy rather than "it parses".

Trade-offs

Constraint removes a class of failures and introduces a quieter one: a model forced into a schema always produces a confident, well-formed answer, even when the input has none. Give it a way out (an "unknown" enum value, an empty string by rule) or it will invent one. Tight schemas are brittle: every new field needs re-evaluation, and large schemas cost input tokens on every call. And none of this fixes the input: a scanned invoice with an unreadable total yields a valid record with a wrong number.

In an interview

Lead with the guarantee and its limit: "I use schema-constrained output or a tool call so the shape is guaranteed by the decoder, then validate the meaning in code and retry once with the error; constraint buys syntax, not truth." At junior depth, know the four approaches and why free text breaks programs. At mid depth, design a flat schema with enums and descriptions and describe the capped validate-and-retry loop. At senior depth, explain constrained decoding, the forced-answer failure mode and the "unknown" escape, and tie retry rate and field accuracy to an evaluation set. Expect "What does the schema not guarantee?" (correct values) and "What happens when the model cannot find the field?" (an explicit unknown, never a guess).