The API surface: messages, roles, parameters

Why it matters

Every framework, agent library and "AI platform" wraps one HTTP call: send a system prompt and a list of messages, receive a completion. Engineers who know that call cold can debug any wrapper; engineers who know only the wrapper are stuck the moment it misbehaves. Interviewers probe the API surface to see whether you know what the model receives, what it returns, and which settings change behaviour rather than being turned out of habit.

The idea

The model is stateless. Each call carries everything: the instructions, the whole conversation so far, the tools on offer and the settings. The provider renders those into the chat template described in the Tokens lesson, runs the model, and returns what it generated with an account of what it cost.

The request

Three parts matter. The system prompt sets role, rules and format and is sent on every call. The messages are an ordered list of turns, each with a role (user or assistant) and content; the last is what the model answers. Content is a list of blocks, so a user turn may hold text and an image, an assistant turn text and a tool call. Tools are declared on the request as JSON schemas with a name and a description; the model may reply with a call to one instead of text. The Function calling lesson covers that loop.

{
  "model": "mid-tier",
  "system": "You are a support assistant. Answer from the context only.",
  "max_tokens": 1024,
  "temperature": 0,
  "messages": [
    { "role": "user", "content": "Order 4812 has not arrived." },
    { "role": "assistant", "content": "Which address was it shipped to?" },
    { "role": "user", "content": "The office one." }
  ],
  "tools": [{ "name": "lookup_order", "description": "Fetch an order by id", "input_schema": {} }]
}

The knobs

Max output tokens is a hard stop on generation and a ceiling on cost. Set it to the longest answer you want, not the model's maximum, so a runaway loop costs a bounded amount.

Temperature scales the probability distribution before sampling; top-p truncates it to the smallest set of tokens whose probabilities sum to p. Both trade determinism for variety. For extraction, classification, code and anything a program parses, set temperature to zero so the same input gives the same output almost every time ("almost" because batched floating-point arithmetic is not perfectly repeatable). Leave the default for drafting. Move one of the two, not both.

Stop sequences end generation when a given string appears: useful when you want one item from a list, or a reply that ends before a delimiter.

Effort or thinking controls set how much hidden reasoning a model spends before answering; the current frontier lineups expose one setting from minimal to maximal. It has the biggest effect on quality and cost for hard tasks, and none worth paying for on easy ones.

The response

The reply mirrors the request: content blocks (text, tool calls, sometimes a thinking summary), a stop reason saying why generation ended (a natural end, the token limit, a stop sequence, or a tool call awaiting your result), and usage with input tokens, output tokens and how many input tokens came from the prompt cache. Read the stop reason every time: an answer cut off by the limit looks fine to a user and is silently wrong for a parser. Log usage every time: it is your cost ledger and your cache-hit metric.

Errors, limits and retries

Providers rate-limit by requests and tokens per minute and return 429 when you cross either. Treat 429 and 5xx as retryable: exponential backoff with jitter, capped attempts. A retry that lands after a slow success bills you twice and, if the call had a side effect through a tool, duplicates it; send an idempotency key where the API supports one and make tool handlers idempotent regardless. Client timeouts must sit above the longest answer you expect, which at high effort can be minutes.

Streaming is a flag on the same request; the response then arrives as events carrying content deltas, with usage at the end. The Streaming and latency lesson covers what that changes.

Trade-offs

Statelessness is simple and expensive: every call re-sends the history, so cost and latency grow with the conversation until you trim it or cache the fixed prefix. Temperature zero buys repeatability within one model version and nothing across versions, so a changed model behind the same endpoint still changes your outputs. A low max-tokens cap truncates; a high one leaves cost unbounded. Retries hide transient errors and double-charge when the first attempt succeeded late. Field names differ between vendors, and a thin adapter of your own is cheaper than a heavy framework whose abstractions leak the moment a vendor adds a feature.

In an interview

Describe the call before the system: "The call is stateless, so I design around what I send each time: a system prompt, trimmed history, tool schemas and a max-tokens cap; and I read the stop reason and the usage block on every response." At junior depth, know the request and response shapes, temperature, and max tokens. At mid depth, add retries with backoff and idempotency, stop-reason handling, usage logging and the effort setting. At senior depth, cover a thin provider adapter, rate-limit budgets shared across tenants, and cost accounting from usage. Expect "Why not temperature zero everywhere?" (variety matters for drafting, and zero is not a guarantee) and "What do you do on a 429?" (back off with jitter, cap attempts, keep the retry idempotent).