Multimodal input

Why it matters

A growing share of what a product needs the model to read is not text: a photo of a receipt, a scanned contract, a dashboard screenshot, a voice note. Frontier models take all of these, and each arrives through the same window at a cost and reliability that differ from text in ways the API does not warn you about. Interviewers ask about multimodal input because the trade-off, send the pixels or send the extracted text, is one most engineers have never made, and the wrong call either doubles the bill or quietly misreads numbers.

The idea

One window, many encoders

Every modality ends up as tokens in the same window at the same per-token prices from the model card; only the encoder differs. Text goes through the tokeniser. An image is resized, cut into tiles, and each tile encoded into a fixed number of tokens, so cost grows with resolution: a thumbnail costs a few hundred tokens, a full page a few thousand, and twenty screenshots can cost more than a long document. Documents are extracted as text (born-digital PDFs, where the characters are in the file), rendered as page images (scans, forms, anything where layout carries meaning), or both. Audio is transcribed to text first by a speech model, or fed natively to a model that accepts audio and can hear tone, overlap and hesitation. Video is sampled frames plus a transcript, the dearest input of all because every frame is an image.

{
  "role": "user",
  "content": [
    { "type": "image", "source": { "type": "base64", "media_type": "image/png", "data": "..." } },
    { "type": "text", "text": "Read the total from this invoice. Answer with the number only." }
  ]
}

Image or text

When the content is prose, send text: a thousand words is about 1,300 tokens as text, several times that as a page image, and the text is exact. When the meaning is in the layout, send the image: a table's columns, a chart's bars, a form's ticked boxes, a user interface's arrangement. Vision models read charts and screenshots the way a person does, which makes them useful for UI testing, dashboard summaries and "what is wrong with this screen" questions.

OCR or vision

Classic OCR gives exact characters cheaply and at scale, and fails on handwriting, rotated scans, charts and anything where position matters. Vision reads layout and context, handles handwriting and charts, and in return hallucinates: a plausible digit where the scan is blurred, a line item that fits the pattern, small text lost when the image was downscaled to the model's input size. The robust pipeline for documents that matter sends both, the OCR text for exact characters and the page image for layout, asks the model to reconcile them, then validates in code (line items sum to the total, dates are in order). Cropping the region that matters and sending it at higher resolution fixes most small-text failures at a fraction of the cost of sending the whole page large.

Audio in and out

A voice agent is a pipeline: speech to text, the model, text to speech, under the tight turn budget from the Streaming and latency lesson. Transcribing first keeps the model choice open and gives you a text log; native audio preserves tone and interruption but ties you to the models that accept it. Audio out is a synthesiser fed sentence by sentence from the model's stream, so speech starts before the answer is finished.

Pitfalls

Hallucinated detail is the dangerous one, because a wrong number in a confident, well-formed answer looks exactly like a right one. Small text vanishes under downscaling. Cost climbs with resolution and count. Injection: text inside an image is still text to the model, so an uploaded screenshot can carry instructions, which the Prompt injection lesson covers. And privacy: images carry faces, addresses and documents that logging and retention rules must treat as such.

Trade-offs

Multimodal input removes a preprocessing step and adds a cost line that is easy to miss until the invoice arrives. Pixels carry layout that text loses; text carries exactness that pixels lose; so the honest choice for important documents is both, at the highest cost. Vision errors are confident and silent, so every extracted number needs a check in code. Resolution is a dial with no right setting: too low loses small text, too high pays for pixels the model does not need; crop and zoom beats either extreme.

In an interview

Frame it as a routing decision: "I send text when the content is prose and pixels when the meaning is in the layout; for documents where a wrong digit costs money I send both and validate the result in code." At junior depth, know that images are tiled into tokens, cost tracks resolution, and charts and UI need vision. At mid depth, compare OCR and vision on tables, charts and handwriting, and design a crop-and-validate pipeline. At senior depth, discuss the voice pipeline and its budget, injection through images, retention rules for uploaded media, and when native audio is worth the lock-in. Expect "Why not always send the image?" (cost and hallucinated detail) and "How do you catch a misread total?" (cross-check against OCR text and the sum of the line items).