Prompt caching, batch APIs and the cost model

Why it matters

A model-backed feature has a bill that nobody modelled before launch, because the demo cost nothing and the price list was in units no product manager thinks in. Then the invoice arrives, it is the largest line in the infrastructure budget, and the questions start: what does one conversation cost, which feature is burning it, and what happens at ten times the users. Interviewers ask for the cost model because an engineer who can decompose a bill into its lines and name the lever for each one can defend a design; one who says "we will use the cheaper model" cannot.

The idea

The four lines

Every request bills four kinds of token. Fresh input at the model's base input price. Cached input, the part of the prompt that matched a recent prefix, read at 2.5 to 10 percent of base on current lineups. Output, at a multiple of the input price. And thinking, the reasoning tokens a model spends before answering, billed as output even though the user never sees them. Two modifiers sit across all four: the batch API halves the price of any request that can wait hours, and the tier sets the base price, which spans about $0.10 to $8 per million input tokens across a lineup.

A worked example

A support assistant carries a 30,000-token stable prefix of system prompt, tool definitions and policy excerpts, plus about 2,000 tokens a turn of retrieved passages and the user's message. A conversation runs ten turns. Price it on the frontier tier at $8 per million input tokens.

Without caching, each turn re-sends the whole prefix: ten turns of 32,000 tokens is 320,000 input tokens, or $2.56 a conversation. At 5,000 conversations a day that is $12,800 a day and about $384,000 a month, on input alone.

With caching and the prefix placed first, 300,000 of those tokens are cache reads. The 20,000 fresh tokens cost $0.16; the cached ones cost $0.24 at a 10 percent read price or $0.06 at 2.5 percent. The conversation now costs $0.40 or $0.22, the day costs $2,000 or $1,100, and the month about $60,000 or $33,000. Output is a separate line: 300 tokens a turn is 3,000 a conversation, small at these lengths, but it is the line thinking inflates. At 2,000 reasoning tokens a turn the output line grows eightfold, and effort becomes the lever that matters.

Making the cache hit

The Prompt templates lesson sets the layout rules: stable material first, volatile material last, nothing in the prefix that varies per request. Two more facts drive the economics. Caches have a lifetime, a default of minutes refreshed by every hit and a longer option at a higher write price, so a prefix must be read several times per lifetime to pay for its write. And caches are per model, so routing and failover start cold. Put the cache hit rate on the dashboard next to latency; a release that drops it is a release that raised the bill.

Batch for anything that can wait

The batch API takes a file of requests and returns results within a window of hours at half price. It is the home of every job with no one waiting: eval runs, index building and re-embedding, backfills when a prompt changes, nightly classification and summarisation. It stacks with caching when the batch shares a prefix, which a backfill always does.

Cost per request in the trace

The cost model is only useful if it is measured. Every trace line carries the four token counts and a cost computed from the price list in code, so the bill can be sliced by feature, tenant, prompt version and model.

req=c71e feature=support prompt=support@22 model=frontier in=2140 cached=30210 out=312 think=0 usd=0.044

Roll it up by feature and the invoice stops being a surprise; roll it up by tenant and the pricing page can be defended.

The three levers, in order

Route down. The tier sets the base price for every line, so moving traffic that does not need the frontier tier to a cheaper one shrinks all four at once; the Model routing lesson covers how. Cache the prefix. Order the prompt so the stable part is read at a tenth or less. Shorten the context. Retrieve fewer, better passages, trim history, drop the tool definitions a feature never uses. After those three come effort and output caps, which attack the output line, and batch, which halves anything offline.

When the model is not the bill

Sometimes the four lines are not the problem. An agent that makes eight tool calls per task prefills the growing prompt eight times, so the loop, not the per-call price, is the multiplier, and the fix is fewer rounds or a cached prefix across them. A retrieval pipeline that embeds, reranks and hosts a vector index for short prompts can cost more than the model it feeds. And a feature with a large fixed prefix and tiny traffic never hits its cache and pays full price for every call; for that one, the fix is a shorter prefix or a smaller model.