Module 17 · Observability · Lesson 2 of 4
Cost and token accounting
Cost is tokens times price in four buckets: uncached input, cached input at a tenth or less, output, and thinking billed as output; compute it on the span, attribute it per request, user, feature, prompt version and run.
Why it matters
A model-backed feature is the first thing most teams have shipped whose marginal cost per request is visible, variable and large. A web request costs a fraction of a cent whatever the user does; a model request costs anywhere from a thousandth of a cent to a dollar depending on the prompt, the model, the cache state and how long the model thinks. Finance notices the bill before engineering notices the cause. Interviewers ask about cost accounting because it is where "we use an LLM" meets "and the unit economics work", and because the answer shows whether you can turn a usage object into a decision.
The idea
Cost is tokens multiplied by price, and both halves vary. The price depends on the model, on whether an input token was read from the prompt cache, and on whether a token is input or output. The tokens depend on what you sent and what came back. The usage object on every response gives the counts, the price sheet gives the rates, and the trace gives the dimensions to attribute by.
The four buckets
Uncached input at the model's base input price, which across the current frontier lineups runs from about ten cents to eight dollars per million tokens. Cached input: prompt-cache reads of a prefix the provider has seen recently cost 2.5 to 10 percent of the base input price. The Prompt caching, batch APIs and the cost model lesson covers how to earn those reads; here they are a separate line because a change in cache hit rate changes the bill without changing a single token count. Output, priced per token at its own, higher rate. Thinking tokens, which the model generates before its answer and which are billed as output, often with none of them visible to you. A hard question at high effort can spend several times more tokens thinking than answering, and that is where cost spikes hide. Batch requests, where the provider may take hours to respond, run at half price on every bucket.
In the worked example, 6,000 input tokens of which 4,000 are cached, plus 400 output and 1,200 thinking tokens, cost about 3 cents on an illustrative price sheet of $3 per million input, a tenth of that for cache reads and $15 per million output. Thinking is 58 percent of it. With no cache hit the request costs 4 cents; with effort turned down so the model does not think, 1.3 cents.
Attribute, then decide
A total is a number; a total per dimension is a decision. Compute the cost on the span the moment the usage object arrives, from a price table keyed by resolved model id, and let the trace's metadata carry the dimensions. Per request, for the engineer reading one trace. Per user or tenant, for fair-use limits and pricing tiers. Per feature, for which feature pays for itself. Per prompt version, so a prompt edit that doubled cost is caught before it reaches everyone. Per agent run, so a loop shows up as one run worth a hundred. The day's sum per feature divided by that feature's conversations is cost per conversation. Set it against revenue per conversation and you have the unit economics question that decides whether the feature survives.
Budgets and alerts
Give each feature a daily budget and alert at a fraction of it on rate, not total, so a loop that starts at 2 a.m. is caught at ten past. Give each agent run a token budget enforced in the call path and record the hit as an attribute. Alert on cost per request leaving its usual band per prompt version: that is the earliest signal of a prompt change, a cache-miss storm or a provider change behind an alias.
The three levers, in order
When cost per conversation is too high, pull these in order. Route easy requests to a smaller model: the price gap between tiers is tens of times, larger than any other lever, and the Model routing and cascades lesson covers how to decide. Cache the prefix: put the stable system prompt, tools and documents first, and the cache read price takes most of the input bill away. Shorten the context: trim history, send fewer and shorter passages, cap output and lower effort where the task is easy. Shortening comes last because it is the lever most likely to cost quality.
Show the cost where engineers look
Print the cost on every span in the trace viewer, next to latency. An engineer who sees "$0.031, 58 percent thinking" on the span they are reading will fix it; one who sees a line on a finance dashboard a month later will not.
| Per request | Per user or tenant | Per feature | |
|---|---|---|---|
| Decision it supports | Why this one cost what it did; which span to fix | Fair-use limits, pricing tiers, abuse detection | Does the feature pay for itself; where to pull a lever |
| How to tag | Cost on each span from usage and the price table | User or tenant id on the root span | Feature name and prompt version on the root span |
| Granularity | One trace, in fractions of a cent | Daily and monthly sums per id | Daily sum divided by conversations |
| Fails when | Nobody reads individual traces | Background and batch calls run with no user id | One prompt serves several features |