Models and the API drill

You own three workloads: a support-ticket classifier at 2 million calls a day (about 300 input tokens each, p95 under 300 ms), a contract summariser at 1,000 contracts a day (about 40K tokens each, results needed by the next morning), and a coding assistant for 400 engineers (first token under one second, repository context of about 30K tokens per session). The total model budget is $600 a day. Pick a model tier and the API settings for each, and show the arithmetic.

A chat assistant runs on a self-hosted open-weight model with a 128K-token window. About 2 percent of sessions, almost all from power users who attach several documents and chat for an hour, fail with a context-length error on the provider side. The product wants no visible failures and no drop in answer quality for everyone else. Design the fix.

An invoice extraction pipeline processes 200,000 documents a day by asking a mid-tier model for JSON in the prompt and parsing the reply. About 3 percent of outputs fail: markdown fences around the JSON, a trailing sentence, a missing field, or a total quoted as a string. The team is retrying the whole call blindly, which has doubled cost on the failing cases without clearing them. What do you change, and what failure rate do you expect afterwards?

A product manager says the chat assistant feels slow: users wait about 8 seconds for an answer. The model dashboard shows generation taking about 2 seconds per request. The assistant retrieves from a document index and calls two tools on a typical turn, and the web client renders the reply when the response completes. Explain where the other 6 seconds go and what you would fix first.