Model routing and cascades

Why it matters

A single provider's lineup spans about $0.10 to $8 per million input tokens, an eighty-fold range, and most products put every request on the dearest end because that was the model the demo used. Most traffic does not need it. Classifying a ticket, extracting a date, rewriting a sentence or answering a question the retrieved passage answers already are jobs a small model does as well as a frontier one, and the hard tenth of requests is where the frontier model earns its price. Routing is the lever that turns that observation into money, and interviewers ask about it because "how would you halve this bill without hurting quality" is the question every team with a model in production eventually gets.

The idea

Tiers and the within-model lever

Think in three tiers. A small tier for classification, extraction, routing itself and anything with a short, checkable output. A mid tier for most conversational and retrieval traffic. A frontier tier for multi-step reasoning, long synthesis, hard code and agent planning. Current lineups add a lever inside each model: an effort or thinking setting that controls how many reasoning tokens the model spends before answering. Low effort on a frontier model and high effort on a mid model overlap in both quality and cost, so the router's output is really a pair, tier and effort, not a model name.

Three ways to route

Rules route on what you already know: the feature that made the call, the input length, the tenant's plan, whether a tool is involved. They are free, deterministic and the right first step.

A classifier predicts the tier from the request itself. It can be a small model asked for one label, a trained head over an embedding, or a few features such as question length and the presence of code. It adds a few milliseconds and a model whose own errors you now own.

A cascade tries the cheap tier first and escalates when the answer looks weak. The signal can be the model's own stated confidence, a schema or validator failure, a cheap judge, or disagreement between two small samples. Escalation costs a second call plus the first one's latency, so a cascade pays off only when most requests stop at the first step.

Measuring the router

A router is a model and is evaluated like one. Keep a per-tier eval: for each tier, the quality of its answers on the slice the router sends it, so a tier that quietly degrades is caught. Then build the confusion matrix: which tier each request was routed to against which tier it needed, judged by a frontier model or a human on a sample. The two expensive cells are opposite corners. Hard requests sent down cost quality and show up in complaints; easy requests sent up cost money and show up on the invoice.

                  needed small   needed mid   needed frontier
routed small           2,610          140                12
routed mid               180        5,400               310
routed frontier           10          260               678

The 310 mid-routed requests that needed the frontier tier are the ones to study; the 180 that needed small are the budget. Track the escalation rate of a cascade as its own metric, because a drifting confidence signal moves it silently.

Fallbacks across providers

Routing also buys availability. Each tier should map to a model on at least two providers, with the prompt and the eval set portable between them, so a rate limit or an outage becomes a route change rather than an incident. The cost is that caches are per model, so a failover starts cold, and that two models at the same tier are never identical, so the per-tier eval must pass on both.

The arithmetic

Take 100 million input tokens a day. All on the frontier tier at $8 per million is $800 a day, around $24,000 a month. Route 30 percent to the small tier at $0.10 per million, 60 percent to a mid tier, and 10 percent to frontier: the small slice costs $3, the frontier slice $80, and the mid slice sits between $6 and $480 depending on where in the lineup that tier is priced. The day's bill lands between roughly $90 and $560 against $800, and the saving is larger still if the mid tier is nearer the cheap end, which it usually is. Output tokens scale the same way, since every tier prices them as a multiple of its input rate.