Routing

Why it matters

A single prompt that handles refunds, outages, password resets and small talk is long, contradicts itself, and runs on the largest model because one of those jobs needs it. Routing splits the traffic first. A cheap classifier sends each input to the prompt, tools and model built for it, so the refund prompt never mentions outages and small talk never touches a frontier model. Interviewers ask because routing is where a demo acquires a cost structure, and because its failure is quiet: a fluent answer from the wrong specialist.

The idea

Classify, then dispatch

A router sits in front of a set of routes. It assigns each input a label from a closed set; the label selects a route. A route is a bundle: a system prompt, a tool set, a model, and a set of evaluation cases. The labels are product decisions: for a support product, billing, technical, account, cancellation and general. Keep the set under ten, with an explicit general label so nothing is forced into a route it does not fit.

Building the classifier

Rules match patterns: keyword lists, a regular expression on an order number, the arrival channel. Free, instant, brittle to phrasing. A small model reads the input with the label definitions and a dozen examples per label and returns a label and a confidence in a few hundred milliseconds for a fraction of a cent. Embedding similarity embeds the input and compares it with a labelled example set; the nearest neighbours vote. Cheap and easy to update by adding examples, but weak on labels that turn on intent rather than topic. Production routers layer them: rules for the unambiguous cases, a small model for the rest.

{
  "type": "object",
  "properties": {
    "route": { "type": "string", "enum": ["billing", "technical", "account", "cancellation", "general"] },
    "confidence": { "type": "number", "minimum": 0, "maximum": 1 },
    "reason": { "type": "string", "description": "One clause naming the cue" }
  },
  "required": ["route", "confidence", "reason"]
}

Separation of concerns

The real gain is what the split does to engineering. Each route has its own prompt, so rewriting the billing prompt cannot regress the technical route. Each route has its own evaluation set, so a change is measured on the traffic it affects. Each route picks its own model and tools: a frontier model with retention tools for cancellation, a small model with none for general. The router becomes the only shared component, and it is measured on its own.

Routing as the cost lever

Prices in 2026 span nearly two orders of magnitude: small models list at about a tenth of a dollar per million input tokens, frontier models at several dollars. Take a million requests a day at 1,000 input tokens each. All on a frontier model at $5 per million: $5,000 a day. If the router sends 70 percent to a small model at $0.10: 700 million tokens for $70, plus 300 million at $5 for $1,500, about $1,570 a day, under a third of the bill. The router adds about $20 a day. No prompt trim, caching trick or batch discount comes close. The lever only works if the router is right.

Measuring the router

Hand-label a sample of real traffic, run the router, and build a confusion matrix: true labels as rows, predicted as columns. Every off-diagonal cell is a mis-route, and they cost different amounts. A technical question sent to billing wastes a turn. A cancellation sent to general loses a customer. Weight each cell by what the mistake costs, then move the thresholds so the expensive mistakes become rare even at the price of more cheap ones. Re-run it on every change to labels, prompt or model, and as the traffic mix drifts.

The mis-route and the fallback

Three defences. Send low-confidence inputs to the general route, a capable model with a broad prompt. Let a route hand back: a specialist told to say when an input is not its job, which triggers one re-route, never a loop. And check the output: a cheap grade on the route's answer that escalates to general on failure. Count every fallback by route; a rising rate on one route is drift.