Adaptive routing by query complexity

Why it matters

The Graph RAG and Agentic RAG lessons ended the same way: the expensive technique pays for a minority of questions and is waste for the rest. That is a routing problem. Most traffic to any assistant is simple: a lookup, a definition, a greeting, a question the corpus cannot answer. A few percent need synthesis across documents or a chain of hops. If every question takes the most capable path, the bill and the latency are set by the hardest question; if every question takes the cheapest path, the hard ones fail. Interviewers ask about routing because it turns a demo into a product with a cost structure, and a candidate who proposes one pipeline for everything has not thought about traffic.

The idea

Classify, then dispatch

Put a router in front of the pipelines. It reads the question and assigns a class; the class picks the path. A workable set: lookup (one fact, one document), synthesis (several documents on one topic), multi-hop (a chain of facts across documents), out of scope (nothing in the corpus can answer it), and chit-chat (no retrieval needed). Each class maps to a tier: no retrieval for chit-chat and out of scope, hybrid RAG for lookups and most synthesis, graph RAG for multi-hop, the agentic loop for research-style questions and for synthesis that fails its grade.

What does the classifying

Three options, cheapest first. Rules: patterns and keyword lists catch greetings, product names, and shapes such as "compare" or "which of"; free, fast and brittle. A small model: a prompt with the class definitions and ten examples per class, run on the cheapest model in the lineup, returns a label in a few hundred milliseconds for a fraction of a cent. A trained classifier: a few thousand labelled questions fine-tune a tiny model that runs in milliseconds, the right choice once traffic is large and the classes are stable.

Routing by corpus too

Classes can also name a source. An assistant with a product manual, a ticket history and a pricing table should not search all three for every question; the router adds a corpus label and the pipeline searches only those. That cuts retrieval cost and, more importantly, keeps irrelevant chunks from the other corpora out of the context. A question that needs two corpora is a synthesis question by definition.

The router as a cost lever

Suppose a tier costs, per question, 0.1 cents for no retrieval, 0.5 cents for hybrid RAG, 2 cents for graph RAG and 5 cents for the loop. Sending everything through the loop costs 5 cents a question. With the split in the diagram the blended cost is 0.03 plus 0.275 plus 0.2 plus 0.25, about 0.75 cents, nearly seven times cheaper, with the hard questions still getting the expensive path. The router adds perhaps 0.05 cents, and every point of traffic it moves to a cheaper tier compounds with volume.

Measuring the router

Label a sample of real questions by hand, run the router, and build a confusion matrix: rows are true classes, columns are predicted ones. The off-diagonal cells are the mis-routes, and they are not equally bad. Sending a lookup to the loop wastes money. Sending a multi-hop question to hybrid RAG produces a confident wrong answer. Weight the cells by cost and by harm and tune the thresholds towards the cheaper mistake. Track cost and answer quality per class in production, because the traffic mix drifts.

Fallback escalation

A mis-route must not be the end of the story. Give each tier a cheap check, such as the relevance grade from the Agentic RAG lesson, and on failure escalate one tier up, once. The escalation rate per class is the second router metric: if 30 percent of lookups escalate, the lookup tier or the router is wrong, and the per-class numbers say which.

	No retrieval	Hybrid RAG	Graph RAG	Agentic loop
Latency	One model call	Retrieval plus one call	Traversal plus one call	Several calls in sequence
Cost per query	Lowest	Low	Moderate	Five to twenty times hybrid
Router sends here when	Chit-chat, out of scope, pure reasoning	Lookups, most synthesis	Multi-hop chains across documents	Research questions, low recall, failed grades
Typical share	Up to a third	Most of the rest	Under a tenth	A few percent