Distillation and small models

Why it matters

The frontier model that makes your demo work is often the wrong model to run it on. At a few million calls a day, a large model's cost and latency dominate the product, while the task itself, classify this ticket, extract these fields, pick which tool to call, is narrow enough that a model a tenth the size could do it, if only it had been taught. Distillation is that teaching: the large model becomes the teacher and the small one the student. Interviewers like it because it ties the whole module together and ends in a number.

The idea

Distillation, in the form teams use, means generating outputs from a large model on your inputs, filtering them with your evals, and fine-tuning a small model on the result. The student learns the teacher's decisions on your distribution. Early research distilled the teacher's full output probabilities; what teams do today is SFT on filtered teacher outputs, sometimes with the teacher's reasoning traces included so the student learns the steps and not only the answers.

Why it works, and where it fails

It works because a narrow task needs a narrow function. A small model lacks the breadth of a large one, but it does not need breadth to map a support ticket to one of forty labels; it needs that mapping, and ten thousand teacher-labelled examples describe it well. Reasoning traces help further: the student learns to write a short chain of working on the way to the label, which makes it more robust than one trained on labels alone.

It fails where the task is not narrow. Open-ended generation, long multi-step reasoning, anything where quality is a matter of judgement: the student imitates the surface and misses the judgement. It also fails off-distribution. Inputs unlike the training set, a new product line, a new language, a new attack, and the student degrades faster than the teacher would, with no warning.

The tasks that suit a small model

Classification, extraction into a schema, routing between tools or tiers, and guardrails (is this input an injection, is this output on topic) are the natural fits: bounded outputs, clear evals and volume. A small model on these runs at a fraction of the cost and latency, on a CPU or one small accelerator, and can sit in the request path where a large model could not.

The pipeline

Start with the large model in the loop on all traffic, logging inputs and outputs. Build the eval set from a judged sample. Generate training data from the teacher, filter it by the evals and a judge, and fine-tune the student, usually with LoRA on a small open-weight base. Shadow the student against the teacher on live traffic until the gap on the eval set is acceptable, then switch the traffic. Keep the teacher for two jobs: escalation, where the student's confidence is low or the input looks unfamiliar, and data generation, labelling a continuing sample so the student retrains as the distribution drifts.

Measuring the gap

The gap is a number on your eval set, not a feeling: accuracy or F1 for a classifier, field-level match for extraction, agreement with the teacher on routing, broken down by slice so you see where the student fails. Watch the escalation rate as a second signal; if it climbs, the distribution moved. Fix drift by generating fresh teacher labels and retraining, which is cheap once the pipeline exists.

The licence question

Using one provider's outputs to train a model you serve, or a model from another provider, is governed by the first provider's terms, and some terms restrict exactly that. Open-weight models carry their own licences, some with conditions on derived models. This is a thing to check before the first training run, with someone who reads contracts, not a thing to assume.

	Large model everywhere	Distilled small model	Small plus escalation
Cost	Highest; scales with every call	A fraction; often CPU-class	Small model cost plus a few percent of calls at large-model price
Latency	Seconds at the tail	Tens to hundreds of milliseconds	Fast for most calls; slow for the escalated few
Quality on hard cases	Best available	Worst; degrades silently off-distribution	Near the large model, if the confidence signal is calibrated
Maintenance	Vendor upgrades for free	Retrain on drift; your eval set is the guardrail	Two models, a router and a retraining loop