Module 19 · Serving and cost · Lesson 3 of 6
Speculative decoding and quantisation
Two ways to spend less memory bandwidth per token: a draft model proposes tokens the large model verifies in one pass, and weights or cache stored in fewer bits; what each buys, what it risks, and how to measure the quality cost.
Why it matters
Decode is bandwidth-bound: each step streams the weights and the cache through memory to produce one token. Batching spreads that read across requests, but a single user's answer still arrives at one token per step. There are two ways to get more out of each read. Produce more than one token per pass over the large model, which is speculative decoding. Or make the bytes smaller so each pass reads less, which is quantisation. Both are standard in every serious serving stack, both are invisible to an API user, and both come up the moment an interviewer asks how you would make a self-hosted model faster without changing its answers. The honest reply is that one of them does not change the answers and the other one does, a little, and you have to measure it.
The idea
Speculative decoding
A small draft model that shares the large model's tokeniser runs several decode steps cheaply and proposes, say, five tokens. The large model then processes those five positions in a single pass, much as prefill processes a prompt, and produces its own next-token distribution at each one. Because decode leaves the accelerator's arithmetic mostly idle, scoring five positions costs about the same as scoring one.
Verification walks the proposals left to right. Where the large model agrees with the draft the token is accepted; at the first position where it does not, the token is resampled from a corrected distribution and the remaining proposals are discarded. The sampling rule is chosen so that the sequence of accepted tokens is drawn from exactly the distribution the large model would have produced on its own. The output is unchanged; only the clock moves.
The gain is set by the acceptance rate. Predictable text, such as code, boilerplate, JSON scaffolding and answers that copy from retrieved context, accepts most proposals and can run two to three times faster. Open-ended prose accepts fewer and the extra draft work can cancel the gain. Variants remove the separate draft model: extra prediction heads bolted onto the large model guess several tokens ahead from its own hidden state, an early exit from its own layers serves as the draft, or an n-gram lookup over the prompt proposes continuations, which suits retrieval-heavy workloads where the answer quotes the context. Gains also shrink as the serving batch grows, because the idle arithmetic that made verification free is no longer idle.
Quantisation
A model's weights are trained in 16-bit floating point. Quantisation stores them in fewer bits, commonly 8 or 4, with a scale factor per group of values so the range is preserved. Memory halves or quarters, so a model that needed two accelerators fits on one and the cache has more room; bandwidth per decode step falls with it, so tokens per second rises. Weight-only quantisation keeps the arithmetic in higher precision and is the common choice for serving. Quantising activations as well lets prefill use the accelerator's integer units, which raises throughput on long prompts. Quantising the KV cache to 8 bits doubles the context that fits in the same memory, which matters for long conversations more than any weight trick.
The quality cost is real and uneven. Eight bits is usually indistinguishable on evals. Four bits is often fine on large models and visibly worse on small ones, on long reasoning chains, on numbers and on languages under-represented in the calibration data. Post-training methods use a few hundred calibration samples to decide scales and to protect the outlier channels that matter most, which is why a quantised checkpoint tuned on English web text can degrade on your domain while scoring the same on a generic benchmark.
Distillation, the training-side cousin
Speculative decoding and quantisation leave the model's knowledge alone. Distillation trains a smaller model on the larger one's outputs and makes the compression permanent, which is where provider "fast" tiers often come from. It belongs to the Fine-tuning module; here it matters only as the third option on the list when a serving trick is not enough.
When each applies, and how to measure
Self-hosters choose: a quantised checkpoint, a draft model matched to the workload, and a KV-cache precision. API users get them implicitly; a provider's cheaper tier is some mix of distillation, quantisation and speculation, and the visible knobs are model tier and effort. Either way the measurement is the same: run the module's eval set against the baseline and the variant, broken out by task, and look at the tails rather than the average.
variant exact match faithfulness p50 latency memory
16-bit base 0.912 0.95 1,840 ms 140 GB
8-bit weights 0.910 0.95 1,210 ms 72 GB
4-bit weights 0.893 0.93 880 ms 38 GB
Speculative decoding needs no quality eval, only a latency one; a draft that proposes badly costs time, not correctness.
| None | Speculative decoding | Quantisation | |
|---|---|---|---|
| Latency gain | Baseline | Up to two or three times on predictable text; none on open prose | Faster decode from fewer bytes per step; modest, steady |
| Throughput gain | Baseline | Small; fades as the batch fills | Large: a smaller model admits more requests and more cache |
| Quality risk | None | None by construction; the output distribution is preserved | Small at 8 bits, measurable at 4; worse on small models, numbers and rare languages |
| Where it applies | Any deployment | Self-hosted serving; providers run it behind the API | Self-hosted checkpoints and KV cache; providers' cheap tiers |