When to fine-tune: prompt, retrieve, then tune

Why it matters

"Should we fine-tune?" is the question every team asks in its second month, usually because a demo disappointed someone and fine-tuning sounds like the serious option. It is also the most expensive wrong turn in the field: months of data work to fix something a paragraph of prompt or a retrieval index would have fixed in a week. Interviewers ask it as a judgement test. They want an order of operations, a clear statement of what tuning can and cannot change, and a cost list before any enthusiasm.

The idea

Three levers change what a model does, and they are ordered by how cheap they are to pull and to undo.

Prompt, then retrieve, then tune

Prompting changes behaviour per request with text: instructions, examples, a schema. It costs minutes, ships without a deploy and reverses with an edit. Retrieval changes what the model knows per request by putting the right documents in the context; it costs an index and a pipeline, and it updates the moment a document changes. Fine-tuning changes the weights, so the behaviour is there on every request without being asked for, at the cost of data, training runs and a model you now own.

The order is not a taste; it follows from what each lever fixes. Most failures are instruction failures or missing-information failures, and the first two levers exist for exactly those. Fine-tuning is for the remainder.

What tuning changes, and what it does not

Fine-tuning is good at behaviour and format: a house style, a fixed output structure, domain vocabulary, a narrow classifier, declining in your way rather than the provider's. A few thousand demonstrations shift those reliably, because the model is learning a mapping it is already able to produce.

It is bad at knowledge. Facts in a small training set are learned weakly, forgotten unevenly and impossible to update without another run, and the model states them with the same confidence whether it absorbed them or not. Retrieval puts the fact in front of the model at request time, with a source, and changes when the document changes. "The model does not know our product" is a retrieval problem. "The model knows but answers in the wrong shape" is a tuning candidate.

Good reasons and bad reasons

Good reasons share a shape: the behaviour is stable, you can show it, and the economics favour baking it in. A fixed output format that must hold at 99.5 percent over millions of calls. A small model doing a narrow task that today needs a large one, which cuts cost and latency at volume. A domain register that no prompt quite captures. A classifier with clean labels.

Bad reasons share a shape too. "Accuracy" with no eval set that says what accuracy is or where it fails. Knowledge that changes weekly. A dataset of fifty examples, which is a prompt with extra steps. A belief that tuning is what serious teams do.

The bill

Count the costs out loud. Data: hundreds to thousands of clean examples, which someone must write or curate. An eval set first, or you cannot tell whether the run helped. Training runs, plural, because the first is never the last. A model to serve, with its own capacity, latency and on-call. Regressions on general tasks, since the weights moved. Re-tuning on every base-model upgrade, because an adapter belongs to the weights it was trained on, and bases improve every few months.

The usual production answer

Teams that tune well tend to land in the same place: a thin adapter, trained with LoRA on a strong open-weight base, serving beside retrieval. The adapter carries format and style; retrieval carries facts; the prompt carries per-request instructions. Each lever does the job it is good at, and the adapter is small enough to retrain in an afternoon when the base changes.

	Prompting	Retrieval	Fine-tuning
What it fixes	Instructions, examples, output schema per request	Missing or changing knowledge, with sources	Stable behaviour, format, style, a narrow task
Data needed	A few examples	A corpus and an index	Hundreds to thousands of clean demonstrations plus an eval set
Time to change	Minutes; edit and redeploy	As soon as the document changes	Days: fix data, rerun, re-evaluate, redeploy
Cost	Tokens per request, cacheable	Index, pipeline, retrieval latency	Data, runs, a model to serve, a retune per base upgrade
Reverses by	Editing text	Removing documents	Rolling back to the previous weights

Trade-offs

Fine-tuning trades flexibility for consistency: a prompt can change this afternoon, a tuned behaviour changes after a data fix and a run. It trades breadth for depth: the model gets better at your task and a little worse at everything else, and nobody notices until a side feature breaks. It trades a vendor dependency for an operational one: you no longer wait on a provider, and you now own a model. And it ties you to a base-model version in a market where the base moves faster than your data.

In an interview

Lead with the order and the distinction: "I prompt first, add retrieval if the problem is missing information, and only fine-tune for a format or behaviour problem that I can demonstrate with data and measure with an eval. Tuning changes behaviour; it does not reliably add knowledge." At junior depth, know the order and the behaviour-versus-knowledge rule. At mid depth, list the costs, name good and bad reasons, and explain why fifty examples is a prompt. At senior depth, describe the adapter-beside-retrieval architecture, the retune on base upgrades, and how to push back in a meeting without sounding obstructive: ask for the eval set, ask to see ten failing examples, ask which prompt variants were tried, and offer a two-week retrieval spike before any tuning budget. Expect: "What if the prompt gets too long?" Prompt caching makes a long prompt cheap; tuning is for when the prompt cannot express the behaviour, not when it is long. "Can tuning replace retrieval?" Not for facts that change.