Post-training: SFT, RLHF, DPO

Why it matters

The difference between a base model and the assistant you call is a few weeks of post-training on a few million chosen examples. It is a small fraction of the compute and almost all of the behaviour: the chat format, the refusals, the tone, the reasoning style. When a model flatters a wrong user, declines a harmless request, or opens every answer with the same cheerful phrase, you are looking at a post-training artefact, and an interviewer who asks "why does the model do that?" wants to hear which stage caused it. This lesson covers what the provider did; the Fine-tuning module in the Operating stage covers doing it yourself.

The idea

A base model continues text. To make it answer, three things have to be taught: the shape of a conversation, what a good answer looks like, and when to hold back. Each has a stage.

Supervised fine-tuning

SFT trains the base model with the same next-token loss on demonstrations: prompts paired with the answer a skilled person, or a stronger model, would write, rendered through the chat template so the model learns the turn markers. Tens of thousands to a few million examples turn a base model into something that answers in the expected shape. SFT teaches format and competence by imitation; it cannot exceed its demonstrations and has no way to say that one answer beats another.

Preference optimisation with a reward model

RLHF adds that comparison. Raters, human and increasingly model, compare two answers to the same prompt. A reward model, usually a copy of the policy with a scalar head, is trained on those pairs to predict which answer a rater would prefer. The policy is then trained by reinforcement learning, classically PPO, to produce answers the reward model scores highly, with a penalty on drifting from the SFT model so it cannot wander into text that scores well and reads badly. This stage shapes helpfulness, tone and refusals. It is expensive: four models in memory, a noisy optimisation, and a reward model only as good as its preference data.

DPO

Direct preference optimisation reaches a similar place without a reward model or an RL loop: it takes the same preference pairs and trains the policy directly, with a loss that raises the probability of the chosen answer over the rejected one, measured against the SFT reference. It is one supervised-looking training run, stable and cheap, which is why it dominates open-weight post-training. Its limit is that it learns only from the pairs it was given; RLHF can explore new answers and score them.

GRPO and verifiable rewards

For maths, code and tool use, the reward need not come from a rater: the unit test passes or it does not, the final number matches or it does not. GRPO samples a group of answers to each prompt, scores each with such a verifiable reward, and pushes the policy towards the answers that beat the group average, which serves as the baseline instead of a separate value model. Trained this way, models learn to write long chains of working, because thinking longer raises the chance that a sample is correct. This is the stage that creates a reasoning model, which the next lesson picks up.

What each stage changes

SFT sets the format and the baseline competence. The preference stage sets helpfulness, verbosity, tone and the refusal boundary. Reasoning RL raises accuracy on tasks with checkable answers and changes how the model spends tokens. Knowledge barely moves at any stage: post-training is steering, not teaching facts, which is why retrieval rather than fine-tuning is the answer to "the model does not know our data".

	SFT	RLHF	DPO	GRPO
Data needed	Prompt plus an ideal answer	Preference pairs	Preference pairs	Prompts with an automatic checker
Reward model	None	Trained separately	None; implicit in the loss	None; a verifier
Stability and cost	Stable, cheap	Fragile, four models in memory	Stable, one run	Moderate; many samples per prompt
What it teaches	Format and imitation	Helpfulness, tone, refusals	The same, from fixed pairs	Reasoning on checkable tasks
Limit	Cannot exceed the demos	Reward hacking	Only the pairs it saw	Needs a verifier

Trade-offs

Each stage has a signature failure. Preference training rewards what raters like, and raters like being agreed with, so models drift towards sycophancy: changing a correct answer when the user pushes back. A reward model is a proxy, and the policy will find its gaps, which is reward hacking: length, confident tone and heavy formatting that score well without being better; with verifiable rewards it appears as code that special-cases the tests. Over-weighting safety preferences gives over-refusal, where the model declines to explain how to kill a Linux process. And every stage costs a little of the base model's breadth, the alignment tax, which is also why a fine-tune on your own data can quietly erode a provider's refusals.

In an interview

Say: "A base model is autocomplete. SFT teaches it the format, preference optimisation teaches it what people prefer, and RL on verifiable rewards teaches it to reason. Most odd behaviours are artefacts of one of those three." At junior depth, name the stages and explain SFT. At mid depth, explain reward models, why DPO can skip one, and sycophancy versus reward hacking. At senior depth, discuss GRPO, the alignment tax, why fine-tuning can weaken safety, and how pushback tests detect sycophancy. Expect: "Why not stop at SFT?" (it cannot express preference or exceed its demonstrations). "Where does refusal behaviour come from?" (the preference stage, which is why you cannot fully prompt it away).