Preference tuning: RLHF, DPO, GRPO

Why it matters

Supervised fine-tuning gets a model to the shape of the task; it does not get it to the taste. After SFT, outputs are correct in format and competent, and still wordy, or too eager, or inconsistent about when to decline. Fixing that needs a signal that says this answer beats that one, which is a different kind of data and a different training step. The Post-training lesson covered a provider doing this at frontier scale; this lesson is about whether and how a team should do it on its own model. Interviewers ask because it is where the expensive mistakes live, and because "should you do this at all" has a short answer that candidates rarely give.

The idea

A preference step starts from an SFT model and a signal about which outputs are better. The methods differ in where the signal comes from and how it reaches the weights.

RLHF: a reward model and a policy

Collect pairs: one prompt, two answers, a rater's choice. Train a reward model to predict the choice, then optimise the SFT model, now the policy, by reinforcement learning, classically PPO, to produce outputs the reward model scores highly, with a penalty on distance from the SFT reference so it cannot drift into text that scores well and reads badly. It is the strongest method because the policy explores: it generates new answers and is scored on them, rather than learning only from pairs it was handed. It is also the most fragile: a reward model to train and validate, four models resident during training, and hyperparameters that decide whether the run converges or collapses.

DPO: the pairs without the reward model

Direct preference optimisation takes the same pairs and skips the middle. Its loss raises the probability of the chosen answer and lowers the rejected one, relative to the reference model, which turns out to be the same as fitting an implicit reward. One supervised-looking run, stable, cheap, no reward model to maintain. The limit is that it learns only from the pairs it saw; it does not explore. For a team with a few thousand pairs and one accelerator, that limit rarely bites, which is why DPO is the usual choice.

GRPO and verifiable rewards

When a program can check the answer, nobody needs to rate it. Unit tests pass or fail; a number matches or does not; a schema validates or does not. GRPO samples a group of answers to each prompt, scores each with the verifier, and pushes the policy towards the ones above the group average, using the group as its own baseline instead of a separate value model. Trained on checkable tasks, models learn to spend more tokens working before answering, which is the mechanism behind reasoning models. It applies when you have a verifier and a task where the answer is more than a style: code against tests, extraction against a schema, maths against a solution. Without a verifier it is RLHF with a cheaper critic and the same risks.

What each teaches

The preference stage shapes helpfulness, verbosity, tone and the refusal boundary. With verifiable rewards it raises accuracy on the checkable task and changes how the model spends output tokens. None of it adds knowledge. If the SFT model is wrong about facts, preference tuning teaches it to be wrong more persuasively.

Where the pairs come from

Human raters: slow, expensive, and the standard for taste. A judge model with a rubric: fast, and it inherits the judge's biases, including a liking for length and confidence. Verifiers: exact, and only available for some tasks. Mixing is normal. The judge produces the volume, humans audit a sample, verifiers cover the checkable slice.

Should a team do this at all

Rarely. Most teams that need better answers need better retrieval, a better prompt or better SFT data. The cases that justify a preference step are specific: a tone or format that SFT cannot stabilise, a refusal boundary that must be yours, or a checkable task with a verifier and a measurable gap. When justified, the typical shape is DPO on a few thousand pairs from a judge plus a human audit, run once, measured on a held-out set with a pushback test, and repeated only when the data changes.

	RLHF	DPO	GRPO
Data	Preference pairs, then on-policy samples	Preference pairs only	Prompts with an automatic checker
Reward model	Trained and maintained separately	None; implicit in the loss	None; the verifier scores
Stability	Fragile; four models and sensitive hyperparameters	Stable; one supervised-looking run	Moderate; many samples per prompt
What it teaches	Helpfulness, tone, refusals, with exploration	The same, from fixed pairs	Accuracy and reasoning on checkable tasks
When	A dedicated team and a validated reward model	A few thousand audited pairs for tone or format	You own a verifier the policy cannot game