Module 20 · Fine-tuning · Lesson 3 of 5
LoRA and QLoRA
Full fine-tuning needs about sixteen bytes per parameter; LoRA trains small low-rank adapters on a frozen base that swap per customer, and QLoRA adds a 4-bit base so a large model tunes on one accelerator.
Why it matters
Fine-tuning moved from research labs to product teams when it stopped needing a cluster. Full fine-tuning of a mid-sized model wants several times the memory of the model itself; a low-rank adapter wants one accelerator and an afternoon, and the result is a file small enough to attach to a ticket. Interviewers ask about LoRA because it is the method almost every team that tunes uses, and because the arithmetic behind it, why full tuning does not fit and why an adapter does, is a clean test of whether you know what training costs.
The idea
Why full fine-tuning is expensive
Training updates every weight, and every weight needs company in memory: the weight itself, its gradient, and the optimiser's running statistics, which for the usual Adam-style optimiser are two more values per weight, often kept at higher precision. The rule of thumb is about sixteen bytes per parameter during training against two for serving. An 8-billion-parameter model that serves in 16 GB needs roughly 128 GB to train in full, before activations; a 70-billion one needs over a terabyte, which is a cluster and a sharding strategy, not a job.
LoRA
Low-rank adaptation keeps the base weights frozen and adds, beside chosen weight matrices, a pair of small matrices A and B whose product has the same shape as the frozen matrix but a low rank r. Only A and B train, and their product is added to the frozen matrix's output. For a 4,096 by 4,096 matrix at rank 16, that is two times 4,096 times 16 values instead of 16.8 million, under one percent. Across the layers you target, trainable parameters land between a few tenths of a percent and a few percent of the model. Gradients and optimiser state exist only for those, so memory drops to the frozen weights plus a small margin, and a mid-sized model tunes on one accelerator.
The adapter is a separate artefact, typically tens to a few hundred megabytes. It can be versioned, rolled back, and trained per customer or per task while the base stays shared.
Two knobs matter. Rank sets capacity: 8 to 16 is enough for style and format, 64 and up for a harder behaviour shift, at more memory and more risk of overfitting a small set. Target modules choose which matrices get adapters: the attention projections alone is the cheap default; attention plus the feed-forward layers tunes more and usually helps.
QLoRA
QLoRA quantises the frozen base to 4-bit before training, so a 70-billion-parameter model sits in about 35 GB, and trains adapters at higher precision on top. Forward passes dequantise blocks on the fly, which costs speed but not memory. A model that needed a cluster to tune in full now tunes on one large accelerator. Quality is close to LoRA on the 16-bit base for behaviour tasks: the quantisation noise lives in the frozen part, and the adapter learns around it.
Serving adapters
Two options. Merge: add the product of A and B into the base weights and serve one ordinary model with no runtime cost; simple, and you now hold a full copy per adapter. Load separately: keep one base in memory and load many adapters beside it, routing each request to its adapter. Servers in the vLLM style batch requests across adapters, so one deployment serves fifty customers' tunes at a small per-token cost. Merge for one adapter at high volume; load separately for many.
A cost sketch
In accelerator hours rather than currency, because prices move. LoRA on an 8-billion model, 5,000 examples of a few hundred tokens each, three passes: roughly one to two hours on one accelerator. QLoRA on a 70-billion model with the same data: several hours on one large accelerator, slower per step from dequantisation. Full fine-tuning of the 70-billion model: eight or more accelerators for a comparable time, plus the engineering to shard it. The run is cheap; the data and the evals around it are where the time goes.
| Full fine-tuning | LoRA | QLoRA | |
|---|---|---|---|
| Memory | About 16 bytes per parameter; a cluster past 10B | Frozen 16-bit weights plus a small margin | Frozen 4-bit weights plus a small margin |
| Trainable parameters | All of them | A few tenths of a percent to a few percent | Same as LoRA |
| Quality | The ceiling; needed for new skills or languages | Close to full for behaviour and format tasks | Slightly below LoRA; noise sits in the frozen part |
| Serving flexibility | One full model per tune | Merge, or load many adapters on one base | Same, served at 4-bit or re-merged into 16-bit |
| Fits on | Many accelerators | One accelerator for mid-sized models | One large accelerator for large models |