Pre-training and scaling

Why it matters

The model behind the API began as a base model: a transformer trained for months on trillions of tokens to predict the next token. Everything the product does is either that ability or a thin adjustment to it. Pre-training answers the questions that come after the demo: why a bigger model is better and by how much, why an open-weight model listing 400 billion parameters can be cheap per token, and why a base model with no post-training answers a question with another question.

The idea

The objective

Pre-training feeds the model a stream of token sequences and, at every position, asks for the next token. The loss is the cross-entropy between the model's predicted distribution and the token that actually came next: low when the model gave the right token high probability, high when it was surprised. Averaged over a held-out slice of text, this one number is what training drives down. Nothing else is specified. Grammar, facts, code, arithmetic and the rules of chess all get learned because each one, somewhere in the corpus, makes the next token more predictable.

The corpus

The corpus is a filtered web crawl plus books, code, scientific text, forum threads and, increasingly, licensed and synthetic data: tens of trillions of tokens after duplicates, boilerplate and unsafe text are removed. Data mixing sets the share of code, maths and each language, and shapes what the base model is good at long before any fine-tuning. The cut-off is simply the last date present in the corpus.

Scaling laws

Loss falls smoothly and predictably as three things grow: parameters, training tokens, and the compute that multiplies them. On log axes, loss against compute is close to a straight line for each model size, and bigger models reach a lower floor, so a lab can fit the curve on small models and predict the loss of one a hundred times larger before spending the budget. Compute-optimal training is the second half: for a fixed budget there is a best split between model size and data, roughly twenty training tokens per parameter. Production models are deliberately trained far past that point, because serving cost scales with parameters and a smaller model trained longer is cheaper to run for its whole service life.

Data over quantity

In 2026 the binding constraint is not compute but good tokens. The high-quality public web has been read, so what moves the curve now is curation: deduplication, quality classifiers, domain mixing, and synthetic data generated by stronger models and filtered for correctness. Two models of the same size and compute can differ sharply from data alone, which is why data recipes are closely held.

Mixture-of-experts

A dense model runs every parameter for every token. A mixture-of-experts (MoE) model replaces each feed-forward layer with several expert copies and a small router that sends each token to one or two of them. Total parameters grow with the number of experts; compute per token stays close to one expert plus the shared layers. That is how an open-weight model can list hundreds of billions of parameters yet cost about as much per token as a dense model a fifth of the size. The catch is memory: every expert must be loaded, so serving needs the full parameter count in GPU memory even though most of it sits idle for any given token.

	Dense	Mixture-of-experts
Total parameters	All of them used on every token	Many experts, most idle for any one token
Active parameters per token	Equal to the total	One or two experts plus shared layers, a fraction of the total
Compute and cost per token	Proportional to total size	Proportional to active size
Serving memory	Total parameters	Total parameters, idle experts included
Operational catch	Simple; expensive at the top end	Routing imbalance, uneven batching, needs several GPUs

What a base model can and cannot do

A base model is a very good autocomplete. Give it the start of a document and it continues in that style; give it a few worked examples and it continues the pattern, which is where few-shot prompting comes from. It has no notion of a user or a refusal: ask it the capital of France and it may answer, add more quiz questions, or reproduce the rest of a page it once saw. Turning it into an assistant is the Post-training lesson.

Trade-offs

Scaling is predictable in loss, not in capability: a smooth curve hides abilities that appear abruptly at some size and others that never appear. A bigger model costs more per token for the rest of its life, which is why the market is converging on medium models trained far past compute-optimal. MoE lowers per-token compute but raises memory, makes batching uneven, and can fail in training when the router collapses onto a few experts. Every model's knowledge is frozen at its cut-off; retrieval exists because retraining is not an update mechanism.

In an interview

Say: "Pre-training is next-token prediction at scale. Loss falls predictably with compute, data and parameters, and a lab's choice of model size is a serving-cost decision as much as a quality one." At junior depth, know the objective, the corpus and the cut-off. At mid depth, explain scaling laws, compute-optimal versus over-trained, and dense versus MoE. At senior depth, argue that data quality is the current lever and weigh MoE memory against compute for self-hosting. Expect: "Why not train the biggest model you can afford?" (serving cost and the supply of good data). "Why does an MoE model need so much GPU memory if it is cheap per token?" (all experts stay resident; only compute is sparse).