Tokens and tokenisation

Why it matters

Every number you will ever be quoted about a language model is in tokens: the price per million, the context window, the output limit, the generation speed. A token is not a word and not a character; it is a piece of text the model's vocabulary happens to contain. Engineers who think in words under-estimate cost by a third, overflow context windows on non-English input, and cannot explain why a model that writes sonnets fails to count the letters in "strawberry". Interviewers use tokens as the first filter: if you cannot say what one is, nothing you say about retrieval or agents lands.

The idea

A tokeniser is a deterministic function from text to a list of integers, and back. The model is trained on those integers; it never sees letters. The function is fixed before training and cannot change afterwards, so the choice of tokeniser is baked into the model for life.

Byte-pair encoding

Almost every current model uses a subword scheme, usually byte-pair encoding (BPE) or a close relative (SentencePiece, unigram). Training the tokeniser is simple: start with single bytes as the vocabulary, count the most frequent adjacent pair in a large corpus, merge it into a new symbol, repeat until the vocabulary reaches a target size (typically 32,000 to 200,000 entries). Frequent words become one token; rare words split into pieces that are themselves common. Encoding at inference time replays those merges greedily on the input.

What this does to everything downstream

Cost. English prose runs at roughly four characters, or three quarters of a word, per token, so a 1,000-word document is about 1,300 tokens. Code, JSON, and languages with non-Latin scripts tokenise worse: a Hindi or Japanese sentence can cost two to four times the tokens of its English translation, and a wall of indentation or UUIDs is almost all tokens. Pricing is per token, so the same product is more expensive for some users than others.

Context. The context window is a token count. A "1M-token" window holds about 750,000 English words, far fewer of Japanese, fewer still of minified code. Budgeting a prompt means counting tokens with the model's own tokeniser, not estimating from characters.

Speed. Generation is one token per forward pass, so output speed is quoted in tokens per second and latency scales with output length in tokens, not words.

Reasoning over characters. Because "strawberry" arrives as two or three ids, the model has no direct view of its letters. Counting letters, reversing strings, rhyming on spelling and arithmetic on long digit strings all suffer from the same cause: the unit of thought is the token. Reasoning models partly work around it by spelling the word out token by token before answering, which is a workaround, not a fix.

Special tokens and chat templates

The vocabulary also contains tokens that never appear in ordinary text: end-of-sequence, padding, and the markers that frame a conversation. A chat template wraps your system prompt and messages in those markers before encoding, which is why the API takes a list of roles and contents rather than one string: the provider renders the template so the model sees turn boundaries the way it did in training. Tool calls and tool results are framed the same way, which matters when you read the Function calling lesson.

	Word-level	Character-level	Subword (BPE)
Vocabulary size	Hundreds of thousands, still misses new words	Under a few hundred	32K to 200K, chosen at training time
Unknown words	Replaced by an unknown token, meaning lost	Never unknown	Split into known pieces
Sequence length	Shortest	Longest; attention cost explodes	In between; tuned for frequent text
Cross-language fairness	Needs a vocabulary per language	Fair but slow	Favours whatever dominated the training corpus
Used by	Older NLP systems	Some speech and byte models	Every current frontier model

Trade-offs

A bigger vocabulary means shorter sequences and cheaper long inputs, but a larger embedding table and rarer, less well-trained tokens. A vocabulary built mostly on English makes other languages slower and dearer, which is a product decision disguised as a technical one. Tokenisers are also sticky: two models from different providers count the same text differently, so cost comparisons must be done in each model's own tokens, and a prompt tuned to sit just under one model's window may overflow another's.

In an interview

Bring tokens up the moment cost or context appears: "I budget in tokens, and I count them with the model's tokeniser because characters are a bad proxy, especially for code and non-English text." At junior depth, know that a token is a subword piece and that prices and windows are in tokens. At mid depth, explain BPE merges, the four-characters-per-token rule of thumb and why it fails for code and other scripts. At senior depth, connect tokenisation to product cost per market, to chat templates and tool-call framing, and to the character-level failures that retrieval and tools cannot fix. Expect the follow-up: "Why does the model get letter counts wrong?" The answer is one sentence: it never sees letters.