Chunking strategies

Why it matters

The chunk is the unit of retrieval: it is what gets embedded, scored against the query, placed in the prompt and pointed at by the citation. Cut too small and a chunk loses the context that gives it meaning; cut too large and the embedding blurs several topics, the prompt fills with text that does not answer the question, and the budget runs out at three chunks. An interviewer who asks how you chunk is asking whether you know what retrieval is matching against.

The idea

Why chunk at all

Three limits force it. Embedding models accept a bounded input and compress it into one vector, so a 20-page document embedded whole becomes a vector about nothing in particular. Retrieval precision depends on a chunk being about one thing, so the query matches it for the right reason. And the prompt budget is finite: five chunks of 400 tokens fit; five whole documents do not.

The strategies

Fixed-size with overlap. Cut every N tokens, typically 256 to 512, with 10 to 20 percent overlap so a sentence cut at a boundary survives in a neighbour. Trivial, format-blind, and it cuts tables, lists and arguments in half.

Sentence and paragraph. Split on natural boundaries and pack them up to a size limit: chunks end at natural stops, at the cost of uneven sizes.

Structure-aware. Use the headings the parser kept: one section is one chunk, split further only above the limit, with the section path prepended to the text. For documentation, policies and contracts this is the biggest single improvement over fixed-size, because the author already decided where topics start and end.

Semantic. Embed each sentence, walk the document, and cut where similarity between consecutive sentences drops. Chunks follow topic shifts rather than formatting, at the cost of an embedding per sentence at index time and a threshold to tune.

Parent-child (small-to-big). Index small chunks of 100 to 200 tokens for precise matching, each with a pointer to its parent (the section or page). Match on the child, return the parent to the prompt: precision of a small unit, context of a large one, at the cost of more prompt tokens per hit and a deduplication step when several children share a parent.

Contextual chunking. Have a model write a one or two sentence summary of the document and prepend it to each chunk before embedding. "The limit is 30 days" becomes "From the returns policy, on sale items: the limit is 30 days", which now matches the questions it answers. The cost is one model call per chunk at index time.

Size, with numbers

Typical chunks run 200 to 1,000 tokens. At 200 to 300 you get precise matching and clean citations but chunks that lack context; at 800 to 1,000 you get self-contained passages but blurrier embeddings and fewer chunks per budget. A common production setting is 400 to 512 tokens on structure boundaries with a parent available on demand. Overlap of 50 to 100 tokens rarely hurts and never substitutes for good boundaries.

How chunking meets reranking and citations

A reranker reads query and chunk together and scores relevance properly, so it tolerates larger, noisier chunks than cosine retrieval does: retrieve 20 to 50 small chunks and keep the top 5. For citations, the chunk is where the user is sent: a 300-token chunk is a span verified in seconds, a 1,500-token parent is a page to scroll.

Measuring it

Tune chunking with context recall: for questions with known source passages, the fraction of those passages that appear in the retrieved context. Pair it with context precision, how much of the retrieved context is relevant, so a recall gain is not bought by stuffing the prompt.

	Fixed-size	Structure-aware	Semantic	Parent-child
Precision	Medium; chunks mix topics at boundaries	High when sections are tight	High on prose with clear topic shifts	High; matches on a small unit
Recall of context	Low to medium; overlap helps a little	High; the author drew the boundaries	Medium; depends on the threshold	High; the parent comes along
Index cost	Lowest	Low; needs a parser that kept headings	High; an embedding per sentence	Medium; two levels to store
Complexity	Trivial	Low	Medium; a threshold to tune	Medium; pointers and deduplication