Ingestion and parsing: PDFs, tables, HTML

Why it matters

Every production corpus is PDFs exported from a design tool, wikis of nested tables, slide decks whose meaning is in the layout, email threads that quote themselves, and a scanned contract from 2009. The parser runs once, offline, and every later stage inherits its output: a table flattened into word soup cannot be chunked, embedded or cited well. Retrieval quality is capped by parsing quality, and interviewers who have run RAG in production ask about parsing before embeddings.

The idea

Ingestion turns a pile of formats into one normalised shape: an ordered list of blocks (paragraphs, headings, tables, code, captions), each with metadata.

PDFs

A PDF is a set of drawing instructions, not a document. A text-layer PDF carries characters and their positions; a scanned PDF carries only pixels. For text-layer files the hard part is reading order: a two-column paper read line by line interleaves the columns, and headers and footers repeat on every page and land mid-sentence. Layout-aware parsers group characters into blocks by position and font, then order the blocks; plain extraction does not. Scans need OCR, which yields text with character errors and no structure, or a vision model that reads the page image and emits structured text, more accurate on tables and handwriting at a higher cost per page.

Tables

A table flattened to prose loses the one thing that made it useful: which number belongs to which row and column. Emit tables as markdown or CSV, repeat the header row in every chunk cut from a long table, and add a one-line caption ("quarterly revenue by region") so the embedding has something to match a question against. A cell without its row and column labels is a number without a meaning.

HTML, slides, email, code

HTML needs boilerplate removal: navigation, cookie banners, footers and related-links blocks are most of the bytes and none of the content; extract the main block and keep heading levels. Slides are sparse text where the structure is the content: one slide is one block with its title, speaker notes beside it. Email threads quote earlier messages, so strip quoted text and keep thread order and participants as metadata. Code is best parsed by syntax: a function with its docstring is a natural block, with the file path as metadata.

Metadata worth keeping

Keep source (a URL or path you can link to), title, section path (the heading chain, "Returns > Sale items"), date (created and modified), and access control labels (who may read this). The section path also goes into the chunk text, so the embedding knows what the chunk is about. The access labels are not optional: without them a retrieval system will show one user another user's document, and the filter must run inside the retrieval query, not after generation.

Deduplication

Corpora repeat themselves: the same policy in three wikis, a PDF and its HTML export, a template copied across 400 pages. Duplicates waste index space, crowd the top-k with copies of one passage and make citations arbitrary. Hash exact duplicates, detect near-duplicates by shingling or embedding similarity above a tight threshold, keep the canonical copy and record the aliases.

Structure is the input to chunking

The parser's output is what the chunker cuts. Kept headings let the chunker cut at section boundaries; tables kept as tables can stay whole; a flat string reduces the chunker to counting tokens. Parsing and chunking are one decision made in two places.

	Text extraction	Layout-aware parsing	Vision model
Tables	Flattened to word order; structure lost	Recovered as rows and cells on most layouts	Read from the image, merged cells included; may invent a cell
Scans	Nothing; no text layer	Needs an OCR pass first	Handles scans and handwriting directly
Cost	Near zero	Low; CPU time per page	A model call per page; dominates for millions of pages
Speed	Milliseconds per page	Tens to hundreds of ms per page	Seconds per page
Use when	Clean born-digital text	Multi-column, tables, headers and footers	Scans, odd layouts, where numbers must be right