Graph RAG

Why it matters

Flat chunk retrieval answers questions whose evidence sits in one place. Ask "which suppliers of the parts in product X are in the same region as our warehouse" and the evidence is spread across a bill of materials, a supplier register and a facilities list. No chunk contains the answer, and no embedding of the question is close to the chunk about the supplier. Interviewers ask about graph RAG because this failure is common and quiet, and because the fix is the clearest case of "add structure, or add a join". A senior answer knows the technique, its bill, and when a join is the honest answer.

The idea

Why flat chunks miss

A chunk is a span of text with an embedding, scored against the question on its own. Facts in two chunks have no joint score, so a chain (X contains part A, A is made by supplier S, S is in region R) is found only if one chunk states it. Multi-hop questions are recall failures, not ranking failures: a larger top-k rarely helps, because the missing chunk is about a supplier and the question is about a product.

Building the graph

Graph RAG adds structure at ingestion. A model reads each document and extracts entities (products, parts, suppliers, regions) and relations between them (contains, made by, located in), each with a pointer back to its source span. Entities are merged across documents so that "Acme Ltd" and "Acme" become one node. Most implementations then cluster the graph into communities and have the model summarise each, so there is a short description of every neighbourhood as well as every fact.

Retrieval by traversal

At query time the pipeline finds entry entities in the question (product X, the warehouse), walks the edges a few hops outward, collects the connected nodes with their source text, and hands that subgraph to the model. The walk does the joining that embeddings cannot. Each hop can be filtered (only "made by" edges) so the subgraph stays small, and the source spans travel with it, so the answer can cite its documents.

Local and global questions

Local questions start from named entities and want their neighbourhood: traversal plus the attached text. Global questions ask about the corpus as a whole, "what are the main supply risks across our vendors", with no entity as entry point. Those are answered from the community summaries: retrieve them, rank them, let the model synthesise. Flat retrieval fails them for the same reason: no chunk is about the whole.

Hybrid graph and vector

The graph does not replace the vector index. Entity nodes and community summaries get embeddings too, so a fuzzy question can land on the right entry node; traversal expands it; a vector search near the subgraph fills in detail the extractor missed.

What it costs

Extraction is the bill. Every document needs at least one model call to pull entities and relations, several for long documents, and one more per community summary, so indexing 100,000 documents is a few hundred thousand model calls before anyone asks a question. Extraction also errs: missed relations, entities merged that should be separate. And the graph drifts. When a document changes, its entities and edges must be re-extracted, re-merged, and the affected summaries rewritten, which is harder than re-embedding a chunk; the Freshness lesson covers that machinery.

When a join is the honest answer

If the facts already live in tables, the right pipeline is a structured query. Products, parts, suppliers and regions are what a company keeps in a database, and a join across three tables answers the warehouse question exactly, cheaply and without extraction errors. Graph RAG earns its place when the relations are buried in prose: contracts, incident reports, research notes, email threads. A metadata filter on the vector index (region equals North) is the middle option when only one hop needs structure.

	Vector RAG	Graph RAG	Structured query
Multi-hop questions	Misses chains; more top-k does not help	Joins by traversal; depth is a tuning knob	Exact joins when the facts are in tables
Build cost	One embedding call per chunk	Several model calls per document plus community summaries	Schema and ETL work; no model calls
Freshness	Re-embed the changed chunk	Re-extract, re-merge, rewrite summaries	Live on every query
Explainability	Top chunks and scores	The path through the graph, with source spans	The query itself
Fits	Single-fact questions over prose	Relations buried in prose	Data someone already structured

Trade-offs

You buy multi-hop recall and explainable paths with a large one-off extraction cost, an ongoing maintenance cost, and a new class of errors that look like facts: a wrong edge becomes a confident wrong answer with a citation. Traversal depth has a cliff on each side; one hop too few misses the chain, one too many floods the context with the whole neighbourhood. And the graph does nothing for questions that were already fine, which is most of them, so measure the share of traffic that is multi-hop before building.

In an interview

Raise it when a question needs facts joined across documents: "Flat chunks score each span alone, so a chain across three documents has no chunk to retrieve. If the data is structured I join it; if the relations are buried in prose I extract a graph and retrieve by traversal." At junior depth, explain why multi-hop fails with chunks. At mid depth, describe extraction, traversal, local versus global questions. At senior depth, put numbers on extraction cost, say how the graph stays fresh, and argue for a join or a metadata filter where it fits. Expect: "How do you know the graph is right?" Sample edges against their source spans; measure recall on a labelled multi-hop set.