The AI engineering stack

Why it matters

An LLM product is a model plus five other things, and most outages, cost blow-ups and bad answers come from the other five. The provider owns the model; you own everything around it. When a feature misbehaves, the first useful question is which layer the fault lives in: the model was handed the wrong context, it called a tool wrongly, nobody noticed for a week, nobody can say whether the fix helped, or a user talked it into something it should never do. Interviewers draw this stack on the whiteboard in the first five minutes and expect you to place each decision in the right band.

The idea

Six layers. Four of them are a stack in the ordinary sense: each one builds on the one below. The top two cut across all of them and are the ones most teams skip.

Models and inference

The model itself, reached through a provider's API or served by you with vLLM or SGLang. The decisions here are which model, how much thinking effort, how many output tokens, and whether a prompt is cached, batched or routed to a cheaper model. The cost model lives here: prefill and decode, the KV cache, prompt caching at a fraction of the input price. Taught by How language models work, Models and the API, Serving and cost and Fine-tuning.

Protocols and tools

How the model reaches the world. Function calling gives it a schema to fill; MCP standardises how a host exposes tools, resources and prompts to a model; A2A lets one agent hand a task to another; a sandbox lets it run code. The decisions are the tool surface, who may call what, and what happens on a failed call. Taught by Tool use fundamentals and Protocols and frameworks.

Memory and knowledge

Everything the model knows that it was not trained on. In the short term that is the context window and what you assemble into it; across a corpus it is retrieval, with embeddings, a vector index, chunking and reranking; across sessions it is memory that must also forget. The decisions are what to retrieve, how to chunk, what to remember and how to keep it fresh. Taught by Prompting and context, the whole Retrieval stage and Memory and state.

Frameworks and SDKs

The orchestration: the agent loop, the workflow patterns, multi-agent topologies, durable execution across restarts. Frameworks package these, and they churn every year, so the track teaches the patterns and compares the frameworks once. Taught by The agent loop, Workflow patterns, Agent types, Multi-agent systems and the Agent frameworks compared lesson.

Evals and observability

Two different questions. Observability asks what the system did: traces, token accounting, drift over time. Evals ask whether it was any good: a golden set, a judge, Ragas-style metrics, a gate before release. Among teams with agents in production, 89 percent have observability and 52 percent have evals, per LangChain's State of Agent Engineering as reported by O'Reilly in 2026. Most teams can see what happened and cannot say whether a change helped. Taught by Measuring retrieval, Evaluation and Observability.

Guardrails and safety

Input filters, output checks, policy, least privilege for tools, PII boundaries and a defence against prompt injection, which sits first in the OWASP Top 10 for LLM Applications 2026. Taught by Safety and security.

Which layer owns the decision

	Owning layer	The decision	Taught in
Answers must cite our internal docs	Memory and knowledge	Chunking, index, how many chunks, reranking	The RAG pipeline; Embeddings and vector search
The assistant books a meeting	Protocols and tools	Tool schema, confirmation step, what a failed call returns	Tool use fundamentals; Protocols and frameworks
Stay under two cents a request	Models and inference	Model choice, prompt caching, routing, output cap	Serving and cost
Prove the new prompt is better	Evals and observability	Golden set, judge, release gate, trace comparison	Evaluation; Observability
Never show one tenant another's data	Guardrails and safety	Permission-aware retrieval, output filter, least privilege	Safety and security; Advanced retrieval

Trade-offs

The layers are a map, not a boundary. Retrieval lives in memory and knowledge, but its token cost is an inference decision and its leak risk is a guardrail decision, so any real feature touches three bands at once. The failure mode of thinking in layers is believing that one vendor per layer adds up to a system. The failure mode of ignoring them is a notebook that works and a product nobody can debug. The two cross-cutting layers are under-built almost everywhere because neither ships a feature on its own, which is exactly why they separate strong candidates from the rest.

In an interview

Draw the six bands before you draw the feature, then place the feature. The sentence to say: "The model is one box; the product is the five around it, and the two I design first are evals and guardrails, because those are the ones most teams skip." At junior depth, name the six layers and place a given feature in the right one. At mid depth, name the decision that lives in each and what breaks when it is wrong. At senior depth, trade across layers: retrieve less and pay for a larger model, or retrieve more and route to a cheaper one, and quote the 89 versus 52 gap when asked what you would build first. Expect the follow-ups: "Where does prompt caching live?" (inference) and "Who owns hallucination?" (grounding in memory and knowledge, detection in evals, the last line in guardrails).