Prompt templates, versioning and caching

Why it matters

The prompt that shipped is the prompt nobody can find. It lives in a string literal, was edited in production last Tuesday, and no one can say which version produced the complaint in the support queue. Meanwhile the bill shows the same 30,000-token system prompt paid for in full on every call. Both problems have the same shape: the prompt is being treated as prose when it is an artefact with an interface, a history and a cost profile. Interviewers raise this because it separates people who have built a demo from people who have run a prompt in production for a year.

The idea

Prompts as code

A production prompt is a template: fixed text with named slots such as {{customer_tier}} or {{retrieved_chunks}}, rendered at request time. The template lives in the repository next to the code that fills it, goes through review, and carries a version. The slots are its interface; changing what a slot expects is a breaking change.

Each version is run against the module's eval set before it ships, and the change is written down: what changed, why, what the eval moved. A changelog per prompt is the cheapest observability a team can add, because the question "what did we change" arrives within the week.

Versions in traces

Every request logs the prompt's version tag alongside the model id, the inputs and the output. That makes two things possible: tracing a bad output back to the exact text that produced it, and A/B testing by version, where a slice of traffic renders version 14 and the rest renders 13 and the comparison is read from the traces. Rollback is a tag change. A trace line is small:

req=8f21c prompt=ticket-router@14 model=... in=31412 cached=30900 out=48 ms=610

Prompt caching

Providers cache the prefix of a prompt. If the first N tokens of this request are identical to the first N tokens of a recent one, the provider reuses the computed state for those tokens and bills a cache read instead of fresh input; on current frontier lineups a cache read costs between a fortieth and a tenth of the base input price. The stable material is exactly what a template fixes: the system prompt, tool definitions, few-shot examples and reference documents. The volatile material is the conversation and the user's turn.

Hence the ordering rule: stable first, volatile last. Any change invalidates the cache from that point onward, so a timestamp or a user's name at the top of the system prompt defeats caching for everything after it. Move it to the end, or to the user turn. Tool definitions are part of the prefix; reordering them is a cache miss. So is a whitespace change.

Caches have lifetimes: a short default measured in minutes, refreshed by each hit, with a longer option on some providers at a higher write price. A prefix read less often than its lifetime never hits; a prefix shared by thousands of requests an hour hits almost every time. Breakeven is a handful of reads per lifetime, because the write costs a little more than a plain input.

The other discount

The batch API takes a file of requests, runs them within a window of hours and returns results at about half price. It is the right path for anything that does not have a user waiting: nightly classification, re-indexing, eval runs, bulk summarisation. It stacks with caching when the batch shares a prefix.

Trade-offs

Templates make prompts reviewable and tempt teams to add slots until the template is a programme nobody can read; keep the slot count small and the logic in code. Versioning adds friction: a one-word fix now needs an eval run, which is the point. Caching couples prompt layout to cost: a harmless-looking edit near the top of the system prompt can multiply the bill, so cache hit rate belongs on the dashboard next to latency. Caching also shares nothing across models, so an upgrade starts cold. Batch trades latency you may not have; a job that must finish by nine cannot sit in a window that closes at ten.

In an interview

Say: "I keep prompts in the repo as versioned templates, tag every trace with the version, order the prompt stable-first so the provider caches the prefix, and push anything without a waiting user through the batch API." At junior depth, know templates, slots and the version in the log. At mid depth, explain cache mechanics, what breaks the prefix and breakeven against the lifetime. At senior depth, run A/B by version, treat cache hit rate as an SLO, and state the layout rules that keep a 40,000-token prefix hot across a fleet. Expect: "Why did the bill double after a release that only changed the greeting?" The greeting was at the top of the prefix.