Drift and quality monitoring

Why it matters

Deterministic software changes when you deploy. A model-backed system changes when the provider updates the model behind an alias, when users start sending a new kind of input, when the retrieval index is rebuilt, when a tool's API changes its response shape, and on a bank holiday when the traffic mix shifts. None of those appear in your deploy log; teams find out from a support ticket a week later. The Evaluation module gives you the eval set and the release gate; this lesson is about what you watch between releases, how you tell a real change from noise, and how you find the cause. Interviewers ask because "how do you know it still works" is the question behind every production system.

The idea

Drift is a change in behaviour with no change in your code. It has four usual sources, and each leaves a different fingerprint.

Where drift comes from

Provider model updates. If you call an alias, the model behind it can change: outputs get longer or shorter, a new refusal pattern appears, a tool is called less often. Pin to a dated model version where the provider offers one, and treat moving the pin as a release with the eval gate run against it. If you must use an alias, record the resolved model id on every span so the change is at least visible.

Input distribution shift. A new customer segment, a marketing campaign, a new document type in the knowledge base, a holiday. The system is unchanged; the questions are not. The fingerprint is quality falling on one segment only, or retrieval scores dropping because the index has nothing on the new topic.

Retrieval index changes. A re-index, a new chunking, a new embedding model, a stale document corrected upstream. The model is fine; it is reading something different. Context precision and recall drop while model latency stays flat.

Upstream tool changes. A partner API renames a field or starts returning an empty list, and the model reasons over the new shape without complaint. The tool error rate may not move at all; the answers go wrong.

What to watch

Keep a small set of signals and watch all of them, because each catches a different source. Eval scores on sampled traffic: run the judge from the Evaluation module over a sample of production traces, say 2 percent, scored hourly. User feedback rate: thumbs down, edits, re-asks. Escalation and retry rate: handoffs to a human, the same question sent again. Refusal rate, which jumps on a model update. Output length, which moves on a model update or a prompt change. Tool error rate and tool result size. Latency percentiles, p50 and p95, which move on a provider change or an effort change. Cost per request, from the previous lesson. All of these come from span attributes except the eval score, which costs a judge call per sample.

Baselines, thresholds and segments

A signal without a baseline is a number. Take the baseline per signal over a trailing window, two weeks is usual, and alert when the current window leaves a band: for a rate, more than two or three standard deviations; for a judge score, a drop of a few points sustained over several hours, not one bad hour. Then segment every signal by prompt version and resolved model id, and by user segment, because drift on one segment averages away in the whole. A dashboard with one global quality line tells you something moved a day after a segmented one would.

The weekly read

Numbers find that something changed. Reading traces finds what. Once a week, a rotating engineer reads twenty sampled traces: ten flagged by a signal, ten at random. It takes an hour, and it is where new failure modes are first named and where the eval set gets its next cases, as the Building an eval set from production traces lesson describes.

Closing the loop

Monitoring is the slow loop. The Evals as release gates and runtime guardrails lesson describes the fast one, where a runtime check scores or blocks a single response before the user sees it. The two share the judge and the metric: the guardrail's block rate is itself a drift signal, and a drift alert is the trigger to add a guardrail where one was missing.

	Eval-score sampling	User feedback	Operational metrics
Lag	Hours; bounded by sample size and judge cost	Days; users must notice and bother	Minutes; computed from every span
Noise	Low per sample; judge bias is systematic, so trends matter more than levels	High; biased to the unhappy and the vocal	Low, but moves for reasons unrelated to quality
What it catches	Wrong answers, faithfulness drops, drift on one segment	Failures the judge was never told to look for	Provider changes, tool breakage, cost and latency shifts
Cost	One judge call per sampled request	Free, plus a feedback control in the UI	Nearly free