Module 21 · AI interview playbook · Lesson 6 of 6
Blueprint: diagnosing a failing RAG or agent system
The diagnosis interview worked through 45 minutes: ratings fell from 4.2 to 3.1 with no deploy; segment, sample, classify, rank five suspects, test each from traces and evals, fix the largest bucket, prove it and prevent a repeat.
Why it matters
The fourth interview shape is not a design. You get a running system, a symptom and some evidence, and the board starts full rather than blank. What is graded is structure under uncertainty: whether you gather evidence before guessing, rank causes by likelihood, choose tests that can rule each one out, and prove the fix. The Diagnosing a badly rated RAG system lesson gives the core procedure; this blueprint runs it across 45 minutes on a harder case, a product with an agent in it and a decline with no deploy to blame.
The idea
Scope (0 to 5)
The prompt: a help assistant that retrieves from a documentation corpus and calls three tools (account lookup, order status, ticket creation). Average rating fell from 4.2 to 3.1 over six weeks. Nobody deployed in that time. You have traces of every request (query, retrieved chunks with scores, prompt, tool calls and results, answer, and the model version the provider reported), a nightly golden-set eval, and thumbs with free-text feedback.
Ask four questions. Was the fall a slide or a step on one date? Which segments fell? Did volume or traffic mix change? What is pinned, and what changes on someone else's schedule? "No deploy" means no deploy by this team. The provider, the documents, the users and the services behind the tools all kept changing.
How you will know (5 to 10)
State the test for a cause before hunting for one: a real cause explains the timing, the segment and the failure type at once. The symptom metric is the daily rating per segment; the controlled measurement is the golden set, because its questions did not change. If the golden set held steady while ratings fell, the system did not get worse at the old questions; something about the traffic or the world changed.
Segment, sample, classify (10 to 20)
Plot daily ratings by question type, user cohort, language, corpus area and tool path. Find the segment that is both worst and growing. Read 20 to 30 of its worst traces end to end and classify each by failure point: retrieval miss, ignored context, stale content, tool failure, wrong format, out of scope. Twenty is enough for the largest bucket to show.
Five suspects, tested in order (20 to 35)
The shape of the curve sorts them. A step on one date points to an event; a six-week slide points to drift.
Provider model change. A model behind an unpinned alias was updated. Test: the model version in the traces changed on a date, and the golden set stepped down on that date while retrieval metrics stayed flat. Usually a step, so less likely here, and quick to rule out.
Index drift. The nightly ingestion has been failing silently, or new documents were embedded with a newer embedding model than the old ones, mixing two vector spaces. Test: the date of the last good build, context recall on the golden set across six weeks, and the distribution of top retrieval scores sliding down.
Document changes. The help centre was restructured, or new content arrived as PDFs with tables the parser flattens. Test: the failures cluster on recently changed documents, and their stored chunk text is unreadable.
A new user segment. A launch brought a new market, a new language or a new kind of question the corpus does not cover. Test: ratings inside each old segment are flat while the new segment's share of traffic grew; the average fell from mix alone. A slow slide with a steady golden set fits this best.
A tool change. A service behind a tool changed its response shape. Test: tool error and empty-result rates by date in the traces, and longer agent loops ending in apologies.
Then fix the largest bucket, prove it on the golden set plus a new slice built from the failing segment, and ship behind a flag to that cohort before widening.
| Provider model change | Index drift | New user segment | |
|---|---|---|---|
| Symptom pattern | Step on one date across all segments | Slow slide, worst on recent content | Slow slide in the average; old segments flat |
| Test | Model version in traces; golden set steps while retrieval is flat | Last good build; context recall trend; retrieval score distribution | Ratings per segment; traffic share over time |
| Fix | Pin the previous snapshot; re-tune prompts against the new one | Repair ingestion; re-embed with one model; re-index | Add content, language support or out-of-scope handling for the segment |
| Prevention | Pin dated versions; canary eval before any switch | Freshness and score-drift alerts | Segment-mix alerts; golden set refreshed from new traffic |
Follow-ups (35 to 42)
"What if the fix regresses something else?" The golden set covers every segment, results are compared per slice rather than on average, and the flag makes rollback a switch. "How do you stop it recurring?" Pin model and embedding versions to dated snapshots, never aliases. Alert on drift: index freshness, retrieval score distributions, tool error rates, segment mix. Run the golden set daily against the production configuration, so a change on the provider's side shows up as an eval drop before it shows up in ratings. Refresh the golden set from complaints so it keeps up with the traffic.
Close (42 to 45)
Say what happens on Monday: the segment chart, twenty traces read, the two most likely suspects tested, and the before figure written down.