Requirements and the eval-first framing

Why it matters

The first ten minutes decide the interview. A candidate who draws a pipeline at minute two is designing for a problem nobody has described, and every later number floats free. A candidate who asks eight questions, turns the answers into numbers and states how the system will be judged has already shown the two things being graded: judgement about what matters, and the habit of measuring.

The idea

Eight questions that change the design

Each question earns its place because the answer moves a box.

• Who uses it? Internal staff tolerate rough edges and can be trained; customers cannot. Who also decides permissions.
• How many? Users, requests a day and the peak. Volume decides cost and whether caching matters.
• What data? Formats, size, how messy. Scanned PDFs and tables move the effort into parsing.
• How fresh? Minutes, or a nightly rebuild. Freshness decides the ingestion design.
• What latency? To the first token and to the complete answer. A voice product and an overnight report are different systems.
• What budget per request? The ceiling that picks the model tier and caps the context.
• What happens when it is wrong? A wrong film suggestion and a wrong refund are different failures. This answer sets the guardrails and where a human sits.
• What is out of scope? Saying it aloud protects the forty-five minutes.

Questions that change nothing (which cloud, which vector database) waste the band and signal that you do not know which ones matter.

Turning answers into numbers

Suppose the answers are: 5,000 employees asking about internal policy, four questions each per working day, a complete answer in 3 seconds, a cent per question.

Requests. 20,000 a day. Over an eight-hour day that is about 0.7 a second; plan for a peak of three times, about 2 a second.

Tokens. A 600-token system prompt, five 400-token chunks and a 50-token question: about 2,650 in. An answer of about 300 out.

Cost. At a mid-tier $1 per million input tokens and $5 per million output, that is $0.0027 plus $0.0015, about $0.004 a question and $83 a day. Under budget, with room for a reranker and a judge on sampled traffic.

Latency. Split the 3 seconds across the stages: 50 ms to embed the query, 150 ms to search, 150 ms to rerank, 50 ms to assemble. That leaves about 2.5 seconds for generation, enough for the first token and a 300-token answer at around 150 tokens a second, with little slack. So stream. Every tool call in an agent loop takes its own slice, which is why agents need a looser budget than single-pass retrieval.

Write it on the board, left column, and leave it there:

USERS    5,000 staff, internal, role-based access
VOLUME   20,000 questions/day, peak about 2/s
DATA     policy wiki and PDFs, nightly refresh is enough
LATENCY  3 s to full answer, streamed
BUDGET   $0.01/question (estimate $0.004)
WRONG    worst case: cites a stale or unauthorised page
OUT      no write actions, no HR case handling
EVAL     150 golden questions: context recall >= 0.85,
         faithfulness >= 0.9; guardrail: 0 unauthorised citations

The eval-first move

Before the first box, state four things. The metric, matched to the failure that hurts most: context recall for retrieval, faithfulness for generation, goal completion for an agent. The golden set: real questions from logs or from the people who answer them by hand today, with reference answers and source passages, 100 to 200 to start, stratified, including some the system should decline. The threshold that gates release. The guardrail: the failure that must be zero whatever the averages say, such as a citation the user may not see.

Then say the sentence that makes the move pay: "Every box I draw next is there to move that number or protect that guardrail." A component that cannot be tied to either has to justify itself, so over-design is caught before it reaches the board.

Saying "I would measure" without sounding evasive

"I would measure" sounds evasive when it replaces a decision and strong when it follows one. Make the call, predict the number, name the test that would change your mind: "I would start with 400-token chunks; if context recall on the policy slice comes in under 0.8, I move to structure-aware splitting." Commit, predict, test. Never "it depends" without saying on what.