Every failed AI feature we've been asked to rescue had the same origin story. Someone wrote a prompt, tried it on five examples, showed a demo that impressed the room, and shipped. Three months later, nobody could say whether a change made it better or worse — so nobody changed it, and usage slowly drained away.
The fix isn't a better model. It's a test suite for judgment, written before the first prompt.
Why prompts drift
A prompt is code with no compiler. Every edit that fixes one case can quietly break three others, and without a fixed set of graded examples you are tuning against your own memory of what "good" looked like last Tuesday.
If you can't say whether yesterday's change made it better, you don't have a product. You have a demo.
The 200-case set
We start every AI engagement by collecting real inputs — tickets, documents, queries — and grading the ideal answer with the people who'd otherwise do the work. Two hundred cases is enough to catch regressions; two thousand is enough to compare models with confidence.
- Pull inputs from production logs, not from imagination.
- Include the ugly ones: typos, partial data, adversarial asks.
- Grade with a rubric, not a vibe — three to five criteria, each pass/fail.
case: refund-policy-edge-017 input: "bought 31 days ago, box unopened, can I return?" must: [cites_policy_v4, offers_store_credit, no_promise_of_cash] must_not: [invents_exception]
Evals in CI
The set runs on every pull request, the same as unit tests. A drop of more than two points blocks the merge. Cost per query and p95 latency are tracked alongside, because a model that's 1% better and 4x slower is rarely the right call.
Then pick a model
Only now does model choice become a straightforward engineering decision: run the suite against three candidates, plot score against cost and latency, and pick. Most of the time the answer is cheaper than anyone expected. See how we run this on AI and machine learning projects.
- Evals
- LLMs
- Product