Skip to content

Write the evalbefore the prompt.

The habit that separates AI features that survive real users from those switched off in month three — plus the 200-case template we start every project with.

CureCoding teamHead of AI · · 8 min

Every failed AI feature we've been asked to rescue had the same origin story. Someone wrote a prompt, tried it on five examples, showed a demo that impressed the room, and shipped. Three months later, nobody could say whether a change made it better or worse — so nobody changed it, and usage slowly drained away.

The fix isn't a better model. It's a test suite for judgment, written before the first prompt.

Why prompts drift

A prompt is code with no compiler. Every edit that fixes one case can quietly break three others, and without a fixed set of graded examples you are tuning against your own memory of what "good" looked like last Tuesday.

If you can't say whether yesterday's change made it better, you don't have a product. You have a demo.

The 200-case set

We start every AI engagement by collecting real inputs — tickets, documents, queries — and grading the ideal answer with the people who'd otherwise do the work. Two hundred cases is enough to catch regressions; two thousand is enough to compare models with confidence.

  1. Pull inputs from production logs, not from imagination.
  2. Include the ugly ones: typos, partial data, adversarial asks.
  3. Grade with a rubric, not a vibe — three to five criteria, each pass/fail.
case: refund-policy-edge-017
input: "bought 31 days ago, box unopened, can I return?"
must: [cites_policy_v4, offers_store_credit, no_promise_of_cash]
must_not: [invents_exception]

Evals in CI

The set runs on every pull request, the same as unit tests. A drop of more than two points blocks the merge. Cost per query and p95 latency are tracked alongside, because a model that's 1% better and 4x slower is rarely the right call.

Eval score by build
#212 · 81%#219 · 85% blocked#224 · 94%

Then pick a model

Only now does model choice become a straightforward engineering decision: run the suite against three candidates, plot score against cost and latency, and pick. Most of the time the answer is cheaper than anyone expected. See how we run this on AI and machine learning projects.

  • Evals
  • LLMs
  • Product

Keep reading

All articles →
Business · 5 min

How much does custom software development cost in 2026?

DevOps · 5 min

DevOps for startups: a practical CI/CD, Kubernetes and Terraform roadmap

Mobile · 5 min

React Native vs Flutter vs native: choosing a mobile app stack