LLM evals with a small test set: 30-100 cases, run on every prompt change

A feature that calls an LLM can change behaviour every time you edit the prompt, switch models, or the provider ships a new version. Without a test set, you find out it broke when a user tells you. A small eval set of 30-100 cases that runs in minutes does the job a unit test suite does: it tells you what this change improved and what it broke.

Tested on Python 3.12 (the script uses only the standard library). Every statistic in the article was recomputed for it; the grading advice was checked against the Anthropic and OpenAI docs and the MT-Bench paper (Zheng et al., 2023).

What an eval is, and why poking at it is not one

Typing three or four questions and deciding it “looks fine” is not testing: next time you will ask different questions and you won’t remember the old answers. An eval has three parts, all written to files:

1. Cases: real inputs the system must handle, each with a permanent id
2. Expected results, or the grading criterion for each case, written before the run
3. A grader: code that returns pass/fail per case, summed into a pass rate

As files, any run can be repeated, diffed against the previous one, and kept in git next to the prompt.

Where cases come from

Anthropic’s docs put it plainly: cases should mirror the real task, edge cases included (typos, sarcasm, long rambling input). OpenAI recommends logging from the start of development so you can mine good cases. The mix we use:

Kind	Rough share	Example (ticket triage)
Normal cases from real logs	60%	“My order hasn’t arrived, it’s been 5 days”
Edge cases	25%	mixed Thai and English, typos, two requests in one message
Cases that actually broke	15% and up	every wrong answer a user hit becomes a permanent case

The last row matters most. When a user hits a wrong answer, add that case before you touch the prompt, the same way you write a regression test before fixing a bug. Strip personal data before saving it.

How to grade: code first, judge later

Anthropic’s docs rank grading methods and say to pick the fastest, most reliable, most scalable one that works:

Method	Good for	Weakness
Code (exact match, regex, JSON parse, numeric checks)	categories, extracted fields, output format	cannot grade free text
LLM-as-judge with a rubric	summaries, tone, “does it answer the question”	biased; must be checked against humans
Human review	seed labels, auditing the judge	slow, expensive

The trick is designing the output so code can grade it. If the feature returns JSON with clear fields (the structured output article), half your eval becomes a plain ==. The same docs add that many automatically graded cases beat a few carefully hand-graded ones.

LLM-as-judge and its biases

When you need a judge, follow the three tips in Anthropic’s docs: a detailed rubric (“must mention the order number, otherwise fail”), a measurable answer (pass/fail or 1-5), and reasoning before the score (thinking on). And use a different model from the one that wrote the answer.

The MT-Bench study (Zheng et al., 2023) measured three judge biases. It used 2023-era models; today’s numbers differ, but the kinds of bias still apply:

• Position bias: favouring an answer because of where it sits. Even GPT-4 gave the same verdict after the two answers were swapped in only 65% of cases
• Verbosity bias: favouring longer answers, even when the extra length is repetition
• Self-enhancement: possibly rating its own answers higher (the authors say their data was too limited to be sure)

Countermeasures: when comparing two answers, judge twice with the order swapped and count a win only if it wins both times. When you don’t need a comparison, grade one answer pass/fail instead. Spot-check 20-30 cases by hand against the judge before trusting its numbers. OpenAI’s guide says the same: scale up a judge only once it agrees consistently with human labels.

The grader needs testing too. A judge never compared with humans produces numbers that look precise and mean nothing.

The smallest eval script that is actually useful

Keep cases in cases.jsonl, one per line, e.g. {"id": "c001", "input": "...", "expect": "refund"}. This script is standard library only; predict is the function that calls your AI feature.

import json, sys
from math import comb

def run(cases_path, predict, out_path):
    results = {}
    for line in open(cases_path, encoding="utf-8"):
        c = json.loads(line)
        got = predict(c["input"])
        results[c["id"]] = got.strip().lower() == c["expect"].lower()
    json.dump(results, open(out_path, "w"))
    print(f"pass {sum(results.values())}/{len(results)}")

def compare(old_path, new_path):
    old, new = json.load(open(old_path)), json.load(open(new_path))
    broke = [k for k in old if old[k] and not new[k]]
    fixed = [k for k in old if not old[k] and new[k]]
    n, k = len(broke) + len(fixed), min(len(broke), len(fixed))
    p = min(1.0, 2 * sum(comb(n, i) for i in range(k + 1)) / 2**n) if n else 1.0
    print(f"broke {len(broke)} {broke}\nfixed {len(fixed)} {fixed}\np = {p:.3f}")

if __name__ == "__main__":
    compare(sys.argv[1], sys.argv[2])

# diff the previous run against this one (dev machine or CI)
python evalrun.py results/old.json results/new.json

compare does not stop at the total. It lists broken cases (passed before, fail now) and fixed cases (failed before, pass now), then computes an exact McNemar p-value, which is a binomial test on the cases that changed (the same as exact=True in statsmodels). On simulated data, the old run passed 85/100 and the new one 80/100. The totals suggest a small dip; underneath, 17 cases broke and 12 were fixed, p = 0.458. The change reshuffled the whole set without improving it, and those 17 cases need reading.

Worked example: how many cases to see a 10-point drop?

Say the feature passes 85%, and a prompt edit drops it to 75%. How big must the set be to tell that apart from chance?

Step 1: the uncertainty of a single run. Standard error = √(p(1−p)/n); at p = 0.85 that is √(0.1275/n). The 95% interval is about ±1.96 × SE.

Cases	SE	95% interval
30	6.5 pts	±12.8 pts
50	5.0 pts	±9.9 pts
100	3.6 pts	±7.0 pts
250	2.3 pts	±4.4 pts

At 30 cases the noise band is wider than the drop you want to catch, and one case is worth 3.3 points: a 10-point drop is just 3 cases.

Step 2: comparing totals as two independent samples. The two-proportion sample-size formula at α = 0.05 (two-sided, z = 1.96) and 80% power (z = 0.84):

n = (1.96 + 0.84)² × (0.85×0.15 + 0.75×0.25) / 0.10² = 7.84 × 0.315 / 0.01 ≈ 247 cases

Step 3: comparing case by case on the same set. This works because both runs use the same cases, so only the cases that changed carry information. At 100 cases, a 10-point drop can happen several ways:

Broke	Fixed	Net	p (exact McNemar)
10	0	−10	0.002
12	2	−10	0.013
15	5	−10	0.041

All are below 0.05, so 100 cases compared case by case catch a 10-point drop. At 30 cases, a 10-point drop is 3 broken and 0 fixed: p = 0.25, still not conclusive.

In practice: 30 cases catch outright breakage and obvious bugs; around 100 with a per-case diff catch a 10-point change; anything subtler needs more cases or repeated runs.

The model’s own noise

LLMs don’t answer the same way every time. Before trusting any result, run the baseline twice with no changes and use compare to see how many cases flip on their own. If 5 cases flip per run, a change that breaks 5 cases tells you nothing. Run frequent flippers 3 times and count a pass by majority, or loosen the grader where it is fussy about details that don’t matter.

When to run it and how to track it

Run on every prompt edit, model switch, temperature or tool change, SDK upgrade (choosing a model should be decided by this set). Save each run’s results under the commit hash and watch the pass rate over time.

It costs less than people expect. 100 cases at 1,000 input and 200 output tokens each on Claude Sonnet 5.5 ($2 / $10 per million tokens, as of October 2026): 100 × (1,000 × $2 + 200 × $10) / 1,000,000 = $0.40 a run. A judge roughly doubles that. Far cheaper than a bug users find.

Common mistakes

• Tuning the prompt until it memorises the set (overfitting). Patching case by case to 100% while production gets worse. Keep 20-30% as a holdout you don’t look at while tuning; run it only when deciding
• A stale set. Real inputs moved on, the cases are three months old. Top up from logs weekly
• Watching only the total. As above, 85 → 80 hid 17 broken cases
• Trusting an unchecked judge, or letting a model grade itself
• Personal data in cases. Strip names, phone numbers, account numbers before saving; eval sets live in git longer than you think
• No attack cases. If the feature reads outside text, include prompt injection cases (defending at the system level)

A good eval does not answer “how good is it?” It answers “which cases did this change break?”

Sources

• Define success criteria and build evaluations — Claude Docs
• Evaluation best practices — OpenAI API Docs
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Zheng et al., arXiv
• mcnemar — statsmodels
• Models overview (pricing) — Claude Docs