Evals: how not to fool yourself

Feynman famously warned that the person easiest to fool is yourself. He wasn't talking about language models, but he might as well have been. LLM evaluation is full of small, honest ways to convince yourself that something improved when it didn't.

Your eval set is smaller than it feels

A hundred hand-written test cases feels like a lot of work, and it is. Statistically, it's not much. For a pass rate around 70% on 100 independent items, the standard error is:

sqrt(0.7 × 0.3 / 100) ≈ 0.046   →   a 95% interval of roughly ±9 points

That 4-point improvement is well inside the noise. This doesn't mean small eval sets are useless — it means you should read them carefully:

Compare paired results, not two totals. Both versions ran on the same items, so look at the items that flipped. If 9 went from fail to pass and 5 went the other way, you have a much more honest picture than "+4%" — and a list of examples to read.
Run more than once. If you sample at any temperature above zero, run each version three to five times. The spread between runs of the same prompt is your noise floor.
Grow the set where it matters. When a result is borderline, add cases in that category rather than re-running the same ones.

LLM-as-judge: useful, biased, measurable

Grading open-ended answers by hand doesn't scale, so most teams ask a strong model to grade them. That works far better than it has any right to — but judges have well-documented biases:

Bias	What happens	Mitigation
Position	In A-vs-B comparisons, the judge favours whichever answer comes first (or second)	Judge both orders; count only consistent verdicts
Verbosity	Longer answers score higher, even when they add nothing	Grade against a rubric; penalise unsupported content explicitly
Self-preference	A judge rates outputs from its own model family higher	Use a different family as judge, or at least check agreement with one
Rubric drift	Vague criteria like "helpfulness" get interpreted differently per item	Concrete, checkable criteria; one criterion per question

The most important step is the boring one: calibrate the judge against yourself. Grade 50 items by hand, run the judge on the same 50, and compute how often you agree. If agreement is 90%, trust it for trends. If it is 65%, you don't have an eval — you have a random number generator with good manners.

Prefer checks that can't be argued with

Whenever the task allows it, replace judgment with verification:

def check(case, output):
    data = json.loads(output)                   # valid JSON at all?
    assert set(data) == {"name", "date", "total"}
    assert data["total"] == case.expected_total  # exact value
    assert parse_date(data["date"]) == case.expected_date

Extraction, classification, code that has tests, SQL that can be executed against a fixture database — these give you deterministic pass/fail signals. Keep the LLM judge for the genuinely open-ended parts, and keep those parts as small as you can.

Slice the results

An overall score hides trade-offs. A prompt change that improves short factual questions by 10 points and breaks multi-step questions by 10 points can show up as "no change". Tag every case with a category and always look at the per-category table. Most real regressions I've caught were invisible in the headline number.

Keep a regression set

Every bug report that turns out to be a model failure becomes a test case. Over a few months this set becomes the most valuable eval you own, because it is made of exactly the things your users actually hit. Run it on every change, and treat a newly failing case there as a blocker, not a statistic.

And read the outputs

The habit that beats every metric: after each eval run, read twenty raw outputs — ten random, ten from the items that changed. Five minutes of reading regularly reveals problems no score would have surfaced: a new refusal pattern, a formatting tic, a judge that is grading the wrong thing.

Numbers tell you whether something changed. Only the outputs tell you what changed. You need both, and of the two, the reading is the one people skip.