Evaluation harness

Testing tells you whether the agent is broken. Evaluation tells you whether a change made it better, and without it every prompt edit is a guess dressed as an improvement.

How it works

Collect real tasks, not invented ones. Twenty real ones beat two hundred synthetic.
Write the expected outcome as a check, not as a target string.
Freeze the set. Add to it, never edit it to make a score improve.
Run it on every prompt, model, or tool change, three times each, and record the pass rate.
Track the score over time next to the change log, because that pairing is the whole point.

Optional: load a real model

See it work

When it pays, and when it does not

Use it when	Skip it when
You change prompts or models regularly	The agent is a one-off script nobody will touch again
Quality is hard to eyeball	Correctness is obvious in one glance every time
Several people can change behaviour	One person owns it and runs everything by hand
You need to justify a model upgrade	Cost and latency are the only things that matter to you

How it fails

Editing cases to pass

The fastest way to build a suite that always passes and means nothing.

Scoring with a model

Model judges have their own biases and drift with model versions. Use them only where code cannot judge, and calibrate against humans.

One run per case

Agents are nondeterministic. One run is an anecdote, three is a measurement.

What it costs

Setup	A day for the harness, and a week of collecting real cases, which is the part people skip.
Per run	Tokens times cases times repeats. Budget for it or it will get switched off.
Maintenance	Low, provided you only ever add.

Never edit a case to make the number better. The moment the set bends to the score, you have replaced measurement with reassurance.

Related: Testing agents · Tracing · Choosing a model · all patterns · agent jobs

Free from AI School - no signup, everything runs in your browser.