Eval Confidence Interval Calculator
Your model scored "87% accuracy" on the eval set. But on 30 examples, that could easily mean anywhere from 70% to 95%. This tool shows the honest range behind any accuracy number, using the Wilson score interval.
- observed accuracy
-
How many more evals do I need?
Pick a target margin of error and see how large your eval set must be (at the observed accuracy and selected confidence level).
- total evals needed
-
Why small eval sets mislead: accuracy on an eval set is an estimate of true performance, and its uncertainty shrinks only with the square root of the sample size. Ten evals give a margin of roughly plus or minus 26 points at 95% confidence; you need about 100 for plus or minus 9, and about 1,000 for plus or minus 3. This is why "prompt v2 scored 90% vs v1's 85% on our 20-case suite" is usually meaningless - the intervals overlap almost entirely. The Wilson score interval used here is preferred over the naive "p plus or minus z*sqrt(p(1-p)/n)" formula because it stays inside 0-100% and behaves sensibly for small n and extreme rates (even 20/20 correct does not mean 100% guaranteed). When comparing two variants, use the A/B Test Calculator instead of eyeballing overlapping intervals.