A/B Test Calculator for Prompts and Models
Comparing prompt v1 vs v2, or model A vs model B? First find out how many test cases you actually need. Then, once you have results, check whether the difference is real or just noise.
1. Plan: how many test cases per variant?
-
test cases needed per variant
2. Check: are my results significant?
Enter how many test cases passed (successes) out of how many you ran (trials) for each variant.
Variant A (baseline)
Variant B (challenger)
- two-sided p-value
Why this matters for AI teams: a prompt change that "wins 6 out of 10 test cases" tells you almost nothing - with samples that small, coin-flip noise dominates. The planner uses the standard normal-approximation sample size formula for comparing two proportions, and the checker uses a two-proportion z-test with a pooled standard error. Rules of thumb: decide the sample size before running the test, do not stop early the moment p dips under 0.05, and remember that statistical significance is not the same as practical significance - a 0.5-point gain can be significant and still not worth the migration cost. The normal approximation is reliable when each cell (successes and failures per variant) is at least about 10.