Free Engineering Workbench calculator
Eval suite size
Find how many eval cases you need for a pass rate to be precise enough to act on.
AI evaluationHow it's worked out
- n = z² × p(1 − p) ÷ margin². Two-version comparison uses the two-proportion sample size at 80% power
- Independent cases; normal approximation
NoteSmall eval sets give noisy pass rates: with 100 cases, a 5-point change can be chance. Running the same cases on both versions, as in the regression check, needs fewer.
Source: Standard sample size formulas (Fleiss, Statistical Methods for Rates and Proportions)
Need help acting on this number?
VVnT SeQuor's AI Testing & Evaluations practice helps teams turn this metric into a plan.