Free Engineering Workbench calculator
Eval regression check
Compare a new model, prompt or agent version with the current one on the same eval cases.
AI evaluationHow it's worked out
- McNemar test on discordant pairs: χ² = (|b − c| − 1)² ÷ (b + c)
- Same cases run on both versions
NoteLook at the cases that got worse even when the overall change is an improvement; they often show a new failure mode.
Source: McNemar (1947); Dietterich (1998) on comparing classifiers
Need help acting on this number?
VVnT SeQuor's AI Testing & Evaluations practice helps teams turn this metric into a plan.