Human review doesn’t scale to the volume of outputs a production LLM application generates. Using a second LLM to score the first one’s outputs scales fine — the question is whether the resulting scores are actually trustworthy, which depends entirely on how the judge is set up.
Automated metrics like BLEU or ROUGE measure textual overlap, not whether an answer is actually…
Position bias — when comparing two outputs side by side, judge models show a measurable…
Pairwise comparison (which of these two outputs is better, on a specific stated criterion) tends to…
Automated metrics like BLEU or ROUGE measure textual overlap, not whether an answer is actually correct, helpful, or appropriate — they work poorly for open-ended generation. Human review captures quality well but doesn’t scale to continuous evaluation across thousands of outputs. LLM-as-a-judge sits between the two: an LLM, prompted with clear criteria, scores another model’s outputs at a volume and speed human review can’t match, while capturing more nuance than overlap-based metrics.
Pairwise comparison (which of these two outputs is better, on a specific stated criterion) tends to produce more consistent judgments than absolute scoring (rate this 1-10) for the same comparison. Explicit, detailed rubrics outperform vague quality prompts. Running comparisons in both orders and averaging addresses position bias directly. And periodically validating the judge’s scores against a sample of human-reviewed ground truth — not setting it up once and trusting it indefinitely — catches drift as the underlying judge model or the application being evaluated changes.
LLM-as-a-judge is strongest for well-defined, criteria-based evaluation at scale — factual accuracy against a reference, adherence to a format, tone consistency. It’s weaker for genuinely subjective or high-stakes judgment calls, where a periodic human-in-the-loop spot review (the same practice recommended for hallucination testing generally) remains the check that catches what the automated judge’s own biases might miss.
It’s technically possible but introduces self-preference risk — the judge can favor outputs that match its own stylistic tendencies. Using a different model, or at minimum validating self-evaluation scores carefully against human review, reduces this risk.
On any meaningful change to the model being evaluated, the judge model itself, or the evaluation criteria, plus a periodic baseline check (monthly is a reasonable starting cadence for most production applications) even with no changes, since judge model providers update their models in ways that can shift scoring behavior.
Treat it as one signal among several for safety-critical use cases, not the sole gate — combine it with rule-based checks, human review, and the broader evaluation dimensions (covered in our piece on evaluating an LLM application before launch) rather than relying on judge scores alone for the highest-stakes decisions.