A judge reviews 100 outputs. Ninety-five are acceptable. Five contain failures that should block release. The judge approves everything.
Its agreement with the labels is 95%. It catches zero failures.
Hamel Husain’s evaluation guide, updated September 1, 2026, uses this class-imbalance problem to explain why agreement alone is a poor judge-quality metric. Here is the accounting made explicit.
Choose the positive class before naming the rates
In this article, failure is the positive class. A positive prediction means “the judge detected a failure.”
| Human label | Judge detects failure | Judge accepts |
|---|---|---|
| Failure | TP = 0 | FN = 5 |
| Acceptable | FP = 0 | TN = 95 |
agreement = (TP + TN) / total = 95 / 100 = 95%
failure recall = TP / (TP + FN) = 0 / 5 = 0%
false acceptance among actual failures = FN / (TP + FN) = 5 / 5 = 100%Also report false rejection among acceptable outputs: FP / (FP + TN). In this example it is zero. That looks good only until you notice the judge never rejects anything.
“False acceptance among actual failures” is not the fraction of accepted outputs that are bad. The latter has a different denominator, FN / (TN + FN), and equals 5% here. Put the denominator in the metric name or documentation; otherwise two teams can use the same phrase for different risks.
The Python lab computes all four cells and the first three rates. The labels and judge decisions are hypothetical. No model was called, and this is not a measured result for a named provider.
A transcript can claim success without achieving it
An agent says, “Your booking is confirmed.” The booking store has no corresponding reservation.
A language judge reading only the final response might reward clarity and confidence. A deterministic assertion against the store would reject the task. The Anthropic agent-evaluation guide distinguishes transcripts from environment outcomes for precisely this reason.
Make the assertion specific: correct customer, requested date, expected state, and no duplicate or unauthorized side effect. A nonempty table is only the toy demonstration. In a real test, isolate the environment per trial or restore it reliably so one run’s successful booking cannot make the next run pass.
Use model judgment for dimensions that need it, such as whether an explanation is misleading or addresses an ambiguous request. Use ordinary assertions when the question is whether a required state transition happened.
Test the judge before trusting the release gate
Start with domain-expert labels and a rubric that explains concrete failure modes. Keep prompt-development examples separate from a held-out set. If you repeatedly inspect the held-out set and tune the judge against it, it has become development data.
Report counts as well as rates, broken down by severe failure types. Five known failures are enough to demonstrate an always-pass judge’s blindness; they are not enough to certify reliable detection of rare production failures. A deliberately balanced diagnostic dataset helps find blind spots but does not estimate real-world failure prevalence by itself.
Freeze the candidate outputs while comparing judge versions. Otherwise a change in the agent, retrieval data and judge can all move the score at once. Record dataset, rubric, candidate, environment and judge versions with each result.
Inspect disagreements with the expert. The label can be wrong; the rubric can be ambiguous; the judge can rely on superficial phrasing. That investigation produces better evidence than asking a second model to break every tie automatically.
A different provider is an experiment, not ground truth
The self-preference study provides empirical evidence of model preferences linked to text familiarity. It supports testing for bias. It does not establish a universal five-to-ten-point score penalty for same-family judging, or prove that a different provider eliminates bias.
Compare candidate judges against the same held-out expert labels. Randomize answer order where relevant, test paraphrases that preserve correctness, and look for failure categories they jointly miss. Two judges can agree for the same wrong reason.
The updated AI engineering evaluation chapter covers drift and suite design. Production Agents: Testing & Evaluation covers operational regression checks. This lab adds the smaller question underneath both: does the instrument detect the failure your release process cares about?