Calibrate an LLM judge against human-labelled cases

An LLM judge can scale evaluation, but it is another model with biases and failure modes. Compare it against a clear rubric and human-reviewed cases before using it in a release gate.

Calculate disagreement explicitly

The retrieval evaluation lab compares eight authored binary label pairs.

from retrieval_cases import judge_calibration_case

result = judge_calibration_case()
assert result["cases"] == 8
assert result["agreement"] == 0.75
assert result["false_pass"] == 1
assert result["false_fail"] == 1
assert result["llm_judge_executed"] is False

The judge fixture agrees on six cases and makes one error in each direction. No judge model ran, and the result is not a performance estimate for any model.

Build the human reference carefully

Write claim-level criteria with examples for support, correctness, citation and abstention. Use at least two reviewers on an informative sample, preserve disagreement and adjudicate with reasons. Human labels are not automatically perfect.

Then run the judge with pinned model/prompt/settings. Report agreement, confusion counts and performance by slice. False passes on critical unsupported claims matter more than a headline agreement rate.

Limit judge authority

Combine deterministic checks with calibrated semantic review. Do not let a model-judge score override missing authorization, invalid JSON or a wrong numerical total. Sample judge passes/fails for ongoing human audit and reevaluate after model or prompt changes.

Watch for position, verbosity and self-preference bias. Blind candidate names and randomize order where the rubric permits. Keep the evidence supplied to the judge identical across candidates.

The Generative & Agentic AI course treats judge calibration and disagreement review as required evaluation evidence.

Exercise

Double-label 50 cases, adjudicate them, then evaluate a pinned judge. Report false passes/fails by risk slice and define where human review remains mandatory.

Continue learning

This article is part of the Retrieval quality and grounded answers sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Test a knowledge assistant on unanswerable questions.
• Continue with Evaluate retrieval on numerical tables.

Reference: OpenAI graders documentation.