Measure disagreement between automated graders

Two graders can produce the same average score while disagreeing on the cases that matter. Inspect row-level disagreement and compare each grader with human reference labels before using it for a release gate.

Locate disagreement

The LLMOps and security lab contains authored rule, model-judge and human labels for four cases.

from llmops_cases import grader_disagreement_case

result = grader_disagreement_case()
assert result["rule_model_disagreements"] == ["G2", "G3"]
assert result["model_human_agreement"] == 0.5
assert result["labels"][1]["human"] == "fail"
assert result["model_judge_executed"] is False

The “model” labels are fixtures; no judge ran. With only four rows, 50% is illustrative rather than a reliable estimate.

Match graders to criteria

Use deterministic rules for schemas, exact required fields, citations that exist, forbidden tool calls and authorization invariants. Use human judgement for context-dependent usefulness, tone or source support when rules cannot encode them. A model judge may scale some rubric decisions, but it needs a precise rubric, representative calibration set and a pinned version.

Build a confusion matrix for each grader against adjudicated human labels. Report agreement by slice and class, not only accuracy. For ordinal scores, measure how far labels differ. Review systematic patterns: one grader may pass fluent unsupported answers or fail concise correct ones.

Route rule/model disagreement, low judge confidence and high-impact cases to human review. Preserve the input, candidate output, evidence, rubric, grader versions and reasons. If a grader changes, rescore a stable calibration set before comparing new and old system runs.

Protect the release gate

A model judge must not override deterministic business or security failures. Treat judge output as evidence with known error, not ground truth. Keep a human-audited random sample even when agreement appears high.

The Generative & Agentic AI course uses grader disagreement to design sampling and defensible evaluation reports.

Reproducible exercise

Label fifty local outputs with a rule and two blinded reviewers. Compute confusion matrices and Cohen’s kappa where appropriate, adjudicate conflicts, then define which decisions can be automated.

Continue learning

This article is part of the LLMOps, security and operational evaluation sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Create a human-review sampling policy.
• Continue with Write an AI incident postmortem from trace evidence.

Reference: OpenAI Evals API.