Create a human-review sampling policy

Reviewing only model-flagged cases measures what the model already suspects. Reviewing only random cases can miss rare high-consequence failures. A practical policy combines deterministic risk coverage, uncertainty or novelty triggers and a random baseline.

Apply the policy to a queue

The LLMOps and security lab selects all high-risk, confidence-below-0.5 and new-intent cases, plus one authored random sample.

from llmops_cases import human_review_case

result = human_review_case()
assert result["review_ids"] == ["H1", "H2", "H3", "H5"]
assert result["random_sample_id"] == "H5"
assert result["all_high_risk_reviewed"] is True
assert result["queue"][3]["review"] is False

The five records demonstrate selection logic. Their confidence values and random choice are fixtures rather than production output.

Define strata and capacity

Review 100% of actions that could move money, change access, disclose protected data or create legal/physical consequences. Sample lower-risk output by task, language, user group, model release, tool path and outcome. Add cases with low confidence, grader disagreement, novel embedding clusters, user complaints and policy flags.

Retain a truly random slice to estimate failures outside known triggers. Use a deterministic seeded selection or logged sampling probability so the sample is auditable. Weight estimates by inverse selection probability when combining strata; otherwise the reviewed set will overstate deliberately oversampled risks.

Give reviewers a rubric with examples, source evidence and an “uncertain/escalate” option. Blind them to candidate identity when feasible. Measure inter-reviewer agreement and adjudicate material disagreements. Protect user data through minimal fields, role access and retention limits.

Close the loop

Send confirmed failures to owners, regression sets and threat models. Track queue age and reviewer capacity. A policy that selects more records than humans can examine creates delayed risk rather than oversight.

The Generative & Agentic AI course integrates sampling with graders, incident response and release gates.

Reproducible exercise

Generate 500 labelled fixture events, stratify by risk and language, sample with logged probabilities, then estimate the population failure rate with and without weighting. Explain the bias in the naive estimate.

Continue learning

This article is part of the LLMOps, security and operational evaluation sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Define a rollback rule after a model change.
• Continue with Measure disagreement between automated graders.

Reference: NIST AI RMF Generative AI Profile.