Abstention: let a classifier decline uncertain cases

A classifier need not automate every row. An abstention policy can route uncertain cases to review while making automatic decisions only in lower-risk regions. The trade-off is between automated coverage, automated error and human workload.

Define three disjoint regions

Our validation illustration uses two prespecified probability boundaries. Scores below 0.10 become auto-negative candidates. Scores from 0.10 up to 0.35 go to manual review. Scores at or above 0.35 become auto-positive candidates.

Among 1,500 validation rows:

• 1,205 are auto-negative candidates;
• 264 require manual review; and
• 31 are auto-positive candidates.

The automated regions cover 82.4% of rows. Comparing their assigned binary decisions with synthetic labels gives an automated error rate of 4.77%. That number excludes reviewed rows by design.

from imbalance_cases import abstention_case

r = abstention_case()
assert r['auto_negative'] + r['manual_review'] + r['auto_positive'] == 1500
assert abs(r['automated_coverage'] - (1205+31)/1500) < 1e-12
assert r['manual_review'] == 264
print(r)

Run it in the imbalanced-model lab. The boundary values and action labels are teaching choices, not a deployed risk policy.

Do not treat review as an oracle

Manual review has its own sensitivity, specificity, delay, cost and disagreement. Reviewers may use information unavailable to the model, or they may share the same weak labels. Evaluate the combined system: automated outcomes, review outcomes, unresolved cases and downstream action.

Coverage must have a denominator and period. A low automated error can be achieved by abstaining on nearly everything. Plot risk or loss against coverage and include the review queue. Segment-level coverage matters because uncertainty may concentrate on a particular population.

Probability calibration is useful when boundaries represent risk levels, but calibrated averages do not guarantee a specific error rate inside every slice. If the abstention score is distance from 0.5, remember that class imbalance and unequal costs can make 0.5 operationally irrelevant.

Choose boundaries on validation data under capacity and error constraints. Freeze them for final assessment. Define fallbacks for unavailable reviewers, score ties, missing features and out-of-distribution cases. Log the score, model version, route and final decision for audit.

Exercise: vary the two boundaries over a predeclared grid. Plot automated coverage, automated false-negative count and review workload. Add a reviewer with simulated 85% sensitivity and 95% specificity, then compare end-to-end errors while clearly labelling those review rates as assumptions.

NeuraPath's Data Science course frames abstention as a system design problem. A useful policy measures both what the model decides and what happens to the cases it declines.

Continue learning

This article is part of the Imbalance, calibration and decision thresholds sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Compare group error rates with sample-size context.
• Continue with Tune multiclass thresholds without using the test set.