Tune multiclass thresholds without using the test set

Argmax assigns every row to the class with highest score. Operations may need different evidence for each class: one category may trigger a costly action, while another can be handled automatically. Per-class thresholds allow different operating points, but independent decisions can overlap or abstain.

Expose conflicts with a tiny construction

Our six-row teaching table has three classes and authored probability vectors. For each class, candidate thresholds are 0.3, 0.4 and 0.5. Using the same tiny table for demonstration, we choose maximum recall, then precision, then fewer selected rows.

All three classes select threshold 0.3. Class zero selects three rows with precision two thirds and recall one. Class one selects four with precision one half and recall one. Class two selects two with precision and recall one.

Every row receives at least one class, but three rows exceed the threshold for more than one class. Independent thresholding therefore does not define a unique multiclass prediction.

from imbalance_cases import multiclass_case

r = multiclass_case()
assert r['candidate_thresholds'] == [.3, .4, .5]
assert all(row['threshold'] == .3 for row in r['chosen_by_class'].values())
assert r['rows_with_no_class'] == 0
assert r['rows_with_multiple_classes'] == 3
print(r)

Run the construction in the imbalanced-model lab. It intentionally tunes and diagnoses on six authored vectors, so it is an algorithm illustration rather than a performance estimate.

The scikit-learn precision-recall example discusses one-vs-rest extension to multiclass settings. Each class curve still needs a validation-only operating policy.

Specify the decision semantics

If classes are mutually exclusive, define how to resolve multiple passing thresholds: highest margin above threshold, highest expected utility, a priority order or manual review. Define what happens when none passes. If labels are genuinely multilabel, multiple outputs may be correct and evaluation must reflect that.

Use a disjoint validation set or nested cross-validation to tune thresholds and conflict rules together. Searching each class independently and inventing the resolver after inspecting test conflicts leaks test behaviour into the policy. The untouched test set evaluates the entire frozen mapping once.

Report per-class positive support, precision, recall, selected count, overlap and abstention. Probability calibration can differ by class and may be distorted by one-vs-rest normalization. Changed class prevalence can invalidate the chosen boundaries.

Exercise: implement three conflict rules for the six vectors and show which rows change. Then create separate validation and test tables, choose thresholds and the resolver on validation, and evaluate exact-match accuracy, per-class recall, overlap and abstention on test.

NeuraPath's Data Science course connects multiclass metrics to a complete decision function. Thresholds are only half the policy; conflicts and abstentions must be reproducible too.

Continue learning

This article is part of the Imbalance, calibration and decision thresholds sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Abstention: let a classifier decline uncertain cases.
• Continue with Detect label noise in an imbalanced dataset.