Cost-sensitive classification with an explicit loss table

A classifier threshold encodes a decision trade-off. Accuracy hides that trade-off when false positives and false negatives have different consequences or when a team can act on only a limited number of cases. An explicit loss table makes the assumptions reviewable.

Start with frozen validation predictions

This worked example reuses the 80 validation predictions from our evaluated synthetic inactivity model. The model and probabilities remain frozen; this exercise does not retrain or alter the reference experiment.

We compare three declared thresholds: 0.3, 0.5 and 0.7. The illustrative loss assigns one unit to a false positive and five units to a false negative. Correct decisions receive zero units in this simplified table.

Without a capacity constraint, 0.3 has the smallest observed validation loss among these candidates. With capacity for at most 30 selected cases, that threshold is infeasible because it selects 64. Threshold 0.5 becomes the lowest-loss feasible candidate and selects 24.

The scikit-learn model-evaluation guide documents confusion-matrix metrics and threshold-dependent evaluation. The calculation here keeps the loss assumptions alongside those counts.

from decision_cases import cost_case

r = cost_case()
assert [row['illustrative_loss'] for row in r['candidates']] == [59, 145, 235]
assert r['unconstrained_threshold'] == .3
assert r['capacity_constrained_threshold'] == .5
assert next(row for row in r['candidates'] if row['threshold'] == .5)['selected'] == 24
print(r)

Run this in the supervised-model lab. It reads the stored validation predictions and recomputes each confusion matrix. The code never reads test rows for threshold selection.

Treat the numbers as policy inputs, not discovered truth

The costs are authored units. They are not rupees, measured customer impact or causal estimates of an intervention. If probabilities are calibrated and actions have constant expected utilities, an unconstrained theoretical threshold for FP cost 1 and FN cost 5 is 1/(1+5), about 0.1667. Our candidate set does not include it, and real capacity creates a separate optimization problem.

A false negative cost can vary by customer, time and available action. Contacting a flagged person might help, do nothing or cause harm. Classification data reveal association with an outcome; they do not by themselves estimate the effect of acting. A deployment decision therefore needs calibrated probabilities, action-effect evidence, operational cost and fairness review where applicable.

Capacity must share the same time grain as scoring. “Thirty reviews” is incomplete without saying per day, per queue and at what service level. Ties at a threshold also need a deterministic policy. After choosing on validation data, assess the full policy once on untouched, mature outcomes or through an appropriate prospective design.

Exercise: add a cost of two units for every reviewed case, including true positives. Recompute the table, define whether capacity is a hard constraint or a penalty, and state how you would handle two rows with identical scores at the cutoff.

NeuraPath's Data Science course connects model scores with explicit decisions, denominators and operational constraints. A loss table makes the argument inspectable before a threshold becomes policy.

Continue learning

This article is part of the Supervised learning methods sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Multi-output regression with shared and separate models.
• Continue with Learning curves: decide whether more data may help.