Threshold selection with unequal error costs

A single false-negative cost assumes every missed event has the same consequence. Many decisions are heterogeneous: missing a high-amount transaction may matter more than missing a small one, while review consumes a roughly fixed amount of analyst time. Row-specific costs can make the threshold rule more realistic, provided the numbers are defensible.

Optimize a declared validation loss

Our synthetic validation table assigns five cost units to reviewing a negative row. A missed positive receives a synthetic cost derived from its authored log amount. These values are teaching inputs, not measured rupee losses.

We evaluate 80 thresholds from 0.02 to 0.80. At each threshold, total loss is:

sum(review_cost for selected negatives) + sum(miss_cost for unselected positives).

The lowest validation loss occurs at threshold 0.37544. It selects 28 rows and has total loss 820.60 units. The grid point near the default 0.5 threshold is 0.50380; it selects nine rows and has loss 925.43.

The scikit-learn cost-sensitive threshold example demonstrates threshold tuning with constant and transaction-dependent business values. The split used for threshold tuning must remain separate from final evaluation.

from imbalance_cases import unequal_cost_case

r = unequal_cost_case()
assert abs(r['chosen']['threshold'] - .3754430379746836) < 1e-12
assert r['chosen']['selected'] == 28
assert r['chosen']['loss'] < r['threshold_half']['loss']
print(r)

Run it in the imbalanced-model lab. The code searches validation only. This article does not present a test cost because adding it after the search requires a predeclared final decision and a separate reporting step.

Distinguish classification loss from intervention value

A predicted positive does not automatically prevent the miss cost. Review can confirm, dismiss, delay or escalate a case. The causal value of action depends on what reviewers do and what would have happened without the action. Historical labels alone do not identify that effect.

Costs may include analyst time, customer friction, delayed legitimate activity, recovered value and downstream escalation. Some are asymmetric and some depend on score, amount or segment. Document units, source, effective date and stakeholder owner. Run sensitivity analysis because disputed cost ratios can reverse the selected threshold.

Probability calibration matters when a decision uses expected cost formulas. Direct empirical threshold search can still overfit validation outcomes, especially across a dense grid. Use nested validation or conservative candidate sets, and evaluate the entire frozen policy once on new data.

Hard capacity constraints cannot always be represented by a fixed per-row cost. If only 30 cases can be reviewed, optimize under that constraint or formulate the queue as a ranking decision.

Exercise: replace the synthetic miss cost with three plausible scenarios and add a hard 40-case capacity. Report whether the same threshold remains feasible and optimal, then identify which cost assumptions require causal evidence.

NeuraPath's Data Science course connects thresholds to explicit loss tables and operational constraints. A cost-sensitive model is credible when its costs can be challenged independently of its code.

Continue learning

This article is part of the Imbalance, calibration and decision thresholds sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Measure false-positive workload before deploying a classifier.
• Continue with Compare group error rates with sample-size context.