Evaluate rare-event models with confidence intervals

A rare-event metric can move substantially when a small number of positives change. Reporting only average precision to three decimals hides that sampling variation. The resampling unit must match dependence in the data.

Bootstrap accounts instead of individual transactions

Our synthetic test period has 1,500 transactions from 500 accounts, three rows per account. The fitted model's average precision is 0.2411.

We draw 500 accounts with replacement for each bootstrap replicate and carry all their test rows into the sampled dataset. Across 2,000 valid replicates, the percentile 95% interval for average precision is [0.1745, 0.3387].

Resampling individual rows would treat three transactions from one account as independent. The account bootstrap preserves their within-account grouping and allows an account to appear multiple times, as required by sampling with replacement.

from imbalance_cases import bootstrap_case

r = bootstrap_case()
assert r['account_clusters'] == 500
assert r['replicates_used'] == 2000
assert r['percentile_95'][0] < r['point_average_precision'] < r['percentile_95'][1]
print(r)

The local lab fixes the random seed and recomputes average precision for every sample. The scikit-learn average precision documentation defines the metric used inside the bootstrap.

State the uncertainty target

This interval conditions on the already fitted model, fixed features, test period and label process. It represents variation from sampling accounts similar to these test accounts. It does not include retraining variability, hyperparameter selection, future temporal shift, label revisions or uncertainty in the business effect of acting on predictions.

The percentile bootstrap is one choice. With very few positive clusters, its coverage can be poor. Stratified, balanced or parametric alternatives make different assumptions. Repeated temporal evaluation may be more relevant when the main uncertainty comes from changing periods rather than account sampling.

Intervals do not repair biased evaluation. Leakage can produce a precise estimate of the wrong quantity. A random split can understate uncertainty when the deployment challenge is new accounts or future months. Define the estimand, split and cluster before choosing the interval method.

For model comparisons, resample the same clusters and calculate paired metric differences. Separate intervals around two models throw away their paired structure. If threshold selection is part of the workflow, repeat it inside each resample or clearly condition on a frozen threshold.

Exercise: compare a row bootstrap with this account bootstrap and a bootstrap of paired model differences. Count positives and positive accounts per replicate. Explain which deployment population each scheme approximates.

NeuraPath's Data Science course teaches confidence intervals with their resampling unit and exclusions. The interval is useful because its limits are explicit.

Continue learning

This article is part of the Imbalance, calibration and decision thresholds sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Optimize recall subject to a minimum precision constraint.
• Continue with Prior probability shift: when yesterday's probabilities mislead.