Compare models with uncertainty instead of one lucky score

A lower observed error is evidence about the evaluated sample. It does not establish that the same advantage will hold across other samples, future periods or repeated training runs. State the comparison and the uncertainty source you are estimating.

Our frozen synthetic inactivity model has test log loss 0.702749, compared with 0.754582 for its training-prevalence baseline. Define the difference as model minus baseline: approximately -0.051833. Negative values favor the model.

Preserve the paired comparison

Both methods predict the same eighty test snapshots. Each of forty customers contributes two rows. Calculate each row's model loss minus its baseline loss, then average within customer.

Resampling customer groups keeps the two observations together. It also keeps both methods' errors paired on the same observations. Resampling the model and baseline independently would discard that pairing and estimate a different variability structure.

Every customer has two rows in this fixture, so the average customer difference equals the average row difference. With unequal group sizes, explicitly choose whether the target gives equal weight to customers or observations.

Reproduce the conditional interval

The evaluation lab draws 10,000 bootstrap samples of forty customer groups with replacement, using integer seed 20261006. It records the 2.5th and 97.5th percentiles of the resampled mean differences.

import numpy as np
from metric_cases import paired_customer_bootstrap

r = paired_customer_bootstrap()
assert r['customers']==40 and r['rows_per_customer']==2
assert r['replicates']==10000
assert np.isclose(r['difference_model_minus_baseline'],-.051832875243986276)
assert np.allclose(r['percentile_95_interval'],[-.10330985269481638,.0014551507376661576])
assert r['percentile_95_interval'][0] < 0 < r['percentile_95_interval'][1]
print(r)

The interval is approximately [-0.10331, 0.00146]. Although the observed mean favors the model, this percentile interval includes zero. Report that result rather than rounding the upper endpoint to zero or describing the improvement as decisively established.

SciPy's bootstrap documentation provides primary context for resampling intervals and paired observations. Our implementation explicitly resamples whole customer groups; it is not a copied library example or a claim that ordinary row resampling handles dependence.

Name the uncertainty this does not include

The fitted model and baseline stay fixed. This calculation therefore does not include changes caused by drawing a new training set, selecting a different hyperparameter or refitting preprocessing. It is conditional on the chosen models and the two test dates.

Customer-level resampling also does not model an arbitrary future market shift or a shared time shock affecting every account. The fixture is a synthetic conditional snapshot simulation, not evidence that real customers are exchangeable independent clusters.

A nominal 95% bootstrap interval is not a 95% guarantee about deployment success. With few clusters, unusual group structure or a biased sample, the interval can itself be unreliable.

Connect statistical and operational relevance

Before comparison, define what improvement would justify the added maintenance and decision consequences. A narrow interval around a tiny advantage may still be operationally unimportant. A wider interval can indicate that more appropriate evaluation evidence is needed.

Do not keep changing seeds until an interval excludes zero. Preserve the seed, resampling unit, statistic and interval method in the experiment record. If comparing many candidates, account for the selection procedure rather than presenting the most favorable comparison as if it were the only one attempted.

Exercise: recompute an ordinary row bootstrap and compare it with the customer bootstrap. Explain which dependence each retains. Then propose an evaluation design that would also investigate training variability without reusing the final test set for model selection.

NeuraPath's Data Science course connects model comparisons with uncertainty and decision costs. A strong project preserves an inconclusive interval when that is what the executed analysis shows.

Continue learning

This article is part of the Machine learning workflow and evaluation sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Build a scikit-learn pipeline that prevents preprocessing leakage.
• Continue with Choose MAE or RMSE from the decision cost.