Nested cross-validation: separate tuning from assessment

Nested cross-validation puts model selection inside an outer assessment loop. Each outer held-out portion evaluates a selection procedure that has not used its labels, rather than reporting the same score that selected the hyperparameter.

Our original example uses only the 160 January snapshots from forty fictional customers. It holds out customer identities, so it asks about unseen customers within that January setting. It does not replace the separate chronological March test for future predictions on existing customers.

Give the two loops different jobs

The outer loop has five group folds. Each holds out eight customers and their 32 rows, leaving 32 customers and 128 rows for development.

Inside each outer development portion, three group folds compare C=0.1,1 and10 using log loss. The full imputation, scaling, encoding and logistic-regression pipeline is fitted inside each inner training portion. After selecting C, the search refits on that outer development portion and predicts its untouched outer assessment rows.

The scikit-learn nested-validation example explains this separation. Our account fixture, grouping and results are original.

Preserve every outer result

Mean outer log loss is approximately0.676864 for the selection procedure and0.683843 for the training-prior baseline. The candidate performs worse in two folds. A report that shows only the best fold would materially misrepresent the experiment.

Run and check the nested procedure

import numpy as np
from assessment_cases import nested_groups

r = nested_groups()
assert len(r['folds'])==5
assert all(row['development_customers']==32 and row['assessment_customers']==8 for row in r['folds'])
assert all(row['selected_C']==.1 for row in r['folds'])
assert np.isclose(r['mean_outer_log_loss'],.6768643997908723)
assert np.isclose(r['mean_baseline_outer_log_loss'],.6838429740660891)
assert sum(row['outer_log_loss']>row['baseline_outer_log_loss'] for row in r['folds'])==2
print(r)

The lab includes the complete implementation. Group identifiers are supplied to both split levels. The GridSearchCV reference documents the search, scoring and refit behavior used here.

Avoid a new form of selection leakage

Choosing the most favorable outer fold or revising the candidate grid after repeatedly inspecting outer scores makes the outer assessment part of development. Record the full procedure and any subsequent changes.

The outer training sets overlap, so their scores are not independent observations in the simple sense required by an ordinary independent-sample confidence calculation. A standard deviation across five fold scores is descriptive spread, not automatically a valid confidence interval.

Nested validation also inherits the assumptions of its splitters. Group folds do not impose time order, and clean loops do not repair future-derived features or incorrect labels. Our synthetic snapshots do not establish real-world event-history consistency.

Decide what to fit after assessment

The nested loop evaluates the declared tuning procedure; it does not automatically deliver a single universally best final model. A deployment workflow can apply that procedure to its allowed development data, freeze the resulting artifact and assess it under the intended population and time conditions.

Exercise: add a second model family to the inner candidate procedure without changing the outer grouping. Explain which comparisons are now being selected and why the outer results assess the enlarged procedure rather than an individual hand-picked configuration.

NeuraPath's Data Science course connects hyperparameter search with credible assessment. The result should describe the whole selection process and preserve its unsuccessful folds.

Continue learning

This article is part of the Machine learning workflow and evaluation sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Precision-recall versus ROC curves for rare outcomes.
• Continue with Build an error taxonomy for a trained model.