Feature selection nested inside validation

Feature selection uses the target to choose columns. If selection happens before cross-validation, every fold's labels influence the chosen feature set. Cross-validation then evaluates a decision already tuned on its holdout labels.

Run a falsification test with no real signal

We generate 300 rows, 800 independent normal-noise features and independent random binary labels. There is no predictive relationship by construction. A univariate selector keeps the 20 features with the largest F statistics.

When the selector fits once on all 300 labels before five-fold cross-validation, mean ROC AUC is 0.8045. The procedure mined chance correlations involving every holdout fold.

When selection sits inside a pipeline and refits on each fold's training rows, mean AUC drops to 0.5445, much closer to chance. The remaining departure comes from finite sampling and the specific seed, not known signal.

The scikit-learn feature-selection guide and Pipeline documentation support fitting selectors within the validation workflow.

from feature_cases import nested_selection_case

r = nested_selection_case()
assert r['noise_features'] == 800 and r['selected'] == 20
assert r['leaky_mean_cv_auc'] > .8
assert r['nested_mean_cv_auc'] < .56
assert r['leaky_mean_cv_auc'] > r['nested_mean_cv_auc'] + .2
print(r)

Run the falsification case in the feature-engineering lab. Its intentionally impossible success is a leakage alarm.

Nest every supervised choice

Hyperparameter tuning, missing-value decisions based on outcomes, target encoding, threshold selection and feature elimination belong inside the appropriate training fold. If cross-validation chooses among entire pipelines, use an outer assessment loop or untouched test period for final performance.

Unsupervised filters such as constant-column removal can sometimes fit globally without target leakage, but they still inspect validation distribution and may violate strict deployment simulation. Keeping all learned transformations inside the pipeline is safer and easier to audit.

Record selected features per fold. Large variation indicates instability even when mean score looks good. Domain review should catch forbidden future fields and proxies that a statistical selector cannot understand.

Exercise: vary the number of noise features and selected columns across predeclared values. Plot contaminated and nested AUC. Then add one true signal feature and measure how often each fold retains it.

NeuraPath's Data Science course teaches feature selection as part of model fitting. Random-label falsification is a powerful test that the validation boundary is real.

Continue learning

This article is part of the Feature engineering and data quality sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Create time-based features without looking into the future.
• Continue with Permutation importance with correlated features.