Train, validation and test sets: assign each a separate job

Training data fit the model and its learned preprocessing. Validation data support declared choices such as hyperparameters. Test data assess the procedure after those choices are fixed. Naming three folders is not enough; the actual flow of information must preserve these roles.

Our original synthetic account project uses chronological sets because it asks about later inactivity predictions for existing customers. Its 160 January, 80 February and 80 March snapshots provide a concrete example of the separation.

Assign decisions to the right set

The experiment contract fixes a tie rule too: exact validation-loss ties prefer the smaller C. That removes one otherwise undocumented choice after seeing results.

The reference does not refit on combined training and validation data. Such a refit can be part of another declared procedure, but it changes fitted preprocessing and coefficients. Evaluate the procedure you actually intend to use, rather than mixing the selected model's validation score with an unrecorded replacement fit.

Check that labels existed when they were used

Each target covers seven future days and has one additional reporting day. January training labels are available by the February 1 freeze. February validation labels are available by March 1, before the first March test decision. The March assessment waits until March 20.

import pandas as pd
from evaluation_core import load,contract,run

data = load()
cfg = contract()
for split,key in [('train','train_freeze'),('validation','selection_freeze'),('test','evaluation_as_of')]:
    rows = data[data['split']==split]
    assert (rows['label_available_at']<=pd.Timestamp(cfg[key])).all()
assert data[data['split']=='train']['decision_at'].max() < data[data['split']=='validation']['decision_at'].min()
report,_,_ = run()
candidate = min(report['candidate_validation'],key=lambda row:(row['validation']['log_loss'],row['C']))
assert candidate['C']==report['selected_C']==.1
print({'split_counts':report['split_counts'],'selected_C':report['selected_C']})

The lab includes all inputs and outputs. Its availability metadata is a teaching declaration, not independent evidence of a real ingestion system.

Recognize indirect test-set tuning

Choosing a model after comparing test scores is tuning. So is changing a feature because it improves test performance, searching thresholds on test labels, or repeatedly revising a cleaning rule in response to the final score.

Ordinary bug correction is still necessary, but record what changed and why. After substantial test-informed development, obtain another untouched assessment period or clearly state that the original test set has become development evidence.

Scikit-learn's cross-validation guide explains the separation between model selection and held-out assessment. The date boundaries, candidate grid and result here are original teaching choices, not copied benchmark results.

Choose roles before choosing percentages

A standard percentage split may leave too few positive cases, fail to respect customer grouping or mix future and past conditions. The right split follows the deployment question and available evidence. These sets intentionally share customer identities because the target population is existing customers at later times.

Exercise: write down every decision you made in a recent model project and the data that influenced it. Identify any test-driven choice. Propose a corrected assessment design and state whether it changes the target population or time period being evaluated.

NeuraPath's Data Science course connects fitting workflows with evaluation discipline. A clear experiment record lets a reviewer follow which observations influenced the model and which supplied its final assessment.

Continue learning

This article is part of the Machine learning workflow and evaluation sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Build a baseline before choosing a complex model.
• Continue with Choose a data split that matches the deployment setting.