Time-aware validation for a changing business process

A time-aware evaluation should recreate what could have been known when a model was fitted and used. Sorting by prediction date is necessary for many forecasting settings, but it does not ensure that the training labels were available at the simulated fitting time.

In our original inactivity fixture, a target covers seven future days and takes one additional reporting day to become available. A January 8 prediction therefore has a label available January 16. That label cannot train a model supposedly fitted for a January 15 decision.

Split on availability, not only on observation date

For a validation window starting January 15, only the January 1 training snapshots have matured labels. January 8 snapshots occurred earlier but their outcomes are not yet available.

For a January 22 validation start, January 1 and January 8 snapshots qualify. January 15 outcomes remain unavailable until January 23. The correct eligible training counts are therefore forty and eighty, respectively.

import pandas as pd
from evaluation_core import load

d = load()
january = d[d['split']=='train']
summary=[]
for date,expected in [('2026-01-15',40),('2026-01-22',80)]:
    start = pd.Timestamp(date,tz='UTC')
    fit = january[(january['decision_at']<start) & (january['label_available_at']<=start)]
    held = january[january['decision_at']==start]
    earlier_only = january[january['decision_at']<start]
    assert len(fit)==expected and len(held)==40
    assert len(earlier_only)==expected+40
    assert (fit['label_available_at']<=start).all()
    summary.append({'validation_start':date,'eligible_training_rows':len(fit),
                    'earlier_but_immature_rows':len(earlier_only)-len(fit)})
print(summary)

Run in the evaluation lab. This block audits eligibility; it does not fit additional models or present fold scores that were never measured.

Decide how the model would actually be refreshed

An expanding window retains all eligible history. A rolling window retains a defined recent interval. Either can be reasonable depending on the process and deployment plan, but comparing them is model selection and belongs in development evidence.

Record the fitting schedule, label delay, feature availability, retraining window and evaluation horizon. If deployment retrains monthly, a simulation that refits before every observation may assess a different system.

Scikit-learn's TimeSeriesSplit reference documents ordered splits and a sample-count gap. A gap measured in rows is not automatically the required number of elapsed days, especially when many entities share a timestamp or observations arrive irregularly. Explicit timestamp conditions make the requirement visible here.

Inspect changing conditions without adapting to the test set

The fixture's training, validation and test positive proportions are 43.125%, 58.75% and 68.75%. These are synthetic period differences. They explain why a constant training-prevalence baseline may become poorly matched to later outcomes, but they do not prove a particular real-world cause.

Report performance by a meaningful time window, with sample and positive counts. A single aggregate can hide deterioration or a changed population mix. Preserve the final assessment period while using earlier windows to choose a retraining or recalibration policy.

Check the data-history assumption

Historical tables may contain corrections made after the simulated prediction time. A current extract with old event dates can still leak later knowledge. A real implementation needs as-of feature reconstruction or trustworthy historical snapshots, plus coverage and identity checks.

Our rows are authored conditional snapshots, not a continuous event reconstruction. Their timestamps exercise the validation contract; they cannot establish that an actual source system preserves historical availability correctly.

Exercise: increase reporting delay to three days and include a decision every day. Recompute training eligibility at two validation starts. Explain why a fixed one-row gap would fail and document the fitting schedule your revised evaluation represents.

NeuraPath's Data Science course connects time-dependent data with model assessment. A defensible evaluation recreates the information boundary around each simulated deployment decision.

Continue learning

This article is part of the Machine learning workflow and evaluation sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Cross-validation with repeated customers or patients.
• Continue with Build a scikit-learn pipeline that prevents preprocessing leakage.