Time-series train-test splits: reproduce a real forecast date

A time-series split should reproduce a decision that could have happened on a named date. Randomly shuffling monthly demand mixes later conditions into earlier training data. The resulting score answers how well a model interpolates randomly hidden rows, while a planner needs to know how the model would have behaved when the future was still unavailable.

The practical unit is the forecast origin: the cutoff after the last observation that the model may use. For a forecast issued on 1 January 2023, training ends on 1 December 2022. The validation target begins on 1 January. Every lag, rolling average, scaler and imputer must respect that boundary.

A reproducible split contract

The local time-series lab contains 144 original monthly observations. The first 96 months are training data, the next 24 are validation data, and the last 24 are a locked test period.

Run the same boundary check locally:

from timeseries_cases import split_case

result = split_case()
assert result["forecast_origin"] == "2023-01-01"
assert sum(part["rows"] for part in result["splits"]) == 144
print(result["frequency"], result["forecast_origin"])

The output is month start 2023-01-01. Those two values matter. “Monthly” alone does not say whether a timestamp labels the start or end of a month, and an integer such as row 96 cannot be reconciled easily with a planning calendar.

What belongs on each side of the cutoff

Training may use observations available by 31 December 2022. Validation targets occur after that date. A feature for January 2023 may use December 2022 demand, but it cannot use January’s completed sales, a full-year 2023 average, or a promotion outcome entered after the campaign closed.

Fit transformations inside the same rule. Estimate a mean, scale, category vocabulary or missing-value policy on training data and apply that fitted operation to validation. If a statistic sees validation first, moving the target column out of the feature matrix does not prevent leakage.

Keep the test partition closed while comparing ARIMA orders, lag windows or algorithms. Repeatedly choosing the model with the best test score turns the test period into another validation period. The final number then understates the cost of model selection.

Stronger evaluation uses several origins

One fixed split remains useful for a reproducible final comparison, but it represents one historical state. A rolling-origin backtest repeats the decision at several earlier cutoffs. It can reveal a model that works during ordinary months yet fails after a policy change. Keep the final held-out period even when rolling validation is used.

The fixture is synthetic and its dates are teaching dates, so its scores are not evidence about NeuraPath, a client or a public company. Its value is that every row boundary can be inspected and rerun. The Data Science course connects this split discipline to feature engineering, model evaluation and deployment work.

Exercise

Move the validation origin forward by 12 months. Write down which rows each transformation may fit, which horizons remain scoreable, and whether the test period must move. Reject any design that cannot name its origin before model fitting.

Further reading: Forecasting: Principles and Practice, time-series cross-validation and the scikit-learn time-related feature engineering example.

Continue learning

This article is part of the Forecasting and time-series analysis sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Continue with Seasonal naive forecasting as a serious baseline.
• Then apply it in Rolling-origin backtesting with multiple forecast horizons.