Detect data leakage in holiday and promotion features

Calendar features look safe because dates are known in advance. The attached business fields may not be. A planned promotion flag can exist at the forecast origin, while realized spend, final discount, campaign success and corrected event labels arrive after demand occurs. A model trained on the final table can quietly learn from the outcome it claims to predict.

Leakage is therefore about availability at a historical decision time, not merely column names.

Reproduce an obvious leak

The time-series lab includes a deliberately blunt audit case. An available model uses time and the planned promotion flag. A leaky model also receives realized_demand, copied from the completed-period target.

from timeseries_cases import promotion_leakage_case

result = promotion_leakage_case()
available = result["models"]["available"]["test_rmse"]
leaky = result["models"]["leaky_realized_outcome"]["test_rmse"]
assert result["leaky_field"].startswith("realized_demand")
assert leaky < available
print(round(available, 2), leaky)

The nearly zero error is a symptom, not an achievement. Production cannot populate that field before the outcome. The example is intentionally direct so the contract can be tested; real leaks are often disguised as campaign_tier, final_status or an aggregate recomputed after the period.

Build a point-in-time feature audit

For every feature, store event time, availability time, ingestion time and revision time where possible. At each backtest origin, query the value that would then have been visible. Ask:

• Was the holiday date published before the origin?
• Was the promotion approved, or only proposed?
• Could its discount or budget change before execution?
• Does a category encode campaign performance assigned afterward?
• Was a rolling aggregate recomputed using the target period?

A date dimension built today may include renamed holidays or corrected closures unavailable in old operations. A versioned snapshot is safer than assuming static reference data.

Use tests that fail loudly

Shift target-derived fields forward and confirm model performance collapses. Recompute every rolling feature inside each training fold. Assert that a feature’s maximum source timestamp is earlier than the target timestamp, with a documented exception for genuinely known future schedules. Compare offline feature code with the production request path; training-serving mismatch can resemble leakage during development and failure after release.

An unexpectedly strong validation score deserves investigation, especially when it appears after a new business status field. Remove one feature family at a time and trace lineage. Do not rely on correlation alone: a leaked variable can be subtle, while a legitimate known calendar can be highly predictive.

The Data Science course combines feature engineering with point-in-time validation and reproducible model evidence.

Exercise

Create an availability ledger for holiday date, planned campaign, final discount, realized spend and post-campaign label. Implement an assertion that rejects any field whose availability timestamp exceeds its forecast origin, then rerun the backtest.

Continue learning

This article is part of the Forecasting and time-series analysis sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Reconcile forecasts across product and regional totals.
• Continue with Forecast evaluation during structural breaks.

Reference: scikit-learn’s common pitfalls on data leakage.