Define a machine learning prediction target without future leakage

A prediction target needs a clock. Specify who is scored, when the prediction is made, which future interval defines the outcome and when that outcome becomes reliably observable. Without these details, a high model score can describe information that arrives after the decision it supposedly supports.

Our original teaching project predicts seven-day inactivity for existing accounts. It uses 320 synthetic snapshots across forty fictional customers. The rows are simulated snapshots, not reconstructed continuous event histories, so they demonstrate modelling mechanics rather than a production event pipeline.

Write a target contract before selecting features

The machine-readable contract fixes these choices. An inactive account might remain subscribed; a cancelled account might still have recorded activity. Calling this label “churn” would silently change its meaning.

Separate feature availability from outcome availability

Days since activity, historical ticket count, tenure and plan are the allowed predictors. future_active_days helps construct the label but cannot be a predictor. Customer and snapshot identifiers are also excluded from this model's feature list.

Scikit-learn's leakage guidance explains why unavailable information undermines evaluation. A pipeline helps isolate learned preprocessing, but it cannot discover that an apparently ordinary business field was populated after the event.

The synthetic metadata declares features available before each decision. A real implementation must demonstrate this through source timestamps, ingestion timing and as-of joins. A field's business event time alone does not prove that the prediction service could access it then.

Test a future-arriving feature

Run this from the evaluation lab:

import pandas as pd
from evaluation_core import load,validate,FEATURES,TARGET

data = load()
assert len(data)==320
assert TARGET not in FEATURES and 'future_active_days' not in FEATURES
assert ((data['future_active_days']==0).astype(int)==data[TARGET]).all()
changed = data.copy()
changed.loc[0,'features_available_at'] = changed.loc[0,'decision_at']+pd.Timedelta(seconds=1)
try:
    validate(changed)
except ValueError as error:
    assert str(error)=='future_feature'
else:
    raise AssertionError('future feature was accepted')
print({'snapshots':len(data),'target':TARGET,'future_feature_rejected':True})

The guard rejects even a one-second violation. This checks the declared timestamps; it is not independent proof that the metadata is truthful or that every upstream field obeys the same availability rule.

Do not turn incomplete observation into a negative label

A seven-day outcome cannot be finalized after only three days. The last test decision in the fixture is March 12 at midnight UTC. Its window ends March 19, and the extra reporting day makes its label available March 20.

In a real event dataset, zero recorded activity can also mean failed telemetry or missing identity resolution. Confirm coverage before assigning zero. Decide how late events, deleted records and revised labels affect the training snapshot and evaluation version.

Exercise: design the same contract for fourteen-day inactivity. Update both horizon and label availability, identify which rows would be immature at the existing evaluation cutoff, and explain why simply changing the target column name is insufficient.

NeuraPath's Data Science course connects model building with data-quality and evaluation decisions. A complete target definition should let another person reconstruct both the positive label and the information boundary around each prediction.

Continue learning

This article is part of the Machine learning workflow and evaluation sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Continue with Build a baseline before choosing a complex model.
• Then apply it in Train, validation and test sets: assign each a separate job.