Choose a data split that matches the deployment setting

Choose a split by completing this sentence: the model will predict for which entities, at what future point, using what information? A random split, a customer-group split and a chronological split answer different evaluation questions.

In our original synthetic project, forty existing accounts receive repeated predictions. The intended assessment is later inactivity risk for those existing accounts. January trains the model, February selects regularization and March supplies the test period.

Match the question to the boundary

These are starting points, not automatic guarantees. Dependence, duplicated observations and changing measurement processes can require additional restrictions.

Audit the actual project split

Run in the evaluation lab:

from sklearn.model_selection import GroupKFold
from evaluation_core import load,TARGET

d = load()
train = d[d['split']=='train']
test = d[d['split']=='test']
assert train['decision_at'].max() < test['decision_at'].min()
shared = set(train['customer_id']) & set(test['customer_id'])
assert len(shared)==40
fit,held = next(GroupKFold(n_splits=5).split(train,train[TARGET],train['customer_id']))
group_shared = set(train.iloc[fit]['customer_id']) & set(train.iloc[held]['customer_id'])
assert not group_shared
assert set(train.iloc[fit]['decision_at']) == set(train.iloc[held]['decision_at'])
print({'chronological_test_shared_customers':len(shared),
       'group_fold_shared_customers':len(group_shared),
       'group_fold_is_a_future_period':False})

The chronological test shares all forty customer identities with training. That is intentional for this existing-customer question. The group fold holds out identities but includes the same January decision dates on both sides. It therefore does not evaluate a future period.

The GroupKFold reference documents group separation. It does not promise a chronological split simply because groups happen to be customer identifiers.

Do not classify every overlap the same way

Customer overlap can be appropriate when future predictions genuinely concern known customers and all historical features are available at scoring time. Duplicate copies of the same prediction instance crossing the boundary are a different problem. Check the unit of prediction, not just the presence of an identifier in both sets.

Conversely, removing customer IDs from the feature columns does not remove all customer dependence. Stable behavioral patterns and repeated measurements can still connect rows from the same entity. If the claim concerns unseen customers, split the entities themselves.

The lab rows are conditional synthetic snapshots rather than a reconstructed continuous event log. They demonstrate these split mechanics, while a real dataset needs additional checks of time-consistent features, source coverage and cross-period identity rules.

State what remains outside the evidence

The March test does not establish performance for a new market, a new product plan or a changed intervention policy. A model's reported assessment should name the population and period it actually covers.

Exercise: redesign the split for new customers first scored in March. Specify which customers and dates may enter training, how labels must mature before fitting, and how you will avoid selecting the split after comparing candidate scores. Report the row and positive-label counts remaining in each set.

NeuraPath's Data Science course connects validation design with deployment context. The strongest split explanation states both the question it answers and the populations it has not assessed.

Continue learning

This article is part of the Machine learning workflow and evaluation sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Train, validation and test sets: assign each a separate job.
• Continue with Cross-validation with repeated customers or patients.