Evaluate recommendations with temporal holdouts

A random interaction split can train on a user's future and test on their past. Recommendation systems serve forward in time, so an offline split should reproduce what the system knew before each held-out interaction.

Verify ordering user by user

Our synthetic dataset has 60 users and five ordered positive interactions each. Times one through four create 240 training rows. Time five creates one test row per user, for 60 test rows.

The validator compares the maximum training time with the held-out time for every user. There are zero ordering violations. A five-component SVD recommender retrieves the held-out item within its top five for 88.33% of users.

from unsupervised_cases import temporal_case

r = temporal_case()
assert r['users'] == 60
assert r['train_rows'] == 240 and r['test_rows'] == 60
assert r['time_order_violations'] == 0
assert r['svd_hit_rate_at_5'] == 53/60
print(r)

Run the split and model in the unsupervised lab. The fixture provides interaction order rather than real timestamps, and its hit rate is a code reference only.

Reconstruct the serving state

A global cutoff trains on all interactions before a date and tests afterward. A leave-last-out design creates a user-specific cutoff. Both must build features, item eligibility and popularity from information available before recommendation time.

Remove or separately handle items unavailable at the cutoff. Avoid calculating user profiles, item frequencies or negative samples from the complete history. Deduplicate retries and define the event grain. If an interaction can be revised later, retain the as-of version used by the evaluation.

Offline held-out positives show that a user interacted under the historical exposure policy. They do not reveal whether unshown items were irrelevant, and they can reward reproduction of the old recommender. Report candidate-set construction, exposure availability and warm/cold eligibility.

Compare simple popularity and recent-item baselines. Add precision or recall at k, NDCG, coverage and segment results. Final policy value needs an online experiment or a defensible causal design with guardrails.

Exercise: compare a random interaction split with the temporal split. For each random test row, count later interactions that leaked into its training profile. Explain why a higher random-split score can be less informative for deployment.

NeuraPath's Data Science course connects recommender evaluation to point-in-time feature correctness. A temporal holdout starts with an auditable event order.

Continue learning

This article is part of the Clustering, reduction and recommendations sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Collaborative filtering and the cold-start problem.
• Continue with Ranking metrics: compare precision at k and NDCG.