Handle late-arriving events in a training dataset

An event can occur on Monday and arrive next week. Features based on event time should place it in Monday’s history, while the pipeline only learns about it at ingestion. A late-data policy defines how long outputs remain revisable and which partitions to recompute.

Measure lateness before choosing a window

The data-engineering lab calculates ingestion minus event time for 3,000 invented events.

from data_engineering_cases import late_event_case

result = late_event_case()
assert result["allowed_lateness_days"] == 3
assert result["late_rows"] == 150
print(result["late_share"], result["partitions_to_recompute"])

Five percent arrive more than three days late. They affect 72 event-date partitions. This intentionally broad impact shows why blindly reopening every touched daily partition can be expensive.

Choose policy from business cost

Options include ignoring after cutoff, correcting versioned historical outputs, maintaining a rolling recomputation window or issuing a backfill. Report the excluded share and affected features. A training snapshot should record its as-of ingestion cutoff so another analyst can reproduce what was known.

For labels, late arrival can create immature negatives. Separate event occurrence, source update and availability time. Do not let a corrected future table leak into an earlier backtest.

Bound the correction

Group affected dates, estimate rows and dependencies, recompute into versioned output, validate and publish atomically. Notify model owners if historical features or labels change materially. Preserve the previous version for rollback.

The Data Science course links late data to delayed labels, backfills and forecast cutoffs.

Exercise

Plot the lateness distribution and compare 1-, 3-, 7- and 14-day policies. Estimate missed events, recompute cost and model-impact risk before choosing a window.

Continue learning

This article is part of the Data engineering for data science sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Incremental ingestion with deduplication and watermarks.
• Continue with Build a point-in-time correct feature join.

Reference: Apache Beam programming guide on watermarks and late data.