Design delayed-label monitoring for a deployed model

Many outcomes arrive days or months after prediction. Scoring immature records as negatives creates biased monitoring and can make a model look better or worse depending on the delay. Store prediction time, label-available time and an as-of date.

Reconcile maturity

The deployment lab creates 100 synthetic prediction IDs on 1 September with outcome delays of 7–30 days. At the authored as-of date 21 September:

from deployment_cases import delayed_label_case

result = delayed_label_case()
assert result["mature_labels"] + result["immature_labels"] == result["predictions"]
assert result["unique_prediction_ids"] == result["predictions"]
print(result["mature_labels"], result["immature_labels"])

Fifty-six outcomes are mature and 44 remain immature. Performance denominators include only the 56 under the declared policy.

Store point-in-time identifiers

Log prediction ID, entity ID where permitted, prediction timestamp, model version, score, threshold and action. Join to a versioned outcome with event and availability timestamps. Prevent duplicate or revised outcomes from silently changing earlier reports.

Report maturity rate and delay distribution beside performance. Slice delays by class and cohort; positive outcomes may take longer to appear, creating informative censoring. Use survival or windowed methods when a fixed maturity cutoff is inadequate.

Monitor before labels arrive

Schema, freshness, latency, errors, prediction mix and input drift remain available. They support technical monitoring but do not replace performance labels. Sample manual reviews where appropriate and label their selection bias.

The Data Science course links label joins to model versioning and retraining evidence.

Exercise

Build a point-in-time performance table with 30-, 60- and 90-day maturity. Test revised labels, missing joins and duplicated predictions, then define when a performance alert is eligible.

Continue learning

This article is part of the Model deployment and MLOps sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Concept drift versus data drift: choose the right response.
• Continue with Model retraining triggers based on evidence.

Reference: Google Cloud model monitoring concepts.