Monitor model input drift without treating every alert as failure

Input drift means the serving feature distribution differs from a reference. Performance can remain stable under some shifts and fail without a large marginal shift. Treat drift as an investigation signal, then confirm impact with labelled outcomes or reviewed samples.

Two strong synthetic shifts

The local deployment lab adds 40 units to monthly spend and changes the plan mix.

from deployment_cases import input_drift_case

result = input_drift_case()
assert result["alert"] is True
assert result["performance_label_available"] is False
print(result["spend_psi"], result["plan_share_change"])

Spend population-stability index is 1.75. Basic and plus plan shares fall by 30.2 and 31.2 percentage points, while pro rises by 61.4 points. The configured alert fires, but no performance labels are available. Calling the model failed would exceed the evidence.

Monitor data quality first

Check missingness, range, schema, units and freshness. A currency conversion bug can resemble business drift. Compare within stable segments so a product-mix shift does not obscure feature behavior.

Select metrics for feature type and keep bins or reference definitions versioned. PSI thresholds are conventions, not universal statistical truths. Report counts and trend over time, not only red or green.

Attach an action policy

An alert can trigger sample review, label acceleration, slice analysis or temporary human review. Retraining should require fresh representative labels and candidate evaluation. Automatic retraining on shifted unlabelled data can preserve or amplify a problem.

Track prediction and confidence distributions while awaiting labels. Once outcomes mature, measure the decision metric by cohort and model version.

The Data Science course links drift monitors to delayed-label and retraining policies.

Exercise

Create data-quality, benign mix-shift and performance-degrading scenarios. Set reference windows and actions, then measure alert precision after labels arrive.

Continue learning

This article is part of the Model deployment and MLOps sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Canary releases with a measurable rollback condition.
• Continue with Concept drift versus data drift: choose the right response.

Reference: Evidently data-drift methods documentation.