Compare offline metrics with online decision outcomes

An offline model metric measures prediction under historical labels. An online outcome measures what happens when a product or operations process acts on predictions. A model can rank risk well while an intervention has no effect, reaches the wrong people or creates another cost.

Prediction and intervention in one fixture

The deployment lab reports held-out AUC 0.785 for its synthetic risk model. It separately creates 150 randomized assignments to an intervention that has no authored treatment effect.

from deployment_cases import offline_online_case

result = offline_online_case()
assert result["online_randomized_rows"] == 150
print(result["offline_test_auc"], result["absolute_rate_difference"])

Observed outcomes are 23.2% in treated and 16.0% in control, a +7.1 percentage-point raw difference despite no simulated effect. Random variation in a small sample can look like lift. The lab does not claim statistical significance or business improvement.

Define the decision chain

Write how a score becomes eligibility, treatment, exposure and outcome. Log each step. Offline evaluation should match the population and threshold considered for launch. Online analysis should use the experiment unit, assignment and exposure definitions fixed in advance.

AUC does not identify a threshold or action value. Add capacity, expected cost, calibration and slice constraints. If treatment effects vary, a risk model is not automatically an uplift model.

Protect experimentation

Randomize when appropriate, predefine primary outcome and guardrails, estimate uncertainty and account for sample size. Check allocation and exposure. Avoid stopping when a favorable point estimate appears.

Monitor latency, errors, complaints and workload alongside the main outcome. A statistically positive outcome can still fail operational or fairness gates.

The Data Science course links offline modeling to an explicit online decision and evidence plan.

Exercise

Design an experiment for one score-triggered action. Specify randomization unit, eligibility, sample-size assumptions, primary outcome, guardrails and the offline threshold report.

Continue learning

This article is part of the Model deployment and MLOps sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Build a lightweight experiment tracking convention.
• Continue with Shadow deployment before switching model traffic.

Reference: Microsoft experimentation platform guidance.