Separate offline evaluation from production success metrics

Offline tests answer whether a version handles controlled cases. Production metrics answer what happens in a live workflow with real traffic, integrations and human behaviour. Combining them into one score destroys that distinction.

Keep two measurement families

The LLMOps and security lab stores authored offline and production-like fields separately and joins them with release dimensions.

from llmops_cases import offline_online_case

result = offline_online_case()
assert result["metrics_kept_distinct"] is True
assert result["join_dimensions"]["release_id"] == "rel-9"
assert result["causal_claim"] is False
assert result["fixture_metrics"] is True
assert "task_success" in result["offline"]
assert "completion_rate" in result["production"]

No production system was observed. The values only demonstrate a measurement schema.

Use offline evidence for controlled change

Representative fixed cases support repeatable comparisons of task acceptance, citation support, tool correctness, abstention and adversarial behaviour. Slice results by intent, language, risk and input condition. Protect sealed holdouts from prompt iteration and pin the complete system bundle.

Production measures include completion, escalation, repeat contact, correction, abandonment, user-reported problems, latency, cost and incidents. Define the denominator and window. An escalation can be a safe success for an unsupported high-risk request, so interpret metrics in the task context.

Join both families by release, prompt/model/tool/index/policy versions and eligible task slice. Monitor distribution shift between the evaluation sampling frame and current traffic. Privacy-minimize production telemetry and keep raw content opt-in and restricted.

Test causal questions deliberately

A quality score and completion rate moving together does not show that one caused the other. Traffic mix, UI changes, outages and policy changes are confounders. Use randomized experiments where appropriate and safe, or staged comparisons with concurrent controls and explicit limitations. Safety invariants remain gates rather than experiment trade-offs.

Offline regressions can block a release; production outcomes can trigger rollback and supply new evaluation cases. Neither replaces the other.

The Generative & Agentic AI course closes the loop from evaluation design through observability, release and incident learning.

Reproducible exercise

Create separate offline-results and production-metrics tables, join by release manifest, simulate a traffic-mix shift and show how an unsegmented before/after comparison gives a misleading conclusion.

Continue learning

This article is part of the LLMOps, security and operational evaluation sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the preceding task in Build a release gate that blocks a known regression.
• Return to the cluster foundation in Build an evaluation dataset that represents real users.

References: OpenAI Evals API, OpenTelemetry GenAI conventions and NIST AI RMF Generative AI Profile.