Define a rollback rule after a model change

Rollback decisions become political when thresholds are invented after a bad graph appears. Write the rule, window, data source and authority before exposing the new model route.

Evaluate multiple guardrails

The LLMOps and security lab compares authored candidate metrics with fixed minimums and maximums.

from llmops_cases import rollback_case

result = rollback_case()
assert result["breaches"] == ["task_success", "unsafe_rate"]
assert result["candidate"]["p95_ms"] < result["rules"]["max_p95_ms"]
assert result["decision"] == "rollback"
assert result["fixture_metrics"] is True

Faster latency does not compensate for lower task success and a non-zero unsafe rate. These values are fixtures, not production measurements.

Make the rule executable

Use critical invariants for unauthorized actions, sensitive disclosure and corrupted output. One verified critical failure can stop a rollout regardless of averages. Add statistical or count-based thresholds for task success, abstention, error rate, latency, cost and escalation. Specify minimum sample size, observation window, slices and treatment of missing telemetry.

Tie signals to one release manifest and compare a canary with a concurrent control where possible. Seasonality or traffic mix can move metrics without a model change. Do not infer causality from a before/after dashboard alone.

The rollback artifact must exist: previous prompt/model/tool/index versions, deployment procedure, credentials, data migrations and cache invalidation plan. Test it in staging. If an irreversible schema change makes rollback impossible, use forward recovery and feature isolation instead of pretending.

Define ownership and recovery

Name who can trigger rollback, who investigates and what evidence is preserved. Automate clear critical rules; allow reviewed judgement only where written. After rollback, verify health and replay the failed cases. Turn confirmed failures into regression tests before another attempt.

The Generative & Agentic AI course links rollback to canary telemetry, incident records and version bundles.

Reproducible exercise

Create a local rollout controller with three metrics and a minimum sample. Inject a safety failure, prove immediate reversal, then show the prior release manifest and cache key are restored.

Continue learning

This article is part of the LLMOps, security and operational evaluation sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Release an AI feature using shadow evaluation.
• Continue with Create a human-review sampling policy.

Reference: NIST AI RMF Generative AI Profile.