Design model fallback without silently lowering acceptance criteria

Fallback improves availability only if it preserves the task contract. Returning an uncited answer because the preferred model is unavailable converts an outage into a quieter correctness failure.

Apply one acceptance function

The LLMOps and security lab evaluates three authored routes. The available model fallback fails the citation requirement, so the workflow selects a constrained static response.

from llmops_cases import fallback_case

result = fallback_case()
assert result["candidates"][0]["available"] is False
assert result["candidates"][1]["accepted"] is False
assert result["selected"] == "safe_static"
assert result["criteria_changed"] is False
assert result["fixture_outputs"] is True

No model was called. The result demonstrates policy ordering, not relative model quality.

Specify the fallback matrix

List triggers such as timeout, capacity, regional outage and provider error. For each task class, record eligible models, supported features, data region, context limit, tool compatibility and maximum attempts. Keep schema validation, source support, abstention, authorization and prohibited-content checks outside the model route.

If a fallback cannot meet the contract, narrow the function explicitly. It can return verified static guidance, create a pending job, transfer to a human or state that the task is unavailable. Make that state visible to users and telemetry. Do not disguise a reduced-capability answer as normal success.

Test prompt and tool compatibility because model changes can alter structured output, tool selection and context handling. Pin versions and evaluate the full route on the same cases. Prevent fallback loops with a total deadline and attempt budget. For write-capable workflows, reuse idempotency keys so retries cannot duplicate effects.

Observe the route separately

Track fallback activation, success, critical failures, latency and cost by task slice. Alert on sustained fallback even if requests appear successful; degraded operation can hide a primary-service incident.

The Generative & Agentic AI course links routing to eval gates, tool contracts and incident response.

Reproducible exercise

Implement three local adapters with different capabilities. Force the first to time out and the second to omit a citation. Prove the shared validator rejects both and the workflow returns the documented constrained response.

Continue learning

This article is part of the LLMOps, security and operational evaluation sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Define latency budgets for retrieval, generation and tools.
• Continue with Semantic caching: test incorrect cross-user reuse.

Reference: NIST AI RMF Generative AI Profile.