Design an abstention response for missing evidence

“I don’t know” prevents fabrication but may leave the user without a next step. A useful abstention identifies the unsupported decision, states why available evidence is insufficient and routes the user to an allowed source or owner.

Make abstention a valid output

The offline foundations lab creates an empty-evidence case.

from foundation_cases import abstention_case

result = abstention_case()
assert result["evidence_count"] == 0
assert result["response"]["status"] == "abstain"
assert result["unsupported_answer_emitted"] is False
assert result["passes"] is True

The response says no approved evidence supports the requested exception and directs the user to the policy owner or current exception record. It emits no answer. This is an authored contract test, not an LLM run.

Define abstention triggers

Triggers can include no eligible source, evidence below a retrieval threshold, contradictory approved versions, missing mandatory fields, failed authorization or an out-of-scope request. Do not use a model’s self-reported confidence as the sole trigger; validate evidence and business rules outside the generation step.

The structured response can contain status, reason code, human-readable explanation, missing evidence, safe next step and trace ID. Avoid leaking the existence of sources the user cannot access.

Evaluate helpfulness and safety separately

Create answerable and unanswerable cases. Measure unsupported-answer rate, correct-answer rate and appropriate-abstention rate. Review over-abstention too: a system that refuses every question is safe from hallucination but fails its purpose.

Test conflicts, stale documents, near matches and adversarial requests that instruct the assistant to guess. Preserve the retrieved source IDs and rule results for review. When a human supplies missing evidence, rerun through the same authorization and validation path rather than appending untrusted text directly.

The Generative & Agentic AI course requires unsupported and contradictory cases in the evaluation set for grounded assistants.

Exercise

Write five reason codes and 20 labelled cases. Set a release gate for unsupported answers and over-abstention. Have a second reviewer judge whether the next steps are accurate and permitted.

Continue learning

This article is part of the LLM fundamentals and prompt design sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Handle an LLM refusal in an application workflow.
• Continue with Compare models using task quality, latency and cost.

Reference: NIST Generative AI Profile.