Grounded answers: distinguish unsupported and incorrect statements

Unsupported means the supplied eligible evidence does not establish the claim. Incorrect means the claim conflicts with the applicable truth or source. The distinction changes the repair: improve retrieval or abstention for unsupported content; correct interpretation, versioning or generation for contradicted content.

Apply a claim taxonomy

The retrieval evaluation lab labels three authored refund claims.

from retrieval_cases import grounded_case

result = grounded_case()
assert result["labels"]["supported_correct"] == 1
assert result["labels"]["unsupported"] == 1
assert result["labels"]["contradicted"] == 1
assert result["unsupported_distinct_from_incorrect"] is True

The current policy supports receipt or order ID. Passport acceptance is absent and labelled unsupported. “Only receipt” conflicts with the current rule and is contradicted. These are fixtures.

Label the evidence boundary

Break answers into atomic claims and attach source spans. Use categories such as supported-correct, supported-but-ineligible/stale, unsupported, contradicted and unverifiable judgment. Do not ask a grader to infer world truth when only source support can be assessed.

An unsupported claim might coincidentally be true elsewhere; it still fails a source-grounded assistant. A source-supported historical statement may be wrong for a current question. Keep support and applicability separate.

Route failures to the right stage

If relevant evidence was absent from top-*k*, diagnose retrieval. If evidence was present but unused, inspect prompt/context. If the answer added a new claim, strengthen abstention and claim-level validation. If sources conflict, apply version rules or escalate.

Report denominators by claim and answer. Calibrate human or model graders with hard negatives and disagreement review. A fluent answer-level score should not override a contradicted critical claim.

The Generative & Agentic AI course uses this taxonomy in retrieval traces and release reports.

Exercise

Create 30 answers with 80 atomic claims. Label support, correctness and source eligibility independently, then map each failure to ingestion, retrieval, generation or policy handling.

Continue learning

This article is part of the Retrieval quality and grounded answers sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Measure citation correctness separately from answer quality.
• Continue with Create hard-negative examples for a RAG benchmark.

Reference: NIST Generative AI Profile.