Measure citation correctness separately from answer quality

A statement can be factually true while its citation does not support it. A cited source can also support a statement that is obsolete for the requested date. Evaluate claim correctness, source entailment and source eligibility separately.

Label three combinations

The retrieval evaluation lab uses three authored statements.

from retrieval_cases import citation_case

result = citation_case()
assert result["correct_statements"] == 2
assert result["supported_citations"] == 2
assert result["both"] == 1
print(result["statements"])

One false historical rule is supported by its cited old document; one true travel statement cites the wrong clause. Only one statement is both correct and supported. These are labelled fixtures, not generated answers.

Evaluate at claim level

Split an answer into atomic claims. For each, label whether the cited span entails it, whether the source is authorized/current, and whether the claim is correct under the task. Check citation completeness too: material claims without a source should not disappear from the denominator.

Use exact source version and span, not a homepage or document family. Numerical/table claims need the correct row, unit and condition. When multiple sources are required, label the complete set.

Report distinct failure rates

Track unsupported citation, wrong source, stale source, missing citation, contradicted claim and overbroad interpretation. A single “groundedness” score can hide these operationally different repairs.

Calibrate automated citation graders against human labels and preserve disagreements. Deterministic source-ID and quote checks can complement semantic judgment but do not prove entailment alone.

The Generative & Agentic AI course requires citation support checks beside retrieval and answer metrics.

Exercise

Label 50 claims from 20 answers for correctness, support, eligibility and completeness. Calculate each rate and create one regression example for every failure class.

Continue learning

This article is part of the Retrieval quality and grounded answers sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Evaluate retrieval recall at k with labelled evidence.
• Continue with Grounded answers: distinguish unsupported and incorrect statements.

Reference: RAGAS metrics documentation.