Evaluate citations in an AI-generated analytical answer

Evaluate an analytical citation by asking whether the source exists, whether the version applies and whether it supports the specific claim. A valid link beside a sentence does not establish that every number or conclusion in the sentence came from that source.

For calculated business results, also inspect the computation evidence. A metric-definition document can explain the rule while containing none of the transaction values used in the answer.

Separate four checks

Source existence asks whether the cited document can be found. Version validity asks whether it was approved and effective for the requested context. Metric alignment asks whether it discusses the measure being reported. Claim support asks whether its content actually justifies the sentence.

The first three are partly amenable to structured checks. Arbitrary prose support requires more than a matching document ID. A sentence may cite the correct definition and still invent a causal explanation.

The original glossary lab deliberately makes this boundary visible: its citation function reports metadata validity and always retains a semantic-review requirement.

Execute the metadata checks

from glossary import check_citation
from calculator import calculate

metric = 'completed_order_amount'
good = check_citation('completed-orders-v1',metric)
stale = check_citation('completed-orders-v0',metric)
unknown = check_citation('invented-document',metric)
wrong_metric = check_citation('refund-order-share-v1',metric)
assert good['metadata_checks_passed'] is True
assert good['semantic_support_review_required'] is True
assert good['proves_calculated_number'] is False
assert all(not result['metadata_checks_passed']
           for result in (stale,unknown,wrong_metric))
evidence = calculate()
assert evidence['value']==sum(row['amount_paise'] for row in evidence['source_rows'])
assert evidence['value']==104000
print({'current_definition_metadata_passes':True,
       'rejected_citation_cases':3,'calculated_amount_paise':104000,
       'definition_alone_proves_amount':False})

The stale definition has both an effective-period problem and a different historical metric ID. The invented document fails existence. The refund definition exists and is current, but it is the wrong source for the completed-order amount definition.

Review claims at sentence level

Consider three statements. “The current definition includes unmatched customer orders” is supported by the current glossary. “January's eligible amount is 104,000 paise” is supported by the source-row calculation. “The campaign caused the amount to increase” is supported by neither.

Combining all three into one sentence and attaching the glossary citation does not repair the unsupported parts. Split the claims and attach evidence appropriate to each.

For a calculated number, preserve the source version, contract, period, unit and eligible identifiers. For a quotation or definition, preserve the exact document version and relevant passage. A hash can establish which bytes were used; it cannot establish that those bytes are true or sufficient.

Avoid rewarding citation quantity

Ten citations do not necessarily provide more support than one relevant source. Measure whether important claims are supported and whether the cited material actually entails them. An answer can have perfect citation formatting and poor evidence coverage.

For a small evaluation set, label each material claim as supported, contradicted or not established by the supplied evidence. Keep reviewer notes for disagreements. A numeric claim with the wrong unit should not pass merely because the underlying amount appears somewhere in the source.

This lab does not implement a general natural-language entailment grader. Its automated checks are intentionally narrow, and the articles do not report a fabricated citation-accuracy score for any model.

Make unsupported answers easy to repair

When a citation fails, identify the repair: retrieve the current definition, use the correct metric source, rerun the calculation or remove an unsupported conclusion. If the required evidence is unavailable, state the limitation instead of attaching a loosely related document.

Exercise: write a sentence containing one supported definition and one unsupported causal claim. Mark the boundary between them and explain why a single valid citation should not cause both claims to pass review.

NeuraPath's Data Analytics with Generative AI course connects retrieval, calculation and analytical communication. Citation review is useful when it helps a reader trace a claim to evidence that actually supports it.

Continue learning

This article is part of the Generative AI for verified analyst work sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Build a business glossary retrieval assistant.
• Continue with Know when a spreadsheet formula is better than an AI agent.