Multimodal RAG: connect an image claim to its source

A citation to a long PDF does not establish which diagram, label or region supports an image-derived claim. Multimodal RAG needs evidence coordinates that a reviewer can inspect.

Validate claim provenance

The adaptation lab checks two authored claims.

from adaptation_cases import image_source_case

result = image_source_case()
assert result["citable_indices"] == [0]
assert result["unsupported_indices"] == [1]
assert result["claims"][0]["page"] == 4
assert result["claims"][0]["region"] == [120, 80, 260, 150]
assert result["image_model_executed"] is False

Only the pressure claim has asset, region, page and source version. The inspection claim must abstain. No image model ran.

Preserve multimodal evidence

During ingestion, assign stable document, page and asset IDs; store coordinates, captions, nearby text, checksum, version and permissions. Keep tables and figures attached to their labels. Retrieval should return the region and its context rather than an untraceable embedding match.

Break answers into atomic claims and require one or more eligible evidence objects for each. Validate that cited regions belong to the user’s tenant and current document version. Render a cropped preview for human verification, but retain coordinates against the immutable source.

Evaluate retrieval of the correct asset separately from visual interpretation and answer support. Include visually similar figures, old revisions, missing captions, unreadable images and claims absent from all sources.

The Generative & Agentic AI course joins image provenance with RAG authorization and grounded-answer evaluation.

Exercise

Create a five-page local PDF with two similar diagrams. Index regions and version IDs, ask ten image questions and prove every supported claim resolves to the exact crop while unsupported claims abstain.

Continue learning

This article is part of the Model adaptation and multimodal tasks sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Evaluate OCR on difficult document layouts.
• Continue with Speech transcription evaluation beyond one accuracy number.

Reference: NIST AI RMF Generative AI Profile.