Evaluate retrieval on numerical tables

Table questions require the right row, column, unit and condition. A response can quote the correct number from the wrong band and still look plausible.

Check fields exactly

The retrieval evaluation lab uses a two-row authored table.

from retrieval_cases import table_eval_case

result = table_eval_case()
assert result["cases"] == 2
assert result["expected"] == [12000, 18000]
assert result["predicted"] == [12000, 18000]
assert result["exact_field_accuracy"] == 1.0
assert result["unit"] == "INR"

The exact-match result is a deterministic fixture, not evidence from PDF extraction or a model.

Label the complete evidence unit

Each question should identify table, row key, column, value, unit, applicable date and supporting page. Include range boundaries such as exactly 500, just above 500, missing values and footnote exceptions.

Evaluate two stages: did retrieval return the correct table/row, and did extraction/answering return the exact normalized field? Keep string formatting separate from numeric equality. Reject unit conversions unless the rule and rate source are explicit.

Add hard negatives

Use nearby rows with similar values, same number in a different unit, old table versions and prose summaries that omit exceptions. Measure whether multiple retrieved rows confuse the answer.

For high-impact calculations, recompute totals outside the model and cite inputs. A semantic judge should not replace exact arithmetic or range validation.

The Generative & Agentic AI course connects table-preserving ingestion to field-level retrieval and answer checks.

Exercise

Label 30 table questions covering exact values, ranges, units and footnotes. Report row retrieval and field accuracy separately, then inject three wrong-row answers that share the expected number.

Continue learning

This article is part of the Retrieval quality and grounded answers sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Calibrate an LLM judge against human-labelled cases.
• Continue with Diagnose a RAG failure from its intermediate evidence.

Reference: pypdf text extraction.