Extracting budget tables with an LLM, then proving the numbers

Source documents

PRS publishes an annual state budget analysis for many states. Its Delhi listing runs from 2016-17 to 2026-27, and Uttar Pradesh and Haryana are also covered. Each analysis page links to a PDF. The 2026-27 Delhi analysis has "Table 1: Budget 2026-27 - Key figures (in Rs crore)" with columns for the previous year's actuals, the current year's BE and RE and the next year's BE.

That table holds identities you can test. From the real 2026-27 Delhi table:

• Fiscal Balance = Net Receipts − Net Expenditure. For 2026-27 BE, 82,480 − 99,446 = −16,966, which matches the table.
• The same identity holds for 2024-25 actuals (62,281 − 56,557 = 5,724) and 2025-26 RE (72,379 − 94,668 = −22,289).
• Total Expenditure − Net Expenditure equals debt repayment in the 2026-27 BE: 1,03,700 − 99,446 = 4,254. The analysis text gives debt repayment as ₹4,254 crore.

Every year's table may not carry exactly these rows, so confirm which identities apply to each document before you rely on them.

Why this is employable work

Finance, consulting and research teams are adopting document extraction fast. What they lack is people who can say how often it is wrong. A harness that separates verified, contradicted and unverifiable values is the difference between a demo and something a compliance team will allow near a report. It also shows judgement about when not to use a model: if the PDF has a clean text layer, a parser may beat the model.

Pipeline

1. 1Fetch the PDF and record its URL, file name and download date.
2. 2Extract with the model, asking for JSON with a fixed schema: item, column, value_text exactly as printed, value, unit. Keeping value_text makes the text-layer check possible.
3. 3Parse Indian digit grouping deterministically in your own code. Do not let the model do arithmetic on "1,00,000".
4. 4Check identities within each column.
5. 5Match the text layer. Search the PDF's extracted text for each value_text. A value that appears nowhere in the document is suspect.
6. 6Label each value: verified by identity, found in text only, or unverified.

Code: the verification harness

The model response below is synthetic. It is a hand-written JSON string shaped like a model's output, with invented figures and one planted transposition error. No model was called, and the numbers are not from any budget.

import json
import re
import pandas as pd

model_json = """[
 {"item": "Net Receipts",     "column": "BE", "value_text": "71,240"},
 {"item": "Net Expenditure",  "column": "BE", "value_text": "84,910"},
 {"item": "Fiscal Balance",   "column": "BE", "value_text": "-13,670"},
 {"item": "Net Receipts",     "column": "RE", "value_text": "68,300"},
 {"item": "Net Expenditure",  "column": "RE", "value_text": "83,150"},
 {"item": "Fiscal Balance",   "column": "RE", "value_text": "-14,580"},
 {"item": "Total Expenditure","column": "RE", "value_text": "1,02,300"}
]"""
# Synthetic stand-in for text pulled from the PDF (the true RE fiscal balance is -14,850)
pdf_text = "Net Receipts 71,240 68,300 Net Expenditure 84,910 83,150 Fiscal Balance -13,670 -14,850 Total Expenditure 1,02,300"

def parse_inr(s: str) -> float:
    s = s.strip().replace("−", "-")
    neg = s.startswith("-") or (s.startswith("(") and s.endswith(")"))
    digits = re.sub(r"[^\d.]", "", s)
    return -float(digits) if neg else float(digits)

df = pd.DataFrame(json.loads(model_json))
df["value"] = df["value_text"].map(parse_inr)
df["in_pdf_text"] = df["value_text"].map(lambda v: v in pdf_text)

w = df.pivot(index="column", columns="item", values="value")
w["identity_gap"] = w["Net Receipts"] - w["Net Expenditure"] - w["Fiscal Balance"]
print(w[["Net Receipts", "Net Expenditure", "Fiscal Balance", "identity_gap"]].to_string())

ok_cols = set(w.index[w["identity_gap"].abs() < 1])
identity_items = {"Net Receipts", "Net Expenditure", "Fiscal Balance"}

def status(r):
    if r["item"] in identity_items:
        if r["column"] in ok_cols:
            return "verified: identity"
        return "contradicted: identity fails" if not r["in_pdf_text"] else "identity fails, value in text"
    return "found in text only" if r["in_pdf_text"] else "unverified"

df["status"] = df.apply(status, axis=1)
print(df[["item", "column", "value_text", "in_pdf_text", "status"]].to_string(index=False))
print(df["status"].value_counts().to_string())

The identity pinpoints the RE column, and the text-layer check pinpoints the value: "-14,580" appears nowhere in the text, while the other two RE values do. Together they identify the transposed digits without a human rereading the table. On real documents, report the share of values in each status, and hand-check a random sample of "verified" values too, since identities can pass when two errors cancel.

Analysis plan

• Pick three states and five years of analyses, about fifteen PDFs.
• Run extraction, then the harness, and record each value's status.
• Hand-label a random sample of values against the PDF to estimate the true error rate in each status group.
• Compare against a non-model baseline: a text-layer parser for the same tables. Report accuracy, time taken and failure types for both.
• Write up where the model helped (odd layouts, merged cells) and where it added risk.

Deliverables

• Status table: document, item, column, printed text, parsed value, status.
• Error summary: counts by status, and hand-checked error rates with sample sizes.
• Comparison chart: model against parser, accuracy and failure types.
• README: documents and fetch dates, model name and version, the exact prompt and schema, identities used per document, the privacy note that only public PDFs were sent, and known limits.

Interview questions

• Why store the printed text as well as the parsed number?
• Which errors can an accounting identity never catch?
• When would you choose a deterministic parser over a model?
• How would you estimate the error rate among values your harness marked verified?
• Your model's output changes between runs. How does your pipeline handle that?

The Delhi budget decade project builds the panel this pipeline would feed. Structured extraction and evaluation are part of our Generative & Agentic AI add-on, and the programmes page shows how it fits with the core tracks.