Catch denominator errors in AI-written business summaries

To verify a rate in an AI-written summary, inspect the numerator and denominator separately before checking the division. Both must refer to the intended population, grain and observation window. A mathematically correct percentage can still describe the wrong business measure.

Refund analysis makes this problem visible because a single order can have several refund transactions. Counting refunds and counting refunded orders answer different questions.

Name the measure precisely

In the original synthetic commerce fixture, eight January orders have completed status. Three of those orders appear in the supplied refund ledger: O1001, O1005 and O1007. O1005 has two refund records, giving four refund records in total.

For this exercise, define the measure as the share of eligible January completed orders with at least one refund in the supplied, undated ledger. The numerator is three distinct eligible orders, and the denominator is eight eligible orders. The result is 37.5%.

The phrase “undated ledger” matters. The fixture cannot tell us when those refunds occurred or whether every order has had an equal observation period. Do not label the result a January refund-transaction rate, a mature cohort rate or a future refund probability.

Reproduce the numerator as a set

import csv
from fractions import Fraction
from pathlib import Path
from calculator import calculate

eligible = set(calculate()['evidence_order_ids'])
with Path('../commerce-sql/refunds.csv').open(encoding='utf-8',newline='') as f:
    refunds = list(csv.DictReader(f))
matching = [row for row in refunds if row['order_id'] in eligible]
refunded_orders = {row['order_id'] for row in matching}
rate = Fraction(len(refunded_orders),len(eligible))
assert len(matching)==4
assert refunded_orders=={'O1001','O1005','O1007'}
assert len(eligible)==8 and rate==Fraction(3,8)
assert float(rate)*100==37.5
assert Fraction(len(matching),len(eligible))==Fraction(1,2)
assert sum(int(row['refund_paise']) for row in matching)==17500
print({'eligible_orders':8,'orders_with_ledger_refund':3,
       'refund_records':4,'order_share_percent':37.5,
       'refund_record_count_divided_by_orders':0.5})

The calculation of four divided by eight produces 0.5 refund records per eligible order. Describing it as “50% of orders were refunded” changes its meaning and is wrong for this dataset.

Reject plausible alternative denominators

The source orders table contains ten rows, but cancelled and pending orders are not in this measure's eligible population. Three divided by ten gives 30%, answering a different question.

Using only refunded orders as the denominator gives three divided by three, or 100%. That tautology tells us nothing about refund prevalence among completed orders.

A refund-value ratio is different again: 17,500 refund paise divided by 104,000 completed-order paise is approximately 16.83%. Its numerator and denominator are monetary amounts, not order counts. It cannot replace the 37.5% order share simply because both involve refunds.

Make the assistant show its working

Ask for the metric name, eligibility rule, numerator count, denominator count, unit, observation limitation and source identifiers. Then recompute the rate from those fields. Avoid accepting an isolated percentage with a citation attached.

For a real reporting system, verify that the cited rows actually satisfy the rule and that the source extraction is complete. A well-formed answer can still be based on a partial ledger. The small fixture establishes the calculation for its supplied rows only.

The answer-verification lab illustrates the broader principle of checking structured facts separately from prose. Its original amount checker does not automatically validate every new rate; a new metric needs its own explicit contract and reference calculation.

Use a sentence that preserves the limitation

A defensible summary is: “Three of eight January completed orders have at least one refund in the supplied undated ledger, a 37.5% share. Refund timing and cohort maturity cannot be assessed from these files.”

That wording gives a manager the measured fact and the missing information needed before comparing periods or setting a target.

Exercise: add another refund record for O1005 in a temporary copy. Predict which quantities change. The refund-record count and value should increase; the number of refunded orders should remain three.

NeuraPath's Data Analytics with Generative AI course connects SQL grain, business metrics and AI review. Denominator discipline turns a fluent percentage into a measure that can be reproduced and interpreted correctly.

Continue learning

This article is part of the Generative AI for verified analyst work sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Build a small evaluation set for an analyst assistant.
• Continue with Separate retrieval facts from calculated numbers.