Use AI to propose data checks and verify them independently

Use AI-generated data checks as proposals. Validate each rule against the business contract, a known-good fixture and deliberately broken examples before adopting it. A check that rejects valid business records can damage a report just as surely as a check that misses corrupt data.

Independent verification means the expected behavior comes from the data contract and reviewed examples, not merely from the same assistant agreeing with its own code.

Classify the proposed rule

For the original synthetic orders fixture, unique order IDs and nonnegative integer amounts are valid requirements. Completed-order eligibility is determined by status and period. A matched customer record is not required for the current amount measure.

An assistant might suggest dropping every order with an unmatched customer. That could be a useful diagnostic for a customer-dimension process, but it is the wrong exclusion rule for this report. O1009 remains eligible and contributes 9,000 paise.

Distinguish a blocking contract violation, a warning requiring investigation and an observation that should not alter eligibility. Do not make every unusual value a reason to discard data.

Test both rejection and acceptance

The source calculator validates the fixture before calculating its amount. The following example creates temporary copies and checks behavior without modifying the original dataset.

import csv
from copy import deepcopy
from pathlib import Path
from tempfile import TemporaryDirectory
from calculator import calculate,SOURCE,FIELDS

with SOURCE.open(encoding='utf-8',newline='') as handle:
    original = list(csv.DictReader(handle))

def write_rows(path,rows):
    with path.open('w',encoding='utf-8',newline='') as handle:
        writer = csv.DictWriter(handle,fieldnames=FIELDS)
        writer.writeheader()
        writer.writerows(rows)

with TemporaryDirectory() as folder:
    path = Path(folder)/'orders.csv'
    write_rows(path,original)
    good = calculate(path)
    assert good['value']==104000 and 'O1009' in good['evidence_order_ids']
    duplicate = deepcopy(original)+[deepcopy(original[0])]
    negative = deepcopy(original)
    negative[0]['order_total_paise']='-1'
    for rows,wanted in [(duplicate,'duplicate_or_missing_order_id'),
                        (negative,'invalid_amount')]:
        write_rows(path,rows)
        try:
            calculate(path)
        except ValueError as error:
            assert str(error)==wanted
        else:
            raise AssertionError('Broken fixture was accepted')
print({'valid_fixture_accepted':True,'mutations_rejected':2,
       'legitimate_unmatched_order_retained':True})

The positive case matters. A validator that rejects every file would pass rejection-only tests while being useless. The unmatched-order assertion protects a legitimate business exception from an overbroad cleaning rule.

Challenge checks that merely repeat the implementation

“The total equals the sum returned by the total function” is not an independent expected result when both sides use the same logic. Preserve a small hand-audited reference with known eligible IDs and amounts.

Likewise, counting the output rows after an inner join cannot establish that the join retained every required source row. Compare the source eligibility set with the output set. This catches omissions that a plausible aggregate may conceal.

Boundary cases are especially useful: an order exactly at the period start, one exactly at the exclusive end, a repeated amount belonging to a different order and an absent optional field. Each should have an expected behavior grounded in the metric contract.

Ask the assistant for counterexamples

A productive prompt is: “For each proposed check, state the business rule, one valid record it must accept, one invalid record it must reject and whether failure blocks the report or creates a warning.”

Then inspect those examples yourself. The assistant may invent requirements not present in the source definition, such as treating every missing discount as zero without authorization or requiring every customer join to succeed.

The code in this article tests two specific mutations. It does not establish comprehensive data quality, statistical anomaly detection or correctness of every source timestamp. Extend the test set when a new rule or failure mode is introduced.

Keep a reason for every exclusion

If records are rejected, report the count, reason and effect on the measure. Do not silently remove failures and present the remaining total as complete. The appropriate response may be to stop publication until the source issue is resolved.

Exercise: propose a check for duplicate order amounts. Explain why duplicate values are valid when order IDs differ, then create a test that prevents a mistaken SUM(DISTINCT amount) repair.

NeuraPath's Data Analytics with Generative AI course connects AI suggestions with reproducible data-quality work. The analyst remains responsible for deciding which rules the business actually requires.

Continue learning

This article is part of the Generative AI for verified analyst work sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Document model uncertainty in a management narrative.
• Continue with Build a business glossary retrieval assistant.