Build a reusable data-quality report in Python

Return a structured quality report from reusable checks, keeping each check's population, severity and result explicit. A single quality score can hide a failed key constraint behind many passing cosmetic checks.

Separate row-level validity from source completeness and business approval. A perfectly typed dataframe can still omit half the day's orders.

Start with a narrow contract

The pandas quality lab includes quality.py with profile_orders(frame). It expects a typed order dataframe, nullable integer amounts in paise and parsed calendar dates.

It checks missing required keys, duplicate order-key rows, missing dates, negative amounts and unsupported statuses. It also reports Paid order count, observed/missing amount counts and the known monetary subtotal.

import json
from build_and_verify import load_orders
from quality import profile_orders

report = profile_orders(load_orders())
assert report['rows'] == 8 and report['paid_orders'] == 7
assert report['paid_observed_amounts'] == 6
assert report['paid_missing_amounts'] == 1
assert report['paid_known_subtotal_paise'] == 57000
assert all(value == 0 for value in report['hard_check_counts'].values())
assert report['row_quality_status'] == 'partial_missing_amounts'
assert json.loads(json.dumps(report)) == report
print(json.dumps(report, indent=2))

The result is intentionally partial even though all hard checks pass: one Paid amount remains unknown. JSON serialization is checked so another tool can consume the report without parsing formatted console tables.

Give each count a precise meaning

duplicate_order_key_rows counts all rows participating in duplicated keys. If one key appears twice, the count is two, not one extra row. That definition matters when a reviewer compares the result with a replay count.

Likewise, missing amounts count records, not currency. You cannot infer the monetary value of missing amounts from their frequency alone.

The pandas isna documentation describes missing-value detection, and the duplicated reference supports the duplicate-row check. The severity rules are authored for this fixture.

Test that failures change the report

import pandas as pd
from build_and_verify import load_orders
from quality import profile_orders

orders = load_orders()
duplicated = pd.concat([orders, orders.iloc[[0]]], ignore_index=True)
duplicate_report = profile_orders(duplicated)
assert duplicate_report['hard_check_counts']['duplicate_order_key_rows'] == 2
assert duplicate_report['row_quality_status'] == 'failed'
negative = orders.copy()
negative.loc[negative['order_id'].eq('P01'), 'amount_paise'] = -1
negative_report = profile_orders(negative)
assert negative_report['hard_check_counts']['negative_amounts'] == 1
assert negative_report['row_quality_status'] == 'failed'
assert orders.loc[orders['order_id'].eq('P01'), 'amount_paise'].iloc[0] == 10000
print('Quality checks detect injected defects without mutating the source frame')

These controlled mutations demonstrate that the checks respond to defects. A report that always prints accepted is not a useful validation system, even if its normal fixture looks correct.

Keep detection separate from repair

profile_orders does not fill amounts, delete duplicates or rewrite statuses. That separation lets a reviewer inspect the problem before choosing an authorized correction.

If a later cleaning step repairs data, run the same checks afterward and retain both reports with input/output identifiers. A lower error count should be explainable through documented transformations, not just a smaller dataset.

Add operational context without changing the checks

A run wrapper can attach a source hash, code revision, reporting period and execution timestamp. Keep secret values and unnecessary raw customer fields out of the summary. Detailed exception records can live in a separately controlled artifact linked by stable record keys.

The current helper does not validate customer-dimension completeness, date coverage or every possible business rule. Extend it with named checks when the reporting contract requires them, rather than describing it as universal data validation.

Exercise: add a reference-integrity check for customer IDs using the valid dimension. Report the unmatched P07 separately from C03's missing region, and define whether each condition blocks publication or remains a visible warning.

NeuraPath's Data Analytics with Generative AI course connects reusable Python with accountable data preparation. A useful quality report tells the next person exactly what passed, what failed and what was never checked.

Continue learning

This article is part of the Pandas wrangling and data checks sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Pandas merge_asof for time-aligned event data.
• Continue with Compare two dataframe snapshots with stable keys.