Validate a pandas pipeline against a tiny golden dataset

A golden dataset is a small input with independently specified expected results. Use it to test the complete transformation at its business grain, including unusual records that expose plausible mistakes.

The expected table should encode the contract, not be regenerated automatically by the same pipeline under test. Otherwise, an incorrect change can update both sides and remain invisible.

Choose cases that challenge the pipeline

The pandas quality lab includes an identical replay, an unmatched customer, a matched customer with a missing region, a zero amount, a missing amount and a Pending order.

The expected Paid regional report has three rows. North contains three orders, two observed amounts and a 15,000-paise known subtotal. West has two orders and 35,000 paise. Missing region has two orders and 7,000 paise. One North amount remains unknown.

Assert the complete result

import pandas as pd
from build_and_verify import load_orders, load_customers

def regional_report(orders):
    if not orders['order_id'].is_unique:
        raise ValueError('order_key_not_unique')
    paid = orders.loc[orders['status'].eq('Paid')]
    joined = paid.merge(load_customers(), on='customer_id', how='left', validate='many_to_one')
    joined['region_label'] = joined['region'].fillna('Missing region')
    group = joined.groupby('region_label', observed=True)
    result = group.agg(order_rows=('order_id', 'size'), observed_amounts=('amount_paise', 'count'))
    result['known_subtotal_paise'] = group['amount_paise'].sum(min_count=1)
    result['missing_amounts'] = result['order_rows'] - result['observed_amounts']
    return result.sort_index()

expected = pd.DataFrame({
    'region_label': ['Missing region', 'North', 'West'],
    'order_rows': [2, 3, 2], 'observed_amounts': [2, 2, 2],
    'known_subtotal_paise': [7000, 15000, 35000], 'missing_amounts': [0, 1, 0],
}).set_index('region_label')
orders = load_orders()
actual = regional_report(orders)
pd.testing.assert_frame_equal(actual, expected, check_dtype=False, check_index_type=False)
assert all(pd.api.types.is_integer_dtype(dtype) for dtype in actual.dtypes)
pd.testing.assert_frame_equal(regional_report(orders.sample(frac=1, random_state=7)), actual)
try:
    regional_report(pd.concat([orders, orders.iloc[[0]]], ignore_index=True))
except ValueError as error:
    assert str(error) == 'order_key_not_unique'
else:
    raise AssertionError('Duplicate order key accepted')
print(actual)

The expected values are written explicitly from the fixture's business controls. The first comparison allows pandas' nullable versus ordinary integer storage and string-index representation to differ; separate assertions still require integer-valued result columns. The shuffled-input comparison is exact because it compares the same pipeline contract and runtime.

The assert_frame_equal reference documents comparison controls. Relax an option only when the contract permits that difference, not merely to make a failure disappear.

Test intermediate assumptions too

The loader has separate checks for identical and conflicting replays. The merge validates many-to-one cardinality. The final golden table verifies status filtering, retained missing regions, subtotals and amount coverage together.

An end-to-end failure tells you the contract changed; focused intermediate checks help locate why. Neither layer replaces the other.

Use invariants beyond fixed expected values

Input row order should not change this report. Adding an exact raw replay should not change accepted business results under the loader's policy. Changing a Pending amount should not alter a Paid-only subtotal.

These invariants catch errors that one static table might miss. They should follow the business definition rather than arbitrary implementation details.

Keep the fixture small enough to inspect

A golden dataset is not a performance benchmark and cannot represent every production edge case. Its advantage is that a reviewer can explain every row and expected result without trusting the pipeline itself.

When a production defect is discovered, add the smallest representative synthetic case that would have caught it. Preserve the documented reason for the new case.

Exercise: change the merge to inner in a disposable copy and predict the failing expected row. Then drop missing regions during grouping and explain how the same 7,000-paise loss arises through a different defect.

NeuraPath's Data Analytics with Generative AI course connects pandas pipelines with evaluated projects. A reviewable project includes the tiny evidence set that demonstrates what its report is supposed to mean.

Continue learning

This article is part of the Pandas wrangling and data checks sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Profile null patterns across customer segments.
• Continue with Export analysis results with a data dictionary and manifest.