Profile null patterns across customer segments

Report missing-value counts and rates by relevant segment, retaining the segment size and any unknown-segment group. A global missing rate can hide concentration in one source or customer population.

A pattern is evidence to investigate, not proof of why values are missing. Small segments especially require cautious interpretation.

Build a segment-level coverage table

The pandas quality lab contains seven Paid orders. North has three orders, one with a missing amount; West has two observed amounts; two other orders have missing region context.

from build_and_verify import paid_enriched

paid = paid_enriched()
paid['amount_missing'] = paid['amount_paise'].isna()
profile = paid.groupby('region', dropna=False, observed=True).agg(
    order_rows=('order_id', 'size'), missing_amounts=('amount_missing', 'sum'))
profile['missing_rate'] = profile['missing_amounts'] / profile['order_rows']
assert profile.loc['North', 'order_rows'] == 3
assert profile.loc['North', 'missing_amounts'] == 1
assert abs(profile.loc['North', 'missing_rate'] - 1/3) < 1e-12
assert profile.loc['West', 'missing_rate'] == 0
assert profile.loc[profile.index.isna(), 'order_rows'].iloc[0] == 2
assert int(profile['order_rows'].sum()) == 7
assert int(profile['missing_amounts'].sum()) == 1
print(profile)

The overall missing-amount rate is 1/7, while North's is 1/3. Averaging segment rates without weighting by their row counts would not recover the overall rate.

The pandas grouping guide documents named aggregations. Here, the denominator is Paid order rows, not unique customers or all orders.

Keep missing segment values visible

Dropping null regions would exclude two Paid orders from the profile. Their amount coverage is still relevant even though their geographic classification is unavailable.

The merge indicator further distinguishes C03, a matched customer with no region, from C99, a customer absent from the dimension. Those causes may require different owners or repairs.

from build_and_verify import paid_enriched

paid = paid_enriched()
amount_missing = paid['amount_paise'].isna()
unmatched_customer = paid['_merge'].eq('left_only')
matched_missing_region = paid['_merge'].eq('both') & paid['region'].isna()
assert paid.loc[amount_missing, 'order_id'].tolist() == ['P08']
assert paid.loc[unmatched_customer, 'order_id'].tolist() == ['P07']
assert paid.loc[matched_missing_region, 'order_id'].tolist() == ['P04']
any_issue = amount_missing | unmatched_customer | matched_missing_region
assert int(any_issue.sum()) == 3
print({'missing_amount': int(amount_missing.sum()),
       'unmatched_customer': int(unmatched_customer.sum()),
       'matched_missing_region': int(matched_missing_region.sum()),
       'unique_orders_with_any_issue': int(any_issue.sum())})

These three conditions happen to affect different records in the fixture. In another dataset, one order could satisfy several conditions, so summing issue counts would overcount unique affected orders.

Investigate source and time as well as segment

Compare missingness by import batch, collection channel, schema version and date. A sudden increase after a source migration may explain more than the customer segment itself.

Do not infer a causal customer characteristic from a collection-system defect. If one region uses an older form, region-level missingness may reflect that system rather than customer behavior.

The fixture is intentionally tiny and does not support a statistical claim about real regional differences. Its purpose is to make denominator and grouping choices reproducible.

Connect the profile to analytical bias

If complete-case analysis removes North's missing-amount order, the retained population has a different segment composition. Whether that changes a conclusion depends on the question and missingness mechanism, which the rate table alone cannot establish.

Keep counts before and after exclusions, and explain any imputation or sensitivity analysis separately. A low global null rate is not automatic permission to discard affected records.

Exercise: make P07's amount missing in a copied dataframe. Recalculate individual issue counts and the union count. Show why the sum of issue counts increases while the number of unique affected orders can remain three.

NeuraPath's Data Analytics with Generative AI course connects pandas profiling with representative analysis. A useful null report shows who is represented, what is missing and which explanations still need investigation.

Continue learning

This article is part of the Pandas wrangling and data checks sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Create a timezone-aware hourly activity report.
• Continue with Validate a pandas pipeline against a tiny golden dataset.