Handle missing values without inventing customer behaviour

Choose a missing-value treatment from the field's meaning and the analysis question. A missing purchase amount is not a zero purchase. An unmatched customer is not necessarily a new customer. A blank region does not reveal where someone lives.

Preserve the original missingness and distinguish its causes before filling, dropping or estimating values.

Separate three different problems

The pandas quality lab includes P08 with an unknown amount, P04 with a known zero amount and a matched customer whose region is missing, and P07 with an unmatched customer ID.

from build_and_verify import paid_enriched

paid = paid_enriched().set_index('order_id')
assert paid.loc['P04', 'amount_paise'] == 0
assert paid.loc[['P08'], 'amount_paise'].isna().all()
assert paid.loc['P04', '_merge'] == 'both'
assert paid.loc['P07', '_merge'] == 'left_only'
assert paid.loc[['P04', 'P07'], 'region'].isna().all()
assert paid['amount_paise'].count() == 6 and len(paid) == 7
print('Zero, unknown measure and missing dimension context are distinguishable')

The missing region column alone cannot distinguish P04 from P07. The merge indicator retains evidence about whether a customer record existed.

The pandas missing-data guide explains missing-value representations and detection. Use isna rather than equality comparisons to identify missing values across supported dtypes.

Show how zero filling changes interpretation

from build_and_verify import paid_enriched

paid = paid_enriched()
observed_mean = paid['amount_paise'].mean()
zero_filled_mean = paid['amount_paise'].fillna(0).mean()
assert observed_mean == 9500
assert abs(float(zero_filled_mean) - 57000 / 7) < 1e-9
assert paid['amount_paise'].isna().sum() == 1
print('Observed mean:', observed_mean, 'Zero-filled mean:', zero_filled_mean)

The observed mean uses six known amounts. Filling P08 with zero creates a seventh amount and lowers the mean to approximately 8,142.86 paise. No new evidence about P08 was obtained; the result changed because an assumption was inserted.

If zero filling is an approved modeling choice, preserve an imputation flag and explain it. Do not present the filled value as an observed customer action.

Keep monetary coverage visible

The seven Paid orders have 57,000 paise of known value and one unknown amount. That is not enough to state the complete paid total. Report the known subtotal, observed count, missing count and any available source-completeness information together.

sum(min_count=1) helps prevent an entirely missing group from appearing as zero, but a partially observed group still produces a subtotal. Coverage must remain explicit.

For an operational report, it may be better to hold publication until the missing amount is resolved. For exploratory work, showing a clearly labeled partial result may be acceptable. The decision depends on the report's intended use.

Avoid inferring behavior from absence

A missing order in an incomplete extract does not establish that the customer did not purchase. A missing survey response does not establish dissatisfaction. A missing customer attribute may reflect an optional form field rather than a meaningful segment.

Compare missingness across source, period and segment before choosing an estimation strategy. If one channel systematically lacks an attribute, complete-case analysis can change the represented population.

The fixture is too small to estimate a real missingness mechanism. It demonstrates how to retain the evidence needed for a larger investigation.

Make repairs reversible

Keep raw and cleaned columns or a transformation log with stable record keys. When filling a display label such as Unknown region, preserve the original region and the reason category separately. A convenient chart label should not erase the difference between an unmatched customer and a matched blank attribute.

Exercise: produce a quality table with counts for missing amount, unmatched customer and matched customer with missing region. Allow a record to appear in more than one relevant quality category, and explain why those counts should not automatically be summed into a unique-row total.

NeuraPath's Data Analytics with Generative AI course connects pandas cleaning with honest interpretation. A useful analysis distinguishes what was observed, what is unknown and what was assumed.

Continue learning

This article is part of the Pandas wrangling and data checks sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Pandas duplicated: choose the business key before the method.
• Continue with Reshape survey data with melt and pivot.