Detect outliers without automatically deleting valid sales

Use an outlier rule to identify records for investigation, not to declare them incorrect. A large sale can be a legitimate bulk order, a duplicated transaction, a unit mismatch or a data-entry error. The unusual value alone does not determine which explanation applies.

Keep the original record, attach a flag and inspect supporting evidence before changing the reporting population.

Build a case where the largest sale is valid

This synthetic example uses ten orders in INR. Nine amounts range from 100 to 180; the final order contains twenty units at INR 100 each. The amount is large but reconciles to its line calculation.

import pandas as pd

orders = pd.DataFrame({
    'order_id': [f'O{i:02}' for i in range(1, 11)],
    'quantity': [1] * 9 + [20],
    'unit_price_inr': list(range(100, 190, 10)) + [100],
})
orders['amount_inr'] = orders['quantity'] * orders['unit_price_inr']
q1, q3 = orders['amount_inr'].quantile([.25, .75], interpolation='linear')
iqr = q3 - q1
lower, upper = q1 - 1.5 * iqr, q3 + 1.5 * iqr
orders['iqr_flag'] = ~orders['amount_inr'].between(lower, upper)
assert (q1, q3, upper) == (122.5, 167.5, 235.0)
assert orders.loc[orders['iqr_flag'], 'order_id'].tolist() == ['O10']
assert int(orders['amount_inr'].sum()) == 3260
assert int(orders.loc[~orders['iqr_flag'], 'amount_inr'].sum()) == 1260
assert orders.loc[orders['order_id'].eq('O10'), 'amount_inr'].iloc[0] == 2000
print(orders.to_string(index=False))

The Series.quantile reference documents the interpolation choice. Stating that choice makes this small-sample threshold reproducible.

The code is also runnable from the pandas quality lab, whose execution record identifies the tested pandas version.

Quantify the consequence of deletion

Removing the flagged order reduces the total from INR 3,260 to INR 1,260. That is not merely smoothing a chart; it deletes INR 2,000 of valid synthetic business activity.

The ordinary mean is INR 326 and the median is INR 145. Those describe different aspects of this skewed population. Reporting the median alongside the total can be more informative than deleting the large order to make the mean look typical.

This fixture is designed to demonstrate a review decision. It does not establish an appropriate threshold for every sales dataset.

Investigate the record's provenance

Check quantity, unit price, currency, discounts, invoice identity, customer type and source version. A value entered in rupees where the field expects paise can look like a distributional anomaly but requires a unit correction, not statistical trimming.

Look for duplicate business keys and verify whether a large order spans multiple lines. A legitimate enterprise purchase may belong to a different segment from small consumer orders. That segmentation should follow business meaning rather than being invented solely to remove inconvenient observations.

Separate detection from treatment

Possible outcomes include accepting the value, correcting a documented source error, excluding an invalid record with a reason, or using a robust statistic for a specific analytical question. Preserve the original and the treatment decision.

Winsorization caps values rather than deleting rows, but it still changes the measure. It may support a modeling experiment under a declared protocol; it should not silently replace an actual revenue total.

If a threshold is used in predictive modeling, estimate it from the appropriate training data and evaluate its effect without leaking future information. A threshold chosen after seeing the desired outcome can bias the result.

Report sensitivity honestly

Show the primary result under the approved population and, when useful, a clearly labeled sensitivity result. Include the number and value of affected records. Do not select whichever treatment produces the most persuasive chart.

Exercise: create a second version where O10's quantity is 2 but its amount remains 2,000. Flag the arithmetic inconsistency separately from the IQR flag and explain what evidence is needed before correcting the source.

NeuraPath's Data Analytics with Generative AI course connects pandas exploration with analytical judgment. An outlier flag starts an investigation; it does not authorize rewriting the business history.

Continue learning

This article is part of the Pandas wrangling and data checks sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Calculate customer cohorts in pandas and reconcile with SQL.
• Continue with Pandas merge_asof for time-aligned event data.