Handle outliers before deciding to cap or remove them

An extreme value can be a data error, a valid rare case, a different population or the observation that matters most. Capping and removal are modelling decisions that should follow investigation.

Put two different extremes behind the same threshold

Our synthetic regression follows approximately y=2*x. We add two training extremes. The first is (8,16), legitimate and exactly on the rule. The second is (9,-20), an authored corrupted response.

Keeping all 350 training rows pulls the fitted coefficient to 1.2936 and gives test RMSE 0.8602. Removing every row with |x|>5 removes both extremes, estimates coefficient 2.0125 and achieves RMSE 0.2974. Capping both x values at five keeps the corrupted target and changes the legitimate relationship; coefficient is 1.6957 and RMSE 0.4774.

from feature_cases import outlier_case

r = outlier_case()
assert r['audited_extremes'][0]['type'] == 'legitimate_on_rule'
assert r['audited_extremes'][1]['type'] == 'corrupted_target'
assert r['strategies']['keep']['train_rows'] == 350
assert r['strategies']['remove_abs_x_gt_5']['train_rows'] == 348
print(r)

Run it in the feature-engineering lab. The lower removal RMSE in this seed does not justify a universal threshold; it happens to discard a harmful corruption and a valid informative extreme together.

Investigate source and consequence

Trace extreme rows to source records, units, joins and timestamps. Compare related fields. A thousand-fold amount jump may be paise-versus-rupee error; a real enterprise customer can also be legitimately large.

If values are impossible, correct from evidence or reject with lineage. If legitimate but influential, consider robust models, transformations, separate populations or domain caps. Fit any cap on training data and log how many rows it changes at serving.

Evaluate overall and extreme-subgroup performance. A model that improves average error by failing every high-value customer may be unacceptable. Keep raw values for audit even when the model receives a transformed copy.

Exercise: add a validation split containing legitimate extremes. Compare robust regression, winsorization and separate-segment models. Report coefficient, average error and extreme-subgroup error with the number of altered rows.

NeuraPath's Data Science course teaches outlier handling as data investigation plus model validation. Distance from the median is not a deletion reason.

Continue learning

This article is part of the Feature engineering and data quality sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Data validation for model inputs with realistic edge cases.
• Continue with High-cardinality categories: compare hashing and encoding.