Build an error taxonomy for a trained model

An error taxonomy groups failures so that investigation can lead to specific, testable changes. Begin with observable facts and keep proposed explanations separate. A label such as “missing historical ticket count” describes a condition; “the imputer caused the failure” requires additional evidence.

Our synthetic inactivity model has eighty frozen March test predictions at threshold0.5. The confusion counts are33 false negatives, one false positive,24 true negatives and22 true positives. These are prediction instances, not eighty distinct customers; forty fictional customers each contribute two rows.

Preserve the prediction grain when adding context

Join predictions to the source using the unique snapshot identifier, and validate a one-to-one relationship. A customer-only join would match multiple snapshots and could duplicate errors.

The error audit retains snapshot ID, customer ID, target, probability, plan, ticket missingness, activity band and outcome category. These fictional identifiers are safe teaching fixtures; real reviews should expose only the data needed for the investigation.

import pandas as pd
from assessment_cases import error_rows

errors = error_rows()
assert len(errors)==80 and errors['snapshot_id'].nunique()==80
assert errors['outcome'].value_counts().to_dict()=={'FN':33,'TN':24,'TP':22,'FP':1}
table = pd.crosstab(errors['activity_band'],errors['outcome'])
assert table.loc['under_7_days','FN']==19
assert table.loc['7_or_more_days','FN']==14
assert table['FN'].sum()==33
assert table.loc['under_7_days','TP']==0
print(table)

Run from the evaluation lab. The source merge checks that every prediction finds exactly one snapshot, preserving the denominator before any grouping.

Use two layers of labels

The first layer is mutually exclusive prediction outcome: TP,FP,FN,TN. It must reconcile to all evaluated rows. The second layer records investigation conditions, such as missing input, activity band or plan. A row can belong to several such conditions, so their counts must not be summed as if they were a single partition.

In the current test set, the under-seven-days activity band contains nineteen positives and eighteen negatives. The threshold selects none of them, producing nineteen false negatives. The seven-or-more-days band contains the remaining fourteen false negatives and all22 true positives.

That pattern suggests reviewing score behavior and threshold suitability in the recently active group. It does not prove that a particular feature is causally responsible or that a lower threshold is operationally preferable.

Turn a pattern into an investigation

For each important category, record the count, affected decision, representative snapshot IDs, candidate explanation and evidence needed to test it. Examples of candidate explanations include a changed population, insufficient historical signal, label ambiguity or a threshold chosen for a different cost tradeoff.

Inspect correct predictions from comparable conditions too. Reviewing only errors can make a common feature look like a failure cause simply because it appears frequently throughout the dataset.

A real project should also audit source quality and label coverage. Our rows are conditional synthetic snapshots, not reconstructed event histories; the exercise cannot establish that real inactivity labels are complete or historically available.

Keep debugging separate from final assessment

Use earlier development evidence to design changes. If the test error review informs substantial model revisions, treat it as development knowledge and obtain fresh assessment evidence for the revised procedure.

Exercise: write an investigation ticket for the nineteen recently active false negatives. Include an observed fact, a hypothesis, a proposed development-set check and an acceptance condition. Reject wording that presents the hypothesis as an established cause.

NeuraPath's Data Science course connects model evaluation with structured debugging. A useful taxonomy produces a reproducible investigation while keeping the original error counts and uncertainty visible.

Continue learning

This article is part of the Machine learning workflow and evaluation sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Nested cross-validation: separate tuning from assessment.
• Continue with Evaluate a model by meaningful data slices.