Classification metrics when accuracy hides failure

A classifier can be highly accurate while missing every case that motivated the project. When positives are rare, report the class counts and confusion matrix before interpreting an accuracy percentage.

Our original teaching fixture contains 1,000 observations, twenty positive and 980 negative. A rule that predicts every observation as negative is correct 980 times: 98% accuracy. It identifies none of the twenty positives, so its recall is zero.

This fixture uses authored labels and scores, not a trained model or real fraud, medical or customer performance. It is separate from the account-inactivity experiment, whose positive outcome is much more common.

Calculate the operationally relevant counts

A second authored score rule, classified at a threshold of 0.5, produces sixteen true positives, ten false positives, four false negatives and 970 true negatives. Accuracy is 98.6%, precision is 16/26, approximately 61.54%, and recall is 16/20=80%.

The accuracy improvement is only 0.6 percentage points, but the second rule finds sixteen positives that the all-negative rule misses. Whether its ten false alarms are acceptable depends on the action and cost.

Reproduce the comparison

import numpy as np
from metric_cases import rare_case

r = rare_case()
baseline = r['all_negative']
candidate = r['original']
assert baseline['accuracy']==.98 and baseline['recall']==0
assert baseline['precision'] is None
assert [candidate[k] for k in ['tn','fp','fn','tp']]==[970,10,4,16]
assert np.isclose(candidate['precision'],16/26)
assert candidate['recall']==.8 and candidate['accuracy']==.986
print({'all_negative':baseline,'candidate':candidate})

Run from the evaluation lab. Precision for the all-negative rule is undefined because there are no positive predictions. The report uses null rather than silently presenting an arbitrary replacement as an observed fraction.

The scikit-learn evaluation guide describes the classification metrics used here. Preserve the positive-class definition when calculating and reporting them.

Translate a confusion matrix into workload

If every positive prediction enters a review queue, the candidate produces 26 reviews in this sample. Sixteen contain actual positives under the fixture labels. This is a workload description, not evidence that review would prevent a loss or improve an outcome.

At larger scale, capacity can matter as much as a summary metric. Report the expected number of selected cases and the uncertainty in that expectation. A high recall rule that overwhelms reviewers may require a different operating threshold or workflow.

Avoid replacing one single-number habit with another

Precision alone can look excellent when a rule selects only a few easy cases. Recall alone can be perfect when every case is labelled positive. F1 combines precision and recall under a particular harmonic-mean weighting, but it does not include true negatives or automatically express the decision's costs.

Also inspect meaningful slices and probability quality where the scores are intended as probabilities. The authored scores here illustrate ranking and thresholds; they are not demonstrated to be calibrated.

Exercise: calculate the candidate's balanced accuracy and F1, then explain one operational consequence that neither number includes. Propose a reporting table that lets a stakeholder see workload, missed positives and false alarms without reconstructing them from a headline score.

NeuraPath's Data Science course connects classification metrics with decisions. A credible result makes the positive class, denominator, threshold and error counts visible.

Continue learning

This article is part of the Machine learning workflow and evaluation sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Choose MAE or RMSE from the decision cost.
• Continue with Confusion matrices at more than one threshold.