Evaluate image models beyond aggregate accuracy

Aggregate accuracy assigns the same weight to every image and error. It can hide a failed minority class, capture device or low-quality slice. A complete image-model report begins with a reconciled confusion matrix and adds class and deployment-relevant slices.

Verify the denominator

The local lab evaluates 60 noisy stripe images.

from deep_learning_cases import image_evaluation_case

result = image_evaluation_case()
matrix = result["confusion_matrix"]
assert sum(map(sum, matrix)) == result["test_rows"]
print(matrix, result["class_recall"])

The matrix is [[29, 0], [0, 31]]: all 29 vertical and 31 horizontal images are correct. Accuracy and balanced accuracy are both 100%. Splitting images at median pixel standard deviation also gives 100% accuracy in each half.

These perfect values show that the authored two-pattern fixture remains easy even with added noise. They do not demonstrate deployment readiness. A strong report says the evaluated population was not difficult enough to expose a failure.

Add slices with a reason

Choose slices connected to acquisition or action: camera type, site, lighting, object size, occlusion, demographic group where appropriate, or time since deployment. Define them before test inspection. Report row count and uncertainty; a 100% result on two examples is weak evidence.

For multi-class work, include precision and recall per class, macro averages, and the confusion pairs that dominate operational cost. If confidence scores drive review, evaluate calibration, selective coverage and errors at the chosen threshold. For imbalanced data, accuracy can remain high while the important class has zero recall.

Review individual failures

Create an error gallery with stable IDs, labels, predictions, confidence and relevant metadata. Check annotation ambiguity and data corruption before changing the network. Compare failures with the baseline; a new model may fix one slice while regressing another.

Avoid declaring fairness or safety from a few slices. Group definitions, sample adequacy and consequences require domain review. Preserve raw counts so rates can be audited.

The Data Science course treats aggregate metrics as the start of error analysis, not its conclusion.

Exercise

Add occlusion and contrast slices until at least one exposes a failure. Report confusion matrices and class recall with denominators. Set an acceptance rule for critical slices before training a replacement.

Continue learning

This article is part of the Deep learning and computer vision sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Detect near-duplicate images across train and test sets.
• Continue with Handle class imbalance in an image dataset.

Reference: scikit-learn classification metrics.