Inspect a model failure without overstating saliency maps

A saliency map highlights pixels whose small local changes affect a model output. It can help inspect a failure, but it does not prove which pixels caused the real-world outcome, what the model “understands,” or whether the model would behave reliably after a larger intervention.

Pair sensitivity with perturbation

The image lab finds a misclassified noisy stripe image and estimates the absolute derivative of its predicted probability with respect to each of 64 pixels using central finite differences.

from deep_learning_cases import saliency_case

result = saliency_case()
assert result["was_misclassified"] is True
assert result["maximum_absolute_saliency"] >= 0
print(result["predicted_probability"], result["top_five_deleted_probability"])

The true label is vertical (0), but the predicted horizontal probability is 0.5049. The largest absolute local sensitivity is 0.0169. Setting the five most sensitive pixels to zero moves the probability to 0.4858, crossing the threshold.

This change supports local sensitivity for this input. Deleting pixels also creates an image that may be outside the training distribution, so it remains a diagnostic rather than a causal proof.

Start with the ordinary evidence

Confirm the source, label, preprocessing and model version. Compare the failure with nearest training examples and the baseline. Check confidence calibration and the class confusion. A saliency map should not replace these steps.

Run sanity checks: randomize model parameters, randomize labels in a controlled experiment, and see whether the visualization changes. Compare more than one explanation method where stakes warrant it. Evaluate explanation stability under small label-preserving transforms.

Use precise language

Say “pixels with high local output sensitivity” rather than “the pixels the model used” unless the method supports that stronger statement. A gradient at one point can be zero because of saturation, not because a feature is globally irrelevant. Correlated pixels and nonlinear interactions further limit individual attribution.

If explanations affect a decision, test that decision with user studies or task metrics. Attractive overlays are not evidence that reviewers become more accurate.

The Data Science course places explanation tools inside failure analysis with falsification and scope notes.

Exercise

Compare finite-difference saliency before and after parameter randomization. Delete and blur top-ranked pixels, then perturb random pixels as a control. Report output changes and whether transformed images remain plausible.

Continue learning

This article is part of the Deep learning and computer vision sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Handle class imbalance in an image dataset.
• Continue with Sequence models: distinguish padding from real observations.

Reference: PyTorch Captum attribution library documentation.