Backpropagation with a small numerical gradient check

Backpropagation applies the chain rule from a loss through each operation to the model parameters. A compact numerical gradient check can catch a wrong transpose, missing activation derivative or incorrect normalization before the same bug is hidden inside a large training run.

For parameter w, central finite differences approximate its derivative as:

[L(w + epsilon) - L(w - epsilon)] / (2 * epsilon).

This approximation is slow, so use it on a tiny deterministic network and a few parameters. Compare it with the analytic backpropagation gradient at float64 precision.

Executed two-layer check

The local lab uses two inputs, two tanh hidden units, one sigmoid output and binary cross-entropy. It checks all four first-layer weights with epsilon = 1e-6.

from deep_learning_cases import gradient_check_case

result = gradient_check_case()
assert result["epsilon"] == 1e-6
assert result["relative_error"] < 1e-7
print(f'{result["relative_error"]:.3e}')

The relative error is approximately 2.586e-10. One analytic derivative is -0.1144728155; its numerical counterpart is -0.1144728156 after rounding.

The relative error divides the norm of the difference by the combined gradient norms. An absolute threshold alone behaves poorly when gradients are naturally very small or large.

Make the check stable

Disable dropout and other randomness. Use a tiny batch and deterministic forward pass. Avoid points where ReLU is exactly nondifferentiable. Check parameters from several layers, including biases. Run in double precision because float32 rounding can dominate small perturbations.

Epsilon also matters. Too large measures curvature rather than a local derivative; too small suffers cancellation. Sweep several powers of ten if the check fails. Inspect individual parameters instead of accepting one aggregate silently.

Know what passed

This check supports the first-layer derivative in one small implementation. It does not validate data loading, loss reduction across distributed workers, optimizer updates or an entire neural framework. Add shape assertions, a one-batch overfit test, loss-decrease check and integration test for the complete training path.

A gradient can be mathematically correct while the model solves the wrong target or leaks validation data. Numerical agreement is necessary evidence for custom calculus, not evidence of business validity.

The Data Science course uses small numerical checks to connect matrix calculus with observable implementation behavior.

Exercise

Check the hidden bias and output weights. Then intentionally remove the tanh derivative and confirm the failure. Plot relative error across epsilon from 1e-2 to 1e-10.

Continue learning

This article is part of the Deep learning and computer vision sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Build a neural network baseline before adding more layers.
• Continue with Activation functions: diagnose saturation and dead units.

Reference: scikit-learn’s neural-network mathematical formulation.