Choose a learning rate using training evidence

The learning rate controls the size of parameter updates. Too small can leave a model under-trained within the available budget. Too large can oscillate, diverge or fit the training data in a way that generalizes poorly. Choose it from a declared experiment, not from the final test.

Hold the rest constant

The local lab trains the same eight-unit tanh network for 80 epochs with batch size 32, momentum 0.9 and a shared initialization seed.

The largest rate has the lowest final training loss but almost twice the validation loss of 0.03. Training fit alone would choose the wrong candidate for this fixture.

from deep_learning_cases import learning_rate_case

result = learning_rate_case()
best = min(result["candidates"], key=lambda key: result["candidates"][key]["validation_log_loss"])
assert result["selection"] == "minimum validation_log_loss"
assert best == "0.03"
print(best)

One run per rate is a diagnostic. Before making a close decision, repeat several seeds and report variability. The fixture’s result does not prescribe 0.03 for another model.

Read the curve, not only the endpoint

Plot loss against optimizer steps as well as epochs because batch size changes the number of updates per epoch. Look for divergence, plateaus and sudden instability. Record gradient norms if exploding or vanishing updates are suspected.

A learning-rate range test can narrow candidates, and schedules can decay the rate after warmup or plateaus. The schedule becomes part of the model specification and must be validated with the same budget. Do not give one candidate more epochs until it wins.

Preserve the test boundary

Learning rate, scheduler, optimizer, batch size and early stopping patience all consume validation information. Once they are fixed, refit under the declared policy and evaluate the locked test once. Save the curve, seed, package versions and selected checkpoint.

The Data Science course treats optimizer settings as evaluated model decisions rather than hidden notebook defaults.

Exercise

Test logarithmically spaced rates between 1e-4 and 1. Predeclare a divergence rule, compare validation distributions over five seeds, and select the simplest schedule whose improvement exceeds your chosen margin.

Continue learning

This article is part of the Deep learning and computer vision sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Activation functions: diagnose saturation and dead units.
• Continue with Batch size: compare throughput and validation behaviour.

Reference: scikit-learn MLP optimization notes.