Dropout and weight decay: different regularization mechanisms

Dropout and weight decay can both reduce overfitting, but they operate differently. Dropout randomly masks activations during training and uses the complete network at evaluation. L2 weight decay penalizes large weights through the training objective or optimizer update. They are hyperparameters to validate, not automatic improvements to stack together.

Isolate each mechanism

The local lab expands a two-feature classification problem with 18 noise features and fits a small linear teaching model. It compares no regularization, L2 coefficient 0.05, and 30% input dropout.

Weight decay reduces the norm as intended, yet both regularized candidates have worse validation loss. Mechanism success is not model-selection success.

from deep_learning_cases import regularization_case

result = regularization_case()
plain = result["models"]["none"]
decay = result["models"]["weight_decay"]
assert decay["weight_norm"] < plain["weight_norm"]
assert result["dropout_train_eval_max_difference"] > 0
print(round(result["dropout_train_eval_max_difference"], 3))

The maximum prediction difference between an evaluation pass and one masked training-style pass is 0.380 on eight examples. This confirms that dropout changes training behavior and must be disabled or scaled correctly at inference.

Compare under equal budgets

Use the same split, initialization distribution, optimizer budget and primary metric. Search regularization strength on validation only. Repeat seeds because dropout introduces randomness. Report training and validation curves: a candidate that harms both may be underfitting, while a widening gap suggests overfitting.

Weight decay is not identical to every implementation of “L2 regularization,” especially with adaptive optimizers. Record the optimizer and whether decay is coupled to the loss gradient. Exclude biases or normalization parameters only through an explicit parameter-group policy.

Test the inference path

A validation loader must put dropout layers in evaluation mode. Test deterministic predictions by running the same input twice. If Monte Carlo dropout is intentionally used for uncertainty, name that separate inference procedure and evaluate its calibration and cost.

The Data Science course uses adverse results like these to teach controlled comparison rather than ritual hyperparameter use.

Exercise

Create a grid over dropout {0, .1, .3, .5} and decay {0, .001, .01, .05}. Predeclare the seed count and choose on validation log loss. Inspect whether the combination adds value beyond either mechanism alone.

Continue learning

This article is part of the Deep learning and computer vision sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Batch size: compare throughput and validation behaviour.
• Continue with Early stopping with a reproducible validation policy.

References: scikit-learn L2 regularization notes and PyTorch Dropout API.