Build a neural network baseline before adding more layers

More layers increase a neural network’s capacity, tuning surface and operating cost. They do not guarantee better future performance. A useful experiment starts with a linear reference, adds one small hidden layer, then accepts a deeper candidate only when validation evidence justifies the change.

One split, three capacities

The inspectable neural lab creates 600 noisy two-moon observations and fixes stratified partitions of 360 training, 120 validation and 120 test rows. Scaling is fitted on training only. Three models see the same data:

The one-layer network clears the linear reference. The deeper candidate adds 608 parameters and loses 1.7 percentage points on validation. It should not be selected merely because “deep” sounds more capable.

from deep_learning_cases import neural_baseline_case

result = neural_baseline_case()
one = result["models"]["one_hidden_layer"]
deep = result["models"]["two_hidden_layers"]
assert result["train_rows"] + result["validation_rows"] + result["test_rows"] == 600
assert one["validation_accuracy"] > deep["validation_accuracy"]
print(one["parameters"], deep["parameters"])

The code prints 33 641. These deterministic fixture results demonstrate the decision process; they are not estimates for another dataset or learner project.

Define the baseline ladder

Start with a trivial class-frequency predictor, then a regularized linear model, then a small neural network. Keep preprocessing, split, primary metric and threshold constant. This ladder shows whether nonlinearity adds value and whether another hidden layer adds value beyond that.

Use validation to choose architecture, regularization and learning settings. The table records test accuracy for audit, but it must not reverse the validation decision. Repeatedly choosing from test results consumes the holdout.

Accuracy is suitable only when its error trade-off fits the decision. For class imbalance or asymmetric costs, predeclare precision, recall, log loss, calibration or a cost function. Report slices and uncertainty when conclusions are close.

Complexity has operating costs

Count trainable parameters, fit time, artifact size and inference latency. More capacity can require more data and makes failure analysis harder. A model that wins by a negligible amount may still lose the deployment decision if it is unstable or expensive.

The Data Science course connects architecture experiments to evaluation, reproducibility and deployment evidence.

Exercise

Repeat the comparison over five declared seeds and report the validation distribution. Keep the test closed. State an improvement margin that a deeper model must exceed before the experiment begins.

Continue learning

This article is part of the Deep learning and computer vision sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Continue with Backpropagation with a small numerical gradient check.
• Then apply it in Activation functions: diagnose saturation and dead units.

Reference: scikit-learn supervised neural networks.