Deep learning interview questions with short derivations

Gradients and backpropagation

1. What is backpropagation?

It is the chain rule applied efficiently. The forward pass computes each layer's output and stores intermediate values. The backward pass starts from the loss and multiplies local derivatives layer by layer, reusing stored values, so gradients for every parameter cost roughly one extra pass instead of one pass per parameter.

2. Derive the gradient of binary cross-entropy with a sigmoid output.

Let z be the logit, p = σ(z), and loss L = −[y log p + (1 − y) log(1 − p)].

• dL/dp = −y/p + (1 − y)/(1 − p)
• dp/dz = p(1 − p)
• Multiply: dL/dz = −y(1 − p) + (1 − y)p = p − y

The result, prediction minus label, is why this pairing trains well: the gradient does not shrink when the sigmoid saturates, as long as the prediction is wrong.

3. How would you check a gradient you derived?

Compare it with a finite difference: (L(z + h) − L(z − h)) / 2h for small h.

import math

def sigmoid(z):
    return 1 / (1 + math.exp(-z))

def bce(z, y):
    p = sigmoid(z)
    return -(y * math.log(p) + (1 - y) * math.log(1 - p))

h = 1e-5
for z, y in [(2.0, 1), (2.0, 0), (-0.5, 1)]:
    analytic = sigmoid(z) - y
    numeric = (bce(z + h, y) - bce(z - h, y)) / (2 * h)
    print(f"z={z:>4}, y={y}: analytic {analytic:+.6f}, numeric {numeric:+.6f}")

The two columns agree to six decimals. Mentioning gradient checking when debugging a custom layer is a good signal in an interview.

Why deep networks are hard to train

4. What causes vanishing gradients?

Backpropagation multiplies one factor per layer. The sigmoid's derivative is at most 0.25, so through many sigmoid layers the product shrinks roughly geometrically, and early layers barely learn. If factors are consistently above 1, gradients explode instead.

max_sigmoid_grad = 0.25
for layers in (2, 5, 10, 20):
    print(f"{layers:>2} layers: gradient scale at most {max_sigmoid_grad ** layers:.2e}")

At ten layers the upper bound is below one in a million, before weights are even considered.

5. How do ReLU, initialisation and residual connections help?

ReLU has derivative 1 for positive inputs, so it does not shrink gradients in the active region. Initialisation schemes such as He (for ReLU) or Xavier/Glorot (for tanh) scale initial weights by the layer's fan-in so activations and gradients keep roughly constant variance across layers. Residual connections add the input of a block to its output, giving gradients a direct path backwards.

Follow-up to expect: "What is a dead ReLU?" A unit whose input is negative for every example outputs zero and receives zero gradient, so it stops learning. Leaky ReLU or a lower learning rate helps.

6. Why not initialise all weights to zero?

Every unit in a layer would compute the same output and receive the same gradient, so they stay identical forever. Random initialisation breaks that symmetry.

Regularisation and normalisation

7. What does dropout do, and what changes at inference?

During training, each unit is zeroed with probability p, which stops units relying on specific partners and acts like averaging many thinner networks. With inverted dropout, surviving activations are scaled by 1/(1 − p) during training, so at inference dropout is simply switched off with no rescaling.

8. Batch normalisation versus layer normalisation.

Batch norm normalises each feature using statistics across the batch, and keeps running averages for inference; it works well for convolutional networks with reasonable batch sizes. Layer norm normalises across features within each example, so it does not depend on batch size and behaves the same in training and inference, which is why transformers use it.

Optimisation

9. SGD with momentum versus Adam.

Momentum keeps a running average of gradients, smoothing noisy updates and speeding progress along consistent directions. Adam adds a per-parameter scale from the running average of squared gradients, so parameters with small gradients take relatively larger steps. Adam usually needs less tuning to start; well-tuned SGD with momentum sometimes generalises as well or better on vision tasks. Say you would try both if training time allows.

10. Your training loss is falling but validation loss started rising after epoch 8. What do you do?

That is overfitting after epoch 8. Use early stopping at the best validation epoch, then add regularisation (weight decay, dropout, data augmentation) or get more data. Before that, check the validation set is drawn from the same distribution as training and is not leaking into training.

Probe: "And if training loss is not falling at all?" Check the learning rate first (too high diverges, too low stalls), then that labels and inputs are aligned, then try to overfit a tiny batch — if the model cannot memorise ten examples, there is a bug.

Preparing

Write out question 2 from memory, then do the same for softmax with categorical cross-entropy, where the gradient with respect to the logits is again prediction minus one-hot label. Run the gradient check for it. Two derivations you can reproduce calmly are worth more than a list of architectures.

The NLP interview bank applies these ideas to transformers, and the ML fundamentals bank covers the classical questions that usually come first. Neural network training and debugging are part of the deep learning module in the Data Science programme.