ML fundamentals interview questions with worked answers

Learning setup

1. Supervised versus unsupervised learning, with an example of each.

Supervised learning uses labelled examples to predict a target — for instance, predicting whether a loan defaults from past loans with known outcomes. Unsupervised learning finds structure without a target — for instance, grouping customers by purchase behaviour with clustering.

2. Why split data into training, validation and test sets?

Training data fits the parameters. Validation data chooses among models and hyperparameters. The test set is used once at the end to estimate performance on unseen data. If you tune on the test set, its score is no longer an honest estimate.

3. What is k-fold cross-validation and when would you not use plain k-fold?

Split training data into k parts, train on k−1 and validate on the remaining one, rotating k times and averaging. Do not use plain random folds for time series (use forward-chaining splits) or when rows from the same customer can land in both train and validation (use grouped folds).

Bias, variance and regularisation

4. Explain the bias-variance trade-off.

Bias is error from a model too simple to capture the pattern; variance is error from a model so flexible it fits noise. Increasing flexibility lowers bias and raises variance. The demo below shows it with nearest neighbours, where small k is flexible and large k is smooth.

import random

random.seed(3)
def make(n):
    pts = []
    for _ in range(n):
        x, y = random.uniform(0, 10), random.uniform(0, 10)
        label = int(y > x + random.gauss(0, 1.5))   # noisy diagonal boundary
        pts.append((x, y, label))
    return pts

train, test = make(200), make(400)

def knn_predict(data, x, y, k):
    nearest = sorted(data, key=lambda p: (p[0] - x) ** 2 + (p[1] - y) ** 2)[:k]
    return int(sum(p[2] for p in nearest) * 2 > k)

def accuracy(data, evaluate_on, k):
    return sum(knn_predict(data, x, y, k) == lab for x, y, lab in evaluate_on) / len(evaluate_on)

for k in (1, 15, 151):
    print(f"k={k:>3}: train acc {accuracy(train, train, k):.2f}, test acc {accuracy(train, test, k):.2f}")

With k = 1, training accuracy is perfect because each point is its own nearest neighbour, but test accuracy is lower: that gap is variance. With k = 151, the model averages over most of the data, and both scores fall: that is bias. A middle value does best on the test set.

5. How do you recognise overfitting?

Training performance much better than validation performance, and validation performance that gets worse as you add complexity or train longer. Fixes: more data, simpler model, regularisation, early stopping, or better features.

6. L1 versus L2 regularisation.

Both add a penalty on coefficient size to the loss. L2 (ridge) penalises squared coefficients and shrinks them smoothly towards zero. L1 (lasso) penalises absolute values and can set some coefficients exactly to zero, which acts as feature selection. With highly correlated features, L1 tends to pick one arbitrarily; elastic net mixes both.

Algorithms

7. What does logistic regression output, and is it a regression?

It models the log-odds of the positive class as a linear function of features, then applies the sigmoid to produce a probability between 0 and 1. It is used for classification; "regression" refers to fitting the linear log-odds. A threshold on the probability turns it into a class decision.

8. How does a decision tree choose a split? Compute one.

It tries candidate splits and picks the one that most reduces impurity, commonly Gini impurity: 1 − Σ pᵢ², where pᵢ is the share of class i in a node.

def gini(counts):
    n = sum(counts)
    return 1 - sum((c / n) ** 2 for c in counts)

parent = [50, 50]                 # 50 defaults, 50 non-defaults
left, right = [40, 10], [10, 40]  # split on "missed a payment last year"
n = sum(parent)
weighted = sum(sum(s) / n * gini(s) for s in (left, right))
print(f"parent Gini {gini(parent):.2f}, after split {weighted:.2f}, reduction {gini(parent) - weighted:.2f}")

Parent Gini is 0.50. Each child has Gini 1 − (0.8² + 0.2²) = 0.32, so the weighted Gini after the split is 0.32 and the reduction is 0.18. The tree compares that reduction with every other candidate split.

9. Random forest versus gradient boosting.

A random forest trains many deep trees independently on bootstrap samples with random feature subsets and averages them, mainly reducing variance. Gradient boosting trains shallow trees in sequence, each fitting the errors of the ensemble so far, mainly reducing bias. Boosting often scores higher on tabular data but needs more careful tuning (learning rate, number of trees, depth) and overfits more readily.

10. Which models need feature scaling?

Models based on distances or gradients: k-nearest neighbours, k-means, SVMs, and regularised linear or logistic regression, where the penalty treats all coefficients alike. Tree-based models split on thresholds and are unaffected by monotonic scaling.

11. What happens if the learning rate in gradient descent is too high or too low?

Too high, the loss can oscillate or diverge because steps overshoot the minimum. Too low, training is slow and may stall on a plateau. Learning-rate schedules and adaptive optimisers help.

Data problems

12. Your classifier shows 97% accuracy on fraud detection. Good?

Not without the base rate. If 3% of transactions are fraud, predicting "not fraud" for everything gets 97%. Report precision and recall for the fraud class, and choose a threshold from the cost of missed fraud versus false alarms.

13. What is the curse of dimensionality?

As the number of features grows, data becomes sparse: points are far from each other and distances become less informative. Distance-based methods suffer most. Remedies are feature selection, dimensionality reduction and regularisation.

14. Parametric versus non-parametric models.

Parametric models have a fixed number of parameters regardless of data size, such as linear regression. Non-parametric models grow in complexity with data, such as k-NN or deep decision trees. Non-parametric models are more flexible but need more data and are more prone to overfitting.

Preparing beyond these answers

For each question, be ready for "why?" twice. "Trees don't need scaling" leads to "why?" (they split on thresholds) and then "what about a tree-based model with a distance feature?" (the feature itself may need care). Running the two examples with different settings — change the split counts, change k — builds the intuition interviewers probe.

Once fundamentals are comfortable, move to model evaluation metrics and feature engineering. These foundations are the core of the first modelling phase in the Data Science programme.