K-nearest neighbours: why scaling changes the prediction

K-nearest neighbours predicts from nearby training observations, so the meaning of “nearby” is part of the model. Changing one feature's units can change distances, selected neighbours and the predicted class even when the underlying records stay the same.

An original two-point example makes the mechanism visible. Training point B=(2,1) has label zero, and C=(1,3) has label one. The query A=(1,1) is one unit from B and two units from C. With one neighbour and Euclidean distance, the prediction is zero.

Convert one coordinate consistently

Multiply the first feature by ten for both training points and the query. The coordinates become B=(20,1), C=(10,3) and A=(10,1).

Now A is ten units from B and two from C. C becomes the nearest neighbour, so the prediction changes to one. We transformed training and prediction consistently; the change arises because the new coordinate scale gives the first feature greater distance weight.

The KNeighborsClassifier reference documents neighbour voting and distance settings. The included model uses one neighbour, Euclidean distance and a KD-tree search, with no parameter tuning.

Verify the selected neighbour and class

from classical_cases import knn_case

r = knn_case()
assert r['original_prediction']==0
assert r['converted_prediction']==1
assert r['original_neighbor_index']==0
assert r['converted_neighbor_index']==1
assert r['original_nearest_distance']==1
assert r['converted_nearest_distance']==2
print(r)

Run from the supervised-model lab. The example links the prediction change to the actual neighbour index and distance, rather than reporting two labels without explaining why they differ.

These are abstract signed-coordinate teaching records, not measured customers or a validated classifier. Neither representation is declared superior for a real task.

Choose a representation that matches the question

If a one-unit difference in the first feature should matter much more than a one-unit difference in the second, a weighted distance may be justified. If unit magnitudes are arbitrary, scaling may prevent one coordinate from dominating unintentionally.

Standardization, robust scaling, feature weighting and row normalization make different choices. Standardization changes each feature according to training statistics; row normalization changes each observation's overall length. They should not be treated as interchangeable rituals.

Fit any learned preprocessing on training data inside the validation workflow. A scaler fitted on all rows lets held-out observations influence the representation, even if the KNN estimator itself only stores the training subset.

Evaluate more than the number of neighbours

The number of neighbours, distance metric and voting weights interact. A larger neighbourhood can smooth local noise but may mix distinct regions. Distance weighting gives nearby observations more influence, which can also make unusual points important.

Ties need attention. Equal distances or equal class votes can make outcomes depend on implementation conventions or training order. Record the relevant behavior when it affects the decision, and do not infer confidence from a single neighbour's label.

In high-dimensional or poorly chosen representations, numerical proximity may cease to reflect useful similarity. Validation and error analysis are necessary even when scaling is mathematically consistent.

Exercise: add a third labelled point near the query and compare one versus three neighbours under both coordinate scales. Record the selected neighbours and vote counts, then explain which representation assumptions you would need to justify for a real application.

NeuraPath's Data Science course connects algorithms with feature representation. A complete KNN explanation includes the geometry that produced the prediction and the evidence supporting that geometry.

Continue learning

This article is part of the Supervised learning methods sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in LightGBM categorical features: verify the encoding assumptions.
• Continue with Support vector machines: choose a kernel with validation.