Hierarchical clustering with a defensible distance metric

A dendrogram does not discover one inevitable hierarchy. The distance metric and linkage define which merges look close. When magnitude and composition answer different business questions, they produce different trees.

Compare six exact profiles

Consider two-component vectors:

• A, B and C are [10,1], [20,2] and [100,10];
• D, E and F are [1,10], [2,20] and [10,100].

Within each letter family, magnitude changes while the component ratio stays constant. We request two clusters using average-linkage agglomerative clustering.

Euclidean distance isolates F and groups A through E together. Its large absolute magnitude dominates late merges. Cosine distance groups A/B/C and D/E/F by direction, ignoring uniform rescaling within a vector.

The scikit-learn agglomerative clustering documentation exposes both metric and linkage. Cosine similarity is the normalized dot product described in the pairwise metrics guide.

from unsupervised_cases import hierarchical_case

r = hierarchical_case()
euclidean_groups = list(r['clusters']['euclidean'].values())
cosine_groups = list(r['clusters']['cosine'].values())
assert ['F'] in euclidean_groups
assert sorted(map(sorted, cosine_groups)) == [['A', 'B', 'C'], ['D', 'E', 'F']]
print(r)

Run this exact case in the unsupervised lab. No outcome selects the “correct” metric; the choice depends on whether absolute volume or composition defines similarity.

Choose metric and linkage as a pair

Euclidean distance is sensitive to units and scale. Manhattan distance changes the geometry. Cosine focuses on direction and is undefined for a zero vector without a policy. Domain distances can combine numeric, binary and categorical information, but every weight needs justification.

Linkage also matters. Single linkage can chain clusters through close pairs. Complete linkage emphasizes the farthest pair and can prefer compact groups. Average linkage balances pairwise distances. Ward linkage minimizes within-cluster variance and requires Euclidean geometry.

Before inspecting the dendrogram, write the unit of analysis, feature transformations, metric, linkage and cut rule. Afterward, test stability under resampling and feature perturbation. Dendrogram heights describe the chosen dissimilarity, not statistical confidence.

Exercise: add a zero profile and define a policy for cosine distance. Compare single, complete and average linkage. Identify which resulting groupings support your intended action and which reflect only a convenient visual cut.

NeuraPath's Data Science course treats hierarchical clustering as an explicit geometry. A defensible dendrogram starts with the meaning of distance.

Continue learning

This article is part of the Clustering, reduction and recommendations sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in DBSCAN: distinguish noise from a bad density setting.
• Continue with Cluster stability under resampling.