Cluster stability under resampling

A clustering can look clean once and still change when the sample changes. Stability analysis asks whether plausible resamples produce similar assignments. It does not prove that stable groups are useful or causally meaningful.

Compare assignments on a common reference set

We standardize the 360-row synthetic customer fixture. For each of k=3 and k=4, we draw 12 bootstrap samples, fit K-means on each sample and use its centroids to assign all 360 original reference rows.

Every pair of the 12 assignment vectors is compared with adjusted Rand index, giving 66 pairwise comparisons. ARI is invariant to arbitrary label permutations, so cluster zero in one fit need not be cluster zero in another.

For k=3, median and minimum pairwise ARI are both 1.0. For k=4, median ARI is 0.8991 and the minimum is 0.7296. The extra split is less stable under sampling in this fixture.

from unsupervised_cases import stability_case

r = stability_case()['by_k']
assert r['3']['replicates'] == 12
assert r['3']['pairwise_comparisons'] == 66
assert r['3']['minimum_ari'] == 1
assert r['4']['minimum_ari'] < .75
print(r)

Run it in the unsupervised lab. Reusing the common reference rows isolates changes in fitted centroids; it does not create independent validation data.

Match the perturbation to the uncertainty

Row bootstrapping represents sampling variation when rows are exchangeable. Repeated customers, stores or households require cluster resampling at that entity level. Time-varying segments need later-period stability, which tests a different and often harder question.

Feature uncertainty matters too. Repeat the analysis across plausible preprocessing rules, missing-value treatments and measurement noise. A clustering that survives row resampling but collapses when one feature is removed depends heavily on representation.

Choose a comparison metric that handles label permutations. ARI compares pairs of assignments and adjusts for chance. Centroid matching can explain how profiles moved, while prediction strength and consensus matrices offer other views. State the reference population and whether unfitted rows can be assigned by the algorithm.

A perfectly stable partition can encode a stable identifier artifact. Review profiles, source quality and downstream action. Stability is necessary for some uses, not sufficient for value.

Exercise: bootstrap at customer level and then perturb each feature by its measurement error. Compare ARI distributions for k=2 through k=5. Predeclare the stability and minimum-size conditions that would stop a segment from receiving an action.

NeuraPath's Data Science course teaches clustering with perturbation tests. A segment earns confidence by surviving the changes the data process can realistically produce.

Continue learning

This article is part of the Clustering, reduction and recommendations sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Hierarchical clustering with a defensible distance metric.
• Continue with PCA for compression without leaking test information.