Text embeddings: evaluate similarity against labelled pairs

An embedding maps text to a vector. Cosine similarity then measures angle, not task relevance. Before using it for deduplication, retrieval or clustering, evaluate labelled pairs that represent what “similar” means for the decision.

A lexical embedding fails on synonyms

The local lab uses character TF-IDF vectors for four relevant and four irrelevant support pairs.

from nlp_cases import similarity_case

result = similarity_case()
assert len(result["pairs"]) == 8
print(result["roc_auc"], result["positive_mean"], result["negative_mean"])

Mean similarity is 0.147 for positive pairs and 0.032 for negative pairs, yet ROC AUC is only 0.50. Two true paraphrases—parcel is late versus shipment has not arrived, and invoice charged twice versus duplicate billing amount—receive zero similarity because they share no useful character fragments.

This is a valuable negative result. The vectors encode lexical form but fail the authored synonym task. Calling them “semantic embeddings” would overstate the evidence.

Build representative pairs

Define relevance by action. Duplicate detection may require near-identical wording; support retrieval may treat different phrases with the same resolution as relevant. Include hard negatives that share words but need different actions and hard positives that use synonyms or another language.

Split pair sources so the same ticket does not appear across training and test pairs. Report score distributions, ROC or precision-recall behavior and threshold metrics. A high average separation can still hide unacceptable hard cases.

Compare candidates under one protocol

Evaluate sparse vectors, a latent semantic model and a versioned neural embedding model on the same pair IDs. Record model version, normalization, maximum length, latency and vector dimension. Select a threshold on validation and assess once on test.

The Data Science course uses labelled pairs to turn representation claims into falsifiable evidence.

Exercise

Create 200 support pairs with paraphrases, shared-keyword negatives and multilingual positives. Double-label a sample, compare three vectorizers and set a threshold for a declared review capacity.

Continue learning

This article is part of the NLP and text analytics sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Named entity recognition with boundary-level evaluation.
• Continue with Semantic search versus keyword search for support tickets.

Reference: scikit-learn cosine similarity API.