Prevent duplicate text leakage across data splits

Copied tickets, quoted email threads, template replies and minor punctuation edits can cross text partitions. A model then appears to generalize while recognizing nearly identical content. Split by source or conversation first, then screen exact and near duplicates.

Layer three checks

The NLP lab compares four training and four test strings with an exact set, lowercase/punctuation normalization and character n-gram cosine similarity.

from nlp_cases import duplicate_case

result = duplicate_case()
assert result["exact_matches"] == 1
assert result["normalized_matches"] >= result["exact_matches"]
assert result["near_matches_cosine_ge_0_8"] == 2
print(result["maximum_similarities"])

Exact and normalized checks each find one match. The simple normalizer misses ORDER-123 versus order 123 because punctuation removal joins one form; normalization itself needs tests. Character similarity at 0.8 identifies two candidates, including a shortened parcel phrase.

Similarity candidates are not automatic duplicates. Review source provenance, labels and intended generalization. Two users can independently send the same short phrase, while a long quoted thread can hide a copied section beneath new text.

Choose the source group

Keep all messages from one conversation, customer case or document family in one partition. For temporal prediction, also respect time. Remove quoted reply history or hash individual segments so repeated boilerplate does not dominate similarity.

If duplicates carry conflicting labels, investigate annotation policy rather than simply deleting one. Report exact same-label, exact conflicting-label, near same-source and near unknown-source counts.

Freeze the test after audit

Run duplicate detection before training and after every data refresh. Fit similarity vectorizers on training or use content-independent hashes. Store reviewed exceptions and fail the dataset build when unexpected exact cross-split copies appear.

The Data Science course connects this audit with group-aware splitting and credible evaluation.

Exercise

Create variants using case, punctuation, quoted threads and paraphrases. Measure candidate precision at several similarity thresholds, then choose a grouping rule based on provenance rather than model score.

Continue learning

This article is part of the NLP and text analytics sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Stemming versus lemmatization for a specific task.
• Continue with Sentiment analysis: define the label before training.

Reference: scikit-learn group-aware cross-validation.