Text features: control vocabulary and fit boundaries

TF-IDF learns a vocabulary and document-frequency weights. Fitting on all documents before a split imports test-language statistics into the representation, even when no target is used.

Track one future-only token

Our three training documents mention Python, SQL, data analysis, joins and machine learning. Two test documents introduce futuretoken.

A proper training-only vectorizer learns seven terms and excludes futuretoken. Transforming test produces a two-by-seven matrix; the unknown token contributes nothing. A leaky vectorizer fitted on all five documents adds futuretoken to its vocabulary and uses test documents in inverse-document-frequency weights.

The TfidfVectorizer documentation combines token counts with learned IDF statistics and exposes a fitted vocabulary.

from feature_cases import text_case

r = text_case()
assert not r['futuretoken_in_proper']
assert r['futuretoken_in_leaky']
assert r['proper_test_shape'] == [2, 7]
assert 'futuretoken' not in r['proper_vocabulary']
print(r)

Run it in the feature-engineering lab. The artificial token makes the distribution leak unmistakable.

Put the vectorizer inside the model pipeline

During cross-validation, fit vocabulary, document frequency and any supervised feature selection on each training fold. Transform fold validation with the stored vectorizer. Persist the final object for serving.

Define tokenization, casing, Unicode normalization, n-grams, stop words, minimum frequency and maximum vocabulary. These choices affect language and subgroup coverage. A word-level vocabulary may fail on misspellings or mixed scripts; character features introduce different trade-offs.

Unknown-token rate and document length should be monitored after deployment. A rise can indicate new topics, upstream formatting or language drift. Do not automatically refit because vocabulary changes alter model inputs and require validation.

Duplicate and near-duplicate text can cross splits and inflate results. Group by source or template and use temporal splits when future documents are the target.

Exercise: add supervised chi-square selection after TF-IDF inside a pipeline and compare with global vocabulary plus global selection. Mutate one validation document and verify the fitted training vocabulary stays unchanged.

NeuraPath's Data Science course treats text vectorization as learned preprocessing. Vocabulary provenance and split boundaries belong in the model record.

Continue learning

This article is part of the Feature engineering and data quality sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in High-cardinality categories: compare hashing and encoding.
• Continue with Build a reusable preprocessing artifact.