Tokenization choices that change a text model

Tokenization decides which patterns a text model can represent. A word tokenizer may split sign-in into sign and in, while character n-grams preserve fragments across punctuation and spelling variants. The choice changes vocabulary size, sparsity, robustness and interpretation.

Inspect the actual analyzer

The local NLP lab injects hyphenated login phrases and order#1042 identifiers, then compares word unigrams/bigrams with character 3–5-grams under the same split and logistic classifier.

from nlp_cases import tokenization_case

result = tokenization_case()
word = result["models"]["word"]
char = result["models"]["character"]
assert word["features"] == 196
assert char["features"] == 1162
print(word["sample_tokens"][:5])

The word analyzer yields cannot, sign, in, order, 1042 and bigrams. The character analyzer creates 1,162 features versus 196 word features. Both reach 100% validation accuracy because the authored categories remain easy; the result does not establish equivalence on spelling noise or new languages.

Design around the input

Inspect contractions, emoji, hashtags, URLs, code, numbers and scripts in the intended population. Lowercasing can merge meaningful case distinctions. Removing punctuation can turn ORDER-123 into a different token boundary. Replacing every number may improve generalization or erase product identifiers needed for the task.

Character n-grams often tolerate misspellings and inflection but increase dimensionality and can learn source artifacts. Word n-grams are easier to inspect and can capture short phrases, yet unseen forms become out of vocabulary. Subword tokenizers used by transformer models have their own vocabulary and normalization contract; compare token lengths and truncation rates on real samples.

Prevent split leakage

Fit tokenizer vocabulary on training only. Version the tokenizer with the model. Do not normalize all text using a dictionary learned from test. Test empty text, unusual Unicode, repeated punctuation and maximum-length inputs.

Tokenization quality is task evidence, not vocabulary elegance. Compare validation metrics, memory, latency and failure slices. The Data Science course joins this preprocessing decision to a reproducible pipeline.

Exercise

Create spelling, punctuation and transliteration slices. Compare word, character and subword token counts and class metrics. Document transformations that may change label meaning.

Continue learning

This article is part of the NLP and text analytics sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Build a text-classification baseline with TF-IDF.
• Continue with Stemming versus lemmatization for a specific task.

Reference: scikit-learn TfidfVectorizer API.