Build a text-classification baseline with TF-IDF

TF-IDF plus a regularized linear classifier is a strong text baseline: fast, inspectable and hard to beat on small datasets with obvious vocabulary signals. A transformer candidate should improve a decision-relevant metric enough to justify its extra latency, artifact size and evaluation burden.

Fit vocabulary on training only

The original NLP lab creates 180 templated support tickets across billing, access and delivery. It fixes 120 training, 30 validation and 30 test rows. Word unigrams and bigrams occurring at least twice create 196 training features.

from nlp_cases import tfidf_case

result = tfidf_case()
assert sum(result["split_rows"].values()) == result["rows"] == 180
assert result["vocabulary_size"] == 196
print(result["validation_accuracy"], result["test_accuracy"])

Validation and test accuracy are both 100%. This reflects an easy synthetic template task with distinct class terms. It does not estimate live support performance. The appropriate response is to add paraphrases, ambiguous tickets, temporal variation and source-group controls rather than promote the number.

The vectorizer learns vocabulary and inverse-document frequencies from training only. Test-specific numeric identifiers remain unknown, yet class words suffice. Fitting TF-IDF on all text would leak document-frequency information from future partitions even without labels.

Define the classification action

Specify whether a prediction routes a ticket, suggests a queue or closes a case. Define multi-intent handling and abstention. A ticket can mention a failed login and a duplicate charge; forcing one class may be a label-policy error rather than a model failure.

Evaluate macro F1 and per-class recall when class volumes differ. Add a confusion matrix and review misclassified examples. Calibrate or tune thresholds on validation if scores control automation. Keep the test closed during vocabulary, n-gram and regularization choices.

Save the complete pipeline

Serialize normalization, vectorizer, classifier, label mapping and threshold together. Test unknown terms, empty text and maximum length. A model weight matrix without the exact vocabulary cannot reproduce predictions.

The Data Science course uses linear text baselines to anchor later NLP experiments in measurable improvement.

Exercise

Add mixed-intent and paraphrased tickets with source IDs. Compare word and character features across grouped splits, report macro F1 and define an abstention queue before evaluating the final test.

Continue learning

This article is part of the NLP and text analytics sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Continue with Tokenization choices that change a text model.
• Then apply it in Stemming versus lemmatization for a specific task.

Reference: scikit-learn text feature extraction.