Compare a transformer with a linear baseline

A credible comparison needs both candidates executed on identical data roles and metrics. If a transformer runtime or artifact is absent, the honest result is “comparison incomplete,” not an estimated improvement borrowed from another dataset.

The evidence gate runs locally

The NLP lab executes the TF-IDF baseline but records the transformer candidate as unexecuted because this project environment has no reviewed transformer runtime or model artifact.

from nlp_cases import transformer_comparison_case

result = transformer_comparison_case()
assert result["linear_baseline"]["executed"] is True
assert result["transformer_candidate"]["executed"] is False
assert result["comparison_complete"] is False
print(result["transformer_candidate"]["reason"])

The linear fixture score is 100% on an easy templated task. There is no transformer score. This prevents a placeholder, framework reputation or published benchmark from becoming local evidence.

Predeclare a fair comparison

Use the same train, validation and test IDs. Freeze text normalization, label policy and evaluation script. Let each model use its own required tokenizer, but fit or configure it without seeing test. Select checkpoints and thresholds on validation.

The lab contract names macro F1 as primary and requires latency and artifact size. Add per-language, length and ambiguity slices, several seeds, memory and calibration where probabilities drive action. Record the exact base-model revision and license.

Decide whether complexity earns release

Set a minimum improvement or critical-slice gate before execution. A larger candidate that improves an easy average by a fraction may not justify slower service and more maintenance. A transformer can earn selection through robust paraphrase or multilingual gains even when overall scores are close, if those slices match the decision.

The comparison becomes complete only after a reproducible candidate artifact and results are saved. This cluster leaves that work for a reviewed transformer environment and later GAAI material.

The Data Science course teaches that “not executed” is a valid evidence state and a stronger outcome than fabricated certainty.

Exercise

Create an isolated, versioned transformer run. Use the manifest’s fixed IDs and report macro F1, slice metrics, median/p95 latency, peak memory and artifact size beside TF-IDF. Preserve a failed candidate.

Continue learning

This article is part of the NLP and text analytics sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Fine-tune a text classifier with a held-out evaluation set.
• Continue with Extract structured fields from text and validate each field.

Reference: Hugging Face text classification task guide.