Sentiment analysis: define the label before training

“Positive” and “negative” are not self-defining labels. A ticket can praise an agent while reporting an unresolved refund. A literal positive word can express sarcasm. Before training, state whose sentiment is being labeled, its target, time scope and how mixed or insufficient evidence is handled.

A four-label teaching contract

The local NLP lab proposes positive, negative, mixed and uncertain labels for sentiment expressed by a ticket author about the support experience.

from nlp_cases import sentiment_contract_case

result = sentiment_contract_case()
assert set(result["labels"]) == {"positive", "negative", "mixed", "uncertain"}
print(result["examples"][0])

“Helpful agent, but the refund is still missing” is mixed because service praise and an unresolved outcome coexist. “Amazing, another reset link that does not work” is uncertain in this guide because likely sarcasm conflicts with the literal word. “The update arrived” is uncertain because it reports an event without evaluation.

This guide is a proposed teaching artifact with review pending. It is not an authenticated production annotation policy.

Write decision rules and exclusions

Specify how much evidence is required, whether emoji count, whose perspective wins in quoted conversations, and whether aspect sentiment is needed. If routing depends on unresolved harm, a single overall sentiment label may be the wrong target. Multi-label aspects such as agent interaction, outcome and effort may be more useful.

Provide positive, negative and boundary examples for every label. Include multilingual, mixed and sarcastic cases from the intended population with appropriate privacy and permissions. Define escalation for threats or sensitive content separately from sentiment.

Measure annotation quality

Train annotators on a pilot, double-label a sample and review disagreements. Agreement does not prove the task is valid, but low agreement can expose an unclear guide. Preserve pre-adjudication labels so ambiguity is visible.

Only after the policy stabilizes should model metrics become meaningful. Report class counts, per-label performance and confusion. Keep uncertain examples rather than forcing them into a class that creates artificial certainty.

The Data Science course joins label design to modeling and error analysis.

Exercise

Write ten boundary examples for mixed and uncertain. Have two annotators label them independently, calculate agreement, revise one rule and repeat on new examples.

Continue learning

This article is part of the NLP and text analytics sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Prevent duplicate text leakage across data splits.
• Continue with Evaluate sentiment on mixed and sarcastic examples.

Reference: scikit-learn text-classification example.