Document annotation guidelines for an NLP project

Annotation converts a business question into labels. If the guide is vague, more rows create a larger inconsistent dataset. Define scope, labels, inclusions, exclusions, boundary examples and an adjudication process before scaling.

Measure a pilot without hiding disagreement

The local lab gives two authored annotators eight sentiment examples across positive, negative, mixed and uncertain.

from nlp_cases import annotation_case

result = annotation_case()
assert result["items"] == 8
assert result["adjudication_status"].startswith("pending")
print(result["agreement"], result["cohen_kappa"], result["disagreements"])

Raw agreement is 75% and Cohen’s kappa is 0.667. The disagreements concern mixed praise/complaint and sarcastic praise—the boundary cases the guide most needs to address. Adjudication remains pending instructor review; code cannot authenticate it.

Make the guide operational

Include the decision target, unit of annotation, label definitions, precedence rules, span boundaries, multi-label policy, uncertain route and examples near every boundary. State what annotators may infer and what requires explicit text evidence.

Train annotators on a pilot and discuss disagreements without punishing honest uncertainty. Version the guide, then annotate a new pilot after changes. Preserve original labels and adjudicated outcome.

Interpret agreement carefully

Kappa adjusts for chance under its assumptions but depends on prevalence and annotator behavior. Report the confusion table and per-label agreement. High agreement can still encode a consistently wrong policy; domain review and connection to the intended action remain necessary.

Monitor disagreement over time and by language, source and annotator. Use gold checks carefully: ambiguous items should not become hidden traps. Protect annotators from sensitive content with training, escalation and access controls.

The Data Science course treats the guide, pilot and disagreement log as model inputs worthy of version control.

Exercise

Draft a two-page guide for one support task. Double-label 50 examples, calculate label-level agreement, adjudicate with reasons, revise the guide and validate on a fresh 50.

Continue learning

This article is part of the NLP and text analytics sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the preceding task in Privacy-aware text preprocessing for a portfolio dataset.
• Return to the cluster foundation in Build a text-classification baseline with TF-IDF.

Reference: scikit-learn Cohen’s kappa API.