Named entity recognition with boundary-level evaluation

Named entity recognition must locate a span and assign its type. Token-level accuracy can look high because most tokens are “outside,” while an extracted name is incomplete or a date includes punctuation. Score exact span and type for downstream extraction.

Three overlaps, one exact entity

The local lab annotates an invented sentence with an order ID, person and date. Predictions find all three regions but shorten Maya Rao to Maya and extend the date to include a period.

from nlp_cases import ner_case

result = ner_case()
assert result["overlap_matches"] == 3
assert result["exact"]["true_positives"] == 1
print(result["exact"])

Overlap matching reports three, while exact precision, recall and F1 are each 0.333. Only NP-1042 has the correct start, end and type.

Define boundaries in the guide

State whether titles, punctuation, legal suffixes and nested entities belong in a span. Give examples for hyphenated IDs, multi-word names, dates and overlapping entities. Choose a token-offset convention and preserve raw character offsets so evaluation can reconstruct the exact substring.

Normalization comes after extraction. Converting a date or amount to a canonical value is a separate field-level task. Report boundary error, type error, missing entity and spurious entity separately.

Test the downstream action

Exact matching may be strict but appropriate when an order ID drives a database lookup. Partial overlap can remain a diagnostic for model improvement. If a fuzzy downstream resolver accepts variants, evaluate the complete extraction-plus-resolution pipeline and record its errors.

Split related documents by source. Repeated templates and identifiers can leak across partitions. Protect personal text and use invented or appropriately governed examples in public portfolios.

The Data Science course connects span metrics with schema validation and operational use.

Exercise

Annotate 50 invented messages with exact spans. Create one-character boundary, wrong-type and missing-entity errors. Implement exact and overlap reports and decide which metric governs release.

Continue learning

This article is part of the NLP and text analytics sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Evaluate sentiment on mixed and sarcastic examples.
• Continue with Text embeddings: evaluate similarity against labelled pairs.

Reference: spaCy linguistic features and entity spans.