Topic modelling: validate topics with human inspection

Topic models produce weighted term patterns. A low reconstruction error or coherent-looking word list does not establish a useful business theme. Reviewers must inspect terms and representative documents, name the theme if justified and test whether another reviewer reaches a similar interpretation.

A model that needs rejection or revision

The local lab fits three nonnegative matrix factorization components to 180 templated support tickets.

from nlp_cases import topic_case

result = topic_case()
assert result["human_review_complete"] is False
for topic in result["topics"]:
    assert topic["review_status"] == "pending"
    print(topic["topic"], topic["top_terms"])

The top terms include generic template language such as request, status, need, assistance, hello and support. They do not cleanly recover billing, access and delivery. Reconstruction error is 11.76, but that number cannot rescue the weak interpretation.

No human labels are inserted into the verification file. Review is pending, so the model cannot be presented as a validated theme system.

Use an inspection protocol

For each topic, show top weighted terms, highest-scoring documents and documents with mixed membership. Ask reviewers to assign a concise label, describe the common action and flag incoherence. Add an intrusion task: can a reviewer identify a document or term that does not belong?

Compare topic counts and seeds. Useful themes should show reasonable stability and coverage. Remove boilerplate using a documented training-only policy, then confirm that the remaining topics improve the downstream task.

Connect themes to action

A theme such as “refund delay” becomes useful only if it supports an owner, metric or investigation. Measure how many reviews it covers, precision of assigned examples and change over time. Do not treat topic proportions as customer prevalence without sampling and data-collection analysis.

The Data Science course uses failed topic outputs to teach human validation and action design.

Exercise

Review the three term lists and ten representative documents each. Revise stop words, repeat five seeds, measure topic matching across runs and accept only themes with a defined owner and reviewer agreement.

Continue learning

This article is part of the NLP and text analytics sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Semantic search versus keyword search for support tickets.
• Continue with Classify multilingual tickets without hiding language imbalance.

Reference: scikit-learn topic extraction with NMF and LDA.