NLP engineer in India: Hinglish tickets, taxonomies and per-class F1

Has the LLM made this role obsolete?

No, but it has changed what the role spends time on. Before large language models, much of the effort went into feature engineering and training task-specific models. Now many first versions are an LLM with a well-designed prompt. The NLP engineer's value has moved to three places:

• Problem definition. What exactly are the categories, and what counts as correct?
• Evaluation. A labelled test set that reflects real traffic, including the messy parts.
• Cost and latency choices. A fine-tuned small classifier can handle millions of short messages per day at a fraction of an LLM's cost; an LLM may handle the long tail better. Often the answer is both, with a router between them.

The artefact, part one: a label taxonomy for support tickets

Taxonomies fail when labels overlap or when "Other" becomes a dumping ground. This is a hypothetical first-level taxonomy for a consumer lending app's support inbox.

The last column is the most important. It tells labellers where to slow down and tells the engineer which pairs to watch in evaluation. The COLLECTION_COMPLAINT label also carries weight beyond routing, because such complaints may need to reach a grievance process quickly.

The artefact, part two: reading a per-class report

Overall accuracy on this inbox might look healthy while a small but important class fails. The script below computes per-class precision, recall and F1 from synthetic true and predicted labels.

from collections import Counter

# Synthetic evaluation: (true label, predicted label) for 20 tickets
pairs = [
    ("EMI", "EMI"), ("EMI", "EMI"), ("EMI", "EMI"), ("EMI", "STATEMENT"),
    ("EMI", "EMI"), ("EMI", "EMI"), ("CLOSURE", "CLOSURE"), ("CLOSURE", "STATEMENT"),
    ("CLOSURE", "CLOSURE"), ("STATEMENT", "STATEMENT"), ("STATEMENT", "STATEMENT"),
    ("STATEMENT", "CLOSURE"), ("STATEMENT", "STATEMENT"), ("COLLECTION", "EMI"),
    ("COLLECTION", "COLLECTION"), ("COLLECTION", "EMI"), ("KYC", "KYC"),
    ("KYC", "KYC"), ("KYC", "KYC"), ("EMI", "EMI"),
]

labels = sorted({t for t, _ in pairs})
tp = Counter(t for t, p in pairs if t == p)
pred_count = Counter(p for _, p in pairs)
true_count = Counter(t for t, _ in pairs)

print(f"{'label':<12}{'support':>8}{'precision':>11}{'recall':>8}{'f1':>6}")
f1s = []
for label in labels:
    precision = tp[label] / pred_count[label] if pred_count[label] else 0.0
    recall = tp[label] / true_count[label]
    f1 = 2 * precision * recall / (precision + recall) if precision + recall else 0.0
    f1s.append(f1)
    print(f"{label:<12}{true_count[label]:>8}{precision:>11.2f}{recall:>8.2f}{f1:>6.2f}")

accuracy = sum(tp.values()) / len(pairs)
print(f"\naccuracy {accuracy:.2f}   macro-F1 {sum(f1s) / len(f1s):.2f}")

Accuracy comes out at 0.75, which sounds acceptable. The per-class table tells a different story: COLLECTION has recall of about 0.33, because two of its three tickets were routed to EMI. Those are exactly the complaints that should reach a grievance team fastest. The fix might be more labelled collection examples, a keyword safety net for phrases about calls and agents, or sending that class to an LLM second opinion. Whatever the fix, it is judged on this class's recall, not on overall accuracy.

A fortnight in the role

• Reviewing 300 misrouted tickets with the support lead to refine two confusing labels.
• Building a code-mixed test set that includes Devanagari, romanised Hindi and English in the proportions actual traffic shows.
• Comparing a fine-tuned small model with an LLM prompt on that test set, including cost per 1,000 tickets and latency.
• Adding a monitoring job that samples live predictions for weekly human review.
• Writing a note for the product manager explaining why the OTHER bucket grew after an app release.

Per-class quality, cost per ticket and drift

Per-class quality on the classes the business cares about, cost and latency in production, stability as language and products change, and the clarity of the evaluation others can trust.

Support leads, annotators and AI engineers

An ML or AI lead, sometimes a data science manager. You work with annotators and their leads, support or operations teams who own the process, product managers, backend engineers, and increasingly AI engineers building LLM applications on top of your evaluation work.

Confused with AI engineer and linguist

An AI engineer typically assembles LLM applications: retrieval, tools, prompts and interfaces. An NLP engineer is more concerned with task definition, data, models and measurement for language specifically. Computational linguists focus on language structure itself; in Indian language work, their input on script, transliteration and dialect is valuable.

Indic, code-mixed and NER signals in postings

• "Text classification, NER, intent detection" — classic NLP tasks, still common.
• "Indic languages, code-mixed, transliteration" — multilingual data is central.
• "Speech-to-text, call analytics" — transcripts, with their own noise.
• "Fine-tuning, LoRA, small language models" — training your own models.
• "LLM evaluation, prompt design" — overlap with AI engineering work.

Test set, weakest class, live sampling

Month one: rebuild the test set and a per-class baseline. Month two: fix the weakest important class. Month three: set up live sampling and a monthly review. Next roles: senior NLP engineer, applied scientist, ML lead or AI engineering lead. Per-class evaluation, confusion analysis and honest test-set design are covered in the Data Science programme before any specialisation in language.