Customer support to GenAI: build an evaluated ticket-triage assistant

Most first attempts go wrong in predictable ways. Here are the five mistakes worth avoiding, with what to build instead.

Mistake 1: a demo chatbot with no labelled test set

A chat window that answers three questions you typed is not evidence. Anyone can make one in an afternoon. What a hiring manager cannot easily fake is a test set: 50 to 200 synthetic tickets, each labelled with the correct category, priority and whether it needs a human.

You have an advantage here. You know how tickets are actually written, with half sentences, mixed Hindi and English, order IDs pasted in the wrong place and two issues in one message. Write synthetic tickets that look like that, never copies of real customer messages.

Mistake 2: reporting one accuracy number

A triage assistant that is 90% accurate overall can still send every refund dispute to the wrong queue. Evaluate per category, and treat some errors as worse than others. The script below is standard-library Python. The predicted column stands in for whatever your assistant returned; in your project you would fill it from the model's output and keep that output saved as a file.

from collections import Counter

# Synthetic labelled tickets: (ticket_id, true_category, predicted_category)
results = [
    ("T01", "delivery_delay", "delivery_delay"),
    ("T02", "refund_dispute", "refund_dispute"),
    ("T03", "refund_dispute", "delivery_delay"),
    ("T04", "account_access", "account_access"),
    ("T05", "delivery_delay", "delivery_delay"),
    ("T06", "damaged_item",   "refund_dispute"),
    ("T07", "refund_dispute", "refund_dispute"),
    ("T08", "account_access", "delivery_delay"),
    ("T09", "damaged_item",   "damaged_item"),
    ("T10", "delivery_delay", "delivery_delay"),
]

categories = sorted({r[1] for r in results})
print("Overall accuracy:", sum(t == p for _, t, p in results) / len(results))
tp = Counter(); fp = Counter(); fn = Counter()
for _, true, pred in results:
    if true == pred:
        tp[true] += 1
    else:
        fp[pred] += 1
        fn[true] += 1

print(f"{'category':<16}{'precision':>10}{'recall':>8}{'support':>9}")
for c in categories:
    precision = tp[c] / (tp[c] + fp[c]) if (tp[c] + fp[c]) else 0.0
    recall = tp[c] / (tp[c] + fn[c]) if (tp[c] + fn[c]) else 0.0
    print(f"{c:<16}{precision:>10.2f}{recall:>8.2f}{tp[c] + fn[c]:>9}")

# Errors that must never happen: a refund dispute routed away from the refunds queue
critical = [t for t, true, pred in results if true == "refund_dispute" and pred != true]
print("Critical misroutes:", critical)

On this synthetic set, overall accuracy is 70%, but refund-dispute recall is only 0.67 and T03 is a critical misroute. Delivery-delay precision is also dragged down because the assistant used it as a default for tickets it did not understand. That pattern, a model defaulting to the most common category, is something support leads recognise immediately, and naming it shows you read results rather than just computing them.

Mistake 3: no rule for when the assistant should not answer

Your escalation matrix already says which issues go to a supervisor, legal, or a specialist team. Turn it into written rules the assistant follows: chargebacks, threats of legal action, safety complaints and anything involving personal data changes go to a human, whatever the model's confidence. Then test those rules with tickets written specifically to trip them. An assistant that escalates correctly is more valuable to an operations team than one that answers more.

Mistake 4: ignoring the knowledge base

Reply-drafting assistants usually retrieve from help articles and macros. If the article on return windows is outdated, the assistant will confidently quote the old policy. Your project should include a small synthetic knowledge base, show which article each drafted reply relied on, and include one test where the article is deliberately wrong so you can show how you would catch it. The underlying method is explained in what is RAG.

Mistake 5: describing the switch as "I used ChatGPT a lot"

Using AI tools daily is common. What is less common is someone who can say how an assistant was evaluated, which failures were acceptable and which were not, and who owns the escalation rules. That is the language to use.

What transfers, and what does not

You will also need working Python, an understanding of how prompts, retrieval and tool calls fit together, and basic cost awareness, since every model call costs money at volume.

Role titles to consider first

Titles vary widely and are not standardised, so read the responsibilities rather than the name. Useful searches include AI support operations, conversational AI analyst, knowledge base specialist (AI) and AI quality or evaluation analyst. These sit close to the support floor and value process knowledge. Engineering-heavy titles come later, once your Python is solid.

Explaining your move

"I have handled customer support for an e-commerce business for three years, including maintaining our macros and the escalation matrix. When our team started trying AI reply suggestions, I noticed nobody was measuring which ticket types it got wrong. I built a triage assistant on a synthetic ticket set I labelled myself, evaluated it per category, and added hard escalation rules for refund disputes and legal threats. The main result was that it defaulted to 'delivery delay' when unsure, which I fixed with an explicit 'needs human' route. I want to work on making these assistants safe to use on a real support floor."

Where to go next

The Generative & Agentic AI add-on covers retrieval, evaluation and agent workflows over three months; ask a counsellor for current batch details. Before that, check how your current experience reads with the free skill check.