Semantic search versus keyword search for support tickets

Semantic search should retrieve relevant text when wording differs. That promise needs query-level relevance judgments and a keyword baseline. A lower-dimensional vector does not become semantic merely because it uses cosine similarity.

A tie on six queries

The local NLP lab indexes six invented support documents and evaluates six queries. Keyword search uses word TF-IDF. The latent candidate applies truncated SVD and normalization to the same corpus.

from nlp_cases import search_case

result = search_case()
assert len(result["queries"]) == 6
print(result["keyword_mrr"], result["latent_mrr"])

Both systems achieve mean reciprocal rank 0.511. They rank access queries with overlapping vocabulary first, but synonym-heavy queries such as “package is late” find the first delivery document at rank five. The latent representation does not improve this tiny corpus.

The comparison rejects a superiority claim. Six documents are also too few for latent structure to become meaningful; the next step is better evaluation data, not a semantic label in the UI.

Define relevance and rank metrics

Label relevant documents per query, including multiple acceptable resolutions. Report recall at the result depth a support agent sees, mean reciprocal rank for first useful result and nDCG when relevance is graded. Include zero-result and latency rates.

Create synonym, misspelling, exact-ID, multilingual and shared-keyword-negative slices. Keyword search may remain best for order IDs and error codes. Hybrid retrieval can combine exact terms and embeddings, but it becomes another candidate to evaluate.

Keep an inspectable retrieval trace

For every test query, save ranked IDs, scores and relevance labels. Do not tune on final test queries. Version documents and remove near duplicates across evaluation partitions. If a reranker is added, evaluate retrieval before and after it separately.

The Data Science course links search architecture to ranking evidence rather than feature names.

Exercise

Expand to 100 documents and 50 queries with at least two relevant results each. Compare keyword, latent, neural and hybrid candidates on recall@5, MRR, nDCG and p95 latency.

Continue learning

This article is part of the NLP and text analytics sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Text embeddings: evaluate similarity against labelled pairs.
• Continue with Topic modelling: validate topics with human inspection.

Reference: scikit-learn TruncatedSVD for latent semantic analysis.