NLP interview questions on tokenisation, embeddings and attention

From text to tokens

1. Why do modern models use subword tokenisation instead of whole words?

A whole-word vocabulary cannot cover every word, so rare or new words become "unknown". Character-level tokens cover everything but make sequences very long. Subword methods such as byte-pair encoding split rare words into frequent pieces ("unbelievably" → "un", "believ", "ably") and keep common words whole, balancing vocabulary size against sequence length.

What they check: the trade-off, not the name of the algorithm.

2. Walk through one byte-pair encoding merge.

Start from characters, count adjacent pairs across the corpus, merge the most frequent pair into a new symbol, and repeat.

from collections import Counter

corpus = {"l o w": 5, "l o w e r": 2, "n e w e s t": 6, "w i d e s t": 3}

def most_frequent_pair(vocab):
    pairs = Counter()
    for word, freq in vocab.items():
        symbols = word.split()
        for a, b in zip(symbols, symbols[1:]):
            pairs[(a, b)] += freq
    return pairs.most_common(1)[0]

for step in range(3):
    (a, b), count = most_frequent_pair(corpus)
    corpus = {w.replace(f"{a} {b}", a + b): f for w, f in corpus.items()}
    print(f"merge {step + 1}: {a}+{b} (seen {count} times) -> {list(corpus)}")

The first merges build "es", then "est", because "newest" and "widest" together contribute nine occurrences. A tokenizer is simply this merge list applied in order to new text.

3. Why does token count matter in practice?

Model context limits and API costs are counted in tokens, not words. Text in scripts or languages under-represented in a tokenizer's training data often splits into more tokens per word, so the same content costs more and fits less. Measure token counts on your actual text before estimating cost.

4. Should you remove stop words and lowercase?

For bag-of-words models on topic classification, often yes. For transformers, no: they were trained on natural text, and words like "not" carry meaning — removing it turns "not good" into "good".

Representations

5. What does TF-IDF reward?

Words frequent in this document but rare across the collection. A word in every document gets a low inverse-document-frequency weight.

import math

docs = ["refund not received for order",
        "order delivered late",
        "refund processed for order"]
tokens = [d.split() for d in docs]
N = len(docs)

def tfidf(term, doc):
    tf = doc.count(term) / len(doc)
    df = sum(term in d for d in tokens)
    return tf * math.log(N / df)

for term in ["order", "refund", "late"]:
    print(term, [round(tfidf(term, d), 3) for d in tokens])

"order" appears in all three documents, so its weight is zero everywhere; "late" appears once and gets the highest weight in document two. Libraries add smoothing and normalisation, but the idea is this.

6. What is a word or sentence embedding?

A dense vector learned so that texts with similar meaning are close together. Unlike TF-IDF, "refund" and "money back" can be near each other without sharing a word.

7. Why cosine similarity rather than Euclidean distance?

Cosine similarity compares direction and ignores vector length, which often reflects text length or frequency rather than meaning. Many embedding models normalise vectors, in which case cosine and dot product rank results the same way.

import math

def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    return dot / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)))

refund = [0.9, 0.1, 0.3]       # toy 3-dimensional vectors, not from a real model
money_back = [1.8, 0.25, 0.55] # similar direction, about twice the length
late = [0.1, 0.9, 0.2]
print(f"refund vs money_back {cosine(refund, money_back):.3f}")
print(f"refund vs late       {cosine(refund, late):.3f}")

The first pair scores close to 1 despite different lengths; the second is far lower.

Models and attention

8. What problem does attention solve?

Earlier recurrent models compressed a whole sequence into one hidden state, losing information over long inputs. Attention lets each position compute a weighted combination of all positions, with weights from query-key similarity, so a word can draw directly on the words that matter to it.

9. Write scaled dot-product attention and explain the scaling.

Attention(Q, K, V) = softmax(QKᵀ / √d) V. Without dividing by √d, dot products grow with dimension d, softmax becomes very peaked, and gradients become tiny.

10. Encoder-only, decoder-only, encoder-decoder — when is each used?

Encoder-only models (BERT-style) read the whole input bidirectionally and suit classification and embeddings. Decoder-only models generate left to right and underlie most chat LLMs. Encoder-decoder models suit input-to-output tasks such as translation and summarisation.

Applied questions

11. You have 2,000 labelled support tickets. Fine-tune a transformer or not?

Start with TF-IDF plus logistic regression as a baseline, and a classifier on top of a pre-trained sentence embedding. Fine-tuning can help, but with 2,000 examples the gain is uncertain and the baseline tells you whether it is worth the cost. Hold out a test set by time so you measure on tickets the model has not seen.

12. Tickets mix Hindi and English, sometimes in Roman script. What changes?

Check tokenisation and embedding quality on real samples: transliterated Hindi ("paisa wapas nahi aaya") may be split poorly and embedded far from its English equivalent. Evaluate per language segment rather than overall, consider a multilingual model, and label a sample of code-mixed tickets specifically, because an overall accuracy can hide poor performance on them.

What they check: evaluation by segment, not a named model.

Retrieval systems build on questions 6 and 7; the RAG interview bank continues from there, and the deep learning bank covers the training mechanics. Text classification and embeddings are covered in the NLP module of the Data Science programme.