RAG interview questions on chunking, retrieval and evaluation

Pipeline overview

Q1. Describe a RAG pipeline in five steps.

Ingest and clean documents; split them into chunks with metadata; embed and index the chunks; at query time retrieve and optionally rerank the top candidates; pass the best chunks with instructions to the model and return an answer with citations.

Q2. When is RAG the wrong tool?

When the task needs reasoning over a whole dataset rather than a few passages — "what is total revenue by region?" is a SQL question, not a retrieval one. Also when the answer must be computed exactly, or when the knowledge is small and stable enough to fit in a well-tested prompt.

Interviewer checks: that you do not reach for RAG by reflex.

Chunking

Q3. How do you choose chunk size?

Small chunks retrieve precisely but lose context; large chunks keep context but dilute the embedding and waste prompt tokens. Start from the document's structure — sections, headings, FAQ entries — rather than fixed character counts, then compare two or three sizes on a retrieval test set. Overlap of a sentence or two helps when answers straddle a boundary.

Q4. A policy PDF has tables. What goes wrong?

Naive text extraction flattens tables into a stream where headers and values separate, so a chunk may contain "₹5,000" with no indication of what it limits. Extract tables with structure, convert each row to a sentence with its headers, or keep the table as one chunk with a descriptive title.

Q5. What metadata would you store with each chunk?

Source document, section heading, version or effective date, and access permissions at minimum. Dates allow filtering out superseded versions; permissions let you filter before retrieval so a user never receives text they cannot see.

Retrieval

Q6. Vector search misses queries containing a product code like "PLN-4471". Why, and what do you do?

Embeddings capture meaning, and codes have little meaning to capture, so similar-looking codes may be treated as near-identical or ignored. Add keyword search (BM25) and merge the two ranked lists — reciprocal rank fusion is a simple method — or route queries that match a code pattern to an exact metadata filter.

Q7. What does a reranker add?

A first-stage retriever is fast but coarse. A cross-encoder reranker reads the query and each candidate together and scores relevance more accurately, at higher cost per pair. Retrieve 20–50 candidates cheaply, rerank, keep the top few.

Evaluation

Q8. How do you evaluate retrieval separately from the answer?

Build a set of questions, each labelled with the chunk IDs that contain the answer. For each question, check whether a relevant chunk appears in the top k (recall@k) and how high the first relevant chunk ranks (mean reciprocal rank).

# Hypothetical labelled evaluation set: question -> relevant chunk IDs
relevant = {
    "q1": {"c12"},
    "q2": {"c40", "c41"},
    "q3": {"c7"},
    "q4": {"c88"},
}
# What the retriever returned, in rank order
retrieved = {
    "q1": ["c12", "c3", "c9", "c30", "c2"],
    "q2": ["c5", "c41", "c40", "c6", "c1"],
    "q3": ["c2", "c8", "c19", "c4", "c7"],
    "q4": ["c11", "c14", "c15", "c16", "c17"],
}

def recall_at_k(k):
    hits = sum(bool(set(retrieved[q][:k]) & relevant[q]) for q in relevant)
    return hits / len(relevant)

def mrr():
    total = 0.0
    for q, docs in retrieved.items():
        rank = next((i + 1 for i, d in enumerate(docs) if d in relevant[q]), None)
        total += 1 / rank if rank else 0
    return total / len(relevant)

for k in (1, 3, 5):
    print(f"recall@{k}: {recall_at_k(k):.2f}")
print(f"MRR: {mrr():.3f}")

Recall@1 is 0.25, recall@3 is 0.50 and recall@5 is 0.75; MRR is 0.425. Question 4 is never retrieved — that failure should be investigated first, because no generation change can fix it. If you pass only the top 3 chunks to the model, recall@3 is the number that bounds answer quality.

Q9. How do you evaluate the generated answer?

Separately score faithfulness (every claim supported by the retrieved chunks), correctness against a reference answer, and appropriate refusal when the chunks do not contain the answer. An LLM grader can scale this, but calibrate it against human labels on a sample and report where they disagree.

Q10. Your answers got worse after re-indexing. How do you debug?

Rerun the retrieval evaluation set. If recall@k dropped, compare what changed — chunking, embedding model, cleaning — and inspect the questions that stopped hitting. If recall is unchanged, the regression is in the prompt or the generation step. Keeping the evaluation set versioned is what makes this a ten-minute diagnosis instead of a week.

Production

Q11. What would you monitor after launch?

Share of queries with no chunk above a similarity threshold (a sign of missing content), user feedback and escalation rate, citation click-through, latency by stage, cost per query, and document freshness. Sample real queries weekly, label them, and add the failures to the evaluation set.

What distinguishes strong candidates

They split every problem into "did we retrieve the right text?" and "did we use it correctly?", and they have a number for each. They also know the non-RAG answer — SQL, a rules table, a smaller prompt — when it fits better.

For the model-side questions that usually precede these, see LLM and generative AI interview questions. Embeddings and similarity are covered in the NLP bank. Building and evaluating a retrieval pipeline is a core project in the Generative and Agentic AI programme.