Evaluate retrieval recall at k with labelled evidence

Generation cannot cite evidence that retrieval never supplied. Recall@*k* measures whether at least one labelled relevant item appears among the first *k* eligible results for an answerable question.

Keep the denominator visible

The retrieval evaluation lab calculates recall from its executed BM25 rankings.

from retrieval_cases import recall_case

result = recall_case()
assert result["denominator"] == 5
assert 0 <= result["recall_at_1"] <= result["recall_at_3"] <= 1
assert len(result["rows"]) == 5
print(result["rows"])

The sixth corpus question is unanswerable and belongs in abstention evaluation rather than retrieval recall. The five-row result is a synthetic unit test.

Label minimum sufficient evidence

For each answerable query, identify one or more acceptable source spans and version/date conditions. Decide whether a query requires all pieces or any one. Use separate metrics for multi-hop requirements.

Apply authorization and effective-version filters before ranking. A restricted or stale relevant document is not eligible evidence for the current user/task. Report recall by query type, language, table, identifier and hard-negative slice.

Choose k from the application

Larger *k* can improve recall while increasing tokens, duplicates and generation confusion. Compare recall@1/3/5 with context cost and answer support. Also inspect rank-sensitive metrics such as MRR where position matters.

Preserve failed query IDs and retrieved results. When labels change, version the task set. Double-review a sample because missing acceptable evidence can make the retriever look worse than it is.

The Generative & Agentic AI course places retrieval metrics before answer-generation scores in the RAG report.

Exercise

Label evidence for 50 answerable and 15 unanswerable questions. Compute recall@1/3/5, break it down by five slices and review every miss before changing the retriever.

Continue learning

This article is part of the Retrieval quality and grounded answers sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Multi-query retrieval with bounded cost.
• Continue with Measure citation correctness separately from answer quality.

Reference: BEIR benchmark paper.