BM25 versus dense retrieval on the same question set

BM25 and dense retrieval fail differently. Lexical ranking can miss a paraphrase; dense retrieval can blur exact identifiers or versions. Compare them under identical conditions rather than quoting unrelated benchmark scores.

Hold the test set fixed

The retrieval evaluation lab executes BM25 and compares it with authored dense rankings.

from retrieval_cases import dense_comparison_case

result = dense_comparison_case()
assert result["same_question_ids"] is True
assert result["dense_model_executed"] is False
assert 0 <= result["bm25_recall_at_3"] <= 1
assert 0 <= result["dense_fixture_recall_at_3"] <= 1

The dense rows demonstrate comparison arithmetic only. They are not embedding-model outputs or benchmarks. BM25 is the only executed retriever in this lab.

Control every other variable

Use the same source snapshot, chunking, metadata eligibility, query set, relevance labels and *k*. Pin the embedding model and normalization for a live dense run. Record index/query latency and vector storage, including external API cost where applicable.

Report per-query transitions. Exact policy numbers may favour BM25; synonyms may favour dense retrieval. Break down results by identifiers, paraphrase, table, language and unanswerable cases. Inspect whether either method retrieves a superseded but topically similar document.

Choose from the workflow

Retrieval recall is necessary but not sufficient. Evaluate source support in the generated answer, context redundancy and authorization. If methods have complementary gains, test a predeclared fusion rule. If lexical clears the release gates, a vector service may not yet justify its cost.

The Generative & Agentic AI course makes model claims conditional on an actual, versioned run. Offline fixtures remain labelled.

Exercise

Run one current embedding model against your BM25 baseline on 50 cases. Preserve exact configurations and compute per-slice recall, latency and storage. Write a decision that can select BM25.

Continue learning

This article is part of the Retrieval quality and grounded answers sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Build a keyword retrieval baseline before vector search.
• Continue with Hybrid retrieval with reciprocal rank fusion.

Reference: Qdrant vector concepts.