Reranking: measure the quality and latency trade-off

A reranker can improve the order of retrieved candidates by scoring query-document pairs more deeply. It also adds compute and latency. Define both gates before deciding.

Preserve a no-winner outcome

The retrieval evaluation lab uses authored results.

from retrieval_cases import rerank_case

result = rerank_case()
assert result["gate"] == {"mrr_min": 0.70, "p95_ms_max": 100}
assert result["eligible"] == []
assert result["selected"] is None
assert result["reranker_executed"] is False

The no-reranker fixture has MRR 0.64 at p95 45 ms; reranking reaches 0.73 but p95 118 ms. Neither clears both gates. These values are teaching fixtures, not measured systems.

Measure on the same candidates

Freeze the first-stage retriever, candidate depth, corpus and labels. A reranker cannot recover evidence absent from its candidate list, so report first-stage recall separately. Measure MRR/nDCG, recall after truncation, critical-query transitions and duplicate context.

Benchmark end-to-end p50/p95 under realistic batch/concurrency and include model loading, network and retries. Calculate cost per successful task, not only per rerank call. Inspect language and document-length slices.

Consider bounded designs

Rerank only when query difficulty or candidate ambiguity justifies it, or use a smaller candidate depth after testing recall. Cache only when query, permissions, corpus and reranker version match. Preserve authorization filters before reranking.

The Generative & Agentic AI course evaluates reranking as a conditional system choice and keeps negative decisions.

Exercise

Run a current reranker on fixed candidates from 50 questions. Predeclare quality and p95 gates, report per-query gains/regressions, and keep “do not adopt” as a valid decision.

Continue learning

This article is part of the Retrieval quality and grounded answers sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Metadata filters: enforce eligibility before similarity ranking.
• Continue with Query rewriting: detect when the rewrite changes intent.

Reference: Sentence Transformers cross-encoder documentation.