Create hard-negative examples for a RAG benchmark

Random unrelated negatives are easy. Hard negatives share vocabulary or topic with the query but fail because of version, clause, entity, date or access. They expose whether retrieval understands the boundary that matters.

Pair positives with plausible mistakes

The retrieval evaluation lab defines two authored pairs.

from retrieval_cases import hard_negative_case

result = hard_negative_case()
assert result["count"] == 2
assert result["all_distinct"] is True
assert result["random_negatives_only"] is False
print(result["rows"])

For refund proof, the positive is current refund-v2 and the hard negative is superseded refund-v1. For travel approval, the wrong clause is rail preference. The corpus is synthetic.

Mine negatives responsibly

Sources include top false results from a baseline, adjacent clauses, old versions, similar product names, another tenant and documents with the right number but wrong unit. Human reviewers must confirm they are truly irrelevant under the query’s date and user scope.

Store the reason. “Same topic, superseded rule” supports a different fix from “same term, wrong entity.” Avoid labelling an additional valid answer negative merely because the original label set was incomplete.

Use them across stages

Measure whether retrieval ranks the positive above the hard negative, whether reranking repairs the order and whether generation cites the correct version. Add critical cases to regression tests. Keep unauthorized negatives out of model context even during evaluation where policy requires it.

Balance difficulty with representativeness; a benchmark made only of adversarial pairs may not reflect daily traffic. Report common and hard slices separately.

The Generative & Agentic AI course uses hard negatives to evaluate chunking, retrieval, reranking and citations.

Exercise

Mine two hard negatives for each of 30 queries. Label failure reasons, double-review ten pairs and compare BM25/dense ranking margins without changing the release set.

Continue learning

This article is part of the Retrieval quality and grounded answers sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Grounded answers: distinguish unsupported and incorrect statements.
• Continue with Test a knowledge assistant on unanswerable questions.

Reference: BEIR benchmark paper.