Build a retrieval regression suite

Aggregate retrieval can improve while a critical policy query regresses to an obsolete source. A regression suite protects named query-to-evidence contracts and records intentional changes.

Fail a critical transition

The retrieval evaluation lab compares baseline and candidate top results.

from retrieval_cases import regression_case

result = regression_case()
assert result["regressions"] == ["Q1"]
assert result["critical_regressions"] == ["Q1"]
assert result["release_pass"] is False

Q1 changes from current refund-v2 to superseded refund-v1, so release fails. The small mapping is a deterministic fixture.

Store contracts, not snapshots alone

For each case, keep query, user/date scope, acceptable evidence IDs/spans, forbidden hard negatives and minimum rank. Include unanswerable cases where no eligible result should trigger an answer.

Run after source, extraction, chunking, embedding, filters, fusion or reranker changes. Report pass→fail and fail→pass transitions plus aggregate metrics. Review label changes as versioned decisions rather than silently updating expected IDs.

Prioritize critical cases

Authorization, current version, safety and required abstention can be hard gates. Lower-risk ranking changes may use thresholds and review. Keep the suite representative; add incident cases without letting it become only a museum of rare failures.

Save complete execution keys and source manifests. Cached results are reusable only when all dependencies match. A passing suite still needs human review and cannot prove coverage of unknown failures.

The Generative & Agentic AI course connects regression suites to versioned release decisions and rollback.

Exercise

Create 40 contracts, including ten hard gates. Change one chunker and one source version separately, preserve transitions and require explicit review for every expected-label update.

Continue learning

This article is part of the Retrieval quality and grounded answers sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Compare chunking strategies without changing multiple variables.
• Continue with Design a RAG release report for a reviewer.

Reference: OpenAI evaluation best practices.