Prevent evaluation leakage during prompt iteration

Every time a developer reads a failing holdout row and changes the prompt for that row, the holdout becomes development data. The next score may show memory of visible examples rather than better performance on unseen work.

Split related records together

The LLMOps and security lab creates two messages for each synthetic account. A naive row split puts account A2 in both partitions. A grouped split removes that overlap.

from llmops_cases import leakage_case

result = leakage_case()
assert result["naive_account_overlap"] == ["A2"]
assert result["grouped_account_overlap"] == []
assert result["holdout_ids"] == ["E7", "E8"]
assert result["holdout_labels_hidden"] is True

The same principle applies to near-duplicate documents, paraphrases of one question, events from one customer, and chunks from one source file. Random row splitting does not protect against these relationships.

Use three evidence zones

Keep a visible development set for prompt changes, a regression set for known failures and a sealed holdout for less frequent release decisions. Assign immutable IDs and store split membership separately from prompts. Only the evaluation owner should expose holdout labels. A release report can show aggregate and slice results without exposing every answer to the prompt author.

Version the prompt, model snapshot, tool schemas, retrieval index, grader and dataset together. If developers inspect holdout failures, record the exposure and retire or reclassify those rows. Replace them from the same sampling frame before claiming an independent result.

Contamination can also enter indirectly. Few-shot examples, fine-tuning data, retrieval documents and model-judge rubrics may reproduce evaluation content. Hash-based duplicate detection catches exact copies; semantic review and provenance records are needed for paraphrases and shared source material.

Decide before seeing the score

Write acceptance thresholds, slices and exclusion rules before the run. Do not remove difficult rows after they fail without a documented data-quality reason. Report the number of iterations made against the development set and the number of holdout looks.

The Generative & Agentic AI course treats evaluation integrity as part of prompt and release engineering.

Reproducible exercise

Generate twenty grouped records, compare random and group-aware splits, then add a duplicate detector. Intentionally leak one entity and prove the gate fails before any prompt comparison is accepted.

Continue learning

This article is part of the LLMOps, security and operational evaluation sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Build an evaluation dataset that represents real users.
• Continue with Design a prompt-injection test pack with expected safe behaviour.

Reference: NIST AI RMF Generative AI Profile.