Evaluate prompt changes on a fixed task set

Testing a new prompt on whichever examples inspired the edit creates selection bias. Freeze representative task IDs and labels first, run every candidate on the same cases and inspect regressions as well as the average.

Compare identical cases

The offline foundations lab uses five authored labels and predictions.

from foundation_cases import prompt_eval_case

result = prompt_eval_case()
assert result["set_frozen_before_comparison"] is True
assert result["scores"] == {"v1": 0.6, "v2": 0.8}
assert result["selected"] == "v2"
print(result["scope"])

Prompt v2 matches four of five labels versus three of five for v1. These are fixture predictions, not outputs from an LLM or evidence that any live prompt performs at 80%.

Define the task set from risk

Include common requests, edge cases, unsupported questions, conflicting sources, adversarial content and important user or language slices. Give every case a stable ID, input version, expected behaviour and label rationale. Separate deterministic rules from judgments that require human review.

Do not expose the final set during every prompt edit. Use a development set for iteration and retain a release set for a less biased gate. When labels are subjective, measure reviewer agreement and adjudicate disputes without silently rewriting the answer key to favour a candidate.

Read the diff, not just the score

Produce a case-level transition table: pass→pass, fail→pass, pass→fail and fail→fail. A one-point gain can conceal a new safety or authorization regression. Weighting may reflect impact, but preserve raw counts and denominators.

Record model snapshot, prompt version, examples, tool configuration, decoding parameters, date, latency and cost. Cache carefully: reuse is valid only when the complete execution key is unchanged. Rerun after provider or retrieval changes.

The Generative & Agentic AI course treats the fixed task set as a deliverable, alongside prompt contracts and version records.

Exercise

Create 30 labelled cases across six slices. Compare two prompts, generate the transition table and require zero regressions on two critical invariants. Have a second reviewer label a sample and document disagreements.

Continue learning

This article is part of the LLM fundamentals and prompt design sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Few-shot examples: choose cases that expose the boundary.
• Continue with Context ordering: test rather than assume.

Reference: OpenAI evaluation best practices.