Version prompts, models, tools and evaluation sets together

“Prompt v5 improved quality” is incomplete if the model alias, tool schema, retrieval index or grader changed during the same comparison. Reproducibility needs a release identity for the whole system.

Hash the dependency bundle

The LLMOps and security lab serializes a fixture manifest and proves that one prompt change produces a new digest.

from llmops_cases import version_bundle_case

result = version_bundle_case()
assert result["complete"] is True
assert len(result["digest"]) == 64
assert result["digest"] != result["changed_digest"]
assert result["change_detected"] is True
assert result["bundle"]["eval_set"] == "support-eval-8"

A digest detects difference; it does not explain whether the change is safe. Store the readable manifest beside it.

Pin every behaviour-changing input

Record application commit, prompt/template and examples, exact model snapshot or provider revision where available, decoding settings, tool contracts and adapters, retrieval corpus/index/embedding versions, policy configuration, evaluator code, evaluation rows and grader. Include dependency locks and feature flags that select runtime paths.

Use immutable artifact IDs rather than mutable names such as “latest.” If a provider exposes only an alias, record the alias, response model identifier and run timestamp, then acknowledge the reproducibility limit. Protect manifests from silent edits and retain the artifact access needed for rollback.

Attach every offline run, shadow comparison and production trace to a release ID. That makes it possible to ask whether one regression began with prompt-5, tool-2.1 or index-12. Treat grader changes separately: rescoring old outputs with a new grader can distinguish system movement from measurement movement.

Make comparison fair

Change one planned variable where possible and run both candidates on identical task IDs. If multiple dependencies change, describe the bundle comparison without assigning improvement to one component. Record exclusions and failed runs.

The Generative & Agentic AI course uses a release manifest as the bridge from development evidence to deployment and rollback.

Reproducible exercise

Build two manifests, change only a tool schema, and create a machine-readable diff. Reject any evaluation record that lacks a complete release digest.

Continue learning

This article is part of the LLMOps, security and operational evaluation sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Test an AI system when its upstream service is unavailable.
• Continue with Release an AI feature using shadow evaluation.

Reference: OpenAI Evals API.