Write a model selection decision record

A leaderboard or one demo cannot explain why a model fits an application. A decision record ties selection to the exact task set, constraints, configuration and evidence available at a date.

Make incompleteness visible

The offline foundations lab creates an authored record.

from foundation_cases import selection_record_case

result = selection_record_case()
assert result["complete"] is True
assert result["approved"] is False
assert result["benchmark_claim"] is False
assert result["record"]["reviewer"] == "pending"
print(result["record"]["limitations"])

The record selects a fictional small candidate for a refund evidence assistant and rejects large because the fixture latency exceeds its gate. It explicitly says no live model ran, there is no production traffic and human review is pending. Completing required fields does not create approval.

Record enough to reproduce the comparison

Include task and user, task-set version, candidates and model snapshots, prompts, examples, tools, retrieval, decoding settings, region, date and execution code. Define quality, critical-invariant, latency, cost, privacy and deployment gates before results.

Report every candidate, not only the winner. Preserve rejection reasons and per-slice regressions. Link to run artifacts rather than copying rounded headline scores without denominators.

State the decision’s lifetime

List assumptions and triggers for reevaluation: provider deprecation, price or context changes, new languages, drift in requests, a security incident or a revised business threshold. Name an owner and approved fallback. If the evidence is an offline fixture, the next action is a controlled live evaluation—not release.

Separate recommendation, human review, approval and production status. Store the chosen complete configuration behind a versioned pointer so rollback does not require reconstructing it from prose.

The Generative & Agentic AI course uses this record to connect fundamentals with evals, LLMOps and capstone defence.

Exercise

Evaluate two current candidates on one fixed set and fill the record. Ask a reviewer to reproduce the choice without speaking to you. Add one adverse result and one trigger that would reverse the selection.

Continue learning

This article is part of the LLM fundamentals and prompt design sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the preceding task in Create a small labelled dataset for an LLM application.
• Return to the cluster foundation in Tokens and context windows: budget a real document task.

Reference: NIST AI RMF playbook.