# Analyst AI verification protocol

Version: original teaching protocol, 20 September 2026. Scope: the synthetic January completed-order exercise and a separate January 5–12 paid-event weekly report. Do not mix their populations, amounts or timezones. Candidate responses are authored fixtures; no live model performance is measured.

## Required evidence and gates

| Gate | Evidence | Local pass criterion | Failure response |
|---|---|---|---|
| Source and contract | Raw files, definitions, hashes and eligible IDs | Reference matches the audited fixture; grain, period and unit explicit | Stop the affected calculation; investigate source or contract |
| Amount and denominator | Independent expected amount/count and eligible sets | Exact integer amounts/counts and complete unique identifiers | Reject candidate; repair calculation |
| Retrieval | Document ID, approved metadata, effective period, metric ID | Exactly one applicable definition or explicit clarification | Do not substitute a stale or arbitrary definition |
| Citation | Source identity plus claim-level review | Metadata valid and each material claim supported by suitable evidence | Correct citation, remove claim or request missing evidence |
| Tool boundary | Allowed and denied SQL cases | Valid aggregate succeeds; forbidden operations and excess output/work fail | Do not expose the tool until the failed boundary is understood |
| Narrative | Human review against source and calculation | No unsupported causal, trend, confidence or accounting claims | Changes requested; structural pass alone is insufficient |
| Review record | Exact packet identity and decision | Current packet matches recorded review; source changes invalidate it | New review required; no automatic carry-forward |
| Distribution | Explicitly authorized action in a real application | Not implemented in this local lab | All artifacts remain local and not sent |

## Executed local commands

Run from `11-Blog-Programme/labs/analyst-ai` with Python 3.12 or a compatible standard-library runtime:

```powershell
python calculator.py
python verify_extensions.py
python weekly_assistant.py
```

The first command checks 11 authored answer cases: nine expected structural rejections, one supported structured answer and one intentional semantic blind spot. The second checks 26 SQL/retrieval/citation/review behaviors. The third checks the separate weekly scaffold, including a seeded unsupported narrative that passes its narrow structural checker. Read the generated JSON results for actual counts.

Matching an expected blind-spot result tests our understanding of the checker; it does not approve that answer. No aggregate average may compensate for a wrong critical amount, an unsupported material conclusion or a failed tool boundary.

## Human review worksheet

For each answer, record the business question, evidence version, numeric checks, citations, claim-by-claim support, missing information and intended use. The reviewer should identify the exact unsupported claim in each seeded causal example and reject it. Real reviewer names and decisions remain pending; synthetic test records are labeled and are not operational approvals.

## Future live-model evaluation

Record model identifier, prompt, tool configuration, source hashes, generation settings, raw responses, tool traces, retries, errors and review labels. Keep held-out cases separate from prompt development. Measure latency and cost from actual runs if relevant. Include missing-data questions with an expected clarification or abstention response. Do not infer a vendor ranking from these authored fixtures.

## Limits

This is a finite teaching suite, not a security certification, privacy certification, real-world error-rate estimate or production release approval. Exact-term glossary retrieval is not semantic search. Local hash binding is not authenticated sign-off. All public demonstrations use invented data; no client outcomes are claimed.