Create an analyst AI verification protocol with pass criteria

An analyst AI verification protocol should say what evidence is required, what constitutes failure and what happens after a failed check. “Review for accuracy” is too vague to reproduce. Define critical gates for calculations, source applicability, supported claims and permitted actions before inspecting the answer.

The supplied protocol covers a small synthetic teaching workflow. Its tests are executed locally, but it does not certify a production system or measure a model's real-world error rate.

Make acceptance observable

For the completed-order exercise, an acceptable structured answer must identify the current metric, January period, Asia/Kolkata interpretation, INR currency, paise unit, eight eligible orders and 104,000-paise amount. Its evidence identifiers must match the eligible set without duplicates.

Those are exact assertions. Narrative support is a different gate: a causal claim about a campaign cannot pass because the dataset contains no causal design. A correct total does not compensate for an unsupported recommendation.

The full protocol records source, calculation, retrieval, citation, tool, narrative and review requirements separately. Distribution is explicitly absent from this local exercise.

Run the available checks

from calculator import seeded_cases,check_candidate
from verify_extensions import verify as verify_extensions
from weekly_assistant import verify as verify_weekly

cases = seeded_cases()
for case in cases:
    assert check_candidate(case['answer'])['structured_checks_passed']==case['expected_pass']
extensions = verify_extensions()
weekly = verify_weekly()
assert len(cases)==11
assert extensions['checks_passed']==26
assert weekly['checks_passed']==12
assert extensions['live_model_calls']==weekly['live_model_calls']==0
print({'authored_answer_cases':11,'extension_checks':26,'weekly_checks':12,
       'real_human_approval':'pending','production_release':'not_evaluated'})

These counts describe three local test groups, not independent samples from a business population. Do not add them together and present the total as a model accuracy denominator.

The answer cases include an intentional semantic blind spot. Matching its expected structural pass confirms the checker's limitation; it does not make the answer acceptable for publication.

Specify the response to failure

A score average should not turn one of these critical failures into an approval. Formatting quality and fluency can be assessed, but they do not offset an incorrect monetary amount or unauthorized action.

Preserve versions and reviewer evidence

Retain the source hashes, contract, tool configuration, candidate answer and test results. For human review, record the material claims, supporting evidence, unresolved questions and intended use of the report.

The local review component binds a record to an exact packet hash. It does not authenticate the reviewer or prove that review occurred. Keep actual sign-off pending until a real reviewer performs it through an appropriate process.

If the source changes, rerun the reference calculation and affected checks. If the model or prompt changes in a future live implementation, rerun the relevant evaluation set. Do not carry forward a pass simply because the report title is unchanged.

Extend the protocol without inventing results

For a real model evaluation, record the model identifier, prompt, generation settings, source version, tools, raw responses, retries and errors. Keep development and held-out cases separate. Measure cost and latency from actual runs rather than filling a comparison table with estimates presented as observations.

The current fixture demonstrates arithmetic errors, stale sources, tool restrictions and review-version handling. It does not cover every document format, database engine or business question. Add cases based on the intended workflow and observed failures.

Exercise: write a pass criterion for an unanswerable refund-timing question. Specify the expected limitation statement and explain why inventing a precise amount should fail even if the response contains a valid glossary citation.

NeuraPath's Data Analytics with Generative AI course connects AI use with explicit acceptance criteria. A protocol is useful when another analyst can reproduce its checks and understand what remains unverified.

Continue learning

This article is part of the Generative AI for verified analyst work sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Use synthetic data when demonstrating AI analysis.
• Continue with Portfolio project: a verified weekly business reporting assistant.