Compare AI assistants on the same analyst task fairly

Compare AI assistants using the same business question, source version, permitted tools and grading rules. Record the exact model configuration and preserve failed or incomplete runs. A polished example from one assistant and a difficult example from another do not form a fair comparison.

This article provides a comparison design and a tested scoring illustration. It does not rank live products: the supplied candidates are authored fixtures, and no model API was called.

Define what is being compared

Decide whether the comparison concerns a base model or an entire analyst workflow. If one system receives a calculator and another receives only prose, the result measures the configured systems, not the models in isolation.

For the synthetic completed-order task, each system should receive the same source files, current metric definition and available tool specification. Keep the expected total in the grading material rather than revealing it as the answer in every test prompt.

Record whether the assistant may ask clarifying questions. A workflow that correctly pauses on an ambiguous request should not automatically lose to one that guesses a plausible number.

Use a multidimensional scorecard

Separate exact metric correctness, eligibility, evidence completeness, supported narrative and appropriate abstention. Record latency, tool calls and cost only when actually measured for the selected system and run conditions.

The following code compares two seeded answers to demonstrate a scoring distinction. The labels identify authored cases, not competing products.

from calculator import seeded_cases,check_candidate

cases = {c['case_id']:c for c in seeded_cases()}
selected = ['correct_structured_answer',
            'semantic_blind_spot_requires_human_review']
# These semantic labels are authored reference judgments for this exercise.
semantic_reference = {
    'correct_structured_answer':True,
    'semantic_blind_spot_requires_human_review':False,
}
rows = []
for name in selected:
    structural = check_candidate(cases[name]['answer'])['structured_checks_passed']
    rows.append({'case':name,'structured_facts_pass':structural,
                 'reference_semantic_pass':semantic_reference[name],
                 'both_dimensions_pass':structural and semantic_reference[name]})
assert all(row['structured_facts_pass'] for row in rows)
assert [row['both_dimensions_pass'] for row in rows]==[True,False]
print({'illustrative_scores':rows,'model_runs':0,
       'latency_measurements':None,'cost_measurements':None})

Both candidates pass the structural checker. Only one has a supported narrative under the authored reference judgment. Reporting a tie based solely on structural checks would hide the reason the second answer is unsuitable.

Freeze the comparison protocol

Before running real systems, record the task set, source hashes, prompt, model identifiers, available tools, generation settings, retry policy and evaluation date. Decide how to count timeouts, tool errors and unavailable features.

Use several tasks that represent intended work: an amount calculation, a denominator-sensitive rate, a missing-data question and a definition lookup with a stale document. Do not infer broad analytical quality from one easy total.

Repeat runs when the workflow can vary. Report the number of runs and the distribution of outcomes rather than selecting the best response. With a small sample, avoid presenting minor differences as stable rankings.

Anthropic's evaluation guidance provides context for designing complementary graders. The comparison here remains an original local protocol, not a reproduction of a provider benchmark.

Prevent favorable grading after the fact

Set critical failure rules before inspecting product identities. A wrong currency unit or an unsupported causal recommendation should not be offset by attractive formatting. Where practical, review narrative support without showing which system produced the answer.

Document disagreements between reviewers and refine the rubric with examples. Do not silently change an expected answer to favor a preferred tool. If a source or contract was wrong, correct it openly and rerun every affected system under the same version.

Make the selection decision local to the task

A system that performs well on these questions may still struggle with large schemas, ambiguous requests or long documents. Choose based on the actual workflow, operating constraints and measured failure modes. Do not turn a small internal trial into a universal “best AI for analysts” claim.

Exercise: design a four-task comparison table and specify the expected behavior for an unanswerable question. Leave cost and latency cells marked unmeasured until a real run supplies them.

NeuraPath's Data Analytics with Generative AI course emphasizes evidence-based tool selection. A useful comparison explains the task, conditions and limitations as clearly as the score.

Continue learning

This article is part of the Generative AI for verified analyst work sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Design a read-only SQL tool for an analyst assistant.
• Continue with Document model uncertainty in a management narrative.