Measure cost per successful task instead of cost per call

A cheap model call can be expensive work. If an answer fails validation, triggers a retry and then needs human correction, cost per call hides the cost of reaching a useful outcome.

Calculate both denominators

The LLMOps and security lab assigns authored cost units to four calls. Only two satisfy the task acceptance rule.

from llmops_cases import cost_per_success_case

result = cost_per_success_case()
assert result["calls"] == 4
assert result["successes"] == 2
assert result["total_cost"] == 0.14
assert result["cost_per_call"] == 0.035
assert result["cost_per_success"] == 0.07
assert result["currency"] == "authored_units"

Cost per successful task is twice cost per call in this fixture. The values are teaching data, not provider prices.

Define success independently

Write acceptance criteria before comparing configurations: correct resolution, supported claims, valid schema, permitted tool use and any latency boundary. A self-reported “done” is not success. Use deterministic business checks where possible, human labels for judgement, and a calibrated grader only for criteria it can assess.

Aggregate every charge attached to one task ID: input and output tokens, cached-token rates, embeddings, reranking, tool APIs, retries, moderation, storage and review time. Avoid double counting shared retrieval or batch work. Report currency, pricing date and whether taxes, infrastructure and labour are included.

Use paired tasks when comparing configurations. A cheaper candidate that receives an easier sample is not a fair result. Report success rate and cost distribution together; averages can conceal a small set of runaway loops. P50, p95 and maximum task cost reveal the tail.

Turn the ratio into a decision

Cost per success supports routing, caching and model-selection experiments, but it cannot replace safety gates. Compare only candidates that first meet critical acceptance rules. If no candidate is acceptable, the metric should not select the least bad one for release.

The Generative & Agentic AI course uses this ratio with evaluation sets, retry budgets and production outcomes.

Reproducible exercise

Add retries and reviewer minutes to ten task records. Recompute total cost, cost per success and p95 task cost, then explain why one apparently cheap configuration loses after failed tasks are included.

Continue learning

This article is part of the LLMOps, security and operational evaluation sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Design a prompt-injection test pack with expected safe behaviour.
• Continue with Trace an LLM request without logging private content.

Reference: OpenAI Evals API.