Define latency budgets for retrieval, generation and tools

An end-to-end latency target tells users what to expect, but it does not tell engineers which stage consumed the budget. Split the objective across retrieval, generation, tools and orchestration before optimization begins.

Find the local breach

The LLMOps and security lab compares authored stage timings against a 1,600 ms budget.

from llmops_cases import latency_budget_case

result = latency_budget_case()
assert result["end_to_end_budget_ms"] == 1600
assert result["observed_total_ms"] == 1515
assert result["breached_stages"] == ["generation"]
assert result["fixture_timings"] is True

The total remains inside its limit while generation exceeds its allocation. That can matter because the unused headroom may be reserved for queueing, network variation or a second tool call. These numbers are fixtures, not measured provider latency.

Start from the user journey

Choose separate objectives for interactive, streaming and asynchronous tasks. Time from accepted request to usable outcome, not only the model API. Include admission control, queueing, authentication, retrieval, reranking, generation, tool round trips, validation, retries and response delivery.

Define percentile and window: “p95 under 2.5 seconds over 30 minutes” is testable; “fast” is not. Also set per-stage timeouts and a total deadline. A child operation should receive only the remaining budget. Cancellation must propagate so timed-out work does not continue spending money.

Measure cold and warm paths, cache hits and misses, different output lengths, tool counts and failure cases. Preserve the trace ID and versions so a percentile shift can be tied to a release. Avoid unbounded metric labels such as raw prompts.

Degrade deliberately

When budget is nearly exhausted, choose a defined behaviour: omit an optional enrichment, return a status for asynchronous work, use a qualified fallback that still passes acceptance, or fail clearly. Never skip authorization or output validation to save milliseconds.

The Generative & Agentic AI course combines latency budgets with cost-per-success, traces and fallback tests.

Reproducible exercise

Add monotonic timers to a local four-stage workflow, inject a 100 ms generation delay and assert both the stage alert and total deadline behaviour. Report p50 and p95 across at least thirty fixture runs.

Continue learning

This article is part of the LLMOps, security and operational evaluation sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Trace an LLM request without logging private content.
• Continue with Design model fallback without silently lowering acceptance criteria.

Reference: OpenTelemetry trace semantic conventions.