Measure queue lag and time-to-completion separately

One end-to-end latency number cannot distinguish a starved queue from a slow model or tool. Queue lag measures time from enqueue to start; processing time runs from start to finish; completion time covers the user’s whole wait.

Define the measurable control

Emit queued, started and terminal timestamps under one job id. Measure distributions and age of the oldest eligible job, not just averages. Slice by priority, tenant and workload. Alert on user impact and saturation together so capacity changes target the correct stage.

The AI reliability lab makes the decision reproducible with authored data:

from reliability_cases import queue_metrics_case

result = queue_metrics_case()
assert result["queue_lag"] == [2, 7, 2]
assert result["time_to_complete"] == [8, 11, 12]
assert result["average_lag"] < result["average_total"]
assert result["reported_separately"] is True

The three jobs have lags of 2, 7 and 2 units, while completion times are 8, 11 and 12. Job b waited longest; job c processed longest. Collapsing them would hide two different interventions.

Challenge the result

Retries can create new queue records and reset apparent age. Preserve the root request id and original enqueue time. Exclude cancelled work only through an explicit outcome, otherwise disappearing jobs make latency look healthier.

Keep a reviewable evidence pack:

• timestamp and identifier schema
• lag, processing and completion distributions
• oldest-job and queue-depth dashboard
• retry/cancellation reconciliation

This work aligns with the evaluation, security, cloud operations, reliability and FinOps sequence in the FDE for Professionals course. The linked course describes the learning pathway; this article’s numbers are synthetic and do not report a model, client, audit or production result.

Practice task

Create five jobs with different waits and runtimes. Diagnose which need capacity, dependency tuning or a product-level deadline change.

Continue learning

This article is part of the Advanced AI reliability and assurance sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Design a secret-rotation drill for an integration service.
• Continue with Load-test an AI workflow with realistic failure rates.

Reference: Google SRE Book: Monitoring Distributed Systems.