Explain model latency percentiles to a product owner

Average latency can hide a slow tail. The 95th percentile is a value at or below which roughly 95% of observations fall under the chosen calculation method. It says that about 5% are slower; it does not say the slowest request took that long.

Read one authored sample

The deployment lab uses 20 invented latencies ranging from 18 to 900 milliseconds.

from deployment_cases import latency_percentile_case

result = latency_percentile_case()
assert result["p99_ms"] >= result["p95_ms"] >= result["p50_ms"]
print(result["p50_ms"], result["p95_ms"], result["p99_ms"], result["maximum_ms"])

The median is 29 ms, p95 is 472.5 ms, p99 is 814.5 ms and maximum is 900 ms under NumPy’s interpolation. Fifteen percent of samples exceed 200 ms. The median experience looks fast while three of twenty requests cross 200 ms.

Tie latency to a journey

State whether timing covers model-only compute, server processing or the full user journey. A product owner needs to know what waits: form submission, fraud review, autocomplete or batch scoring. Different actions justify different objectives.

Report volume with percentiles. A p99 based on 20 samples is unstable. At one million requests, the slowest 1% represents 10,000 events. Slice cold starts, payload sizes, regions, model versions and error states.

Set objectives and budgets

Allocate latency across client, network, service, feature lookup and model. Define an objective over a rolling window and error budget. Monitor timeouts and cancellations, since missing slow requests can bias reported percentiles.

The Data Science course teaches latency as an operating measure connected to user impact.

Exercise

Generate 10,000 latencies with rare cold starts. Compare mean, median, p95, p99 and max; then write a product explanation with affected request counts and proposed investigation.

Continue learning

This article is part of the Model deployment and MLOps sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Load-test a prediction API with realistic payloads.
• Continue with Secure model artifacts and avoid untrusted deserialization.

Reference: Google SRE service-level objectives.