Trace an LLM request without logging private content

An observability system should answer which stage failed, which release was involved and how long it took. It does not automatically need the user’s entire prompt, the generated response, an email address or a bearer token.

Create a content-minimized trace

The LLMOps and security lab starts with a synthetic request containing an email and invoice reference. It emits correlation and version metadata only.

from llmops_cases import private_trace_case

result = private_trace_case()
assert result["forbidden_present"] == []
assert result["raw_content_logged"] is False
assert result["correlatable"] is True
assert result["trace"]["operation"] == "support.answer"
assert result["trace"]["input_chars"] == 26

The pseudonym is an unsalted teaching digest and is not presented as a production privacy control. A real design needs a keyed pseudonym or an access-controlled identity mapping with rotation and retention rules.

Record operational evidence

Use a trace ID and parent/child spans for ingress, policy, retrieval, generation, validation and tools. Helpful low-content fields include operation name, release, prompt/model/tool/index versions, duration, token counts, retry count, result class and stable error code. Keep tenant and user correlation pseudonymous and scoped to the investigation purpose.

Treat prompts, retrieved passages, tool arguments, tool results and system instructions as sensitive by default. OpenTelemetry’s GenAI conventions explicitly flag several content fields as potentially sensitive. Make content capture opt-in, sampled, access-controlled and time-limited if a justified diagnostic case requires it. Apply redaction before export, not after raw values reach every backend.

Test the telemetry pipeline

Inject synthetic emails, tokens and account numbers, then query every sink: application logs, traces, metrics labels, error reports and queues. High-cardinality content also makes poor metric labels. Verify deletion and retention behaviour, access audit records and export destinations.

The Generative & Agentic AI course connects trace design to privacy, incident analysis and release versioning.

Reproducible exercise

Instrument a local three-stage workflow. Allow only an explicit field list, seed each input with a synthetic secret, and make the test fail if any raw content appears in the exported JSON.

Continue learning

This article is part of the LLMOps, security and operational evaluation sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Measure cost per successful task instead of cost per call.
• Continue with Define latency budgets for retrieval, generation and tools.

References: OpenTelemetry GenAI conventions and convention guidance for sensitive attributes.