Build an evaluation dataset that represents real users

An evaluation set can contain hundreds of polished questions and still test the wrong product. Representation starts with the tasks people attempt, the languages and formats they use, and the failure costs attached to those tasks. It does not start with whichever examples are easiest to write.

Compare the intended and test distributions

The LLMOps and security lab uses ten synthetic support cases. Its intent proportions exactly match an authored 50/30/20 mix while five cases deliberately cover ambiguity, adversarial input or an unsupported request.

from llmops_cases import representative_eval_case

result = representative_eval_case()
assert result["intent_delta_points"] == {"account": 0.0, "billing": 0.0, "technical": 0.0}
assert result["languages"] == ["en", "hi"]
assert result["hard_cases"] == 5
assert result["real_user_data_used"] is False

This proves the fixture has its designed composition. It does not prove that 50/30/20 describes NeuraPath, another company or any live user population.

Build from a sampling frame

Define the unit first: conversation, user goal, document or completed task. Sample a time window that includes weekdays, release periods and known seasonal events. Group related messages into one task so a long conversation does not overpower short tasks. Segment by intent, language, channel, user permission, input length and consequence of error.

Frequency alone is insufficient. Reserve explicit slices for rare high-impact cases, unsupported questions, empty context, conflicting evidence, prompt injection and required abstention. Give each row a stable ID, source class, consent/provenance record, expected behaviour and grading rule. Remove or transform personal data before annotation and restrict access to the raw source.

Report slices, not only an average

A 90% overall score can conceal a failed Hindi slice or a zero-pass high-risk category. Publish counts and outcomes for every material slice, plus confidence intervals where sample sizes support them. Record which rows are synthetic, sampled or manually constructed. Refresh the set when task mix, policy, tools or failure reports change, while retaining a sealed regression subset.

The Generative & Agentic AI course connects dataset design to graders, trace analysis and release decisions.

Reproducible exercise

Create a 40-case local dataset for one bounded assistant. Write the intended mix before authoring rows, add ten hard cases, compute slice counts, and document every gap between the target population and available evidence.

Continue learning

This article is part of the LLMOps, security and operational evaluation sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Continue with Prevent evaluation leakage during prompt iteration.
• Then apply it in Design a prompt-injection test pack with expected safe behaviour.

Reference: OpenAI Evals API.