Measure multilingual model quality by language

Pooling all languages rewards the largest slice. Report each language, its sample size and task mix, then show macro and micro summaries with their different meanings.

Compare the averages

The adaptation lab uses authored pass counts for English, Hindi and Tamil.

from adaptation_cases import multilingual_case

result = multilingual_case()
assert round(result["micro_rate"], 3) == 0.740
assert round(result["macro_rate"], 3) == 0.733
assert result["lowest_language"] == "hi"
assert result["fixture_outputs"] is True

Hindi has the lowest fixture pass rate even though Tamil has fewer rows. No multilingual model ran.

Make the slices comparable

Use native-language task definitions and qualified annotators rather than translating an English-only test blindly. Balance intents, difficulty, formality, scripts, code-switching and regional variants. Record whether each row is original or translated and keep source-related variants in one split.

Define semantic acceptance separately from surface wording. Exact match may be inappropriate for valid paraphrases, while names, numbers and tool arguments need stricter checks. Measure task success, unsupported claims, abstention, safety and critical entities per language.

Report row counts, uncertainty and missing coverage. Macro average weights languages equally; micro average weights rows. Neither should replace the language table. Route low-volume or high-impact slices to human review and collect errors under governed consent.

The Generative & Agentic AI course integrates multilingual slices with prompt, adaptation and production monitoring.

Exercise

Create parallel and native examples for three languages, blind-review outputs, calculate both averages and show one conclusion that changes when the row distribution is reweighted.

Continue learning

This article is part of the Model adaptation and multimodal tasks sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Build a human correction loop for extracted documents.
• Continue with Synthetic training data: audit diversity and label quality.

Reference: NIST AI RMF Generative AI Profile.