Validate document encoding and language before indexing

Mojibake and unknown-language content can enter an index without crashing. The result is poor retrieval, damaged citations and misleading claims about multilingual support. Validate bytes and language before chunking.

Reject invalid decoding

The RAG ingestion lab checks three authored byte samples.

from ingestion_cases import encoding_language_case

result = encoding_language_case()
assert result["indexed_ids"] == ["utf8-en", "utf8-hi"]
assert result["rejected_ids"] == ["bad"]
assert result["rows"][2]["encoding_valid"] is False

English and Hindi UTF-8 samples pass the declared support set. Invalid bytes fail rather than being silently replaced. The language labels are authored; no language detector runs.

Preserve bytes and decoding evidence

Hash the original file, detect or obtain the declared encoding, decode strictly and record the result. Replacement characters may be acceptable only under a documented review rule. Normalize Unicode deliberately while retaining the original source for citation.

Run language identification on sufficiently long samples and preserve confidence plus mixed-language status. Calibrate it on your corpus; short names, code and tables can confuse detectors. Route unsupported or uncertain documents rather than assigning the nearest language.

Evaluate multilingual retrieval

Language support needs labelled queries and evidence per language, not only successful decoding. Test same-language and cross-language questions, transliteration, numerals and domain terms. Report denominators and use an embedding/retrieval path evaluated for those slices.

Keep language metadata available for filters and output policy. A source may contain sensitive scripts or personal information even if the model does not support its language; privacy checks still apply.

The Generative & Agentic AI course connects encoding/language validation to multilingual retrieval evaluation and exception queues.

Exercise

Create UTF-8, UTF-16, legacy-encoded and damaged samples in two languages. Decode under explicit policies, compare a language detector with human labels and verify uncertain sources cannot silently reach the active index.

Continue learning

This article is part of the RAG ingestion and document preparation sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Build an ingestion error queue for failed documents.
• Continue with Create a synthetic policy corpus for RAG practice.

Reference: Python Unicode HOWTO.