Handle Devanagari digits and Hindi text in pandas without losing rows

Where this data comes from

Survey tools with a Hindi interface, scheme beneficiary lists, cooperative society registers, field-staff forms filled on phones with a Hindi keyboard: all of them produce files where the same column holds ३४ from one respondent and 34 from the next. Names and places arrive in Devanagari, sometimes typed on different keyboards that encode the same visible letter differently. None of this is wrong data. It is simply data that exact-match tools treat as different.

Three problems you will hit, and their fixes

One detail surprises people: plain Python does understand these digits. int("१२३") returns 123 and float("१२.५") returns 12.5, because Python accepts any Unicode decimal digit. pd.to_numeric is stricter. Relying on the Python behaviour through .map(float) would work for clean cells and then crash on the first "३१ वर्ष", so translation followed by errors="coerce" is the safer pattern.

Converting digits and cleaning the text

The respondents, districts and amounts below are synthetic.

import unicodedata
import pandas as pd

# Synthetic survey export from a Hindi-language form. Some respondents typed
# Devanagari digits, some ASCII digits, and names came from different keyboards.
raw = pd.DataFrame({
    "resp_id": [1, 2, 3, 4, 5, 6],
    "naam": ["राम कुमार", "सीता देवी", "राम\u00a0कुमार", "मोहन लाल ", "गीता", "\u0958ासिम"],
    "zila": ["आगरा", "मथुरा", "आगरा", "आगरा", "मथुरा", "आगरा"],
    "umar": ["३४", "29", "३४", "४१", "२७.५", "३१ वर्ष"],
    "rashi": ["₹१२,५००", "₹8,000", "₹१२,५००", "₹९,७५०", "₹११,०००", "₹7,250"],
})

# pd.to_numeric does not accept Devanagari digits, so map them to ASCII first.
DEV_TO_ASCII = str.maketrans("०१२३४५६७८९", "0123456789")

def to_number(s: pd.Series) -> pd.Series:
    cleaned = (s.astype("string").str.translate(DEV_TO_ASCII)
                .str.replace(r"[₹,\s]", "", regex=True))
    return pd.to_numeric(cleaned, errors="coerce")   # anything else becomes NaN

def clean_text(s: pd.Series) -> pd.Series:
    return (s.astype("string")
             .map(lambda x: unicodedata.normalize("NFC", x), na_action="ignore")
             .str.split().str.join(" "))            # also turns no-break spaces into spaces

df = raw.assign(naam=clean_text(raw["naam"]), zila=clean_text(raw["zila"]),
                umar=to_number(raw["umar"]), rashi=to_number(raw["rashi"]))
print(df.to_string(index=False))
print("\nNot numeric, send back:", raw.loc[df["umar"].isna(), ["resp_id", "umar"]].to_dict("records"))

# Duplicate check only works after cleaning: rows 1 and 3 were different bytes.
print("duplicates before:", raw.duplicated(["naam", "zila", "umar"]).sum(),
      "after:", df.duplicated(["naam", "zila", "umar"]).sum())
print(df.groupby("zila")["rashi"].agg(["count", "sum"]))

# Nukta letters: U+0958 and U+0915 + U+093C look identical but are different
# strings until normalised. NFC turns both into the same two code points.
one, two = "\u0958", "क़"
print("equal raw:", one == two, "| equal after NFC:",
      unicodedata.normalize("NFC", one) == unicodedata.normalize("NFC", two))

# Write a CSV that Excel on Windows opens with Hindi intact.
df.to_csv("survey_clean.csv", index=False, encoding="utf-8-sig")
print(open("survey_clean.csv", "rb").read(3))

Row 3 was typed with a no-break space between the two words, the kind that arrives when text is pasted from a web page. Before cleaning, rows 1 and 3 looked the same on screen but were different strings, so the duplicate check found nothing. After cleaning, it finds one. Row 6 wrote its age as "३१ वर्ष" (31 years); it becomes NaN and goes onto the send-back list rather than being guessed.

Why NFC, and why not strip everything invisible

Some Devanagari letters with a nukta, such as क़, can be stored as one code point (U+0958) or as the base letter plus a nukta sign (U+0915 U+093C). They render identically. Unicode normalisation form NFC maps both to one agreed sequence, so equality, grouping and joins behave. Python's unicodedata.normalize is in the standard library; there is no reason to skip it.

It is tempting to go further and delete every zero-width joiner and non-joiner. Resist that for Indic text. Those characters can legitimately control how a conjunct is displayed, so removing them may change what a reader sees. Clean the problems you have evidence of, and keep a copy of the raw column.

Sorting and filtering Hindi text

Python sorts strings by code point. For Devanagari that follows the order of the Unicode block, which is close to the traditional varnamala order for simple words, but it is not a linguistic collation, and mixed Hindi and English values sort all the English first. For a report where alphabetical order matters to the reader, sort on a numeric key you control, such as a district code, rather than on the name.

Filtering works as expected once the text is normalised: df[df["zila"].eq("आगरा")] matches all Agra rows. If you type the filter value yourself, normalise it with the same function, or the comparison can fail for invisible reasons.

Getting the file back into Excel

The final line of the script writes the first three bytes EF BB BF, the UTF-8 byte order mark. Excel uses it to recognise the encoding when someone double-clicks the file. Without it, the Devanagari in the same file appears as a string of accented Latin characters, and the usual reaction is to "fix" the data by retyping it. If a file has already been received without the mark, open it through Data > From Text/CSV, choose "65001: Unicode (UTF-8)" as the file origin, and load.

For Excel formulas on the Excel side, there is no built-in function that converts Devanagari digits to numbers; a helper column with nested SUBSTITUTE for each of the ten digits, wrapped in VALUE, is the usual workaround, and Power Query's Replace Values step handles it more readably.

A short checklist before analysis

• Translate digits in every numeric column, not only the ones you noticed.
• Normalise text columns with NFC and collapse whitespace.
• List everything that still fails conversion, grouped by the respondent or field officer, and send it back.
• Save with utf-8-sig when the output will be opened in Excel.

The same field data often needs its dates made consistent too; see fix DD-MM-YYYY dates in Power Query with the en-IN locale. Working confidently with multilingual, messy Indian datasets is part of the pandas practice in our Data Analyst course, where sessions run live online with optional in-person labs at our Noida centre.