Read CSV files without corrupting customer identifiers

Read identifiers as text unless the source contract explicitly defines them as numbers. Customer ID 0012 can lose meaningful characters if converted to integer 12, and formatting it later cannot always reconstruct the original identity.

CSV is a text serialization format, not a complete typed schema. Use a parser that handles quoting and newlines, then validate each field according to its meaning.

Read the synthetic source without inference

From the Python reporting lab directory, run:

import csv
from pathlib import Path

source = Path("raw_orders.csv")
with source.open(encoding="utf-8-sig", newline="") as handle:
    reader = csv.DictReader(handle, strict=True)
    rows = list(reader)

assert rows[0]["customer_id"] == "0012"
assert rows[2]["customer_id"] == "0042"
assert rows[9]["amount_inr"] == "1,200.00"
assert len(rows) == 11
print("Identifiers and quoted comma field preserved")

DictReader returns ordinary field text here; the code does not ask it to infer numeric customer IDs. The quoted comma in R10's amount remains inside one field rather than creating an extra column.

Python documents encoding-independent CSV handling and the newline convention in its CSV module reference.

Do not split CSV lines manually

Calling line.split(",") on the R10 record would treat the comma inside the quoted amount as a delimiter. Real CSV can also contain quoted newlines, so one logical record need not equal one physical text line.

Use record numbers for parser-level traceability and retain the original file. If a workflow requires physical line locations, choose tooling that reports them appropriately rather than assuming they match record indices.

The lab uses strict parsing for malformed quoting and separately checks missing or extra fields. A parser can handle the format correctly while the record still violates the business schema.

Validate the header contract

The expected field order is order_id, customer_id, order_date, amount_inr and status. The supplied loader requires that exact schema. A missing, renamed or duplicate header should not silently become an empty value in every downstream record.

Another workflow may permit column reordering, but it should still compare the required and actual field sets and reject ambiguous duplicate headers. Flexibility should be an explicit rule, not an accidental consequence of dictionary access.

Keep schema validation separate from row validation so the error explains whether the whole file is incompatible or a few records are invalid.

Convert measures, preserve identities

Amount and date fields need explicit parsers. CustomerID remains a four-digit text identifier under this fixture's contract. The missing customer on R9 is rejected rather than converted to a default customer.

from pathlib import Path
from report import load_orders

with Path("raw_orders.csv").open(encoding="utf-8-sig", newline="") as handle:
    accepted, rejected, replays, raw_count = load_orders(handle)

assert accepted[0].customer_id == "0012"
assert (raw_count, len(accepted), len(rejected), replays) == (11, 7, 3, 1)
assert {item["order_id"] for item in rejected} == {"R8", "R9", "R10"}
print("Schema and row contracts applied")

The comma-formatted amount is parsed correctly as CSV text but rejected as a business amount because it violates the canonical numeric format. Structural correctness and field validity are different checks.

Check encoding and round trips

The lab uses utf-8-sig to accept UTF-8 input with or without a byte-order mark. Do not assume this handles every legacy encoding; obtain the source's encoding contract when characters are corrupted or decoding fails.

Write output with an explicit encoding and newline handling, then read it back and verify key values. Opening the CSV directly in a spreadsheet application can apply new type inference and remove zeros even if Python preserved them correctly.

Document the receiving application's import rules as part of the handoff. A correct CSV file does not prevent every consumer from misinterpreting its fields.

Exercise: add customer IDs 0012 and 12 to a source copy and define whether both are valid. Show how converting both to integers collapses their representation, then implement the intended validation without losing the original text.

NeuraPath's Data Analytics with Generative AI course connects Python file handling with identity preservation. A reliable CSV workflow validates structure and types while retaining the evidence needed to explain rejected records.

Continue learning

This article is part of the Python foundations for analysts sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Python type conversion: reject invalid numeric records explicitly.
• Continue with Python Decimal for money versus floating-point arithmetic.