Privacy-aware text preprocessing for a portfolio dataset

Text can contain names, contact details, identifiers and sensitive free-form disclosures. A public portfolio should use original synthetic or properly governed data. Pattern redaction is one control, not permission to publish private text.

Test replacements and residuals

The local NLP lab uses an invented sentence with an email, phone number and order ID.

from nlp_cases import privacy_case

result = privacy_case()
assert result["replacement_counts"] == {"email": 1, "phone": 1, "order": 1}
assert not any(result["residual_matches"].values())
print(result["redacted"])

The output is Maya Rao [EMAIL] called [PHONE] about order [ORDER]. The same patterns find no residual email, phone or order ID.

The name remains, illustrating the limit. Names are difficult to detect with regex, and context can re-identify a person even after obvious fields disappear. Real privacy review must cover purpose, lawful access, consent or other basis where applicable, retention, access control and publication risk.

Minimize before modeling

Collect only fields needed for the task. Remove raw text from exported artifacts when aggregate labels or invented examples suffice. Replace stable identifiers with scoped tokens only when linkage is necessary, and protect the mapping separately.

Evaluate redaction precision and recall on representative formats, languages and obfuscations. False negatives expose data; false positives can erase task meaning. Keep a secure review path for uncertain spans.

Prevent memorization and leakage

Split by source and search for duplicated passages. Generative models can memorize rare strings. Do not paste private records into external services without approved controls. Document data source, permissions, transformation, retention and intended audience in the portfolio README.

The Data Science course connects data minimization and provenance with technical preprocessing.

Exercise

Create an entirely synthetic corpus with varied identifier formats. Measure redaction by entity type, add a manual residual review and publish only transformed examples plus the generation method.

Continue learning

This article is part of the NLP and text analytics sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Detect drift in incoming support language.
• Continue with Document annotation guidelines for an NLP project.

Reference: NIST Privacy Framework.