Categorical data: reduce memory without losing unknown values

Categorical storage can reduce memory for repeated labels, but validate the vocabulary before casting. Values outside an explicitly supplied category set can become missing, making a storage change look like a data-quality problem or silently erasing an unfamiliar label.

Measure memory with your actual dtype and cardinality. A short list of repeated labels and a column of mostly unique identifiers have different tradeoffs.

Demonstrate the unknown-category risk

The pandas quality lab contains the unfamiliar category New service alongside Software, Training and Support.

import pandas as pd
from build_and_verify import load_orders

orders = load_orders()
keys = orders['category_raw'].str.strip().str.casefold()
restricted = pd.Categorical(keys, categories=['software', 'training', 'support'])
assert pd.isna(restricted).sum() == 1
assert orders.loc[pd.isna(restricted), 'order_id'].tolist() == ['P07']
unknown = ~keys.isin(['software', 'training', 'support'])
assert keys.loc[unknown].tolist() == ['new service']
display = keys.where(~unknown, 'unmapped')
safe = pd.Series(pd.Categorical(display, categories=['software', 'training', 'support', 'unmapped']))
assert safe.isna().sum() == 0
assert orders.loc[unknown, 'category_raw'].tolist() == ['New service']
print('Unknown label retained in raw data and represented explicitly')

The explicit unmapped category is a display choice, not a claim that New service belongs to an existing business category. Keep the original label and an unknown-value flag for review.

The pandas categorical guide explains category vocabularies, codes and memory behavior. The exact memory result depends on the data and representation.

Measure a controlled repeated-label column

import pandas as pd

labels = pd.Series(['Software', 'Training', 'Support'] * 10000, dtype='object')
categorical = labels.astype('category')
before = int(labels.memory_usage(deep=True))
after = int(categorical.memory_usage(deep=True))
assert categorical.astype('object').equals(labels)
assert after < before
print({'rows': len(labels), 'object_bytes': before, 'categorical_bytes': after})

This example intentionally compares an object-backed column with a categorical representation of the same 30,000 values. It does not establish a universal saving against every string backend or dataset. The assertions check both a reduction in this environment and preservation of values.

Avoid quoting a fixed percentage from this example as a general pandas performance promise. Allocators, versions and label lengths can affect measured sizes, and dataframe memory estimates are not identical to total process memory.

Distinguish storage order from business order

An ordered category can encode a meaningful sequence such as low, medium and high. An unordered product category has no natural ranking merely because Software receives one internal code and Training another.

Do not feed category codes into a model as if their numeric distances represented business similarity. Codes are a storage representation unless you deliberately define an appropriate ordinal meaning.

Keep category contracts consistent across files

Two monthly files may infer different category sets. If downstream processing requires a shared vocabulary, define it explicitly or reconcile categories before combining the data. Validate newly appearing labels instead of assuming every month contains the same values.

Similarly, decide whether grouped reports should include unobserved categories. Specify observed in groupby calls when that choice affects the table. An unobserved label is not automatically proof of complete zero activity.

Evaluate the optimization against correctness

Before and after conversion, compare row count, missing count, distinct raw values and relevant aggregates. A memory reduction that introduces new missing values is not a successful optimization unless that transformation was intended and documented.

Exercise: repeat the memory comparison with 30,000 unique identifiers and then with a nullable string dtype. Report the actual measurements and explain why the best representation can differ from the three-label example.

NeuraPath's Data Analytics with Generative AI course connects pandas efficiency with data preservation. The useful optimization saves resources while keeping unknown values visible and business semantics intact.

Continue learning

This article is part of the Pandas wrangling and data checks sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Pandas string cleaning with reversible mappings.
• Continue with Read a large CSV in chunks with consistent aggregates.