Detect training-serving skew with schema contracts

Training-serving skew occurs when production inputs differ from the data or transformation contract used to train the model. A schema catches structural failures before they become plausible-looking predictions.

Test realistic bad requests

Our strict teaching contract requires exactly amount, count and channel. Amount must be finite numeric, count a nonnegative integer, and channel either store or web.

The valid case passes with no errors. Six cases fail as intended: missing channel, an extra debug field, string amount, negative count, unknown phone channel and infinite amount. The validator does not silently coerce string "10" to numeric or map phone to another category.

from feature_cases import schema_case

r = schema_case()
assert r['cases']['valid'] == []
assert r['cases']['missing']
assert r['cases']['extra'] == ['columns']
assert r['cases']['string_amount'] == ['amount']
assert r['cases']['unknown_channel'] == ['channel']
assert r['cases']['nonfinite'] == ['amount']
print(r)

Run the contract in the feature-engineering lab. It demonstrates rejection behaviour; production schemas need typed errors, versioning and safe logging.

Extend schema checks to semantics and lineage

Column names and types are only the first layer. Record units, grain, time availability, allowed ranges, category policy and null meaning. Ten rupees and ten paise share a numeric type but not a unit. A customer-level value duplicated onto transactions changes weighting.

Use the same transformation artifact in training and serving. Test feature order, unknown categories and default paths. Compare offline and online values for sampled entity-timestamp keys with tolerances appropriate to each feature.

Version schemas with models. Additive fields may be backward compatible for one service and forbidden for another. Define whether unknown fields are rejected or ignored; silent acceptance can hide upstream changes.

Monitor rejection counts and values by source. Avoid logging sensitive raw payloads unnecessarily. A fallback should be explicit and evaluated, not an exception handler that substitutes zero for everything.

Exercise: add currency, event timestamp and feature-as-of timestamp to the contract. Create tests for paise/rupee confusion, future features and missing categories. Define which failures reject the request and which route to a documented fallback.

NeuraPath's Data Science course connects preprocessing to machine-checkable interfaces. A model is only as reproducible as the data contract at its boundary.

Continue learning

This article is part of the Feature engineering and data quality sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in SHAP values: explain a prediction without claiming causation.
• Continue with Feature freshness: prevent stale values at prediction time.