Data lineage: trace a model feature back to its source

Feature lineage explains where a value came from, which rules transformed it and who owns the source. It supports incident analysis, schema review, privacy assessment and model reproducibility.

Reverse one feature path

The local lab traces model.risk_score back through a 30-day spend feature and settled-payment staging to source.payments.amount_cents.

from data_engineering_cases import lineage_case

result = lineage_case()
assert result["complete_to_source"] is True
print(" -> ".join(result["reverse_trace"]))

The path records that staging keeps settled, nonnegative payments and that spend_30d sums cents in a half-open 30-day window before prediction. The source owner is payments.

Record semantic lineage

Table-to-table edges are insufficient when a unit, filter or window can invalidate a model. Store transformation version, columns, units, time semantics, owners, quality checks and output contract. Link the feature definition to training runs and model versions.

Use lineage during change

When amount_cents changes, identify features, models and dashboards downstream. Notify owners, block incompatible publication and backfill only after a plan. During an incident, trace a bad prediction to the exact feature snapshot and source records under appropriate access.

Automated lineage from query plans helps, but dynamic Python and external files may need explicit declarations. Review critical paths manually.

The Data Science course links feature lineage to contracts and deployment ownership.

Exercise

Create a lineage graph for five features. Add owners and semantic rules, then simulate a source unit change and enumerate every affected artifact and action.

Continue learning

This article is part of the Data engineering for data science sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Build a point-in-time correct feature join.
• Continue with SQL versus pandas versus Spark: choose from measured constraints.

Reference: OpenLineage documentation.