Partial dependence: recognize correlated-feature limitations

Partial dependence asks a fitted model to vary one feature across a grid while averaging predictions over the observed values of the other features. That can summarize model behaviour. When features are strongly dependent, the replacement can create combinations the data never contained.

Build two models that agree on observed support

Our deterministic fixture has five rows with x2 = 2*x1 and target 3*x1. Model A is a fitted linear regression using only x1. Model B is a fitted linear regression using only x2. On every observed row, their predictions agree to floating-point precision: -6, -3, 0, 3 and 6.

Now compute a marginal partial-dependence style operation for x1. At each grid value -2, 0 or 2, replace x1 in all five rows while retaining their original x2 values. Four of five constructed rows violate x2 = 2*x1 at every grid point.

Model A's averaged predictions are -6, 0 and 6. Model B ignores replaced x1, so its averaged prediction remains approximately zero. Two models that are indistinguishable on observed support tell different marginal stories off that support.

The scikit-learn partial-dependence guide defines the averaging operation and explicitly warns that correlated features can create unlikely data points. The method explains the fitted response, not a causal intervention.

import numpy as np
from decision_cases import partial_dependence_case

r = partial_dependence_case()
assert r['max_observed_prediction_difference'] < 1e-12
assert np.allclose([row['model_a_partial_dependence'] for row in r['grid']], [-6, 0, 6])
assert np.allclose([row['model_b_partial_dependence'] for row in r['grid']], 0)
assert all(row['off_relationship_rows'] == 4 for row in r['grid'])
print(r)

Run the case in the supervised-model lab. It fits both regressions from the five rows and performs the replacement explicitly, keeping the unsupported-row count visible.

Audit support before interpreting the curve

Start with feature distributions and dependence. For every plotted grid region, ask whether comparable observations exist after accounting for correlated inputs. Overlay rugs, density or counts so sparse regions are visible. Inspect individual conditional expectation curves when interactions matter, while remembering they use the same replacement operation.

Conditional approaches can restrict averages to plausible neighbourhoods, and accumulated local effects can reduce some extrapolation problems. They introduce their own choices and do not create causal identification. A domain-derived feature may also remove a deterministic redundancy, but that decision changes the model and should be revalidated.

The fixture is an extreme perfect relationship to make the mechanism undeniable. Real dependence is often softer; unsupported combinations become a matter of degree. That makes support diagnostics more important, not less. If the intended question is “what happens if we change x1?”, define a feasible intervention and address confounding instead of relabelling a marginal model plot as effect evidence.

Exercise: add small noise to x2 and repeat across several noise scales. Count constructed rows outside a chosen support band, compare both curves, and state how the band choice affects the conclusion.

NeuraPath's Data Science course treats interpretation as a testable workflow. A partial-dependence plot is strongest when the model, support and question are all named precisely.

Continue learning

This article is part of the Supervised learning methods sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Learning curves: decide whether more data may help.
• Continue with Choose a supervised model from the failure you need to avoid.