Naive Bayes: test the independence assumption empirically

Naive Bayes factorizes the feature likelihood conditional on the class. Checking only overall feature correlation examines the wrong condition and cannot establish that this factorization is appropriate.

Our original binary teaching fixture contains fifty copies of each feature pattern: (0,0), (0,1), (1,0) and (1,1). The label is one when the bits differ and zero when they match. This is a balanced, deterministic interaction rule, not a noisy real-world dataset.

Inspect the conditional tables

Across all 200 rows, the feature correlation is zero. Each bit is one half the time, and both are one in one quarter of the rows.

Within class zero, however, the bits always match. Within class one, they always differ. The class-conditional probabilities make the failure visible:

The joint probabilities do not equal the products. Correlations conditional on class are +1 and -1, despite the overall correlation of zero.

The scikit-learn Naive Bayes guide describes the class-conditional factorization. Our count construction directly checks it for two binary features.

Observe what the fitted model loses

Bernoulli Naive Bayes with smoothing alpha=1 learns the same marginal feature probabilities in both classes. It therefore assigns approximately 0.5 to each class for all four patterns. Its class predictions are correct on half of a uniform enumeration of those patterns.

A depth-two decision tree can represent the interaction and predicts all four labels correctly in this constructed rule. This comparison explains a structural limitation; it is not a general claim that trees outperform Naive Bayes.

import numpy as np
from classical_cases import conditional_dependence

r = conditional_dependence()
assert r['overall_feature_correlation']==0
assert [row['joint_11'] for row in r['conditional_tables']]==[.5,0]
assert all(row['independence_product']==.25 for row in r['conditional_tables'])
assert np.allclose(r['naive_bayes_probabilities_on_four_patterns'],.5)
assert r['naive_bayes_accuracy_on_uniform_pattern_enumeration']==.5
assert r['depth_two_tree_accuracy_on_uniform_pattern_enumeration']==1
print(r)

Run in the supervised-model lab. The four-pattern evaluation enumerates the known deterministic rule with equal weights. It is not labelled as a fresh noisy empirical test set.

Use diagnostics without claiming a proof from weak evidence

For a real binary or categorical dataset, inspect class-conditional contingency tables with support counts. For numeric features, examine suitable dependence diagnostics within classes. Zero correlation alone does not imply independence, especially for nonlinear relationships.

With more than two features, pairwise checks also do not establish full joint conditional independence. A failure to detect dependence in a small sample may reflect limited power rather than a valid assumption.

Dependence does not mean Naive Bayes must be useless. Its classification performance can still be competitive for some tasks, so compare it with appropriate baselines and held-out metrics. Probability quality deserves separate attention because correlated evidence can be counted repeatedly by the factorized model.

Exercise: add an explicit interaction feature indicating whether the bits differ. Refit the model and explain how the representation changed the task. Then describe why selecting that feature after inspecting final test labels would require a revised assessment design in a real project.

NeuraPath's Data Science course connects probabilistic assumptions with observable data checks. A useful Naive Bayes explanation tests the conditional relationship the model actually assumes.

Continue learning

This article is part of the Supervised learning methods sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Support vector machines: choose a kernel with validation.
• Continue with Poisson regression for nonnegative event counts.