Chi-square tests: check expected counts before interpreting results

Before interpreting a chi-square test of independence, verify the observational unit, the contingency-table counts and the expected frequencies under the null model. A function returning a p-value does not establish that its approximation is appropriate.

Use counts of independently sampled units in mutually exclusive categories. Percentages without their denominators cannot substitute for those counts.

Calculate expected counts explicitly

This synthetic example has two groups of 100 units. Forty units convert in group A and thirty in group B. Run it in the analyst statistics lab.

import numpy as np
from scipy import stats

observed = np.array([[40, 60], [30, 70]])
result = stats.chi2_contingency(observed, correction=False)
manual_expected = np.outer(observed.sum(axis=1), observed.sum(axis=0)) / observed.sum()
manual_statistic = ((observed-manual_expected)**2 / manual_expected).sum()
assert np.array_equal(result.expected_freq, [[35, 65], [35, 65]])
assert np.allclose(result.expected_freq, manual_expected)
assert np.isclose(result.statistic, manual_statistic)
assert result.dof == 1
print({'chi_square': float(result.statistic), 'p': float(result.pvalue),
       'expected': result.expected_freq.tolist()})

The statistic is approximately 2.198 and the p-value approximately 0.1382. This calculation explicitly disables the continuity correction, making the chosen method reproducible.

The SciPy chi2_contingency reference documents expected counts, correction settings and sparse-table considerations. Expected frequencies are derived from the marginal totals under independence; they are not the observed counts rounded into a convenient shape.

Examine a sparse table

import numpy as np
from scipy import stats

sparse = np.array([[1, 9], [0, 10]])
expected = stats.contingency.expected_freq(sparse)
assert np.array_equal(expected, [[.5, 9.5], [.5, 9.5]])
assert (expected < 5).sum() == 2
exact = stats.fisher_exact(sparse, alternative='two-sided')
assert 0 <= exact.pvalue <= 1
print({'expected': expected.tolist(), 'fisher_two_sided_p': float(exact.pvalue)})

The small expected conversion counts make a routine asymptotic interpretation questionable. A commonly cited count threshold is a screening guideline, not a universal theorem that makes every table above it valid.

For this independent 2×2 teaching case, the Fisher exact-test reference describes an alternative with its own conditional null model. Larger tables or different sampling designs may require other exact, resampling or model-based approaches.

Check independence at the unit level

If one customer generates several visits, counting every visit as an independent customer can understate uncertainty. Repeated measurements, matched pairs and clustered assignments require methods that reflect their structure.

Likewise, before/after conversion indicators for the same people are paired binary outcomes. A generic independent contingency-table test ignores that pairing.

The table's shape does not reveal the design. Keep the unit definition and collection process alongside the counts.

Report the association in useful units

In the first fixture, observed conversion is 40% versus 30%, an absolute difference of ten percentage points. The test addresses compatibility with an independence model; it does not say the difference is operationally unimportant because p exceeds 0.05.

Report effect estimates and suitable uncertainty intervals, and distinguish observational association from a causal claim. Random assignment and implementation quality cannot be inferred from labels A and B.

Avoid repairing the table for significance

Do not combine categories solely to improve a p-value or pass a count rule. Any grouping should have a substantive interpretation established before inspecting the desired result. Preserve the original table and explain the transformation.

Exercise: create a table with the same conversion percentages but ten times as many independent units. Compare the p-value while keeping the effect difference unchanged, and explain why sample size and practical magnitude answer separate questions.

NeuraPath's Data Analytics with Generative AI course connects categorical analysis with study design. A defensible test starts with valid counts and an appropriate model, then communicates the size and uncertainty of the association.

Continue learning

This article is part of the Statistics for analytical decisions sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Paired versus independent observations in before-and-after analysis.
• Continue with ANOVA: what a significant result does not tell you.