Effect size versus statistical significance for analysts

Effect size describes magnitude. Statistical significance describes a test result under a model and decision rule, influenced by both magnitude and precision. A small effect can be statistically detectable with enough independent information, while a meaningful effect can remain uncertain in a small study.

Report the estimate, its uncertainty and a practical decision threshold instead of treating p<0.05 as a complete recommendation.

Hold the observed difference fixed

This teaching calculation assumes two independent normal groups with known standard deviation 10 and an observed mean difference of one unit. It compares the same observed difference at two hypothetical sample sizes.

import numpy as np
from scipy import stats

difference, sigma = 1.0, 10.0
results = {}
for n_per_group in [20, 2000]:
    se = sigma * np.sqrt(2/n_per_group)
    z = difference/se
    p = 2*stats.norm.sf(abs(z))
    margin = stats.norm.ppf(.975)*se
    results[n_per_group] = {'difference': difference, 'p': float(p),
                             'ci95': [difference-margin, difference+margin]}
assert results[20]['ci95'][0] < 0 < results[20]['ci95'][1]
assert results[2000]['p'] < .01
assert 0 < results[2000]['ci95'][0] < results[2000]['ci95'][1] < 2
assert difference/sigma == .1
print(results)

The larger sample makes the one-unit effect more precisely estimated; it does not turn it into a larger effect. The known-SD standardized difference remains 0.1 in both cases.

This is an analytic demonstration with specified observed means, not a simulation claiming those means will occur at every sample size. It runs in the analyst statistics lab. SciPy's normal distribution reference documents the tail and quantile calculations.

Compare with a meaningful threshold

Suppose the hypothetical decision requires at least a two-unit improvement to justify its cost. The n=2,000 interval is approximately 0.38 to 1.62 units, below that threshold despite a small p-value against zero.

The n=20 interval is much wider and includes both negative values and improvements above two units. Calling it no effect would confuse uncertainty with evidence of absence.

The two-unit threshold is an authored scenario assumption. A real threshold should reflect costs, benefits, risk and measurement meaning, established before inspecting the preferred result.

Keep absolute and relative effects distinct

A conversion increase from 4% to 4.5% is an absolute increase of 0.5 percentage points and a relative increase of 12.5%. Both are valid descriptions, but they sound very different without the baseline.

Include counts and denominators so the reader can reconstruct the effect. A relative lift alone can exaggerate the impression of a small absolute change.

For continuous outcomes, retain the original unit alongside standardized measures. A standardized difference helps some comparisons but does not directly state revenue, time saved or another operational consequence.

Avoid interpreting p as the probability of success

A p-value is not the probability that the effect is real, the chance the intervention will succeed or the size of the business benefit. Its meaning depends on the tested null and sampling model.

Repeated analyses, selective reporting, dependence and biased sampling can undermine an ordinary significance interpretation. A precise estimate from an inappropriate design does not become causal evidence.

Make the decision separate from the calculation

An analyst can report whether the interval includes meaningful benefit or harm and explain remaining uncertainty. The final action can also depend on implementation cost, reversibility, guardrails and the value of more information.

Do not move the meaningful threshold after seeing the estimate merely to justify a preferred decision. Record the threshold and rationale in the analysis plan.

Exercise: use the lab's 400/10,000 versus 450/10,000 conversion counts. Report absolute difference, relative lift and an uncertainty interval, then discuss the result against a prespecified one-percentage-point improvement threshold.

NeuraPath's Data Analytics with Generative AI course connects statistical evidence with practical interpretation. A useful result explains how large the effect is, how uncertain it remains and what would make it matter.

Continue learning

This article is part of the Statistics for analytical decisions sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Bootstrap a median delivery-time interval.
• Continue with Multiple comparisons: why twenty tests need a plan.