Statistics interview questions on p-values and confidence intervals

The example, entirely hypothetical: a lending app changed its loan-application form. Before the change, 1,000 visitors saw the old form and 112 applied. After, 1,000 saw the new form and 138 applied.

Opening: what a p-value is and is not

Interviewer: A colleague ran a test on these numbers and got p ≈ 0.08. Explain to the product manager what that means.

Candidate: If the new form truly made no difference, we'd see a gap of 2.6 percentage points or more, in either direction, about 8% of the time just from the random mix of visitors. So the data are somewhat unusual under "no effect", but not so unusual that we'd rule chance out at the usual 5% threshold.

Interviewer: So there's a 92% chance the new form works?

Candidate: No. The p-value is calculated assuming the null is true; it can't then tell us the probability the null is false. Getting that would need prior information and a Bayesian analysis. What I can say is that the data are compatible with a small negative effect, no effect, or a useful positive one — which is why I'd look at the confidence interval.

Checking: the single most common misreading. Candidates who say "8% chance the result is due to chance" lose the point here.

The confidence interval, computed

Interviewer: Compute the 95% interval for the difference.

Candidate: For two proportions with samples this size, the normal approximation is fine:

from statistics import NormalDist
from math import sqrt

x1, n1 = 112, 1000   # old form
x2, n2 = 138, 1000   # new form
p1, p2 = x1 / n1, x2 / n2
diff = p2 - p1

# Unpooled standard error for the confidence interval
se = sqrt(p1 * (1 - p1) / n1 + p2 * (1 - p2) / n2)
z = NormalDist().inv_cdf(0.975)
print(f"difference = {diff:.3f}, 95% CI = ({diff - z*se:.3f}, {diff + z*se:.3f})")

# Pooled standard error for the two-sided z-test
p_pool = (x1 + x2) / (n1 + n2)
se_pool = sqrt(p_pool * (1 - p_pool) * (1 / n1 + 1 / n2))
z_stat = diff / se_pool
p_value = 2 * (1 - NormalDist().cdf(abs(z_stat)))
print(f"z = {z_stat:.2f}, p = {p_value:.3f}")

This gives a difference of 0.026 with a 95% interval of roughly −0.003 to 0.055, and p ≈ 0.08. The interval includes zero, which matches p being above 0.05; the two use slightly different standard errors, so near the boundary they can occasionally disagree.

Interviewer: Interpret the interval.

Candidate: The method that produced it captures the true difference 95% of the time. The plausible range runs from about a 0.3-point drop to a 5.5-point gain. For decision-making, that's the useful part: the upside could be large, the downside is small. Whether to ship depends on the cost of being wrong, not only on whether zero is inside.

Checking: that you can compute it, that you interpret the method rather than saying "95% probability the true value is in this interval", and that you link statistics to a decision.

Standard error, sample size and power

Interviewer: What's the difference between standard deviation and standard error?

Candidate: Standard deviation describes spread of individual observations. Standard error describes spread of an estimate, such as a mean or proportion, across repeated samples. For a mean it's the SD divided by the square root of n, so it shrinks as the sample grows while the SD doesn't.

Interviewer: How many visitors per arm would you need to detect this 2.6-point lift reliably?

Candidate: For 80% power at a two-sided 5% level, the usual approximation for two proportions is:

from statistics import NormalDist
from math import ceil

p1, p2 = 0.112, 0.138
alpha, power = 0.05, 0.80
z_a = NormalDist().inv_cdf(1 - alpha / 2)
z_b = NormalDist().inv_cdf(power)
n = ((z_a + z_b) ** 2 * (p1 * (1 - p1) + p2 * (1 - p2))) / (p2 - p1) ** 2
print("visitors per arm:", ceil(n))

That is about 2,540 per arm, so the 1,000-per-arm test was underpowered for an effect this size. A non-significant result from an underpowered test is weak evidence of no effect.

Checking: that "not significant" is not reported as "no effect", and that you think about power before running a test.

Traps in the second half of the round

Interviewer: Another team tested 20 button colours and one had p = 0.04. Ship it?

Candidate: Not on that alone. With 20 independent tests at a 5% threshold and no real effects, the chance of at least one p below 0.05 is 1 − 0.95²⁰, about 64%. I'd either correct for multiple comparisons — Bonferroni would require p below 0.0025 — or treat it as a hypothesis and rerun that one colour against control.

Interviewer: A result has p = 0.0001 with ten lakh users per arm. Important?

Candidate: Statistically clear, possibly trivial. With huge samples even a 0.05-point difference becomes significant. I'd report the effect size and interval, and ask whether that difference covers the cost of the change.

Interviewer: Average loan amount is ₹48,000 but the median is ₹22,000. Which do you report?

Candidate: Loan amounts are right-skewed — a few large loans pull the mean up. The median describes a typical applicant better; the mean is still needed for total disbursement planning. I'd show both and say why they differ.

Interviewer: Why does the normal approximation work for proportions here?

Candidate: The central limit theorem: the sampling distribution of a proportion from a large sample is approximately normal. A common rule of thumb is at least around 10 successes and 10 failures per group, which we have comfortably. With very small counts I'd use an exact test such as Fisher's.

Quick checks to rehearse

• Type I error: rejecting a true null (false alarm), controlled by alpha. Type II: missing a real effect, controlled by power.
• A 95% interval that excludes zero corresponds to p below 0.05 for the matching test.
• Correlation between app usage and repayment does not show the app causes repayment; ask what else differs between frequent and rare users.
• Report effect size, interval and sample size together; a p-value alone is never the full answer.

The next step from here is designing the test itself; the A/B testing question bank covers randomisation units, peeking and guardrail metrics. Statistics in the Data Analyst programme is taught through business decisions like this one rather than as formula drills.