A/B testing interview questions, from sample size to peeking

The scenario, hypothetical throughout: a food-delivery app wants to test a redesigned checkout page.

Chain 1: what are we measuring?

Q1. What is your primary metric?

Orders completed per checkout-page visitor, measured over the test window. It is closest to what the change is meant to affect. Revenue per visitor is noisier and moves with basket size, which the checkout page barely controls.

Follow-up: "What else would you watch?" Guardrail metrics that must not get worse: payment failure rate, refund or cancellation rate within 24 hours, page load time, and customer support contacts about checkout. A design that raises orders by making cancellation harder is not a win.

Q2. Why not track ten metrics and see which move?

With many metrics and no correction, some will cross p < 0.05 by chance. Fix one primary metric in advance and treat the rest as guardrails or exploratory. Follow-up: "Your guardrail moved significantly but the primary didn't." Investigate the guardrail before anything else; a real harm outranks a missing benefit.

Chain 2: who gets randomised?

Q3. Do you randomise by session, user or restaurant?

By user, logged-in ID where possible. Randomising by session lets one person see both designs, which contaminates behaviour and breaks the independence the statistics assume. Restaurant-level randomisation would be needed only if the change affected restaurants, such as a new order-acceptance screen.

Follow-up: "Your metric is orders per visitor but you randomised users. Any problem?" If a user makes several visits, visits are not independent. Analyse at the user level (orders per user) or use a method for ratio metrics, such as the delta method or bootstrapping by user, so the standard error is not understated.

Q4. When is an ordinary A/B test the wrong design?

When treatment leaks between groups — network effects. Testing a delivery-fee change on half of users in one city can change rider availability for everyone. Options are geographic or time-based switchback designs, and saying that the effect estimate is then less precise.

Chain 3: how big and how long?

Q5. How do you decide sample size?

From four inputs: baseline conversion, minimum detectable effect worth acting on, significance level and power. The minimum detectable effect is a business decision — the smallest lift that would pay for building and maintaining the design — not a statistical one.

Follow-up: "The calculation says six weeks. The PM wants one." Offer the trade: in one week you can only detect a larger effect. Show the detectable effect for one, two and four weeks and let the PM choose knowingly. Also run for whole weeks, since weekday and weekend ordering differ.

Chain 4: did the split work?

Q6. The test was set to 50/50. Control has 10,240 users and treatment 9,760. Worry?

Yes. Check for sample ratio mismatch with a chi-square test:

from statistics import NormalDist
from math import sqrt

control, treatment = 10240, 9760
total = control + treatment
expected = total / 2
chi2 = ((control - expected) ** 2 + (treatment - expected) ** 2) / expected
# With 1 degree of freedom, chi-square is the square of a standard normal
p = 2 * (1 - NormalDist().cdf(sqrt(chi2)))
print(f"chi-square = {chi2:.2f}, p = {p:.5f}")

A chi-square of 11.52 gives p ≈ 0.0007: a split this uneven would almost never happen by chance. Something is dropping users from one arm — a redirect failing, a bot filter applied unevenly, or the new page crashing before logging. Follow-up: "Can you still use the results?" No, not until the cause is found. The missing users are not missing at random, so the comparison is biased in an unknown direction.

Chain 5: when do you stop?

Q7. The dashboard showed p < 0.05 on day three. Stop and ship?

No. Checking repeatedly and stopping the first time the p-value dips inflates the false-positive rate well above 5%. A simulation of A/A tests, where there is no real difference at all, shows it:

import random
from statistics import NormalDist
from math import sqrt

random.seed(7)
rate, looks, per_look, sims = 0.10, 20, 100, 1000
z_crit = NormalDist().inv_cdf(0.975)
stopped_early = 0
for _ in range(sims):
    a = b = na = nb = 0
    for _ in range(looks):
        a += sum(random.random() < rate for _ in range(per_look)); na += per_look
        b += sum(random.random() < rate for _ in range(per_look)); nb += per_look
        pool = (a + b) / (na + nb)
        se = sqrt(pool * (1 - pool) * (1 / na + 1 / nb))
        if se > 0 and abs(a / na - b / nb) / se > z_crit:
            stopped_early += 1
            break
print(f"A/A tests declared 'significant' with 20 peeks: {stopped_early / sims:.1%}")

With twenty looks, roughly a quarter of these no-difference tests get declared winners at some point — about five times the nominal 5%. Follow-up: "So you can never look?" You can monitor guardrails for harm at any time. For the primary metric, either wait for the planned sample size, or use a method designed for repeated looks, such as a sequential test with adjusted boundaries.

Q8. Treatment was well ahead in week one and level by week three. Why?

A novelty effect: users click on something new because it is new. The opposite, a primacy effect, happens when regular users need time to adapt. Look at the effect by days since first exposure, and separately for new versus returning users.

Chain 6: reading and reporting the result

Q9. The result is flat overall but positive for iOS and negative for Android. Ship on iOS?

Be careful: slicing after the fact produces chance differences. If platform was a pre-registered segment and the test was sized for it, it is evidence. If not, treat it as a hypothesis and look for a mechanism, such as a rendering bug on one Android version, before acting.

Q10. How do you reduce variance without more users?

Use pre-experiment data as a covariate — CUPED is the common name — so differences between users that existed before the test are removed from the comparison. It works best when the pre-period metric correlates strongly with the test metric.

Q11. Write the one-paragraph result for the business.

State the decision first, then the evidence: "Recommend shipping. Completed orders per user rose 2.1% (95% interval 0.8% to 3.4%) over four full weeks with 41,000 users; payment failures and cancellations did not change." Numbers here are illustrative. The interviewer is checking that you lead with the decision and include the interval and guardrails, not just a p-value.

What ties the chains together

Every follow-up above asks the same thing in a different form: can the design support the conclusion? Practise by taking any product change you have seen in an app and writing out the metric, unit, sample size inputs, guardrails and stopping rule before thinking about results.

For the statistics underneath these answers, see the p-values and confidence intervals round. Experiment design and readouts for business stakeholders are part of the Data Analyst programme; if you are further along and want the modelling side, the Data Science course goes deeper.