Data analyst interview: diagnose a suspicious conversion uplift

Treat a reported conversion uplift as a claim to investigate. Verify the eligible population, conversion event, observation window and assignment process before deciding whether the difference supports a rollout. An impressive relative percentage can hide a small absolute change or an invalid comparison.

The following hypothetical interview uses authored counts. It is a worked reasoning exercise, not a result from a live experiment or an employer's interview bank.

Translate the headline into counts

Suppose a report says conversion rose from 4.0% to 4.5%, a 12.5% relative uplift. The underlying counts are 400 conversions among 10,000 control users and 450 among 10,000 treatment users.

The absolute difference is 0.5 percentage points. Both descriptions are mathematically compatible, but they answer different communication needs. A manager evaluating benefit needs the absolute change, eligible scale and relevant costs, not just the relative headline.

Ask whether each user appears once, whether conversions are binary per user, and whether the same observation window applies to both groups. Counting events in one numerator and users in the other invalidates the comparison.

Quantify a simple uncertainty check

from math import sqrt

control_success,control_n = 400,10000
treatment_success,treatment_n = 450,10000
p0 = control_success/control_n
p1 = treatment_success/treatment_n
delta = p1-p0
se = sqrt(p0*(1-p0)/control_n+p1*(1-p1)/treatment_n)
lower,upper = delta-1.96*se,delta+1.96*se
assert abs(delta-0.005)<1e-12
assert abs((p1/p0-1)*100-12.5)<1e-10
assert lower<0<upper
print({'absolute_difference_percentage_points':delta*100,
       'relative_uplift_percent':(p1/p0-1)*100,
       'approximate_95_percent_interval_percentage_points':[lower*100,upper*100]})

The approximate unpooled normal interval runs from about -0.059 to 1.059 percentage points and includes zero. It assumes an appropriate independent binary-outcome comparison. It does not repair nonrandom assignment, repeated users, tracking changes or selective stopping.

The analyst statistics lab supplies the broader teaching context for these counts and uncertainty examples.

Investigate the suspicious part

If the uplift appeared immediately after a tracking change, compare event definitions and instrumentation before interpreting behavior. If treatment users came from a different acquisition channel, inspect assignment and audience mix. If one group has less follow-up time, enforce a comparable observation window.

Check for repeated peeking or a headline selected from many metrics or segments. A single conventional interval does not automatically account for those choices. Ask what analysis plan and stopping rule were established before the results were inspected.

Microsoft Research's pre-experiment guidance provides context for defining trustworthy experiments before launch. In this interview exercise, the appropriate response is to request that design evidence rather than infer it from the two counts.

Avoid two common overstatements

“The interval includes zero, so there is no effect” is too strong. The data under this calculation remain compatible with a range of effects. “The point estimate is positive, so the change works” is also too strong, especially if assignment or measurement is questionable.

Separate the observed difference, statistical uncertainty and causal validity. Then ask whether the plausible effect range is meaningful for the decision and whether guardrails such as errors, cancellations or customer experience changed.

Give a conditional recommendation

A defensible interview answer is: “The observed difference is 0.5 percentage points, but this simple interval includes zero. I would verify assignment, unique-user denominators and equal follow-up before interpreting the effect. I would then apply the pre-specified decision and guardrail criteria rather than recommend rollout from the relative uplift alone.”

This answer gives the interviewer a sequence of checks and explains what each resolves. It does not hide behind a p-value or promise that more data will inevitably confirm the desired result.

Exercise: halve the treatment denominator while leaving its conversions unchanged. Explain why the resulting rate cannot be compared responsibly until the reason for the population change is understood.

NeuraPath's Data Analytics with Generative AI course connects metric definitions, statistical reasoning and stakeholder communication. A useful analyst can challenge an attractive uplift while preserving the evidence and the decision context.

Continue learning

This article is part of the Analyst career preparation and interviews sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Data analyst interview: reconcile two conflicting revenue numbers.
• Continue with Turn an Excel-heavy role into evidence of analytical skill.