Diagnose a conversion drop with a segmented checklist

A lower overall conversion rate can result from weaker conversion within comparable groups, a shift toward groups that normally convert less, or a measurement problem. Start by separating those possibilities. An aggregate percentage alone cannot tell you which happened.

Use a fixed conversion contract and a short investigation sequence. The aim is to identify the smallest set of observations that supports a decision, not to search every possible segment until a dramatic story appears.

Verify that the alert compares the same quantity

Check the conversion event, eligible denominator, identity rules, timezone and observation window. Compare mature outcomes with mature outcomes. A recent cohort may look worse because its users have not had enough time to convert.

Then inspect collection: source freshness, duplicate IDs, client changes, consent-related coverage changes and failed joins. An event-name migration can resemble a business decline if the dashboard reads only the old name.

If those checks fail, label the business result as unresolved while repairing measurement. Do not explain a commercial decline that the data cannot yet establish.

Work through a traffic-mix example

This original synthetic example has two segments, each with a stable conversion rate:

Overall conversion falls from 17% to 8%. Neither segment's conversion rate changes. The population shifts toward new users, whose observed rate is lower.

from math import isclose

before = {'returning': (800,160), 'new': (200,10)}
after = {'returning': (200,40), 'new': (800,40)}
def overall(data):
    return sum(c for n,c in data.values())/sum(n for n,c in data.values())
weights = {s:n/sum(v[0] for v in before.values()) for s,(n,c) in before.items()}
standardized_after = sum(weights[s]*after[s][1]/after[s][0] for s in before)
assert isclose(overall(before), .17)
assert isclose(overall(after), .08)
assert isclose(standardized_after, .17)
for segment in before:
    assert before[segment][1]/before[segment][0] == after[segment][1]/after[segment][0]
mix_effect = overall(after)-standardized_after
within_effect = standardized_after-overall(before)
assert isclose(mix_effect+within_effect, overall(after)-overall(before))
print({'before':overall(before), 'after':overall(after),
       'after_at_before_mix':standardized_after,
       'mix_effect_pp':100*mix_effect, 'within_effect_pp':100*within_effect})

At the earlier period's segment weights, the later rate remains 17%. Under this chosen decomposition, the nine-percentage-point decline is entirely a mix effect.

This is descriptive standardization. It does not establish why the mix changed or prove that an acquisition campaign caused it. If both segment rates and weights change, decomposition components also depend on the chosen reference weights; report that convention.

Investigate segments with a reason

Useful candidates include acquisition source, device, geography, account tenure and funnel entry point, provided their definitions are stable and appropriate. Prioritize segments connected to recent releases or known operational differences.

Avoid interpreting dozens of small slices as independent discoveries. Small denominators produce volatile rates, and repeated searching increases the chance of finding an impressive-looking pattern. Preserve counts, compare compatible periods and mark exploratory findings as hypotheses.

Do not define a segment using behavior that occurs after conversion and then treat it as a cause. “Users who reached the success page” is not a sensible baseline segment for explaining conversion success.

Translate diagnosis into the next check

If a device-specific decline remains after measurement checks, inspect the relevant journey and release history. If the aggregate decline is driven by new-user mix, examine acquisition goals and downstream value before deciding that the change is harmful. A larger number of lower-converting visitors can still produce more total customers.

Report absolute conversions alongside rates. In this example, conversions also fall from 170 to 80 because total eligible users remain 1,000. With a different traffic total, that conclusion could change.

The broader risk of aggregate reversals is discussed in Berkeley's experiments and confounding material. Here the raw counts and fixed-weight calculation let a reviewer inspect the explanation directly.

Exercise: increase the later new-user rate from 5% to 6%. Recalculate the standardized rate and explain how within-segment improvement can coexist with an overall decline.

NeuraPath's Data Analytics with Generative AI course connects descriptive statistics with actionable investigation. A useful conversion diagnosis separates what the data show from the mechanism you still need to test.

Continue learning

This article is part of the Customer and product analytics sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Experiment guardrails for revenue and user experience.
• Continue with Measure subscription expansion and contraction revenue.