Product metrics interview case when daily active users drop

The prompt

Interviewer: You're an analyst at a language-learning app. DAU averaged 4,00,000 last week and 3,52,000 this week — down 12%. The head of product has asked you why. Go.

Minutes 0–5: is the drop real?

Candidate: Before looking for a cause, I'd confirm this is a real change in behaviour and not in measurement. Three questions. How is "active" defined — any app open, or a lesson started? Did anything change in logging, like an SDK update or a new app version that fires the event differently? And is the drop in the dashboard only, or also in independent sources such as server-side lesson completions or payment events?

Interviewer: Definition is "opened the app and loaded the home screen". A new app version, 6.2, went out on Android on Tuesday. Server-side lesson starts fell by about 5%.

Candidate: That's useful. If real engagement had fallen 12%, I'd expect lesson starts to fall by a similar share. They fell less than half as much. So part of the 12% may be measurement — possibly the new Android version logging the home-screen event differently — and part may be real.

What this earns: the candidate did not jump to "users are unhappy". Separating a tracking change from a behaviour change is the single most important step in this case, and the independent server-side source is the tool that does it.

Minutes 5–12: where is the drop?

Interviewer: How would you break it down?

Candidate: By dimensions that map to possible causes: platform and app version, new versus returning users, country or region, and acquisition channel. I'd compute each segment's contribution to the total change, not just its percentage change, because a big percentage fall in a tiny segment doesn't explain much.

Here's a query for the platform split, on a small synthetic version of the daily table:

CREATE TABLE dau_by_segment (week TEXT, platform TEXT, app_version TEXT, dau INTEGER);
INSERT INTO dau_by_segment VALUES
 ('last','android','6.1',230000),('last','ios','5.9',120000),('last','web','-',50000),
 ('this','android','6.1', 80000),('this','android','6.2',110000),
 ('this','ios','5.9',114000),('this','web','-',48000);

WITH w AS (
  SELECT platform,
         SUM(CASE WHEN week='last' THEN dau ELSE 0 END) AS last_wk,
         SUM(CASE WHEN week='this' THEN dau ELSE 0 END) AS this_wk
  FROM dau_by_segment GROUP BY platform)
SELECT platform, last_wk, this_wk,
       this_wk - last_wk AS change,
       ROUND(100.0 * (this_wk - last_wk) / last_wk, 1) AS pct_change,
       ROUND(100.0 * (this_wk - last_wk) /
             (SELECT SUM(this_wk) - SUM(last_wk) FROM w), 1) AS share_of_total_drop
FROM w ORDER BY change;

In this example Android accounts for 40,000 of the 48,000 lost users — about 83% of the drop — while iOS and web are down 5% and 4%. Then I'd split Android by version.

Interviewer: Suppose Android on 6.1 is down about 5%, and users on 6.2 show about 30% fewer home-screen events per user than 6.1 did last week.

Candidate: Then the pattern is: a roughly 5% decline everywhere, plus a large extra fall concentrated in the new version. The broad 5% matches the lesson-start figure, so I'd treat that as a real engagement change. The extra fall on 6.2 looks like instrumentation or a bug on the home screen.

What this earns: contribution to change, not just percentages, and a split that separates two effects instead of forcing one explanation.

Minutes 12–18: finding causes for each part

Interviewer: How would you confirm the 6.2 issue?

Candidate: Compare users on 6.2 with users still on 6.1 over the same days. If 6.2 users start lessons at the same rate but log fewer home-screen events, it's a tracking change — for example, the home screen now loads from a cache and the event only fires on a full load. If they start fewer lessons too, it's a real bug, and I'd check crash rates and home-screen load time for that version. I'd also ask the Android team for the release notes; the answer may be written down already.

Interviewer: And the broad 5%?

Candidate: Check the calendar first: a public holiday, exam season or a festival can shift a study habit. Compare with the same week last year if we have it. Then look at the top of the funnel: were fewer new users acquired, perhaps because a marketing campaign ended? If new-user DAU fell and returning users held steady, it's acquisition, not retention. Finally, notification sends — a change in reminder volume moves a habit app's DAU directly.

Interviewer: Marketing confirms a paid campaign ended last Sunday.

Candidate: Then I'd check how much of the 5% comes from users who joined in the previous four weeks. If most of it does, the broad decline is mainly the campaign ending and not a product problem.

Minutes 18–25: the recommendation

Interviewer: Summarise for the head of product in under a minute.

Candidate: "More than half of this week's 12% drop in DAU looks like a measurement change on Android 6.2, not lost users; server-side lesson starts fell only about 5%. The Android team should confirm whether the home-screen event changed. The remaining real decline is concentrated in recently acquired users and lines up with the paid campaign ending. Recommendation: fix or re-baseline the 6.2 event, move the DAU definition to a server-side lesson start so app releases can't move it, and decide separately whether to restart the campaign. I'll send the split by version and cohort today."

What this earns: a quantified split, a clear owner for each action, and a proposal to fix the metric so the same false alarm cannot recur.

Questions to prepare for this case type

• "What if the drop was on all platforms equally?" Look for something shared: backend outage, notification service, a payment gateway, a calendar effect.
• "What single metric would you add to the dashboard?" A server-side engagement metric beside the client-side one, so divergence is visible immediately.
• "DAU rose 15% — do you investigate?" Yes. Bots, duplicate event firing and double-counted users look like growth. Unexplained rises deserve the same checks.
• "How do you know DAU is the right metric at all?" For a learning app, DAU can reward opening without learning. Suggest a metric tied to value, such as users completing a lesson, and DAU/MAU for habit strength.

The same decomposition habit applies to revenue, as the regional sales dip case shows. Metric definitions, segment analysis and stakeholder summaries like this one are practised in the Data Analyst programme.