# Worked example: a conversion experiment analysis plan **Status:** authored teaching plan for review. This is not a historical preregistration or an approved live NeuraPath experiment. Its worked counts are synthetic, and operational guardrail data have not been supplied. ## Question and estimand Compare a hypothetical simplified enquiry flow with the existing flow. The primary estimand is the treatment-minus-control difference in the probability of at least one completed, valid enquiry within seven elapsed days of assignment, among all eligible assigned account IDs. This is an assignment-based comparison; do not condition the primary analysis on clicking the new flow. ## Population and assignment Eligible units are stable account IDs entering the eligible page for the first time in the recruitment window. Exclude documented test/internal accounts using a rule fixed before assignment. One assignment per account persists across visits. For this teaching design, use a pre-generated random permutation of 20,000 labels containing 10,000 labels per arm; assign sequentially to eligible units. Do not reuse that illustrative design without reviewing interference and implementation. ## Horizon and completion Recruit until 20,000 unique eligible units or an administrative cap of 28 calendar days, whichever occurs first. The cap is fixed independently of outcome results. Wait seven elapsed days after the final assignment for outcome maturation, plus the predefined one-day data-arrival grace period. An administrative-cap stop with fewer units is reported as a deviation from the target information size, not extended selectively because the p-value is nearly significant. ## Planning assumptions Baseline 4%, meaningful planning improvement +1 percentage point, two-sided alpha 0.05, desired power 80%, equal independent groups. The approximation in NP-0155 requires 6,745 units per arm; this worked plan chooses 10,000 per arm. The actual population, dependence and traffic feasibility need review before a live plan. ## Data contract Assignment(account_id, variant, assigned_at); outcome(event_id, account_id, event_time, valid_enquiry). Timestamps identify instants. Deduplicate event IDs; conflicting versions fail quality checks. Repeated valid enquiries count once per assigned unit. Events at the seven-day upper boundary are outside the window. No observed enquiry counts as zero only when telemetry coverage is complete. Missing coverage is a quality failure, not an automatic nonconversion. ## Primary analysis Report arm denominators, conversion counts/rates, absolute risk difference in percentage points and relative lift with its baseline. Use a two-sided pooled two-proportion z test for the zero-difference hypothesis and an unpooled normal 95% interval for the risk difference as the declared large-sample teaching approximation. Test and interval are different approximations and need not be exact inverses near a threshold. Review alternatives for sparse or complex data. There is one confirmatory primary outcome. Segment analyses are exploratory and cannot independently authorize a success claim. ## Data-quality checks and guardrails Check unit uniqueness, assignment stability, source completeness, event-window boundaries and assignment-to-analysis counts. Diagnose any sample-ratio mismatch against the actual allocation design; no missingness repair by downsampling. Operational guardrails in this example: enquiry technical-error rate increase above 0.5 percentage points or p95 page-latency increase above 100 milliseconds triggers an operational hold and investigation. These are illustrative business limits, not formal simultaneous statistical acceptance tests. Their definitions, instrumentation and actual results must be reviewed; the aggregate conversion fixture alone cannot establish that they passed. ## Monitoring, deviations and decision Monitor operational incidents and logging quality; do not stop for primary-outcome p<0.05. A safety/quality stop is documented and does not become a completed fixed-horizon success claim. Record every change to eligibility, outcome, exclusions, horizon or analysis after result access. Report whether the effect interval includes zero, meaningful benefit or harm; combine that evidence with cost and guardrail review. Statistical significance alone does not authorize rollout. Save source hashes, code/version information, the quality report and the complete result table, including inconclusive results.