Analyze marketplace seller performance with comparable cohorts

A seller with no recorded refunds may simply have newer orders whose refund windows are still open. Compare outcomes at compatible ages and within relevant product groups before interpreting a seller difference.

Even after those checks, a small observed gap is not a reliable basis for a confident ranking or punitive decision. Show counts, uncertainty and the scope of the comparison.

Define a mature outcome window

The original synthetic seller-cohort fixture records orders delivered on one date and refunds within thirty elapsed days. The analysis cutoff is February 1, 2026.

S1 and S2 each have twenty standard, budget-category orders delivered January 1. S1 has two refunds and S2 has one. S3 has twenty similar orders delivered January 25 with no refunds yet, but its thirty-day window is immature. S4 has ten complex, premium-category orders and two refunds.

S3 cannot be compared as if its zero were a completed thirty-day outcome. S4 differs on the supplied category and price-band fields. The comparable subset under this contract is S1 and S2.

SELECT seller_id,delivered_orders,refunded_within30,
 1.0*refunded_within30/NULLIF(delivered_orders,0) AS refund_rate
FROM seller_cohorts
WHERE date(delivered_date,'+30 days')<='2026-02-01'
 AND category='standard' AND price_band='budget'
ORDER BY seller_id;

The rates are 10% and 5%. The difference is one refunded order. That descriptive result does not establish a stable seller-quality difference or explain the reason for any refund.

Make small-sample uncertainty visible

The following Wilson intervals illustrate uncertainty under a simple independent-binomial model. That model is an assumption; repeated buyers, shared batches or other clustering may require a different analysis. NIST's proportion confidence-interval guide describes the Wilson approach.

from datetime import date, timedelta
from math import sqrt, isclose
from statistics import NormalDist
from build_and_verify import database

def wilson(successes,total,confidence=.95):
    if not 0<=successes<=total or total<=0:
        raise ValueError('invalid counts')
    z = NormalDist().inv_cdf((1+confidence)/2)
    p = successes/total
    denominator = 1+z*z/total
    center = (p+z*z/(2*total))/denominator
    half = z*sqrt(p*(1-p)/total+z*z/(4*total*total))/denominator
    return center-half,center+half

db = database()
rows = [r for r in db.execute('SELECT * FROM seller_cohorts')
        if date.fromisoformat(r[1])+timedelta(days=30)<=date(2026,2,1)
        and r[2:4]==('standard','budget')]
db.close()
assert [r[0] for r in rows]==['S1','S2']
intervals = {r[0]:wilson(r[5],r[4]) for r in rows}
assert isclose(intervals['S1'][0],.0278664812,abs_tol=1e-9)
assert isclose(intervals['S1'][1],.3010336452,abs_tol=1e-9)
assert isclose(intervals['S2'][0],.0088814488,abs_tol=1e-9)
assert isclose(intervals['S2'][1],.2361311934,abs_tol=1e-9)
print(intervals)

The approximate 95% intervals are 2.79%–30.10% for S1 and 0.89%–23.61% for S2. They show how imprecisely each underlying rate is estimated under the stated model. Overlap of individual intervals is not a formal test of the difference; use an appropriate difference analysis if that is the decision question.

Separate seller responsibility from the recorded outcome

Refunds may arise from product defects, delivery problems, buyer preferences or platform processes. The fixture contains no reason or responsibility evidence. A seller score should not attribute every refund to a seller-controlled failure without investigating the event contract.

Category and price-band matching also does not establish complete comparability. Order value, customer mix, geography and fulfillment method may still differ. If important groups have no overlap, a model cannot manufacture a direct comparison without assumptions.

For a large marketplace, repeated comparisons among many sellers create additional selection and uncertainty issues. A dashboard that highlights the most extreme raw rate will often select small, noisy groups. Require adequate evidence and a review process appropriate to the consequence.

Exercise: advance the cutoff until S3's thirty-day window matures, then supply its final refund outcome. Show that eligibility changes because of observation time, while the earlier snapshot remains reproducible.

NeuraPath's Data Analytics with Generative AI course connects cohort design with uncertainty analysis. A defensible marketplace report makes comparisons compatible before turning them into decisions.

Continue learning

This article is part of the Domain analytics and business cases sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Hospitality occupancy and revenue metrics with cancelled bookings.
• Continue with Build a domain analytics capstone from a vague business question.