# Synthetic prediction and evaluation lab

Original instructional account-snapshot dataset, not real account records or a deployed model. Forty fictional customer identities occur at eight decision dates, giving320 snapshots. Rows are simulated conditional snapshots with a shared customer effect; they are **not reconstructed from a continuous event history**. Therefore the fixture supports prediction/split/evaluation mechanics, not verification of cross-week event consistency or real business performance. `future_active_days` is an authored outcome summary; availability timestamps are declared metadata rather than authenticated ingestion evidence.

## Reproduce

From this directory, run `python evaluation_core.py` and `python verify.py`. The included `account-snapshots.csv` is the fixed input. `python make_fixture.py` regenerates it from seed20261004 on the recorded NumPy version; preserve the checked-in input and hashes when comparing environments. Dependencies used: Python3.12.0, NumPy2.4.4, pandas3.0.2, SciPy1.18.0, scikit-learn1.9.0. No download, API key or model service is required.

`experiment-contract.json` fixes the target, features, dates, C grid, model-selection rule and threshold. `experiment-results.json` records the actual comparison and source/code hashes. `predictions.csv` preserves validation and test row-level outputs. Verification checks data contracts, training-only imputation, group overlap, manual log loss and mutation rejection; it does not certify a production system.

## Prediction contract

At each UTC decision time, predict whether the account has zero active days during the next seven elapsed days, with a half-open [decision,horizon_end) interval. Labels become available one additional day after the horizon. This is inactivity, not account cancellation and not a causal estimate of what a reminder would change. A real event-based implementation also needs complete event coverage, identity rules, late-arrival handling and an as-of snapshot process.

Allowed predictors: days_since_activity, tickets_7d, tenure_days and plan. Missing tickets are imputed using the training median, with a missingness indicator. Future active days, target, identities, dates and split labels are excluded. A pipeline learns numeric imputation/scaling and category encoding only from training data. New categories use the encoder's documented unknown-category policy; that does not establish predictive reliability for them.

## Chronological split

| Set | Decision dates | Rows | Label-use cutoff |
|---|---|---:|---|
| Train | Jan1,8,15,22 2026 |160|Feb1 00:00 UTC|
| Validation | Feb5,12 |80|Mar1 00:00 UTC|
| Test | Mar5,12 |80|Mar20 00:00 UTC|

All40 customers appear in every set. This intentionally addresses later decisions for existing customers. GroupKFold on the January rows demonstrates a different question: holding out customer identities. It does not additionally enforce temporal ordering.

## Frozen comparison and actual outcome

The baseline predicts training prevalence69/160=.43125 for every row. LogisticRegression C candidates .1,1,10 are fitted on the160 training snapshots; validation log losses are .686153,.690101,.691314. The declared minimum-log-loss rule selects C=.1. The reference keeps that training-only fit and evaluates the March test set without a train+validation refit or test-driven threshold change. All decisions were specified before the first test report; subsequent runs reproduce that reference.

On validation, model/baseline log loss is .686153/.726907. On test it is .702749/.754582. At the preset .5 threshold, test confusion counts are TN24,FP1,FN33,TP22: accuracy57.5%, precision95.652%, recall40%. The baseline predicts no positives at this threshold, so its precision is undefined, not a measured zero-success fraction. Test prevalence55/80=68.75% differs substantially from training prevalence43.125%.

The model improves the selected probability metric against this baseline, but its low recall and changing prevalence require discussion. No pilot readiness, calibrated probabilities, commercial uplift or superiority on real data is established. Later threshold examples use validation outputs and preserve the original test result.

Primary implementation references: [scikit-learn pipelines and leakage](https://scikit-learn.org/stable/common_pitfalls.html), [DummyClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.dummy.DummyClassifier.html), [GroupKFold](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GroupKFold.html), [log loss](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.log_loss.html). Stable documentation showed1.9.1 when checked; local execution used1.9.0.

## Metric and uncertainty extension

Run `python metric_cases.py` for eight separately recorded checks in `metric-verification.json`; `python render_metrics.py` creates the original ROC/PR SVG and PNG with source coordinates. The predictions input is the frozen `predictions.csv` produced above.

Paired customer bootstrap:40 customers with2 test rows each,10000 resamples,seed20261006. Mean model-minus-baseline log loss=-.0518328752, percentile95 interval[-.1033098527,.0014551507], SD.0267878383. The interval includes zero. Resampling holds models and test dates fixed, preserves paired losses and customer groups, and does not include training/selection uncertainty or arbitrary future time shifts. Equal customer sizes make row-mean and customer-mean estimands agree here.

Separate regression case: five actuals all10. A=[10,10,10,10,20] has MAE2,RMSEsqrt20; B=13 throughout has both metrics3; C=[10,10,10,10,0] matches A's symmetric errors. Hypothetical cost1 per overpredicted unit and5 per underpredicted unit yields totals10,15,50. These are authored score/cost examples, not trained regression results or financial estimates.

Separate rare-outcome case:20 positives with scores ten .9,six .6,four .1;980 negatives with scores ten .8,thirty .4,940 .05. Threshold.5 yields TN970,FP10,FN4,TP16; all-negative accuracy98% with recall0 and undefined precision. Repeating all negative counts tenfold leaves ROC AUC.98877551 unchanged but decreases threshold precision16/26 to16/116 and average precision.75128205 to.55090312. Score distributions conditional on class are held fixed by construction; these are not trained/calibrated probabilities.

Validation threshold sweep for the actual synthetic logistic fit: thresholds .3/.5/.7 give [TN,FP,FN,TP] of [9,24,7,40], [28,5,28,19], [33,0,47,0]. It is development analysis and does not replace the frozen .5-threshold March test report.

## Assessment, provenance and clean reproduction

Run `python assessment_cases.py` for seven additional checks and `error-audit.csv`. Nested January group CV uses5 outer folds and3 inner folds, with C grid[.1,1,10] and training-fold pipelines. Outer losses .768050,.689932,.632938,.660268,.633134 average.676864, versus baseline.683843. Model worse in two folds. This assesses held-out customer identities in January, not future March predictions.

Error audit preserves80 snapshot rows and counts FN33,TN24,TP22,FP1. Plan slices have individual n52/38positives/12TP and team n28/17positives/10TP. Missing tickets has only7rows/3positives. Under-seven-days activity has19positives, all missed at.5. These are descriptive conditions, not verified failure causes.

`requirements.txt` pins the model-lab dependencies, including Narwhals2.22.1. A separate Python3.12.0 virtual environment with system site packages disabled reproduced predictions with maximum absolute difference8.3266726847e-17 against tolerance1e-10 and passed16 contract checks; `pip check` reported no broken requirements. Evidence is in `../../qa/ds-evaluation-clean-environment.json`. Initial dependency resolution selected a newer unpinned Narwhals, then the reference version was explicitly pinned before the reported reproduction. Plotting is separate from the minimal model requirements.

Run `python reproduce.py` to compare with reference outputs without overwriting them. `python provenance.py` verifies16 artifact hashes and exact split membership in `evidence-manifest.json`; `--build` intentionally creates a new manifest and must not be used to hide an unexplained mismatch. Hashes do not authenticate source truth or approval.

The completed [model card](MODEL_CARD.md) and [teaching project/rubric](../../projects/evaluated-inactivity-model.md) preserve the weak recall, interval crossing zero and synthetic-data limitations. Under the project's explicitly illustrative pilot gates, this model is not ready for that pilot even though its educational implementation reproduces.