Build a synthetic fraud benchmark with transparent assumptions

Synthetic data can make a tutorial reproducible without exposing customer records. It can also encode an unrealistically easy problem and produce impressive but meaningless metrics. A useful benchmark documents its grain, generator, split roles and deliberate limitations.

Publish the benchmark contract

Our original fixture has 8,000 unique synthetic transactions and 500 repeated synthetic accounts. Authored periods contain 3,000 train, 1,000 calibration, 1,500 validation, 1,500 test and 1,000 post-policy rows.

Fitted features are a continuous risk signal, log amount and 24-hour velocity. An unobserved account effect influences the generated outcome, which creates dependence across an account's transactions. The post-policy period shifts the risk mix and applies an authored reduction to the target logit for 493 high-risk rows.

Generator probability and label-flip indicator remain in the CSV for teaching audits but are explicitly excluded from fitted features. A separate noisy-label copy makes label-quality exercises possible without corrupting the clean target used by the main reference model.

from imbalance_cases import benchmark_case

r = benchmark_case()
assert r['rows'] == r['unique_transactions'] == 8000
assert r['unique_accounts'] == 500
assert r['split_counts']['train'] == 3000
assert r['excluded_generator_fields'] == ['generator_probability', 'generator_label_flipped']
assert r['post_policy_treated'] == 493
print(r)

Run this check and regenerate the CSV from a fixed seed in the imbalanced-model lab. The result exercises evaluation code; it does not estimate real fraud prevalence or recovered value.

Make leakage easy to detect

Use names that expose forbidden fields instead of hiding them in a broad feature matrix. Validate the exact feature allowlist before fitting. Preserve decision time and label-availability time so a learner can test point-in-time correctness. Keep stable transaction and account IDs for duplicate, grouping and lineage checks.

Assign each split one role. Calibration must not train the base model. Validation selects thresholds and policies. Test evaluates the frozen workflow. The post-policy period is a stress case, not a second chance to tune and claim final performance.

Document what the generator omits: evolving adversaries, investigation feedback, network relationships, merchant hierarchies, missing values, feature outages and real monetary consequences. A simple logistic rule can favour models aligned with that rule. Add challenge variants with interactions or drift, while keeping each mechanism declared.

Version the generator code, seed, schema, output hash and runtime. Regeneration should either reproduce the exact artifact or intentionally create a new benchmark version. Never mix rows from two versions under one result label.

Exercise: add a challenge version with an interaction unavailable to a linear model and a feature that arrives after decision time. Ensure the validator rejects the late feature, compare the baseline honestly and publish the changed generator assumptions before any score.

NeuraPath's Data Science course uses synthetic benchmarks to teach verification, not to simulate market credibility. Transparency about what was authored is the benchmark's main strength.

Continue learning

This article is part of the Imbalance, calibration and decision thresholds sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Detect label noise in an imbalanced dataset.
• Continue with Evaluate a classifier when labels arrive late.