Document dataset provenance in an experiment manifest

An experiment manifest connects a reported result to the data, code, configuration and split membership that produced it. It should let a reviewer distinguish a reproducible reference from a similarly named file that changed later.

Our original inactivity project has a complete evidence manifest. It identifies320 synthetic snapshots, forty fictional customers, the generator seed, target contract, input columns, exact snapshot IDs in each split and hashes for sixteen artifacts.

Describe origin before recording a checksum

The source is an authored conditional-snapshot simulation generated by make_fixture.py. It contains no real account records and does not reconstruct a continuous event history. Stating those facts prevents a technically accurate hash from giving the dataset more credibility than its origin supports.

For a real dataset, record the source system, extraction process, permitted use, coverage period, inclusion rules and known revisions. Avoid putting credentials or unnecessary personal data in the manifest. A reference to an access-controlled source can be more appropriate than copying sensitive records into an evidence package.

Capture the choices that change interpretation

The target contract distinguishes seven-day inactivity from cancellation, defines the half-open outcome window and adds one reporting day before labels become available. It names the allowed predictors and excludes future outcome fields.

Exact split IDs matter because two runs can have the same160/80/80 counts while assigning different observations. The manifest records membership and order, not just percentages. Model selection, threshold and baseline conventions are included in the referenced contract.

The artifact inventory covers the CSV, requirements, generator, model/evaluation code, checks, reference results, predictions and error audit. The manifest itself is excluded from its own artifact hash list to avoid a circular self-hash definition.

Verify the reference and detect a mismatch

import copy,json
from pathlib import Path
from provenance import verify

manifest = json.loads(Path('evidence-manifest.json').read_text(encoding='utf-8'))
result = verify(manifest)
assert result['artifacts_verified']==16 and result['rows']==320
changed = copy.deepcopy(manifest)
changed['artifacts']['account-snapshots.csv']['sha256']='0'*64
try:
    verify(changed)
except ValueError as error:
    assert str(error)=='artifact_changed:account-snapshots.csv'
else:
    raise AssertionError('mismatched checksum accepted')
print(result)

Run from the evaluation lab. The deliberate mismatch changes only an in-memory copy of the manifest; it does not modify the reference dataset.

Separate integrity from authenticity

A matching checksum means the bytes agree with the recorded reference. It does not establish that the source labels are correct, the timestamps reflect historical availability or an instructor approved the result.

Someone able to alter both an artifact and the manifest can create a new internally consistent set of hashes. Authenticated storage, access controls and signatures address different questions. This local teaching manifest makes no claim to provide those controls.

Preserve intentional revisions

When correcting a dataset or changing the experiment, record the reason, affected rows or rules, new evidence and relationship to the prior reference. Do not silently regenerate the manifest merely to make a failed check disappear.

The lab's provenance.py --build command creates an intentional new manifest, while provenance.py verifies the existing one. Keep those operations distinct in the workflow. Numerical reproducibility is checked separately by reproduce.py; a hash check alone does not rerun the model.

Exercise: propose a label correction for one snapshot. Write the revision note and identify which artifacts would need regeneration and review. Explain why preserving only the new dataset hash would be insufficient to reconstruct the changed result.

NeuraPath's Data Science course connects experiments with traceable evidence. A useful manifest records the source's limitations as carefully as the files' identities.

Continue learning

This article is part of the Machine learning workflow and evaluation sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Evaluate a model by meaningful data slices.
• Continue with Reproduce a model result from a clean environment.