Export analysis results with a data dictionary and manifest

Export the result together with a data dictionary and a run manifest. The dictionary explains fields, units, grain and missing values. The manifest identifies the inputs, code, environment and quality controls used to produce the files.

A CSV alone cannot tell its recipient whether 57000 means rupees, paise, a complete total or a known subtotal with missing amounts.

Inspect the supplied bundle

The pandas quality lab includes export_bundle.py and a generated example-export directory. Run python export_bundle.py from the lab to regenerate three local files:

The script overwrites these named files in its example-export directory. It performs sequential local writes and does not claim atomic publication of the bundle.

Verify the bundle in a temporary destination

import hashlib
import json
from pathlib import Path
from tempfile import TemporaryDirectory
import pandas as pd
from export_bundle import export_bundle

with TemporaryDirectory() as directory:
    destination = Path(directory)
    manifest = export_bundle(destination)
    saved = json.loads((destination / 'run_manifest.json').read_text(encoding='utf-8'))
    assert saved == manifest
    for name, expected_hash in manifest['outputs_sha256'].items():
        assert hashlib.sha256((destination / name).read_bytes()).hexdigest() == expected_hash
    dictionary = json.loads((destination / 'data_dictionary.json').read_text(encoding='utf-8'))
    assert dictionary['fields']['amount_paise']['unit'] == 'INR paise'
    restored = pd.read_csv(destination / 'paid_orders.csv',
                           dtype={'order_id': 'string', 'customer_id': 'string', 'amount_paise': 'Int64'})
    assert len(restored) == 7 and restored['order_id'].is_unique
    assert restored['amount_paise'].isna().sum() == 1
    assert int(restored['amount_paise'].sum(min_count=1)) == 57000
    assert manifest['quality']['row_quality_status'] == 'partial_missing_amounts'
    print('Schema, checksums, row count and nullable monetary controls verified')

The to_csv reference documents encoding, index and missing-value output options. The lab writes UTF-8, omits the dataframe index and represents missing fields as empty CSV fields under its dictionary contract.

Explain what a hash proves

A SHA-256 hash identifies file bytes. It helps detect whether a source or output changed, but it does not prove that those bytes represent correct business facts.

The manifest hashes both source CSVs and the three Python modules involved in generation. That makes a changed customer dimension or calculation file visible even if the main order file remains unchanged.

The manifest does not include a hash of itself, which would create a self-reference problem. Its listed output checksums cover the result CSV and dictionary; the manifest can be versioned or signed separately if a production process requires stronger provenance.

Keep quality attached to the export

The bundle contains seven Paid orders, six observed amounts and one missing amount. Its known subtotal is 57,000 paise, equivalent to INR 570.00. The complete Paid total remains unknown.

Including that status prevents the recipient from interpreting a successful file write as complete financial coverage. The unmatched customer indicator also survives the export so missing region causes remain inspectable.

Design for the receiving tool

CSV does not enforce types. A spreadsheet or downstream script may infer identifiers or blank values differently. Provide import guidance and test a round trip using the intended schema.

If exact type preservation is essential, evaluate a typed format supported by the receiving system, while retaining the same dictionary and provenance principles. Do not change formats solely because one is fashionable; the handoff must work for its actual consumer.

Exercise: edit one amount in a copied exported CSV and show that checksum validation fails. Then regenerate the bundle from a deliberately changed source and explain why a new valid checksum still requires reviewing the changed business result.

NeuraPath's Data Analytics with Generative AI course connects analytical output with reproducible handoff. A useful export lets another person understand, verify and correctly interpret the data without relying on the author's memory.

Continue learning

This article is part of the Pandas wrangling and data checks sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the preceding task in Validate a pandas pipeline against a tiny golden dataset.
• Return to the cluster foundation in Pandas merge validation: catch many-to-many joins early.