Make a reporting script safe to rerun

A reporting script is safe to rerun when repeating the same intended operation does not create duplicate effects or silently corrupt an existing result. For local report preparation, that can mean reusing a verified output bundle for identical inputs and creating a distinct version when the evidence changes.

The definition must match the effect. Preparing a file once does not automatically make email delivery, database updates or downstream publication idempotent.

Define what makes one run distinct

The original automation lab computes a run identity from source bytes, source-manifest bytes, canonical configuration, pipeline code and Python version.

Identical evidence resolves to the same output directory. A changed region filter, source correction or code change creates a different identity. The generated timestamp is recorded in the bundle but does not determine the analytical selection or force a new run on every retry.

Python's hashlib documentation describes the hashing primitives used. A hash identifies bytes under the chosen algorithm; it does not prove that the data are complete, correct or authorized for distribution.

Verify reuse in a temporary output directory

import json
from pathlib import Path
from tempfile import TemporaryDirectory
from pipeline import run,verify_bundle

config = json.loads(Path('report-config.json').read_text())
with TemporaryDirectory(prefix='report-rerun-example-') as temporary:
    first,created = run('events.csv','source-manifest.json',config,temporary)
    before = {path.name:path.read_bytes() for path in first.iterdir()}
    second,created_again = run('events.csv','source-manifest.json',config,temporary)
    assert created is True and created_again is False
    assert first==second
    assert before=={path.name:path.read_bytes() for path in second.iterdir()}
    assert verify_bundle(second)['distribution']=='not_sent'
    north_config = dict(config,region='North')
    third,new_version = run('events.csv','source-manifest.json',north_config,temporary)
    assert new_version is True and third!=first
    assert json.loads((third/'metrics.json').read_text())['amount_paise']==1000
    assert first.exists()
    print({'identical_run_reused':True,'changed_filter_new_version':True})

The second call leaves every file byte unchanged, including the original creation timestamp. The changed filter produces a separate 1,000-paise North report while preserving the all-region result.

Stage a complete bundle before exposing it

The implementation writes metrics, selected events and a manifest into a temporary sibling directory. It verifies output hashes and then renames the complete bundle into its final location on the same filesystem.

Injected failures after writing metrics or immediately before the final rename leave no completed bundle in the tested workflow. A downstream reader should consume only a finalized directory with a valid manifest, not an arbitrary file it finds in a staging area.

The Python OS documentation describes rename behavior and platform differences. This local design does not establish power-loss durability, cross-filesystem atomicity or equivalent behavior in object storage. Those require additional mechanisms and tests for the actual deployment environment.

Fail when an existing bundle has changed

Before reuse, the pipeline verifies the recorded output checksums. If a metrics file was edited, the rerun fails rather than overwriting it. That preserves evidence of the inconsistency and avoids silently replacing an artifact someone may already have reviewed.

The manifest itself is not digitally signed. An actor who can rewrite both a file and its expected hash can defeat this accidental-corruption check. Access controls and stronger integrity mechanisms are separate concerns when the environment requires them.

Keep preparation separate from external effects

The lab's manifest says prepared_for_review and not_sent. No email or other external action is performed. A distribution workflow needs its own authorization, recipient scope, delivery identity and retry handling.

Even a stable run ID is insufficient if the code sends a message before recording that it sent it. A crash in that interval can create duplicate delivery on retry. Treat that as a separate transaction and reconciliation problem rather than assuming file idempotency solves it.

Exercise: modify one prepared metrics file and rerun with the original inputs. Verify that checksum validation fails and the modified file is not overwritten. Then document the review process for resolving that discrepancy.

NeuraPath's Data Analytics with Generative AI course connects reusable Python scripts with operational reliability. A safe rerun has an explicit identity, verifies prior evidence and scopes exactly which effects it controls.

Continue learning

This article is part of the Reliable reporting automation sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Automate a weekly report with an explicit reporting cutoff.
• Continue with Detect incomplete source files before generating a report.