Detect incomplete source files before generating a report

A CSV can parse successfully and still be incomplete. A truncated export may end cleanly after a valid row, and an upstream query may omit an entire partition without producing malformed text. Validate source evidence before turning the file into a report.

Use several checks with different purposes: schema, byte integrity, expected row count, key consistency and reporting-period coverage. No single check establishes every form of completeness.

Compare the file with a declared export manifest

The original automation lab supplies an events CSV and a source manifest. The manifest declares seven raw rows, a schema version, the SHA-256 of the data bytes, an event watermark and the extraction time.

The pipeline reads the bytes once for hashing and parsing. If the hash differs from the expected value, it stops before preparing output. A valid hash is followed by header, row-count and event-level checks.

This ordering catches an inconsistent delivery even when the remaining CSV rows are individually valid.

Inject two different export failures

Run this from the lab directory:

import json
from pathlib import Path
from tempfile import TemporaryDirectory
from pipeline import load_source,ReportError

config = json.loads(Path('report-config.json').read_text())
with TemporaryDirectory(prefix='source-validation-example-') as temporary:
    root = Path(temporary)
    data = root/'events.csv'
    manifest = root/'source.json'
    original = Path('events.csv').read_bytes()
    data.write_bytes(original[:-20])
    manifest.write_bytes(Path('source-manifest.json').read_bytes())
    try:
        load_source(data,manifest,config)
    except ReportError as error:
        assert str(error)=='source_hash_mismatch'
    else:
        raise AssertionError('truncated source accepted')
    data.write_bytes(original)
    metadata = json.loads(manifest.read_text())
    metadata['raw_rows']=8
    manifest.write_text(json.dumps(metadata),encoding='utf-8')
    try:
        load_source(data,manifest,config)
    except ReportError as error:
        assert str(error)=='source_row_count_mismatch'
    else:
        raise AssertionError('inconsistent row count accepted')
print({'truncation_rejected':True,'row_count_mismatch_rejected':True})

The first failure changes data bytes without changing the manifest. The second preserves the data hash but declares the wrong row count. Each produces a specific diagnostic rather than a plausible-looking partial report.

Validate records after file-level checks

The pipeline expects exact headers, UTC timestamps, nonnegative integer paise amounts, allowed status and region values, and consistent event chronology. It also rejects conflicting payloads for the same event ID while collapsing an identical replay.

The reference extract has seven raw rows but six unique events. That difference is expected because one event is replayed. A raw-row control and a unique-event control therefore measure different things; do not use one as a substitute for the other.

Malformed or conflicting records stop this bounded workflow. Another system might quarantine records and publish a clearly labeled partial result, but that requires an explicit business rule. Silent row dropping is not an adequate policy.

Check coverage beyond row count

A source watermark earlier than the report end indicates that the declared source coverage does not reach the required period. The pipeline refuses that input even if every available row is valid.

For partitioned data, also compare expected dates, regions or files with those received. A total row count can match by coincidence while one partition is missing and another duplicated. The retail availability capstone demonstrates an expected observation grid that survives missing rows.

Control totals from the source can strengthen reconciliation when they are independently generated and have a documented meaning. Preserve those controls with the extract rather than recomputing every “expected” value from the same potentially incomplete file.

State what the manifest does not prove

This teaching manifest is generated locally with synthetic data. It proves no independent upstream completeness claim. In a real workflow, an incorrectly generated file and an equally incorrect matching manifest can pass these checks together.

Define who produces the manifest, when the source considers a period complete, how late corrections are represented and which independent controls are available. Hashes also do not authenticate the sender unless combined with appropriate trusted delivery or signing arrangements.

Exercise: remove one region's records and generate a matching new hash and row count. Explain why those checks now pass, then add an independent expected-region control that detects the missing partition without inventing zero activity.

NeuraPath's Data Analytics with Generative AI course connects Python validation with reporting reliability. A useful automation verifies source evidence before producing a result and states which completeness questions remain outside its checks.

Continue learning

This article is part of the Reliable reporting automation sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Make a reporting script safe to rerun.
• Continue with Schedule a Python report with observable failures.