Create a backfill plan with bounded impact

A backfill recomputes historical data after corrected logic, late events or a contract change. The transformation may be simple; the operational risk is not. A large rewrite can overload a source, mix old and new logic, publish partial partitions or leave model training with an unexplained population change.

Bound the work before executing it.

Produce a dry-run manifest

The data-engineering lab builds a ten-day example and groups no more than two partitions into a batch.

from data_engineering_cases import backfill_case

result = backfill_case()
assert result["dry_run_only"] is True
assert len(result["partitions"]) == 10
assert len(result["batches"]) == 5
assert result["maximum_partitions_per_batch"] == 2
assert result["total_estimated_rows"] == 10900

This output is a plan, not evidence that a backfill ran. That distinction matters. The manifest identifies dates, estimated rows, batch boundaries, publication method and rollback action without mutating the approved dataset.

A production dry run should also record source version, transformation commit, target location, expected bytes, executor limits, owner, maintenance window and downstream jobs to pause. Estimate the read and write load against actual capacity. If estimates are weak, run one representative partition and revise them.

Make each batch idempotent

Each partition needs a deterministic input window and output name. Retries must replace or merge the intended partition without duplicating records. Write into a versioned staging location, validate it, then publish it atomically where the storage system permits. Never expose a half-built collection of partitions as the approved training source.

For every batch, compare expected and observed row count, unique keys, sums or domain totals, null rates and schema. Reconcile a sample against the source. Record start/end time and status in a run manifest. A failure stops later publication; it does not erase the failed evidence.

Validate model impact separately

Data validity does not prove model equivalence. Rebuild the affected feature window, quantify how many entities and labels change, and run the relevant evaluation suite. If corrected data is supposed to change predictions, require a reasoned acceptance threshold rather than equality with the faulty version.

The rollback in the lab restores the previous approved pointer. It does not try to undo thousands of in-place row edits. This versioned pattern also keeps training reproducible: a model manifest can name the exact data version it consumed.

The Data Science course uses the backfill manifest alongside dataset contracts and model evidence so learners defend both data correctness and operational control.

Exercise

Extend the plan to 90 daily partitions. Set limits for rows, bytes, concurrent batches and source queries. Inject a validation failure on day 17, demonstrate that later partitions remain unpublished, and document resumption plus rollback.

Continue learning

This article is part of the Data engineering for data science sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Schema evolution without silently changing feature meaning.
• Continue with Compare Hadoop-era concepts with modern data-platform needs.

Reference: Apache Airflow backfill documentation.