Python generators for processing a large file incrementally

A generator yields values as a consumer requests them, which can avoid loading every record into a list. It does not guarantee constant memory: deduplication keys, customer sets, sorting and downstream collections can still grow with the dataset.

Design the complete data flow before claiming that adding yield makes a report suitable for a large file.

Stream outcomes instead of hiding invalid rows

The Python reporting lab is intentionally small and reads its source bytes once. The following separate example demonstrates an incremental CSV interface using the same validation contract.

import csv
from decimal import Decimal
from pathlib import Path
from report import FIELDS, parse_order, month_bounds

def outcomes(path):
    with path.open(encoding="utf-8-sig", newline="") as handle:
        reader = csv.DictReader(handle, strict=True)
        if reader.fieldnames != FIELDS:
            raise ValueError("unexpected_csv_schema")
        for record_number, row in enumerate(reader, 1):
            try:
                if None in row or any(value is None for value in row.values()):
                    raise ValueError("malformed_csv_record")
                order = parse_order(row)
            except ValueError as error:
                yield "rejected", (record_number, str(error))
            else:
                yield "valid", order

seen = {}
raw_count = rejected = replays = paid_orders = 0
total = Decimal("0.00")
start, end = month_bounds("2026-01")
for kind, value in outcomes(Path("raw_orders.csv")):
    raw_count += 1
    if kind == "rejected":
        rejected += 1
        continue
    order = value
    if order.order_id in seen:
        if seen[order.order_id] != order:
            raise ValueError("conflicting_order_key")
        replays += 1
        continue
    seen[order.order_id] = order
    if order.status == "Paid" and start <= order.order_date < end:
        paid_orders += 1
        total += order.amount_inr
assert (raw_count, len(seen), rejected, replays) == (11, 7, 3, 1)
assert (paid_orders, total) == (4, Decimal("47.50"))
print("Streamed outcomes reconcile with the reference report")

The generator does not accumulate the raw file. The consumer still retains one validated object per unique order in seen, so memory grows with unique orders. This example demonstrates incremental parsing, not a bounded-memory deduplication system.

Python's generator documentation describes suspension and resumption around yield. The reporting policy remains explicit in the consumer.

Account for state that crosses chunks

An identical replay can occur far from its first appearance. Deduplicating only the current batch misses that replay. A global in-memory set may be sufficient for a moderate dataset; larger workflows can use a database uniqueness constraint or an external sort under a documented conflict policy.

Exact distinct-customer counts also require state. Summing distinct counts from separate chunks double-counts customers who appear in more than one chunk. A running total is additive; a distinct count is not generally additive.

Know when exceptions occur

Calling a generator function creates an iterator; its body runs as the consumer advances it. A malformed CSV record halfway through the file can therefore fail after earlier values were already consumed.

If the consumer has written partial output, define whether that output is provisional, discarded or recoverable. Do not label the report complete before the iterator finishes and reconciliation succeeds.

The with block inside this generator closes the input on normal exhaustion or generator closure. If a caller stops early while retaining the generator, close it explicitly or use a lifecycle pattern that guarantees cleanup.

Measure the whole workflow

Calling list(outcomes(path)) would immediately materialize all outcomes and remove the main streaming benefit. Sorting all yielded records also requires a separate memory or external-processing strategy.

Benchmark with representative row sizes and key cardinality. A file with one repeated customer stresses a distinct-count structure differently from a file where every row has a new customer. The small fixture establishes correctness, not a production memory benchmark.

Exercise: move the repeated R3 record to the end of a much larger synthetic file. Verify that your deduplication policy still catches it, and document which data structures grow with input size.

NeuraPath's Data Analytics with Generative AI course connects Python iteration with reliable reporting. A scalable design explains both what it streams and what it must remember.

Continue learning

This article is part of the Python foundations for analysts sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Debug a Python KeyError in messy business data.
• Continue with Use Git to review changes to an analysis script.