Write useful logs for a scheduled analysis

Useful logs explain what a report attempted, how it progressed and why it failed or completed with limitations. They should help an operator locate the relevant input and output without reconstructing the entire calculation from console prints.

Keep the final data-quality summary as a separate artifact. A log line saying completed does not establish that every input row was accepted or that the report was approved for use.

Define events before choosing formatting

The Python reporting lab emits record_rejected warnings and a report_completed information event. Its summary separately records eleven raw records, seven accepted unique orders, three rejected records and one identical replay.

Those fields answer different questions. The warnings explain individual validation failures. The summary reconciles the entire input. An operator should not need to count warning lines to discover the final rejection count.

For a scheduled version, add a run identifier, reporting month, source identifier, start/end events and elapsed duration. Avoid using a customer's personal details as the run identifier.

Capture a logger without changing the calculation

The example below attaches an in-memory handler to the lab's logger, executes its loader and restores the original logging configuration afterward.

import io
import logging
from pathlib import Path
from report import load_orders, LOG

stream = io.StringIO()
handler = logging.StreamHandler(stream)
handler.setFormatter(logging.Formatter("%(levelname)s %(message)s"))
old_level, old_propagate = LOG.level, LOG.propagate
LOG.addHandler(handler)
LOG.setLevel(logging.INFO)
LOG.propagate = False
try:
    with Path("raw_orders.csv").open(encoding="utf-8-sig", newline="") as handle:
        accepted, rejected, replays, raw_count = load_orders(handle)
finally:
    LOG.removeHandler(handler)
    handler.close()
    LOG.setLevel(old_level)
    LOG.propagate = old_propagate
messages = stream.getvalue().splitlines()
assert len(messages) == 3
assert all("record_rejected" in message for message in messages)
assert sum("invalid_amount_format" in message for message in messages) == 2
assert sum("invalid_customer_id" in message for message in messages) == 1
print("Three rejection events captured with stable reasons")

The check validates event content without relying on timestamps or machine-specific paths. The Python logging guide describes logger levels, handlers and formatting. The fixture's event names and assertions are specific to this reporting workflow.

Choose severity according to action

An invalid row is a warning here because the lab permits a visibly partial report. A conflicting duplicate key is fatal because the program lacks a rule for selecting a winner. In a report that requires all rows to pass, the same invalid row could instead cause the run to fail.

Do not use severity as a substitute for business status. A scheduler should inspect the documented exit code and summary, while logs explain the operational path.

Keep logs useful under repetition

If a million records share the same validation failure, a million full warning messages may obscure the root cause and consume storage. A larger system can log aggregate counts by reason and retain detailed rejection records in a controlled artifact.

Sampling diagnostic examples must not change reconciliation. The rejected-record count should remain complete even if only a few examples appear in logs. Clearly label sampled diagnostics so readers do not interpret them as the whole rejected population.

Similarly, avoid attaching a new handler every time a reusable function runs. Duplicate handlers can produce repeated messages that look like repeated processing. Configure the application's logging boundary once.

Preserve evidence with appropriate scope

The lab logs record numbers and reason codes, not full raw customer rows. The original source and rejection CSV provide more detailed local evidence when needed. Production retention and access controls should reflect the data being processed.

A source hash can help connect an execution to exact bytes, but it does not prove that the source itself was correct. A run identifier connects events; a hash identifies content; a quality status describes the accepted result. Keep those meanings separate.

Exercise: add a run identifier to the logging context and execute January and February separately. Confirm that every event can be assigned to its run, while the business totals remain INR 47.50 and INR 17.00.

NeuraPath's Data Analytics with Generative AI course connects Python reporting with operational diagnosis. The useful deliverable is a report another analyst can investigate when its scheduled run behaves unexpectedly.

Continue learning

This article is part of the Python foundations for analysts sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Python pathlib: make scripts portable across computers.
• Continue with Python comprehensions: readability before cleverness.