Build a runbook for a report that fails overnight

A reporting runbook should help someone identify the failed stage, preserve evidence and choose a bounded recovery action. “Rerun the script” is not enough when the source is incomplete, the configuration targets the wrong period or an existing output has changed.

Write the runbook while the workflow is understood, then test it using failures another analyst can reproduce.

Start with the expected result and scope

The original teaching runbook accompanies a local synthetic report. The expected weekly result is three paid events totaling 3,500 paise, prepared for review and not distributed.

It identifies the intended period, source evidence, output bundle and status. It also states what is absent: no live scheduler, real credentials, external API connection or delivery service is configured.

A real runbook should similarly name the actual owner, backup and operational systems when those decisions exist. Do not invent a contact or escalation path merely to make a document look complete.

Distinguish no attempt from a failed attempt

If no attempt log exists, inspect the scheduling and execution environment: task status, host availability, account access and expected trigger. Application diagnostics cannot explain a process that never started.

If an attempt exists, inspect the exit code and fixed failure diagnostic. Then determine whether any completed bundle exists and verifies. A file in a staging directory is not a completed report.

Keep the intended reporting period visible throughout the investigation. A successful rerun for the wrong week does not resolve the original failure.

Map a diagnostic to the right next action

The runbook maps a stale watermark to source-readiness investigation, a conflicting event ID to identity/version reconciliation, and a checksum mismatch to preservation and inspection of changed output.

Here is a local stale-source exercise:

import json
from pathlib import Path
from tempfile import TemporaryDirectory
from pipeline import load_source,ReportError

config = json.loads(Path('report-config.json').read_text())
source = json.loads(Path('source-manifest.json').read_text())
with TemporaryDirectory(prefix='runbook-example-') as temporary:
    manifest = Path(temporary)/'source.json'
    source['event_watermark']='2026-01-11T23:00:00Z'
    manifest.write_text(json.dumps(source),encoding='utf-8')
    try:
        load_source('events.csv',manifest,config)
    except ReportError as error:
        assert str(error)=='source_watermark_before_period_end'
        action = 'Obtain a consistent source extract whose declared coverage reaches the required report end.'
    else:
        raise AssertionError('stale source accepted')
print({'diagnostic':'source_watermark_before_period_end','next_action':action})

The correct response is not to move the watermark forward by hand. Obtain source evidence that actually satisfies the contract, or make an explicit reviewed decision about a different reporting scope.

Preserve the evidence before attempting repair

Retain the failed configuration, source identity, diagnostic and any affected output version. Avoid overwriting the only copy of the incident input with a new extract before recording what changed.

The lab's build_fixture.py regenerates synthetic data. Its runbook warns against using that command as a repair operation on a real incident extract. A setup command and a recovery command can have very different consequences.

Logs and incident notes should not copy credentials or unrestricted source payloads. Record identifiers and fixed diagnostics, with restricted access to any detailed evidence needed for investigation.

Define recovery acceptance, not just a command

Recovery is complete when the intended source and configuration validate, metrics reconcile, the completed bundle verifies and the attempt returns the expected status. Any corrected version should preserve its relationship to the prior artifact.

If distribution requires review, preparation success does not bypass that step. Approval must apply to the corrected artifact and audience, and delivery requires its own evidence.

A runbook should also state when retrying is appropriate. A transient file-access failure may justify a bounded retry; an invalid business parameter needs correction; an authentication failure needs authorized access repair.

Turn the incident into a prevention check

After recovery, record the cause supported by evidence and add a targeted check where it would have detected the issue earlier. Avoid claiming a root cause when only a symptom is known.

Exercise: give a colleague a deliberately corrupted output bundle and the runbook. Ask them to identify the failure, preserve the changed file and explain the recovery decision without reading your private notes.

NeuraPath's Data Analytics with Generative AI course connects automation with operational ownership. A good runbook makes diagnosis and recovery reviewable for someone who did not write the script.

Continue learning

This article is part of the Reliable reporting automation sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Test a reporting pipeline with failure injection.
• Continue with Measure the time saved by automation honestly.