Document the trade-off between automation and manual review

An automation rate is incomplete without the errors made automatically and the work transferred to people. A model-assisted process needs a route for every score region, ownership for every queue and evidence about reviewer performance.

Start with a reconciled batch

Our synthetic validation policy uses three routes:

The counts reconcile to all 1,500 rows. Automated candidates cover 82.4% of the batch. Their observed synthetic label error rate is 4.77%. Human decisions are required for 264 rows, or 17.6%.

The words “candidate” matter. The example calculates routing. It does not authorize clearing or holding a real transaction.

from imbalance_cases import automation_case

r = automation_case()
assert r['batch_total'] == 1500
assert r['human_decisions_required'] == 264
assert r['actions']['between'] == 'manual_review'
assert r['auto_negative'] + r['manual_review'] + r['auto_positive'] == 1500
print(r)

Run it in the imbalanced-model lab. The same fixture supports label-delay and capacity checks so the routes can be analysed as one system.

Write the operating contract

For each route, record the trigger, action, owner, deadline, override authority and audit fields. Define the unit of work: transaction, account or case. Specify tie handling at boundaries, missing-score fallback and what happens when the review queue is full.

Measure automated false negatives and false positives separately. For manual review, measure turnaround, agreement, escalation and eventual mature outcomes. Reviewers are not a gold standard by default; adjudication and sampling design determine label quality.

Compare end-to-end policies under the same cohort. One policy may automate more but increase customer friction; another may reduce model errors while creating an impossible backlog. Add service-time distributions and peak arrivals rather than relying only on average review count.

Policy changes alter future training data. Automatically cleared cases may receive less investigation, while held cases generate more labels. This selective feedback can bias retraining. Preserve exploration or audit samples where appropriate and document which outcomes are observable under each action.

Exercise: write a one-page decision table for the three routes, add a 200-case review limit and specify an overflow policy. Simulate reviewer accuracy and label delay as stated assumptions, then report end-to-end errors and unresolved cases rather than model-only metrics.

NeuraPath's Data Science course connects modelling to human workflow and auditability. Automation becomes defensible when the manual path is measured with the same care as the model path.

Continue learning

This article is part of the Imbalance, calibration and decision thresholds sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Evaluate a classifier when labels arrive late.
• Continue with Monitor calibration after a business-policy change.