Data science interview: choose a metric from business costs

There is no universally best classification metric. The useful question is which error changes the decision, how often it occurs and what constraint the system must respect. Accuracy, ROC AUC or F1 can summarize behaviour, but none automatically represents money, safety or review capacity.

Calculate one explicit trade-off

The career evidence lab compares two invented candidates. A missed positive costs 1,200 units and an unnecessary review costs 40.

from career_cases import metric_cost_case

result = metric_cost_case()
assert result["total_cost"] == {"A": 16000, "B": 11200}
assert result["selected"] == "B"
assert result["cost_saved_vs_other"] == 4800

Model B produces more false positives but fewer false negatives, reducing the stated total by 4,800. This decision is conditional on the authored counts and cost assumptions. Change the costs, threshold, prevalence or capacity and the choice may reverse.

Translate operations into evaluation

Ask the interviewer or state assumptions:

• What action follows a positive prediction?
• What happens after a false positive and false negative?
• Is the output a probability, class or ranked queue?
• How many cases can people or systems act on?
• When do labels mature, and which outcomes remain unobserved?

For a fixed review budget, precision at *k*, recall at *k* and gain over the current queue can be more direct than a threshold-free curve. For probability-based decisions, assess calibration as well as discrimination. For severe imbalance, always report class prevalence and confusion counts beside rates.

Test the assumptions

Estimate costs with domain owners and show a sensitivity table rather than presenting one value as fact. Break results down by time and relevant groups. Add uncertainty around the cost difference; a small observed advantage may not justify operational change. Account for abstentions and cases where the current policy already intervenes.

In an interview, connect the metric to model selection and the threshold to deployment. The validation set can choose the threshold under the declared cost or capacity rule. The locked test set estimates the final policy; it should not become a threshold-search surface.

The Data Science course makes this reasoning visible through decision briefs, confusion counts, calibrated probabilities and held-out policy evaluation.

Exercise

Recalculate the fixture across false-negative costs from 100 to 2,000 and review capacities from 50 to 200. Plot where the preferred candidate changes, then write the questions needed to replace assumed costs with operational evidence.

Continue learning

This article is part of the Data science careers and portfolio decisions sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in Data science interview: diagnose leakage in a suspicious score.
• Continue with Data science interview: interpret a failed experiment.

Reference: scikit-learn model-evaluation documentation.