Model evaluation metrics interview questions, computed by hand

Start with one confusion matrix

A hypothetical fraud model scored 1,000 transactions, 50 of which were fraud. At a threshold of 0.5 it flagged 60, of which 40 were fraud.

tp, fp = 40, 20        # flagged: 60
fn = 50 - tp           # fraud missed
tn = 1000 - tp - fp - fn

precision = tp / (tp + fp)
recall = tp / (tp + fn)
specificity = tn / (tn + fp)
accuracy = (tp + tn) / 1000
f1 = 2 * precision * recall / (precision + recall)
print(f"TP {tp} FP {fp} FN {fn} TN {tn}")
print(f"precision {precision:.3f} recall {recall:.3f} specificity {specificity:.3f}")
print(f"accuracy {accuracy:.3f} F1 {f1:.3f}")
print(f"accuracy of flagging nothing: {(1000 - 50) / 1000:.3f}")

Precision 0.667, recall 0.800, specificity 0.979, accuracy 0.970 and F1 0.727. A model that flags nothing scores 0.950 accuracy — which is why accuracy alone is the wrong headline here.

Questions, weak answers and strong answers

ROC-AUC by counting pairs

Interviewers sometimes ask what AUC means concretely. It equals the probability that a randomly chosen positive scores higher than a randomly chosen negative, with ties counting half. You can compute it by counting pairs:

scores_pos = [0.9, 0.8, 0.55, 0.4]
scores_neg = [0.7, 0.5, 0.3, 0.2, 0.1]

wins = 0.0
for p in scores_pos:
    for n in scores_neg:
        wins += 1.0 if p > n else 0.5 if p == n else 0.0
auc = wins / (len(scores_pos) * len(scores_neg))
print(f"{wins:.0f} of {len(scores_pos) * len(scores_neg)} pairs ordered correctly, AUC {auc:.2f}")

Seventeen of twenty positive-negative pairs are ordered correctly, so AUC is 0.85. The explanation interviewers want: AUC measures ranking, not probability accuracy, and it is unchanged if you apply any order-preserving transformation to the scores.

Choosing a threshold from costs

Question: "Missing a fraud costs ₹8,000 on average; reviewing a flagged transaction costs ₹150. Where should the threshold be?"

The strong answer computes expected cost at each candidate threshold rather than picking the F1 maximum. With the hypothetical counts below:

missed_cost, review_cost = 8000, 150
# threshold: (TP, FP) among 1,000 transactions with 50 fraud
candidates = {0.7: (30, 5), 0.5: (40, 20), 0.3: (46, 80), 0.1: (49, 300)}

for t, (tp, fp) in candidates.items():
    fn = 50 - tp
    cost = fn * missed_cost + (tp + fp) * review_cost
    print(f"threshold {t}: flagged {tp + fp:>3}, missed {fn:>2}, cost Rs {cost:,}")

The costs are ₹1,65,250 at 0.7, ₹89,000 at 0.5, ₹50,900 at 0.3 and ₹60,350 at 0.1. With these assumptions, 0.3 is cheapest — a lower threshold than the one that maximises F1, because a missed fraud costs more than fifty reviews. Point out that the ranking changes if review capacity is limited, which turns the problem into "top N by score" instead.

Follow-ups to prepare

• Class imbalance: say you would report precision, recall and PR-AUC, and consider class weights; resampling changes the base rate, so recalibrate probabilities afterwards.
• Multi-class: macro averaging weights every class equally; micro averaging weights every row equally and is dominated by large classes.
• Validation design: a metric is only as trustworthy as the split behind it; for time-ordered data, test on a later period.
• Business metric link: end by tying the offline metric to what changes in production — reviews per day, losses avoided, customers contacted.

The same cut-off-from-economics idea appears in the banking and credit case bank, and the ML fundamentals bank covers the model-side questions. Metrics chosen from business costs are practised on case datasets in the Data Science programme.