Classification metrics: DS interview guide

Why classification metrics trip candidates up

You will get a classification metrics question in almost every ML screen at Google, Meta, Stripe, or DoorDash. The bar is rarely whether you can recite the formula for precision — it is whether you can defend a metric choice under business pressure. The interviewer wants to hear you connect TP, FP, FN, and TN to a concrete cost function for the product you are pretending to ship.

Candidates lose points in two predictable ways. They either jump to accuracy on a 1%-positive fraud dataset, or they recite F1 like a magic word without saying which kind of error hurts more. Both signal that the candidate has memorized the slides but has not used the metric on a real model in production. The fix is not more memorization — it is having one or two stories where the metric choice changed the launch decision.

The framing that gets you to senior signal is simple: the metric is a proxy for a business loss function, and your job is to argue the proxy. If a checkout fraud model misses a fraudster, the company eats the chargeback. If it blocks a real shopper, the company loses lifetime value plus support cost. Those two errors are almost never equal, which is why accuracy is almost never the answer.

Confusion matrix, the slow way

The confusion matrix is just a 2x2 (or NxN) table of what the model predicted against what the label actually was. Memorize the layout, because half of the precision/recall confusion in interviews comes from getting the axis wrong on a whiteboard.

                  Predicted Positive   Predicted Negative
Actual Positive   TP                   FN
Actual Negative   FP                   TN

The four cells map cleanly to business outcomes. TP is a caught fraudster, a delivered email that got opened, a churn save. FN is the bad thing you missed — the disease you did not screen for, the click you failed to predict. FP is the false alarm — the legitimate transaction you blocked, the cancer screening that scared a healthy patient. TN is the negative correctly left alone.

Precision and recall

Both come from the confusion matrix, both are ratios, and they always trade off at a fixed model. Precision answers "of everything I flagged positive, how much was real". Recall (also called sensitivity or TPR) answers "of everything that was actually positive, how much did I catch".

precision = TP / (TP + FP)
recall    = TP / (TP + FN)

The trade-off shows up as soon as you move the decision threshold. Lower the threshold and you catch more positives — recall goes up, precision drops because you scoop up more noise. Raise the threshold and precision climbs while recall falls. There is no free lunch on a fixed model; the only way to push both up at once is to train a better model.

The table above is the cheat sheet. If you can produce something like it in the interview, you have already passed the metric question.

F1, F-beta, and the cost trade-off

Once you have precision and recall as two numbers, you want one number to compare models. The naive move is the arithmetic mean. Do not do that — it lets a model with precision 99% and recall 1% score 50%, which is misleading. The harmonic mean penalizes the lower of the two.

F1 = 2 * (precision * recall) / (precision + recall)

F1 treats precision and recall as equally important. That is almost never true in production, but it is a reasonable default when you have not done the cost analysis yet. The general form is F-beta, which lets you weight recall beta times more than precision.

F_beta = (1 + beta**2) * (P * R) / (beta**2 * P + R)

Beta less than 1 weights precision more (use F0.5 for spam filters, ad approval, content moderation false-takedown risk). Beta greater than 1 weights recall more (use F2 for cancer screening, churn-save outreach, fraud where chargebacks are cheap to investigate). Saying "I would use F2 here because a missed churner costs us $1,200 CLV while a wasted save email costs $0.03" is the kind of answer that gets you to the next round.

How to pick a metric in an interview

The honest answer is: it depends on the cost matrix, the class balance, and whether you can tune a threshold. Saying "it depends" is not enough — you have to walk through the dependencies out loud.

If the classes are roughly balanced and the costs of FP and FN are similar, accuracy is fine. This is rare in interesting problems. If the dataset is imbalanced — fraud, rare-disease screening, click prediction at 0.5% CTR — accuracy becomes useless because predicting all-negative scores 99.5%. Move to F1, PR-AUC, or a cost-weighted metric.

If you have not picked a threshold yet and want a threshold-free score, use ROC-AUC when classes are balanced and PR-AUC when positives are rare. PR-AUC is more sensitive to changes in the positive class, which is what you actually care about in fraud and recommendation problems. ROC-AUC can look deceptively healthy on a 99-1 imbalance.

For multi-class problems, you choose between macro, micro, and weighted averaging. Macro-F1 averages F1 across classes equally — it punishes you for ignoring rare classes. Micro-F1 pools all TP/FP/FN globally, which collapses to accuracy in single-label classification. Weighted F1 weights by class support — useful when class frequency reflects real-world prevalence and you do not want the rare class dominating the score.

Common pitfalls

The first pitfall is reporting accuracy on imbalanced data. A churn model on a 5%-monthly-churn book that predicts "no churn" for everyone hits 95% accuracy and zero business value. The fix is to always check class balance before quoting accuracy, and to default to PR-AUC or F1 the moment positives drop below 20% of the data.

A second trap is comparing F1 across different thresholds and pretending the comparison is fair. F1 depends on where you cut the score; the same model can produce wildly different F1 values at 0.3 versus 0.5 versus 0.7. If you are comparing two models, either fix the threshold the same way (precision at 90%, recall at 90%, equal-error-rate point) or compare threshold-free metrics like ROC-AUC and PR-AUC.

The third mistake is forgetting that the test set has to look like production. If you trained on a 1:1 oversampled set but production traffic is 1:99, your recall on the test set is meaningless. The fix is to either evaluate on a held-out set that matches production prevalence or to reweight the test metrics. Interviewers love asking about this — it separates people who have shipped from people who have only done Kaggle.

A fourth pitfall is treating ROC-AUC as the universal goodness metric. ROC-AUC is the probability that a random positive is scored above a random negative. On a 99-1 imbalance, a model can post ROC-AUC of 0.95 and still flag mostly false positives at any usable threshold. Check PR-AUC as a sanity overlay, and look at the precision at the recall level you actually need to ship.

The fifth one is ignoring calibration. A model with great F1 can still output garbage probabilities — 0.7 might mean a 40% true rate. If downstream systems rely on those probabilities (expected-value pricing, fraud scoring with a cost threshold, lead scoring with capacity constraints), you also need Brier score or a calibration plot. F1 says nothing about whether 0.7 really means 0.7.

Related reading

• Class imbalance for DS interviews
• Bias-variance tradeoff for DS interviews
• How to calculate F1 score in SQL
• How to calculate confusion matrix in SQL
• Regression metrics for DS interviews

If you want to drill DS interview questions like this one across confusion matrix, F1, ROC-AUC, and calibration, NAILDD ships with hundreds of classification-metrics problems graded to your level.

FAQ

Is F1 always better than accuracy?

No. F1 is more informative on imbalanced data, but on balanced datasets with balanced costs, accuracy is fine and more interpretable. F1 also hides which side of the trade-off your model is failing on — you can have the same F1 from "precision 0.9, recall 0.5" and "precision 0.5, recall 0.9", and those are very different production stories. Report both alongside F1 if you can.

How do I explain ROC-AUC versus PR-AUC quickly?

ROC-AUC plots TPR against FPR across thresholds; PR-AUC plots precision against recall. ROC-AUC is symmetric in positives and negatives, which makes it look great on imbalanced data even when the model is useless at production thresholds. PR-AUC focuses on the positive class and degrades visibly when the model fails on rare positives. Use ROC-AUC on balanced binary problems and PR-AUC when positives are rare or expensive to miss.

What is the difference between macro, micro, and weighted F1?

Macro-F1 computes F1 per class and averages them with equal weight, so a rare class with awful F1 drags the score down. Micro-F1 pools all TP, FP, and FN across classes and computes one F1 — in single-label problems this equals accuracy. Weighted F1 averages per-class F1 weighted by class support, which is useful when rare classes are rare in production too and you do not want them dominating the score.

Should I report precision-at-K instead of recall?

Yes, when capacity is the binding constraint. Recommender systems, fraud queues, lead scoring at a call center — all have a fixed number of slots K. Precision@K measures how many of your top-K predictions are positive. It is the operational metric in ranking problems where you cannot act on everything the model flags.

How do I handle calibration on top of F1?

Calibrate the probabilities with Platt scaling or isotonic regression on a held-out set, then re-evaluate F1, Brier score, and a reliability plot. If downstream consumers need probabilities (expected-value pricing, capacity-constrained routing), calibration matters more than the last 2 points of F1. Some teams at Netflix and Airbnb routinely accept slightly worse F1 in exchange for better-calibrated scores.

What metric do interviewers actually want to hear at FAANG?

They want to hear a story, not a formula. Open with the cost matrix — "missing a positive costs X, false alarm costs Y" — pick a metric that matches that asymmetry, name the threshold strategy, then mention one or two pitfalls (imbalance, calibration, prevalence shift). If you can defend why F2 beats F1 for the product in 30 seconds, you have already cleared the bar at most teams.