Probability calibration for DS interviews

What calibration actually means

A model is calibrated if its predicted probability matches the empirical frequency of the event. When the model says p = 0.7 across a batch of inputs, roughly 70% of those inputs should turn out positive. Anything else — 45%, 90%, anything in between — and the score is just a ranking signal dressed up in probability clothes.

This matters because half the senior DS interview loop assumes you understand the difference between ranking quality and probability quality. An AUC of 0.95 tells you the model orders examples well; it says nothing about whether 0.7 means seventy percent. Confusing the two is the most common trap on the whiteboard.

Most of the off-the-shelf classifiers you'll meet on the job are not calibrated out of the box:

Group with predicted p ≈ 0.7
  Calibrated model:     ~70% true positives
  Overconfident model:  ~90% (RF/GB/SVM pushed to extremes)
  Underconfident model: ~45% (regularized models compressed toward 0.5)

A quick mental model of which families behave how:

Most candidates assume "logistic regression spits out probabilities" — true, but only when the link function actually matches the data-generating process.

Why interviewers care

The question that splits mid-level from senior DS answers usually sounds like this: "The model has 0.95 AUC, but in production the probability scores are useless for thresholding. What's going on?" The answer is miscalibration. AUC is invariant to any monotonic transformation of the scores, so it cannot detect the gap between 0.7 predicted and 0.4 observed.

You need calibration whenever a downstream decision depends on the absolute value of p, not just the ranking:

• Risk scoring — probability of default, fraud, or churn feeding a hard threshold rule.
• Cost-sensitive decisions where the cutoff moves with business inputs (cost of a false positive, marketing budget, capacity).
• Model ensembles that combine probabilities from multiple classifiers — uncalibrated scores break the mixture.
• Regulated domains — credit, healthcare, insurance — where the score must be defensible as a probability, not just a ranking.

You don't need calibration when you only use argmax, or when the downstream system only cares about ranking (search results, recommendations, ad auctions with relative bidding). The cheap version of the interview question is: would shifting all scores by +0.1 change the decision? If yes, you need calibration. If no, you don't.

Reliability diagrams and metrics

The reliability diagram (also called the calibration curve) is the diagnostic you draw first. Bucket predictions into bins — typically [0.0, 0.1), [0.1, 0.2), ..., [0.9, 1.0] — then plot mean predicted probability on the x-axis against empirical positive rate on the y-axis. A perfectly calibrated model traces the diagonal. Overconfident models bow below the diagonal at the high end; underconfident models sit above it.

from sklearn.calibration import calibration_curve
import matplotlib.pyplot as plt

prob_pred, prob_true = calibration_curve(
    y_test,
    model.predict_proba(X_test)[:, 1],
    n_bins=10,
    strategy="quantile",  # equal-mass bins beat equal-width on skewed scores
)
plt.plot(prob_pred, prob_true, marker="o")
plt.plot([0, 1], [0, 1], "--")  # the diagonal of perfect calibration
plt.xlabel("Mean predicted probability")
plt.ylabel("Empirical frequency")

The numeric companions to the diagram are the metrics you quote in the interview:

• Brier score — mean((p - y)²). Lower is better. Decomposes into calibration + refinement + uncertainty, so a single number captures both probability quality and discrimination.
• ECE (Expected Calibration Error) — bin-weighted mean of |p_bin - y_bin|. Pure calibration signal, but sensitive to the binning scheme.
• MCE (Maximum Calibration Error) — the worst-bin gap. Useful when one tail of the distribution is what you actually care about (rare-event risk models).

A typical interview-grade range: Brier 0.05-0.15 for healthy production binary classifiers; ECE under 0.02 for a well-calibrated model after temperature scaling on a deep net.

Platt scaling

Platt scaling is the simplest calibrator: fit a one-dimensional logistic regression on top of the raw model scores.

p_calibrated = sigmoid(A · score + B)

A and B are learned on a held-out calibration set — or, better, via cross-validation so you don't waste training data. Platt works whenever the miscalibration has roughly a sigmoid shape, which is the default behavior of SVMs, tree ensembles, and many shallow models.

from sklearn.calibration import CalibratedClassifierCV

calibrated = CalibratedClassifierCV(
    estimator=model,
    method="sigmoid",
    cv=5,  # 5-fold CV avoids leakage from re-using training data
)
calibrated.fit(X_train, y_train)
prob = calibrated.predict_proba(X_test)

The downside is the rigid parametric form. If your miscalibration is asymmetric — say, the model is overconfident in the high range but underconfident in the low range — a single sigmoid cannot fix both ends simultaneously. That's when you reach for isotonic.

Isotonic regression

Isotonic regression is the non-parametric calibrator. It fits a monotonic step function minimizing squared error between scores and labels, with no shape constraint beyond monotonicity. More flexible than Platt, hungrier for data.

calibrated = CalibratedClassifierCV(estimator=model, method="isotonic", cv=5)

The trade-offs are predictable: isotonic fits any monotonic distortion (including the asymmetric kind that breaks Platt), but it overfits on small calibration sets. The rule of thumb you can quote in the room is roughly 1,000+ examples in the calibration set before isotonic stops being noisier than Platt. Below that, prefer Platt for the smoother fit.

Isotonic is also harder to extrapolate beyond the calibration data — predictions outside the observed score range get clipped to the boundary values, which can bite you in production if the score distribution drifts.

Temperature scaling for neural nets

For deep classifiers, temperature scaling is the dominant fix. Divide the pre-softmax logits by a scalar T learned on the validation set, then apply softmax as usual. One parameter, one line of code, surprisingly effective.

import numpy as np

def softmax(x, axis=-1):
    e = np.exp(x - x.max(axis=axis, keepdims=True))
    return e / e.sum(axis=axis, keepdims=True)

def calibrated_softmax(logits, T):
    return softmax(logits / T)
# T > 1 softens predictions (less confident)
# T < 1 sharpens predictions (more confident)

The Guo et al. (2017) paper that popularized this trick showed that modern image classifiers — ResNets, DenseNets — typically need T between 1.5 and 3.0 to recover calibration. Temperature scaling preserves the argmax, so accuracy is untouched; only the confidence is rescaled. The catch: it only fixes uniform miscalibration. If different regions of the logit space are miscalibrated in different directions, you need Platt or isotonic on top of softmax.

Choosing a method

Common pitfalls

The first trap is calibrating on the training set. Calibration must run on held-out data, ideally out-of-fold via CalibratedClassifierCV(cv=5). Using the same data that trained the base model leaks information about the labels into the calibrator and produces optimistic ECE numbers that collapse the moment you ship.

The second trap is confusing AUC for a calibration metric. AUC measures ranking quality and is invariant to any monotonic transformation of the scores — including a transformation that destroys calibration. A model with AUC 0.92 and ECE 0.18 is excellent at ranking and terrible at probability, and that combination shows up constantly in production. Brier score and ECE are the right tools here; quote them, not accuracy.

The third trap is ignoring class imbalance. If your training set is 10% positive but production runs at 2%, the base rate shift breaks calibration even when the base rates of features are unchanged. The fix is prior correction — divide and re-normalize using the new base rate — or re-calibration on a sample from the target distribution. Without one of those, your "calibrated" model produces probabilities calibrated against the wrong universe.

The fourth trap is re-calibrating an already calibrated model. Stacking CalibratedClassifierCV on top of a model that already has well-calibrated probabilities (a well-tuned logistic regression, for example) tends to add variance without reducing bias. Calibrate only the raw, uncalibrated scores from the base learner.

The fifth trap is forgetting that calibration drifts. Feature distributions move, the label prior shifts, seasonality changes the population. Calibration measured in January will be off by July. Re-evaluate Brier and ECE on a rolling window and re-fit the calibrator on a fresh slice when the drift exceeds your tolerance. This is also why daily monitoring beats quarterly model reviews for high-stakes scoring systems.

Related reading

• Class imbalance — Data Science interview
• Bagging vs Boosting — Data Science interview
• Classification metrics — Data Science interview
• How to calculate AUC-ROC in SQL
• How to calculate log loss in SQL

If you want to drill probability calibration questions like these the way they actually get asked in DS loops, NAILDD ships with hundreds of ML interview problems across exactly this pattern.

FAQ

Does Random Forest need calibration?

Usually, yes. Random Forests exhibit a well-known conservative bias — predictions get pulled toward the middle of the distribution on noisy examples and toward the extremes on clean groups. The averaging of bootstrap trees acts as a shrinkage operator on raw probabilities. Platt scaling typically gives a substantial Brier improvement, often 10-25%, with minimal effort. Isotonic is worth trying if you have enough calibration data.

Will calibration hurt AUC?

No. Platt, isotonic, and temperature scaling are all monotonic transformations of the score, and monotonic transformations preserve ranking. AUC depends only on ranking, so it is invariant to calibration. If you observe an AUC drop after calibrating, suspect a bug — most likely you re-fit the base model on a smaller dataset, or you accidentally introduced non-monotonicity.

Can I use calibrated probabilities for business decisions?

That is the entire point. Rules like "if P(fraud) > 0.95, block the transaction" or "if P(default) > 0.20, decline the loan" only behave correctly when the probabilities are calibrated. On an uncalibrated model, the 0.95 cutoff is a meaningless threshold on an arbitrary score scale and your block rate will drift with every retrain.

When does temperature scaling fail?

When the miscalibration is non-uniform — different regions of the logit space need different corrections. Temperature scaling is a single scalar, so it cannot fix mismatches that change direction across the score distribution. In that case, fit Platt or isotonic on top of the softmax outputs, or move to vector scaling / matrix scaling for a per-class temperature.

Which sklearn API should I use?

CalibratedClassifierCV is the standard. It handles cross-validated calibration for both method="sigmoid" (Platt) and method="isotonic", and it works with any classifier exposing decision_function or predict_proba. For neural nets, write temperature scaling by hand — sklearn does not ship it, but it's about ten lines of NumPy.

How much calibration data is enough?

For Platt, a few hundred examples can already stabilize the two parameters. For isotonic, aim for 1,000+ at minimum, ideally 5,000-10,000 for production reliability. The relevant quantity is positives in the calibration set, not total rows — a 1% positive rate at 100k rows gives you 1,000 positives, which is borderline for isotonic on a binary task.