Articles

Probability Calibration Interview Questions: Reliability Curves, Brier Score, and Decision Thresholds

Practice probability calibration interview questions with reliability curves, Brier score, ECE binning failures, leak-safe fitting, and cost-based thresholds.

A classifier ranks every positive example above every negative one. Its AUC is perfect on the evaluation sample. Can you trust a predicted probability of 0.9? That ranking result does not answer the question, and it does not tell you which decision threshold minimizes cost.

Probability calibration interview questions separate three claims: which cases rank higher, whether numerical probabilities match observed outcomes, and which action those probabilities justify. Reliability curves and Brier scores help investigate the second claim; a decision policy needs additional assumptions.

The worked numbers are original synthetic fixtures, not candidate reports or employer interview predictions. Library facts and primary research are linked at first use, checked September 30, 2026.

What does it mean for probabilities to be calibrated?

For binary outcomes, calibration asks whether cases assigned probability near p have a positive-outcome frequency near p. If many comparable cases receive 0.8, approximately 80% should be positive under the distribution being evaluated. A single positive or negative outcome cannot establish calibration at that score.

Official documentation: scikit-learn describes calibration curves as comparing predicted probabilities with observed positive fractions. Its curve function groups predictions into bins and returns each bin's mean predicted probability and fraction of positives; empty bins are omitted. Probability calibration, calibration_curve API.

Calibration is distribution-dependent. A model evaluated on last year's population may encounter a different base rate, measurement process, or subgroup mix after deployment. A probability is not permanently certified by one attractive plot.

A constant predictor equal to the population's positive fraction can be calibrated in a simple marginal sense while failing to distinguish individual risk. Useful probabilities usually need both credible numerical values and meaningful separation between cases.

My recommendation is to ask which population, time period, label definition, and prediction horizon the claim concerns. “Calibrated for next-month renewal on recent accounts” is testable. “The model is calibrated” leaves the evaluation target unfinished.

How do you read a reliability curve without overclaiming?

Plot mean predicted probability on the horizontal axis and observed positive fraction on the vertical axis. Agreement lies near the diagonal. A bin around predicted 0.8 with observed frequency 0.5 suggests overconfidence in that region; a bin around 0.2 with observed frequency 0.4 suggests underprediction there.

Those are finite-sample estimates. Add bin counts and uncertainty appropriate to the sampling unit. A point based on three cases deserves different confidence from one based on thousands. If repeated observations come from the same customer, do not automatically treat them as independent customers.

Equal-width bins preserve probability intervals but can contain very few cases. Quantile bins distribute observations more evenly, while changing the probability span of each bin. API fact: calibration_curve supports uniform and quantile binning strategies. Binning options.

Report the binning choice rather than making two models look comparable through differently chosen bins. Also inspect the prediction distribution. An apparently good high-probability region means little if almost no predictions fall there.

Overall agreement can hide segment-specific errors that cancel. Check important slices when sample size and the deployment question support them, and display their uncertainty. Searching many tiny slices until one looks alarming is not a substitute for a planned diagnostic.

What does binary Brier score measure?

For binary labels y_i in {0,1} and positive-class probabilities p_i, this article uses Brier = mean((p_i − y_i)²). Lower is better; under this binary convention the score lies between zero and one. Official Brier score API.

Official documentation: Brier score is a proper scoring rule, but its value reflects calibration and discrimination-related properties as well as outcome uncertainty. A lower score alone is not proof that a model has better calibration. Calibration and proper scoring rules.

Use the original four-observation fixture y = [0, 0, 1, 1]. Model P outputs [0.1, 0.2, 0.8, 0.9]; model Q outputs [0.4, 0.45, 0.55, 0.6]. Both place every positive above every negative, so both have AUC 1 on this sample.

Their binary Brier scores differ: P scores 0.025; Q scores 0.18125. P assigns probabilities closer to the observed labels, although neither four-point result establishes population calibration. The example demonstrates that ranking leaves the numerical probabilities undetermined.

A second useful comparison is the constant base-rate predictor. Evaluate it on the same untouched population and state how its base rate was estimated. Comparisons across populations with different outcome prevalence require care because the underlying prediction problem changed.

Specify positive class, binary versus multiclass convention, and any sample weights when reporting a library result. Otherwise two values named “Brier score” may use different definitions. Do not copy a binary threshold interpretation onto a multiclass aggregate loss.

Why can expected calibration error hide obvious problems?

A common binned expected calibration error, or ECE, is the weighted sum of abs(mean probability − observed positive fraction) within bins. The binning policy is part of that estimate.

In an original 20-observation fixture, ten predictions equal 0.2, with eight positive labels. Another ten equal 0.8, with two positives. Each region is badly wrong: the low-score group has an 80% positive fraction, and the high-score group has a 20% fraction.

Combine all observations into one wide bin. Mean probability is 0.5 and the overall positive fraction is 0.5, so one-bin ECE is zero. Separate the two score groups into two bins and ECE is 0.6.

Binary Brier score is 0.52 for this fixture, compared with 0.25 for a constant 0.5 predictor. The broad-bin cancellation did not make the predictions useful. These exact arithmetic results concern the constructed sample, not a measured production model.

ECE therefore needs its definition, bin count, weighting, and evaluation population. A number without those details is hard to interpret. My recommendation is to retain a reliability curve, counts, a proper score, and decision-level checks alongside the scalar summary.

More bins are not automatically more truthful. Sparse bins create noisy estimates; excessively broad bins can hide opposite errors. Choose a defensible diagnostic policy, disclose it, and investigate whether conclusions survive reasonable alternatives.

How should calibration be fitted without leaking evaluation labels?

Separate model fitting, calibration fitting, threshold selection, and final evaluation. The same physical partition need not be reserved for every step in every method, but the final claim needs data untouched by decisions chosen to improve that claim.

Official documentation: scikit-learn's calibration workflow uses predictions on data not used to fit the corresponding base estimator. Its calibration tools provide cross-validation mechanisms; fitting a calibrator to in-sample predictions can produce biased results. Calibration fitting guidance.

For a simple original interview plan, train the base model on the training set, fit calibration on a separate calibration set, choose a cost or capacity policy on validation data, and report final performance on the held-out test set. Keep preprocessing and hyperparameter decisions within their appropriate training procedures too.

Respect time and entity boundaries. Multiple records from the same customer can leak information across random splits. A future deployment question may need a forward-looking time split rather than a convenient random one.

Sigmoid calibration fits a restricted transformation; isotonic calibration allows a more flexible monotonic relationship and can overfit when calibration data are scarce. Calibration methods. Choose and validate the method rather than assuming greater flexibility guarantees improvement.

This article's arithmetic fixtures do not fit either method. Replacing one score vector with another by hand is a counterexample about metrics, not evidence that a learned calibrator improved a held-out model.

What does temperature scaling preserve, and what does it not prove?

Temperature scaling divides logits by a positive scalar before applying softmax. Primary research: Guo and coauthors evaluate this method as a post-processing approach to neural-network calibration, fitting a temperature on held-out data. On Calibration of Modern Neural Networks.

For one example, dividing all its class logits by the same positive number preserves their ordering and the predicted argmax class. That explains how probabilities can change while the model's chosen class remains the same.

Do not stretch that statement into “all rankings and metrics remain unchanged.” Multiclass probabilities for a particular class depend on the full logit vector; preserving each example's winning class does not guarantee every cross-example class-score ranking stays fixed. Binary positive-temperature scaling does preserve ordering of the binary logit differences.

A fitted temperature also does not certify calibration in every subgroup or after distribution shift. Check held-out reliability and decision outcomes. State which loss was optimized and which population supplied calibration labels.

If an interviewer asks for implementation, define the logits, positive-temperature parameterization, held-out objective, and numerical stability before naming an optimizer. Then verify the transformation does not accidentally refit the base network or touch final test labels.

When do costs imply a decision threshold?

Suppose the deployment probability p is credible, a false positive costs 1, a false negative costs 100, and correct decisions cost zero. Predicting positive has expected cost 1 − p; predicting negative has expected cost 100p.

Choose positive when 1 − p < 100p, or p > 1 / 101, approximately 0.00990. At exact equality the actions have equal expected cost; declare a tie policy.

In this original example, a probability of 0.005 favors negative: costs are 0.995 versus 0.5. A probability of 0.02 favors positive: costs are 0.98 versus 2. A default threshold of 0.5 would answer a different cost problem.

Official documentation: scikit-learn distinguishes probability estimation from the action threshold and warns against tuning a threshold on the same data used to train the classifier. Decision threshold tuning.

The formula assumes binary actions, known costs, credible deployment probabilities, and no capacity constraint. A review queue limited to 100 cases per day needs a constrained policy; three possible actions need a larger cost model. False-positive and false-negative costs may also vary by case.

After calibration, evaluate expected cost, workload, and relevant safeguards on held-out data, then monitor delayed labels and distribution changes after deployment.

What should you verify in code and in practice?

The saved standard-library check ran on Python 3.12.14. It verified both Brier scores, equal AUC from pairwise ranking, one-bin and two-bin ECE, the constant-score baseline, and the cost comparison around 1/101. No fitted calibrator or real dataset was evaluated.

The follow-ups below are original practice suggestions; question details may require Premium and do not establish a universal employer sequence.

Interview question	Calibration-specific follow-up
Derive and implement calibration via temperature scaling	Separate argmax preservation from probability reliability.
Build leak-safe sklearn model with calibration	Identify which labels train the model, calibrator, and policy.
Build and evaluate airline delay prediction model	Evaluate forward in time with calibration and operational costs.
Explain ROC-AUC vs PR-AUC tradeoffs	Explain why a ranking curve cannot validate numerical probabilities.
Choose ML metrics under asymmetric costs	Derive a threshold and state its probability and capacity assumptions.

Return to the temperature-scaling exercise. A complete answer says what the transformation preserves, what evaluation can demonstrate, and what the action policy still needs.