Articles

Imbalanced Classification Interview Questions: Sampling, Metrics, and Deployment Prevalence

Practice imbalanced classification interviews: sampling leakage, precision-recall metrics, probability correction, and deployment prevalence.

A classifier can keep the same recall and false-positive rate while its precision collapses after deployment. The missing variable is prevalence: the proportion of examples that actually belong to the positive class. Strong imbalanced classification interview answers connect that population statistic to training, evaluation, and the final decision.

For machine learning and data science candidates, the useful starting point is three separate choices: how to represent training examples, how to evaluate on a realistic population, and where to set an operating threshold. Balancing the first does not automatically solve the other two.

Evidence boundary: Library behavior below comes from official documentation. No question is presented as a verified report from a particular employer. Start with Build and evaluate imbalanced binary classifier, then use the examples here to justify your decisions.

What should you ask before choosing a sampling method?

Ask what the positive label means, when that label becomes available, and what happens after the model flags an example. A rare event is not necessarily difficult to distinguish; a balanced dataset is not necessarily easy. Feature overlap, label noise, sample size, and the cost of each mistake matter alongside the class ratio.

Consider a synthetic abuse-review system. The label is a confirmed abusive account, and a prediction sends an account to human review. Investigators can process 500 accounts daily. A useful answer asks whether the objective is catching more abuse within that queue, meeting a minimum precision, or minimizing a stated cost. Those objectives can select different thresholds.

Then establish the evaluation unit. If one account generates many events, a random event split can put the same account in training and testing. If deployment predicts future accounts, a chronological split may better represent the task. Stratification preserves class proportions in a random split; it does not prevent entity overlap or future information leakage.

Finally, count positives as well as percentages. A test set with ten positive examples cannot support a stable estimate of 99% recall. One missed positive changes measured recall by ten percentage points. Request the confusion counts and uncertainty before comparing tiny metric improvements.

Why is 99% accuracy an inadequate answer?

In a synthetic population with 1% positives, always predicting negative yields 99% accuracy and zero positive recall. Accuracy correctly describes the fraction of correct decisions, but it hides whether the rare class is detected.

Define the quantities explicitly: precision is TP / (TP + FP), recall is TP / (TP + FN), and false-positive rate is FP / (FP + TN). Here TP means true positives, FP false positives, FN false negatives, and TN true negatives. The denominators answer different questions. Precision concerns the flagged queue; recall concerns all actual positives.

The official scikit-learn precision-recall example explains these definitions and average precision. Average precision summarizes precision across recall increments; it is not necessarily the same number as a trapezoidal area under the plotted curve. Specify which implementation you report.

Interview objective	Useful evaluation	What it leaves unresolved
Find positives within 500 daily reviews	Precision and recall at the review budget	Performance when daily volume changes
Compare ranking across thresholds	ROC-AUC and average precision	Which operating point is affordable
Give both classes equal influence	Balanced accuracy	Unequal financial or operational costs
Make probability-based decisions	Calibration plus an appropriate probability loss	Whether available features separate classes
Choose an action under known costs	Expected cost at a selected threshold	Whether assumed costs remain valid

ROC-AUC is not automatically useless under imbalance. It measures discrimination across class-conditional score distributions. However, a good overall ROC-AUC can coexist with poor performance in the tiny false-positive-rate region your application can tolerate. Show that region and actual counts. F1 also has limits: it combines precision and recall without representing true-negative benefits or every business cost.

How can unchanged model behavior produce worse precision?

Suppose a fixed threshold gives 80% recall and a 5% false-positive rate. Compare two synthetic populations, each containing 100,000 examples. Assume their class-conditional score distributions are unchanged; only prevalence differs.

Quantity	Population A: 10% positive	Population B: 1% positive
Actual positives	10,000	1,000
Actual negatives	90,000	99,000
True positives at 80% recall	8,000	800
False positives at 5% FPR	4,500	4,950
Precision	64.0%	13.9%
Total flagged examples	12,500	5,750

The model has not become worse at separating either class under this assumption. Most flagged examples become false positives because negatives are much more numerous. Compute production workload before celebrating an unchanged recall figure.

Using prevalence p, true-positive rate t, and false-positive rate f, the derivation is:

Expected true positives  = N × p × t
Expected false positives = N × (1 − p) × f
Precision = p × t / (p × t + (1 − p) × f)

These are calculated expectations, not reported production results. At 1% prevalence, achieving 50% precision with 80% recall would require an FPR of about 0.808%. That is a requirement to check against the model's achievable curve, not a promise that moving its threshold can deliver it.

Where do undersampling, oversampling, and class weights belong?

Official guidance: imbalanced-learn documents two problems with resampling before splitting: information can leak across the split, and evaluation can use an artificial class distribution. Put resampling inside the training procedure, including each cross-validation training fold. Keep validation and test distributions appropriate to deployment. See its common pitfalls guide.

Random undersampling reduces training cost but discards majority-class variation. Random oversampling retains examples but repeats minority observations; repetition does not create new independent evidence. SMOTE interpolates between minority examples in a feature space. Whether those synthetic points are meaningful depends on the representation, neighborhood structure, and label quality.

Class weighting changes the loss contribution of examples rather than their count. In scikit-learn's LogisticRegression documentation, class_weight="balanced" uses weights inversely proportional to observed class frequencies. That is a statistical weighting rule, not an automatic encoding of your false-negative cost.

A defensible experiment compares an unweighted baseline, a weighted model, and a sampling approach under identical evaluation rules. Keep preprocessing inside the appropriate folds too. For grouped or chronological data, the split design comes first; a sampler cannot repair an invalid split.

A useful spoken workflow is: “I reserve the final test population, fit transformations and resampling only inside training folds, select the model on validation evidence, then choose calibration and operating thresholds without using the final test labels.” Calibration and threshold selection also need sufficient held-out data or a cross-fitting design; they are not free steps after model selection.

Do not claim that weighting and duplicating observations are universally equivalent. They may resemble one another for a particular empirical loss, yet regularization, optimization, stochastic batches, and the learner's implementation can change the result. Compare the behavior you actually intend to deploy.

Does a probability of 0.8 still mean 80% after balancing?

Not necessarily. A model trained on a deliberately balanced sample may estimate probabilities for that sample's distribution. A ranking that looks useful can therefore produce misleading absolute probabilities for the deployment population.

Here is a synthetic derivation under a narrow assumption: sampling changes only the class prior, and P(features | class) stays the same. Let the sample positive prevalence be q and deployment prevalence be p. If the sampled posterior is correctly estimated, Bayes' rule gives:

Deployment odds = Sample odds × [p / (1 − p)] / [q / (1 − q)]
Probability = odds / (1 + odds)

With q = 0.5, p = 0.01, and sampled probability 0.8, sample odds are four. Corrected odds are 4 / 99, giving deployment probability 4 / 103, approximately 3.88%. A strong answer explains the assumption before applying the formula.

Prior adjustment is an established research topic, including Saerens, Latinne, and Decaestecker's paper. This example uses known priors; it does not implement their iterative procedure for estimating unknown priors.

The correction is unsafe as a universal remedy. Feature-dependent sampling, changed fraud behavior, new acquisition channels, or synthetic samples that alter class-conditional structure can violate its premises. An inaccurate sampled posterior also remains a problem. Use representative labeled calibration data where available and inspect reliability across relevant cohorts. The official calibration guide explains why fitting and calibration data must be separated appropriately.

How should you choose a deployment threshold?

Separate ranking quality from action selection. Scikit-learn's official threshold-tuning guide makes this distinction and warns against using the same data to fit a model and tune its decision threshold.

For a simplified binary decision, assume calibrated deployment probability s, false-positive cost C_FP, false-negative cost C_FN, and zero cost for correct decisions. Predicting positive costs (1 − s) × C_FP in expectation; predicting negative costs s × C_FN. Choose positive when:

s > C_FP / (C_FP + C_FN)

This is a derived decision rule under stated assumptions. It does not handle limited review capacity, heterogeneous case values, or interventions that change outcomes without additional modeling. Under a fixed queue budget, selecting the highest-value cases can be more appropriate than maintaining an unchanging probability threshold.

Avoid reporting only the threshold number. Include its validation period, confusion counts, queue size, calibration evidence, and the intended population. If prevalence is uncertain, recompute expected precision and workload over several plausible values. That sensitivity analysis exposes a brittle launch plan before deployment.

What should you monitor after launch?

Track missing features, cohort proportions, score distributions, flagged volume, and queue delays. Later, measure precision, recall where ascertainable, and calibration using labels with comparable maturity.

A change in score distribution alone does not prove concept drift. It can reflect changed prevalence, a pipeline bug, or a legitimate change in traffic. Conversely, stable aggregate scores can hide a failing small cohort.

Selective labeling deserves special attention. If investigators label only flagged accounts, their decisions cannot directly establish recall across all accounts. A carefully designed audit sample can help estimate missed positives. Record how examples receive labels and preserve that sampling information when estimating population metrics.

When asked what you would ship, give an evidence-backed operating plan: the model, threshold or review policy, expected workload, relevant uncertainty, and a measurable rollback condition. “Use SMOTE and optimize F1” leaves most of that decision unanswered.