Conformal Prediction Interview Questions: Coverage, Exchangeability, and Distribution Shift

Conformal prediction can wrap almost any point predictor with intervals or prediction sets. The attractive phrase “distribution-free coverage” often causes the interview mistake: promising every user a calibrated 90% interval under any future distribution. The standard guarantee is finite-sample and powerful, but it is marginal and depends on exchangeability.

This guide calculates a split-conformal regression interval by hand, then breaks its assumption with a distribution-shift example. For a related model-diagnostics prompt, open Explain bias-variance, calibration, and model drift and keep probability calibration separate from prediction-set coverage.

Evidence boundary: Coverage and exchangeability claims use author tutorials, peer-reviewed research, and current MAPIE documentation. The residual fixture, monitoring thresholds, and interview wording are original preparation material.

What an interviewer is testing

A strong answer resolves six urgent questions:

1. What set or interval is the system trying to construct?
2. Which nonconformity score measures how unusual an outcome is?
3. How is the finite-sample quantile calculated?
4. Is the promised coverage marginal, subgroup, or conditional?
5. Why are calibration examples exchangeable with future examples?
6. What operational response follows when coverage or width drifts?

The article-specific thesis is: conformal prediction calibrates errors from representative held-out examples into a set-valued prediction. The guarantee concerns a future draw under exchangeability; it does not certify every subgroup, every input, or a shifted production stream.

Split conformal regression step by step

Start with observations split into three roles:

1. a training set fits a point predictor f;
2. a calibration set measures nonconformity without refitting f;
3. a test or future example receives an interval.

For basic symmetric regression, use absolute residual scores:

s_i = |y_i - f(x_i)|.

With n calibration scores and target miscoverage alpha, choose the corrected empirical quantile at rank:

k = ceil((n + 1) * (1 - alpha)).

Using the k-th smallest score q, return:

C(x_new) = [f(x_new) - q, f(x_new) + q].

Under exchangeability, the rank of the new score among the n + 1 calibration-plus-new scores is symmetric. That rank argument, rather than a Gaussian error assumption, drives the finite-sample marginal coverage guarantee. Tie handling and exact software quantile conventions must be implemented consistently.

Worked numerical question

Suppose nine calibration residuals, already sorted, are:

[1, 1, 2, 2, 3, 3, 4, 5, 7].

For 90% coverage, alpha = 0.10 and:

k = ceil((9 + 1) * 0.90) = 9.

The ninth score is q = 7. If the base model predicts 42 for a new case, the conformal interval is:

[42 - 7, 42 + 7] = [35, 49].

The maximum residual appears because the calibration sample is tiny. Small samples make the attainable coverage grid coarse. With the same nine scores and a 95% target, the requested rank is ceil(10 * 0.95) = 10, which exceeds the available calibration scores. A conservative implementation may return an unbounded interval rather than pretend that nine examples support that finite-sample target.

This is an important interview result: wider intervals or an infeasible target can be the correct output. Do not quietly switch to a lower uncorrected percentile to make the result look useful.

What the coverage guarantee says

For a conformal set C(X_new), the standard statement is:

P{Y_new in C(X_new)} >= 1 - alpha.

The probability averages over the calibration sample and a new exchangeable example. It is marginal coverage. A nominal 90% procedure can cover 90% overall while covering one subgroup 98% and another 70%. It also does not mean that a particular interval has a 90% probability of containing its fixed outcome under a frequentist interpretation.

Conditional coverage, such as coverage for every value of X, is a stronger target and generally cannot be assumed from basic split conformal prediction. Group-conditional or localized methods can pursue narrower guarantees, but they need enough representative data and add assumptions or efficiency costs. Report which population and condition the guarantee actually covers.

Coverage alone is also insufficient. An interval spanning every possible outcome covers perfectly and helps no decision. Measure efficiency with width for regression and set size for classification. Stratify both coverage and width by relevant groups, predicted difficulty, geography, device, time period, and other product slices.

Symmetric absolute-residual intervals also have constant width, even when error variance changes with X. A normalized score such as |y - f(x)| / sigma_hat(x) can adapt width to a separately estimated difficulty scale, while conformalized quantile regression starts from learned lower and upper quantiles and calibrates their misses. These methods can improve efficiency for heteroscedastic data, but the difficulty or quantile models need their own leakage-safe training, and the final conformal calibration still needs representative exchangeable examples.

Exchangeability is the load-bearing assumption

Exchangeability means that reordering calibration examples and the new example does not change their joint distribution. Independent and identically distributed observations are a familiar sufficient case, though exchangeability is slightly broader.

Ask what the observational unit is. Multiple rows from one patient, customer, machine, or document may be dependent. Randomly splitting rows can put related examples in both training and calibration, producing optimistic residuals. Split at the independent unit or use a method whose assumptions match the dependence.

Time creates another warning. Tomorrow's demand, sensor behavior, or fraud tactics may not be exchangeable with last year's calibration data. Random calibration can hide seasonality and regime changes. A time-ordered evaluation can show what happens at the intended prediction horizon, even if the standard split guarantee no longer applies directly.

The base model must not be fitted on the calibration labels used to compute scores in ordinary split conformal. Reusing data makes the residuals optimistic. Cross-conformal and jackknife-style methods use data more efficiently, but their procedures and guarantees differ; do not describe every resampling variant as identical to split conformal.

Distribution shift: the guarantee can fail

Suppose the calibration residuals came from an indoor temperature sensor with stable errors near 1–7 units. After deployment, the device moves outdoors and extreme weather increases errors to 8–15. The interval still adds q = 7, so coverage falls.

Conformal prediction did not violate its theorem. The new cases are not exchangeable with the calibration cases. “Distribution-free” means the method does not require a parametric distribution under its sampling assumption; it does not mean distribution-shift-free.

Monitor a labeled stream when outcomes arrive:

• rolling empirical coverage at each supported confidence level;
• interval width or prediction-set size;
• coverage by prespecified high-impact slice;
• distribution of nonconformity scores;
• input and base-prediction drift;
• label delay and missing-label selection;
• frequency of empty, singleton, or all-class sets.

MAPIE's documentation emphasizes that tests can find evidence against exchangeability but cannot prove that every violation is absent. A detected break should trigger investigation, recalibration on representative data, retraining of the base model, a wider or abstaining policy, or rollback. Define that response before launch.

Recent or weighted conformal methods may address particular forms of shift, and adaptive approaches can target long-run behavior in streams. Their assumptions and guarantees must be stated separately. Adding a recency weight does not preserve the ordinary exchangeable split-conformal theorem by default.

Classification and adaptive scores

For classification, a simple nonconformity score is 1 - p_y(x), the complement of the predicted probability assigned to the true class. Calibrate a threshold, then include labels whose scores meet it. More adaptive scores use cumulative ordered probability mass so easy examples can receive smaller sets and difficult examples larger ones.

Probability calibration and conformal coverage are related but distinct. A classifier can have imperfect probability calibration and still yield valid conformal sets under exchangeability because the empirical score quantile repairs marginal coverage. Better probability estimates often improve set efficiency, but they are not the source of the rank guarantee.

In multiclass deployment, inspect class-conditional coverage and set size. A model that returns {common_class} for most inputs and all labels for rare cases might meet aggregate coverage while imposing poor service on the rare classes.

Define how the product consumes the set. A medical-review queue may escalate when the set contains several labels; an automated routing system may abstain when the interval crosses a decision threshold. Evaluate the end-to-end policy, including review capacity and the cost of wide sets. The conformal layer supplies uncertainty evidence, while the application still owns the action.

Common wrong answers and precise repairs

Wrong answer	Why it fails	Better response
“Distribution-free means no assumptions.”	Exchangeability is essential.	Name the sampling unit and why calibration matches deployment.
“90% coverage holds for every subgroup.”	Basic split conformal is marginal.	Report slice coverage and pursue conditional goals explicitly.
“Use the ordinary 90th percentile.”	Finite samples need rank correction.	Compute ceil((n+1)(1-alpha)).
“Conformal makes the model more accurate.”	It wraps predictions with uncertainty sets.	Improve the base model to reduce width or set size.
“A coverage check proves exchangeability.”	One diagnostic cannot establish all forms of sameness.	Monitor scores, inputs, time, and outcome-based coverage.
“Recalibrate whenever width grows.”	Wider sets may correctly reflect harder cases.	Tie response to coverage, efficiency, shift, and product cost.

A concise interview answer template

Begin: “I will define the observational unit and justify why calibration cases are exchangeable with future cases before claiming coverage.” Then:

1. Separate training, calibration, and final evaluation data.
2. Define a task-appropriate nonconformity score.
3. Compute the finite-sample corrected quantile.
4. State the marginal coverage guarantee exactly.
5. Evaluate coverage and efficiency overall and by important slice.
6. Simulate temporal or population shift and define abstention or recalibration.
7. Monitor labeled outcomes and version the base model, scores, and calibration set.

That answer shows that uncertainty is an operational contract, not a confidence percentage pasted onto a point prediction.

Sources and Further Reading

• Angelopoulos and Bates: A Gentle Introduction to Conformal Prediction — practical theory, finite-sample coverage, and extensions.
• Lei et al.: Distribution-Free Predictive Inference for Regression — regression methods and theoretical treatment.
• Shafer and Vovk: A Tutorial on Conformal Prediction — foundational online and exchangeability perspective.
• MAPIE: Conformal Prediction Theory — current library explanation of marginal coverage and workflow.
• MAPIE: Exchangeability Testing — shift diagnostics, limitations, and monitoring context.
• MAPIE: Conformal Prediction Metrics — coverage, width, set size, and conditional diagnostics.

Sources checked September 13, 2026. All residuals, predictions, sensor scenarios, and response thresholds are synthetic.