Nested Cross-Validation Interview Exercise: Repair a Biased Model-Selection Result

Your teammate searches several model settings with cross-validation, reports the highest score, and calls it expected production performance. The code may run perfectly while the conclusion is biased. In this nested cross-validation interview exercise, the repair is to evaluate the entire selection procedure on outer folds that never participated in that fold's tuning.

For a data scientist or machine learning engineer, explaining that boundary matters more than remembering two nested loops. You need to identify what was selected, what data each decision inspected, and which performance claim the remaining evidence supports.

The data are synthetic, not a hiring benchmark or an estimate of any company's model quality.

Nested cross-validation: the answer you should defend

Nested cross-validation separates selection from evaluation. Within each outer training partition, an inner search chooses settings and refits its winner. That fitted winner is evaluated on the corresponding untouched outer test partition. The outer results describe the specified training-and-selection procedure under the chosen splitting assumptions.

Official mechanism: scikit-learn's nested-versus-non-nested example explains why reporting the best tuning score can be optimistic. It does not establish that nesting repairs an inappropriate split or gives an exactly unbiased estimate of future deployment performance.

The interview answer should name three separate outputs: an inner selection score, outer evaluation scores, and a final fitted artifact. Treating those as interchangeable is the defect this exercise asks you to repair.

Find the biased claim before changing the code

Here is the problematic pattern. Assume X and y contain all available development observations:

search.fit(X, y)
reported_performance = search.best_score_

The search used validation outcomes to select a winner. Reporting the same criterion as independent evidence rewards settings that fit both genuine signal and favorable variation in those outcomes. A fixed model evaluated with cross-validation and a model selected by maximizing cross-validation are different procedures.

The foundational Cawley and Talbot paper, published in 2010, analyzes overfitting to model-selection criteria. It supports the principle, not a universal correction factor. You cannot repair the result by subtracting five percentage points or quoting the difference from an unrelated dataset.

Ask what the teammate tried beyond the displayed grid. Feature sets, model families, missing-value strategies, thresholds, and random seeds can all become selection decisions. If the notebook contains twenty experiments and the final cell shows one grid, the evaluated procedure includes more choices than that cell reveals.

A runnable repair with fold-local preprocessing

The following original fixture uses independent synthetic rows, three regularization candidates, three inner folds, and three outer folds. It is deliberately small enough to trace. Installations may produce somewhat different numeric results; the assertions check the evaluation contract rather than a preferred score.

import numpy as np
from sklearn.base import clone
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV, StratifiedKFold
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X, y = make_classification(
    n_samples=240, n_features=14, n_informative=5,
    n_redundant=3, flip_y=0.08, random_state=17
)
pipeline = Pipeline([
    ("scale", StandardScaler()),
    ("model", LogisticRegression(max_iter=2000, random_state=17)),
])
grid = {"model__C": [0.01, 0.1, 1.0]}
inner = StratifiedKFold(3, shuffle=True, random_state=11)
outer = StratifiedKFold(3, shuffle=True, random_state=29)
template = GridSearchCV(
    pipeline, grid, scoring="roc_auc", cv=inner,
    refit=True, error_score="raise", n_jobs=1
)

non_nested = clone(template).fit(X, y).best_score_
receipts = []
seen_test_rows = []
for fold, (train_idx, test_idx) in enumerate(outer.split(X, y), 1):
    assert set(train_idx).isdisjoint(test_idx)
    search = clone(template)
    search.fit(X[train_idx], y[train_idx])
    receipts.append({
        "fold": fold,
        "inner_score": search.best_score_,
        "chosen_C": search.best_params_["model__C"],
        "outer_score": search.score(X[test_idx], y[test_idx]),
    })
    seen_test_rows.extend(test_idx.tolist())

assert sorted(seen_test_rows) == list(range(len(y)))
outer_scores = np.array([r["outer_score"] for r in receipts])
assert np.isfinite(outer_scores).all()
assert ((outer_scores >= 0) & (outer_scores <= 1)).all()
print("Selection score:", non_nested)
print("Outer scores:", outer_scores.tolist())
print("Outer mean:", outer_scores.mean())
print("Selection receipts:", receipts)

 # Only after the procedure is fixed: fit the deliverable on all development rows.
final_search = clone(template).fit(X, y)
final_model = final_search.best_estimator_

Each outer fit creates a fresh search. It receives only that fold's training rows. Because scaling is inside the pipeline, the scaler is learned again within each inner training split, then learned on the outer training partition during refit. The outer test rows are transformed using that fitted scaler, never used to fit it.

scikit-learn's common-pitfalls guidance recommends pipelines to keep learned preprocessing inside the appropriate training boundary. A pipeline does not prevent you from passing a target-derived feature that should never have existed; feature availability still needs a separate audit.

Before accepting the run, deliberately break one boundary in a separate test fixture. For example, duplicate an outer test index into its training index and confirm that the disjointness assertion fails. Then restore the correct indices. An assertion that never sees a failing case may be checking the wrong object. Also record the library version and the exact search space with the receipts. Do not assert that the non-nested score must exceed the outer mean on every synthetic sample: selection optimism is a property of the evaluation procedure, not an ordering guarantee for one realization.

Read the selection receipt, not just the mean

The fixture produces one receipt per outer fold. Before discussing whether the mean looks impressive, explain what each field means:

Receipt field	What it records	What it does not establish
inner_score	Criterion used to choose a setting on outer-training data	Independent performance of the selected winner
chosen_C	That fold's selected regularization setting	A universal setting or the final artifact's setting
outer_score	Evaluation on that fold's untouched test rows	Performance on a different deployment population

Different folds may choose different settings. That is not automatically a bug: their training samples differ. The object being evaluated is “run this search on the available training sample,” not “always deploy the setting selected in fold two.”

Do not select the final setting by voting across outer winners unless that voting rule was itself part of the declared procedure. The final search in the example uses all development data and the same inner selection rule. Its best score remains a selection score, even after the outer evaluation has been completed.

Why a pipeline alone cannot repair the original result

There are two different failures to inspect. Learned preprocessing outside folds exposes validation information during fitting. Reporting a maximized validation criterion as independent performance exposes evaluation to selection. Fixing one does not necessarily fix the other.

A useful follow-up is: “What if I scaled all rows first but nested the classifier afterward?” The answer is that the scaler still crossed both inner and outer boundaries. Move the scaler into the searched estimator, not merely inside a surrounding outer loop after it has already seen every row.

Another follow-up concerns threshold selection. ROC AUC measures ranking across thresholds; it does not choose an operating threshold for you. If the task requires a threshold optimized for cost or recall, declare that selection step and perform it using only the current outer training partition. The untouched outer fold should evaluate the resulting decision rule.

Changing a threshold after looking at outer false negatives would make the outer fold development evidence. You may learn from that analysis, but you cannot keep presenting it as untouched confirmation of the changed procedure.

Repair the split unit in both loops

The synthetic example assumes independently sampled rows. Suppose your real table contains two visits for each patient. Row-level splits may place one visit in training and the other in testing, making “new visit for a known patient” look like “new patient.” Nesting that split repeats the wrong experiment more carefully.

For an original counterexample, patient A has rows 0 and 1; B has 2 and 3; C has 4 and 5. Training on 0, 2, and 4 while testing on 1, 3, and 5 gives no row overlap but complete patient overlap. A disjoint-index assertion passes while the intended generalization boundary fails.

Choose group-aware partitions when the target is unseen entities, and verify group disjointness in both outer and inner partitions. Inner groups must be restricted to the current outer-training rows. Do not accidentally feed a full-length group array to a smaller training table.

For forecasting, preserve time order and the relevant horizon. A random inner split can leak future conditions even when the outer split respects time. The scikit-learn cross-validation guide documents grouped and time-series splitters; choosing one still requires knowing the prediction question. The adjacent time-series cross-validation article covers horizon and gap choices in more detail.

Account for cost without weakening the claim

With three candidates and three inner folds, each search performs nine candidate fits plus one winner refit. Three outer searches therefore require thirty fits. The final full-development search adds ten more: forty fits for nested evaluation and the deliverable, excluding the ten-fit non-nested comparison.

This arithmetic assumes no failed fits, additional threshold search, or extra preprocessing experiments. It is a planning estimate derived from this fixture, not a runtime benchmark. More cores can reduce wall time, but nested parallelism may increase memory pressure and resource contention.

If compute is limited, reduce the declared candidate budget or use an untouched holdout for the final check. Be explicit about the trade-off. A holdout that has already influenced fifty design choices is no longer a fresh confirmation set merely because its variable is named test.

Keep search budgets comparable when comparing procedures. A much larger search can discover a better configuration, but it also changes cost and selection pressure. Report the budget alongside performance so the comparison remains interpretable.

Report uncertainty and the final refit honestly

Keep the outer scores, split definitions, sample sizes, metric, and search space with your result. A mean without those details hides whether one partition failed or whether a rare class disappeared from evaluation.

The standard deviation across folds is descriptive variation, not automatically a confidence interval. Outer training sets overlap, so the scores are not independent experimental replications. Repeating a protocol with additional seeds can help assess instability, but repeated folds do not create new patients or extend the observation period.

Outer models also train on less data than the final refitted model. The estimate concerns the declared procedure at those training sizes. Do not assert that the final artifact must achieve exactly the outer mean, or that a larger final training set guarantees improvement.

A concise report can say: “We fixed preprocessing and search before outer evaluation, retained the fold scores, and refit the same selection procedure on all development data. This estimates within-population generalization under our split assumptions; deployment shift remains untested.” That wording repairs the claim as well as the implementation.

Sources and Further Reading

• scikit-learn: Nested versus non-nested cross-validation
• scikit-learn: Common pitfalls and recommended practices
• scikit-learn: Cross-validation and splitting strategies
• Cawley and Talbot: On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation

Research checked October 4, 2026. The paper is from 2010; library behavior is grounded in the linked current documentation. The original synthetic fixture does not establish a universal bias amount or hiring outcome.