Grouped Cross-Validation Interview Questions: Patients, Users, and Duplicate Entities

A medical model can appear accurate because images from the same patient occur in both training and validation. A recommendation model can look strong because several sessions from the same user cross the split. In both cases, ordinary row-level cross-validation answers an easier question than deployment will ask.

Grouped cross-validation keeps related observations together. The hard interview work is identifying the group, choosing a splitter that matches deployment, and proving there is no overlap. Start with Explain K-Fold Cross-Validation and Its Trade-Offs, then add the entity constraint.

Evidence boundary: Scikit-learn officially documents grouped splitters and their mechanics. The patient, user, and duplicate examples below are original teaching fixtures.

What problem does grouped cross-validation solve?

Cross-validation estimates performance on held-out observations. That estimate is optimistic when the held-out row is not meaningfully independent of the training rows. Repeated measurements from one patient share biology and acquisition conditions. User sessions share identity, preferences, device, and history. Augmented images share almost all pixels. Duplicate entities can let the model memorize a signature instead of learning a relationship that transfers.

GroupKFold assigns each group wholly to one test fold. Scikit-learn documents that a group appears exactly once in a test set across the folds. The number of distinct groups must be at least the number of folds. This controls group overlap; it does not automatically preserve time, labels, sites, or class balance.

The first interview question is therefore not “Which class do you import?” It is “What future case are we trying to generalize to?”

How do you choose the group key?

Choose the identity whose recurrence would make a validation example familiar. For a patient diagnosis model, patient_id is usually safer than visit_id; multiple visits from one patient may share stable signals. For a marketplace risk model, you may need seller, household, device cluster, or payment instrument rather than account ID. For document classification, near-duplicate documents or source templates may define the true group.

Map the data-generating process. Ask why multiple rows exist, which attributes persist, how identities merge or split, and what the production population contains. If deployment predicts future visits for existing patients, holding out entire patients may be pessimistic for that specific use case. If deployment targets new patients, it is the correct test.

A defensible answer names both the unit and the scenario: “I group by patient because the product is evaluated on unseen patients; I keep a later-time test because clinical practice also changes.”

Why can a random split look excellent?

Imagine 100 patients with ten measurements each. A random 80/20 row split puts observations from most patients on both sides. A model can exploit patient-specific baselines or acquisition artifacts. The reported metric then blends two abilities: recognizing a known patient and generalizing to a new one.

Create a simple diagnostic: train a deliberately high-capacity baseline on entity-associated features and compare random K-fold with grouped K-fold. A large gap is evidence that entity reuse matters, though it does not by itself identify every leakage channel.

Also search for exact and near duplicates before splitting. Grouping by patient_id does not help if the same record appears under two IDs. Hash canonicalized rows or media, use domain-specific similarity, and trace upstream joins. Duplicate removal is a data-quality decision; grouped validation is an evaluation decision. They often work together.

Which splitter should you choose?

Use GroupKFold when every group must be held out once and approximate balance by sample count is adequate. Use StratifiedGroupKFold for classification when you also want label proportions to be as similar as feasible while groups remain disjoint. Scikit-learn warns through its behavior and documentation that stratification is constrained by the groups; perfect balance can be impossible.

Use GroupShuffleSplit when you need randomized group-level holdouts or repeated experiments. Its test_size refers to groups, not rows. Because different randomized splits can reuse groups across test sets, it is not the same partitioning contract as K-fold.

Use LeaveOneGroupOut when each group represents a small set of meaningful domains, such as hospitals, devices, or years, and you want each domain tested separately. That can expose variation hidden by an average, but it may be expensive and unstable with few groups.

If both group and time matter, do not assume a stock splitter satisfies both. Construct an explicit forward-chaining policy over groups or choose a final out-of-time, group-disjoint test set.

What code and invariants should you show?

from sklearn.model_selection import GroupKFold

cv = GroupKFold(n_splits=5, shuffle=True, random_state=42)
for fold, (train_idx, valid_idx) in enumerate(cv.split(X, y, groups=patient_id)):
    train_groups = set(patient_id[train_idx])
    valid_groups = set(patient_id[valid_idx])
    assert train_groups.isdisjoint(valid_groups)
    print(fold, len(train_idx), len(valid_idx), len(valid_groups))

The overlap assertion is more valuable than printing a high score. Record row count, group count, positive rate, and important segment counts for each fold. If one hospital or large customer dominates a fold, an unweighted mean across rows may hide the instability.

Fit every learned transformation inside the fold. Grouping prevents entity overlap, but preprocessing on all rows can still leak validation information. Pass groups to the splitter and keep imputers, encoders, feature selection, and the estimator inside a pipeline.

How do group size and class imbalance interact?

Suppose one patient contributes 1,000 negative rows and ten patients contribute one positive row each. Keeping patients disjoint may produce folds with very different row counts and labels. That is not necessarily a splitter bug; the dataset may not support the evaluation you hoped for.

Report both entity-weighted and row-weighted metrics when they answer different questions. A row-weighted metric lets prolific users dominate. An entity-weighted metric gives each person equal influence but requires an aggregation rule when an entity has multiple predictions. State the business unit of the decision.

If positives occur in fewer groups than folds, stratified grouped K-fold cannot place positives in every test fold. Reduce the number of folds, collect more independent groups, use repeated group holdouts, or report uncertainty rather than manufacturing balance by splitting an entity.

What does “no overlap” fail to prove?

It does not prove that time is respected. The same global campaign, hospital protocol, or item catalog can leak across groups. It does not prove that target-derived preprocessing is safe. It does not prove that hyperparameters were untouched after inspecting the test set.

Use nested evaluation when tuning materially influences the model: an inner grouped split chooses hyperparameters, and an outer grouped split estimates performance. Keep a final untouched test set if the project has enough independent groups. With very few groups, show fold-level results and confidence limits rather than a precise average.

Also check whether groups are inferred using future information. A household graph built from relationships observed after prediction time can leak even if the resulting household IDs are split cleanly.

How should you explain results to an interviewer?

Do not say grouped validation is always “more accurate.” Say it estimates a different deployment scenario. Compare the random-split score and grouped score, explain the gap, and identify the population each estimate represents.

Then discuss variance. Fewer independent groups mean less information than the row count suggests. A model evaluated on 100,000 rows from 30 patients does not have 100,000 independent validation units. Report the number of groups, fold distribution, and per-group error. Bootstrap by group rather than row when using a bootstrap to represent entity sampling.

End with an operational check: the production feature pipeline must identify entities consistently. If group identity changes or merges after deployment, offline assumptions can break.

Practice related validation decisions

Question	Practice focus	Why it helps
Explain K-Fold Cross-Validation and Its Trade-Offs	CV mechanics	Establishes the baseline before grouping
Evaluate K-Fold Cross-Validation for Model Selection	Model selection	Separates tuning from final estimation
Design cross-validation; explain bias–variance	Split design	Connects resampling to uncertainty
Train GradientBoostingClassifier with 5-Fold Cross-Validation	Implementation	Makes fold-contained training concrete
What Cross-Validation Reduces: Bias, Variance, Type I or Type II Error	Interpretation	Prevents overclaiming what CV proves

The compact interview rule is: define the deployment unit, group by the identity that would make a held-out row familiar, assert disjointness, and show fold-level evidence.

Work through an impossible-fold example

Suppose six patients produce twelve rows. Patients A and B contain all four positive cases; C through F contain only negatives. A five-fold grouped classification split cannot place a positive patient in every fold because only two independent groups contain positives. No shuffle seed can overcome that constraint.

State the options instead of forcing a misleading result. Use two or three folds, repeated group holdouts that report which test sets lack positives, or a fixed group-disjoint test with uncertainty over patient-level outcomes. Collect more positive groups if the decision warrants it. Do not split A's visits across folds to make the chart look balanced; that changes the estimand and reintroduces familiarity.

Now add time: A has visits in January and June. If deployment predicts a later visit for known patients, a forward split within patient may match the product while a new-patient holdout answers a second question. Report both only when both scenarios matter. The interview signal is recognizing that “the right cross-validation” is not a universal object. It is an experiment tied to a defined future population.

Before fitting, print a fold manifest with patient IDs, row counts, label counts, date ranges, and source sites. Review the manifest as carefully as the score. A surprising fold composition often explains metric variance faster than changing the model.

Keep the group rule in the tuning loop too. Selecting hyperparameters with row-level folds and reporting grouped outer-fold performance still lets tuning prefer entity-specific shortcuts. Use a grouped inner splitter or fixed validation population, then reserve outer groups for estimation. Record which group IDs informed each decision.

For reporting, show the distribution rather than only the mean. List per-fold and per-group metrics, the worst important segment, and the number of independent entities. If one large group changes the conclusion, say so. “Insufficient independent positive groups” is more useful than a precise decimal produced by invalid folds. Preserve that limitation in the model decision record.

Sources and Further Reading

• scikit-learn: cross-validation with grouped data
• scikit-learn: GroupKFold
• scikit-learn: StratifiedGroupKFold
• scikit-learn: GroupShuffleSplit