Articles
Target Encoding Interview Questions: Cross-Fitting, Rare Categories, and Unseen Values
Practice target encoding with a six-row cross-fitting trace, rare-category smoothing, unseen-value fallbacks, and leakage-safe validation.
Target encoding is easy to describe and surprisingly easy to leak. Replacing a category with its historical target mean sounds like ordinary feature engineering until the row being predicted helps calculate its own feature. In an interview, the strongest answer traces who knew each label at each point in training, validation, and deployment. This guide walks through that trace, a rare-category calculation, and an unseen-category decision you can defend.
Official documentation: scikit-learn's TargetEncoder uses internal cross-fitting for training data in fit_transform; its documentation warns that fit(X, y).transform(X) is not equivalent. Candidate reports: this article does not claim a particular employer asked these questions. Original exercise and inference: the rows and hand calculations below are invented for practice; a request to explain cross-fitting is best answered as an information-flow problem, not a library trivia question.
What information may an encoded training row contain?
For a categorical value such as merchant ID, a basic target mean is the average observed label among rows with that merchant. If R occurs once and its label is one, its raw encoding is one. A downstream model can effectively memorize that training label. With thousands of nearly unique IDs, a feature that appears predictive on training data may have no useful signal for the next transaction.
The first interview distinction is between label leakage into a training feature and holdout leakage into model selection. Training-row self-influence occurs when a row's own target contributes to the statistic used to encode that row. Holdout leakage occurs when validation or test labels influence any encoding, smoothing parameter, category filter, or model choice before evaluation. Cross-fitting within training addresses the first problem; an outer train/validation boundary addresses the second. Neither replaces the other.
Ask for the prediction contract. Is the category known when the prediction is made? Is the target available only later? Could the same customer, household, or device occur in both folds? A random fold split might still leak future information in a temporal or entity-based problem. The fold design must follow the deployment boundary, with grouped or time-aware splits when needed. Do not promise that generic K-fold is safe for every data set.
The scikit-learn TargetEncoder API documentation says fit_transform encodes training data using internal cross-fitting, whereas fit().transform() on those same rows uses encodings learned from all of them. The second path can make a rare row's label visible in its own feature. This is a version-specific library fact; the broader information-flow argument applies to any target encoder.
Work a six-row cross-fitting example by hand
Original practice data, not a candidate report: six labeled transactions are divided into two folds. Fold A has (merchant A, 1), (merchant A, 0), and (merchant R, 1). Fold B has (merchant A, 0), (merchant B, 0), and (merchant B, 1). The full-data target mean is 3/6 = 0.5. Merchant R appears only once.
To encode fold A, learn category means only from fold B. In fold B, A has mean zero, B has mean 0.5, and the global training-fold mean is 1/3. Therefore the two A rows in fold A receive zero under an unsmoothed illustrative encoder. R is absent from fold B, so its encoded value must come from an explicitly declared fallback, here the fold-B prior of 1/3. It must not become one merely because its own label is one.
To encode fold B, learn only from fold A. A has mean 0.5, while B is unseen. The fold-A prior is 2/3. Thus the fold-B A row receives 0.5 and both B rows receive 2/3 under the same illustrative rule. These are deliberately different from encodings based on all six rows. The row-level table makes the leakage boundary visible:
| Encoded row | Training fold for its statistic | Illustrative output |
|---|---|---|
| A rows in fold A | Fold B; A mean 0 | 0 |
| R row in fold A | Fold B; R unseen | 1/3 prior |
| A row in fold B | Fold A; A mean 1/2 | 1/2 |
| B rows in fold B | Fold A; B unseen | 2/3 prior |
These numbers do not claim to reproduce scikit-learn's default output. Its smoothing and split behavior are configurable, and the default automatic smoothing changes the arithmetic. The useful interview asset is the dependency graph: each encoded row depends on labels in other folds, never its own fold. After training, fit a final encoder on the permitted training population for inference on genuinely new rows.
One follow-up asks whether leave-one-out encoding is the same. It also removes a row's own label from its category statistic, but it does not automatically solve grouped leakage, time ordering, hyperparameter tuning, or the instability of one-count categories. Define the estimator and split before claiming a guarantee.
How should rare categories be smoothed?
Rare categories have noisy means. A common illustrative smoothing formula is (category positive count + m × global mean) / (category row count + m), where m is the prior's equivalent sample size. For the full six-row training set, R has one positive out of one row and the global mean is 0.5. With m = 2, R becomes (1 + 2 × 0.5)/(1 + 2) = 2/3, rather than the raw value one. A large category receives less shrinkage because its observed count dominates the prior.
The interviewer may then point out that the calculation still used R's own target. Correct: smoothing reduces variance, but smoothing alone does not remove self-influence. In training-row cross-fitting, both category count and the prior must be computed from the allowed other-fold data. For R in our fold A, the other fold contains no R, so the fallback remains its other-fold prior of 1/3. If you cross-fit category means but compute a global prior over all six rows, you have reintroduced a smaller label dependency.
There is no universal “rare means fewer than ten” threshold. Test the frequency distribution, model class, cost of high-cardinality features, and stability of the category over time. Collapsing small groups to an OTHER bucket is one option, but the bucket definition must itself be learned only from training data at each validation boundary. A smoothing strength chosen after observing the outer test score is another form of leakage through model selection.
For a classification task, verify which target is encoded. In binary classification, the target mean can be read as the observed positive rate before smoothing. In multiclass classification, an encoder may emit more than one feature per category; check the exact library API rather than claiming every target encoder returns a single number. Regression targets can also be encoded, but then the prior and loss have different units and interpretations.
What happens to unseen or missing categories at inference?
An unseen merchant cannot have a learned category mean. In scikit-learn's documented behavior, categories absent during fitting map to the global target mean. This is a defined inference fallback, not a newly measured outcome for that merchant. The same documentation treats missing values as a category; confirm the exact estimator/version and missingness policy before relying on this in production.
An interview answer should quantify unseen-category frequency. Suppose 18% of requests contain merchant IDs absent from the training window. A model that scores well only on familiar merchants may underperform on the real serving mix. Report validation slices for seen, rare, and unseen categories. For time-evolving IDs, simulate future unseen rates with a temporal holdout, rather than letting the model see category membership from the future. If the fallback prior shifts, monitor calibration by arrival cohort and retrain through a documented schedule.
Do not substitute a magic sentinel such as -1 without checking the model's semantics. A numeric learner may interpret it as an extreme low response rate. A separate unknown indicator can be useful, but it should be created from information available at inference, and its performance should be tested against a simple global-prior fallback. Sometimes one-hot encoding, hashing, or dropping an unstable identifier is more robust than target encoding.
How do you validate a target encoder without optimistic scores?
The clean workflow has three boundaries. First, reserve a final holdout that neither encoder nor model sees during feature development. Second, inside the remaining training data, choose appropriate cross-validation splits and fit the encoder only on each training partition. Third, for each training partition, use out-of-fold encodings for its training rows; transform its validation rows using statistics learned from that partition only. The exact mechanics depend on the pipeline and library, so inspect the calls rather than assuming an API name guarantees this nesting.
Use a reproducible test: permute labels while leaving categories intact. If the model still reports an implausibly strong held-out score, inspect whether encoded features or identifiers carry labels across the split. A second test compares training and validation performance as category cardinality grows; a large gap concentrated in singleton IDs is a leakage or variance warning. These checks are diagnostics, not proof that all confounding or temporal leakage has been removed.
Track the input schema and fallback at serving time. A transformed category should never require access to its future label. If a batch scoring job accidentally recomputes encodings from labels in the same batch, the model has become a retrospective estimator rather than a predictor. Log encoder version, fit window, target definition, and category coverage with the model artifact so an apparent gain can be reconstructed.
A concise interview answer and the follow-up to expect
“I would reserve a holdout before computing target statistics. For each training row, I would construct its encoding from labels outside that row's permitted fold, with grouped or time-aware folds if the prediction setting needs them. I would smooth small categories toward a fold-specific prior, map unseen categories to a documented fallback at inference, and report results separately for seen, rare, and unseen values. I would verify the exact library behavior because fit_transform and fit().transform() are not interchangeable for scikit-learn's TargetEncoder.”
If the interviewer asks why a random cross-fit can still be wrong, give one concrete counterexample: two transactions from the same customer land in different folds, and the target encoding of customer ID reveals the other transaction's outcome. Grouping by customer or using a forecast-origin split may be necessary. If asked whether target encoding “solves leakage,” say it addresses a specific path of label reuse when implemented correctly; the holdout, target timing, entity boundary, and selection process still require separate controls.
Practice explaining the Encode Categorical Variables Across Model Families with the six rows above. A convincing answer exposes every statistic's allowed inputs before it recommends an estimator.