Articles
Feature Selection Interview Questions: Filters, RFE, and Stable Model Decisions
Practice feature selection interviews: compare filters and RFE, prevent leakage, map selected names, and explain correlated features and stable decisions.
Feature selection interview questions rarely end at naming filter, wrapper, and embedded methods. The harder task is deciding what the selector may learn, how much computation it can spend, and whether its chosen columns support a stable production decision.
A smaller feature set can still leak validation information, discard an important interaction, or alternate unpredictably between interchangeable predictors. Your answer should connect the selection rule to the prediction target, evaluation split, model family, and cost of collecting features.
This guide separates official scikit-learn behavior from original examples and interview recommendations. Executed examples use scikit-learn 1.7.2; the linked documentation matches that version. No candidate-process claims are made.
Start with the decision, not the selector
Our original practice case predicts a customer action using information available at a fixed prediction time. There are twenty eligible numeric features, some correlated, and a budget for five training folds. The team wants a useful model that is also inexpensive to serve and explain.
First define eligibility. A field populated only after the action occurs is unavailable at prediction time, even if it ranks first in every selector. Remove it because of its information timing, not because an importance score happens to be low.
Then establish the reference model: all eligible features, appropriate preprocessing, and the intended validation split. Compare selection against that reference on the same held-out observations. Otherwise, you cannot tell whether improvement comes from selection, changed rows, changed tuning, or a different evaluation population.
Ask which cost matters. Reducing twenty cached columns to five might barely change serving latency. Removing one expensive remote lookup could matter more than removing fifteen cheap arithmetic features. Prediction quality, operational cost, and explanation needs should be explicit decision criteria.
Match filters, RFE, and embedded selection to the task
Official facts: VarianceThreshold removes features below a variance threshold; its default removes constant columns. SelectKBest retains a specified number of features using a scoring function. SelectFromModel thresholds fitted model importance. Feature selection guide.
Our recommendation is to compare methods through their assumptions rather than treating those categories as a quality ranking:
| Method | Useful starting point | Question your answer must resolve |
|---|---|---|
| Variance filter | Remove constants cheaply | Was the threshold fitted only on training data, and do units make a nonzero threshold meaningful? |
| Univariate filter | Screen many columns under a tight budget | Can useful information appear mainly through interactions? |
| RFE wrapper | Refit a compatible estimator as columns disappear | Is its importance meaningful, and is the repeated fitting affordable? |
| Embedded/model-based rule | Use sparsity or importance from a fitted model | How sensitive is the selected set to regularization and correlated substitutes? |
A low-variance feature is not automatically useless. A rare event indicator might be important for the relevant population. Similarly, a large variance caused by measurement units does not establish predictive value.
For an original interaction counterexample, take the four equally represented binary pairs (0,0), (0,1), (1,0), and (1,1). Let the target be XOR: zero, one, one, zero. Each individual column has the same mean in both target classes. Our executed univariate F-test returns zero for both, although the pair determines the target exactly. A suitable interaction-capable model could use that information; a simple marginal screen misses it.
Official constraint: the chi-squared scoring function requires nonnegative features. Scoring-function guidance. Do not apply it blindly after centering numeric values around zero. Our negative-input check confirms rejection, not a general assessment of all scoring functions.
Explain RFE as repeated fitting
Official behavior: RFE fits an estimator, removes features with the weakest current importance, and repeats until the requested size is reached. It requires compatible importance, such as coefficients or feature_importances_. Its support mask identifies selected columns; selected columns receive ranking one. RFE reference.
The importance is conditional on the estimator and current feature set. Removing a column changes the next fitted model. RFE is therefore not equivalent to sorting the first model's coefficients once and taking the top five.
For a coefficient-based estimator, our recommendation is to scale eligible continuous features within the training procedure. Otherwise, coefficient magnitude can reflect units as well as predictive contribution. Scaling is a modeling choice; it does not make coefficient rankings causal.
Do not interpret ranking one as a total ordering among retained features. If five columns are retained, all five can have rank one. Nor does a discarded feature's ranking prove that it is irrelevant for every model or dataset.
Count the fitting budget before choosing step
For the original twenty-to-five exercise, integer step=5 fits at feature counts twenty, fifteen, ten, and five. That includes the final fit on the retained set. Our instrumented scikit-learn 1.7.2 run verifies those four estimator fits.
With step=1, the same reduction fits at every size from twenty through five: sixteen fits. The elimination setting changes both cost and the path taken through feature subsets.
| Original RFE setting | Estimator fits per training fold | Across five folds |
|---|---|---|
| Twenty to five, step five | 4 | 20 |
| Twenty to five, step one | 16 | 80 |
These counts cover the estimator fitted inside RFE. They exclude a separate final classifier, hyperparameter candidates, preprocessing costs, and a final refit on all permitted training data. If your pipeline trains another classifier after RFE, count that work too.
The larger step is cheaper in this example, but it can discard several features before their relative usefulness is reconsidered. Do not promise equal selected sets or equal prediction quality. Under a fixed budget, compare a coarse RFE configuration with a cheap filter and the all-feature baseline before spending everything on fine-grained elimination.
Fit every learned selection step within training folds
Official guidance: preprocessing and feature selection should learn from training data, while held-out data receives the fitted transformation. Pipeline supports that separation during cross-validation. Common pitfalls.
The original failure is fitting a selector on all rows, then cross-validating only the classifier. Validation labels or feature distributions have already influenced which columns survive. Moving the classifier into cross-validation afterward does not undo that choice.
Our numeric example keeps a variance filter, scaling, RFE, and classification in one estimator:
from sklearn.feature_selection import RFE, VarianceThreshold
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
model = Pipeline([
("variance", VarianceThreshold()),
("scale", StandardScaler()),
("rfe", RFE(LogisticRegression(max_iter=2000),
n_features_to_select=2, step=1)),
("classifier", LogisticRegression(max_iter=2000)),
])
model.fit(X_train, y_train)
predictions = model.predict(X_test)
Pass this whole pipeline to the cross-validation or tuning procedure. Do not prefit its selection stages outside that procedure. The snippet assumes finite numeric inputs, enough surviving columns, and a valid classification target; it omits categorical routing and imputation deliberately.
A useful original regression test makes the fourth column constant in training but variable in held-out rows. The training-fitted filter drops it. Changing that held-out column leaves predictions unchanged in our synthetic fixture. A deliberately leaky filter fitted on training plus held-out rows retains it. This exposes the changed learning boundary without requiring a misleading accuracy claim.
Map the selected mask back to the right names
A selector's mask applies to the columns it actually receives. In the example, RFE sees only the columns that survived VarianceThreshold. Applying its shorter mask directly to the original names can fail or mislabel your reported features.
For the original four-column fixture, the first mask is [True, True, True, False]. Map names through that mask before applying RFE's mask:
import numpy as np
surviving = np.asarray(original_names)[
model.named_steps["variance"].get_support()]
selected = surviving[
model.named_steps["rfe"].get_support()]
Our fixture retains signal and its near-duplicate, rather than forcing one representative from each conceptual family. This is an executed example of this configuration, not a rule that RFE always chooses redundant columns.
If preprocessing expands categorical columns, report the transformed names and their original feature families. A count of five encoded columns may represent only one raw field, while one raw text field may create thousands of candidates. Keep those units consistent in both the compute budget and the operational decision.
Separate set stability from prediction stability
Consider this constructed stability ledger, not a measured experiment. Four training resamples select {A,C,D}, {B,C,D}, {A,C,D}, and {B,C,D}. A and B represent two highly correlated measurements of the same underlying signal.
C and D appear in all four sets; A and B each appear in two. The Jaccard overlap between the first two sets is 2 / 4 = 0.5, because their intersection contains C and D and their union contains A, B, C, and D.
A changing name can therefore coexist with similar predictions if interchangeable measurements substitute for each other. Test that possibility on held-out data instead of declaring either perfect stability or failure from the selection frequencies alone.
Our recommendation is to compare three levels: individual selected names, selected feature families, and predictions or task metrics. If the team needs one consistent field for an explanation or collection contract, evaluate a representative chosen for availability, cost, missingness, and meaning. If prediction is the main goal, compare a regularized all-feature model as well.
Selection frequency is a descriptive stability measure for your resampling procedure. It is not a probability that a variable causes the outcome. Predeclare any frequency threshold you intend to use; tuning that threshold after inspecting the final test set makes it another unvalidated model choice.
Keep RFECV selection separate from final evaluation
Official behavior: RFECV uses cross-validation to choose the number of retained features. RFECV reference.
Our evaluation recommendation is to reserve an outer evaluation procedure or a separate untouched test set. RFECV's best internal score has already influenced the chosen size; treat it as selection evidence rather than an independent final generalization result.
Choose the split to match deployment. Repeated customers may require grouped separation; future predictions may require chronological separation. A pipeline cannot repair a split that places related records on both sides or lets future information enter a historical prediction.
Freeze the selected policy, preprocessing, and classifier before final evaluation. Preserve the feature schema with the fitted artifact. A missing feature at serving time is a contract failure to diagnose, not permission to silently rerun selection on live requests.
Sources and Further Reading
- scikit-learn 1.7 feature selection guide
- scikit-learn 1.7 RFE reference
- scikit-learn 1.7 RFECV reference
- scikit-learn 1.7 common pitfalls and leakage guidance
Research checked October 4, 2026. Eight local synthetic correctness checks pass; no production improvement, real-data selection stability, or latency benchmark is claimed.