ColumnTransformer Interview Questions: Missing Columns, Feature Names, and Sparse Output

A model still returns a number, but the input frame changed. One field is absent, another arrives in a different position, and a categorical value was never seen during fitting. The engineer says, “ColumnTransformer handles preprocessing,” yet cannot name the five columns actually fed to the estimator. That is the interview problem: reconstruct the fitted input-to-output contract before changing a single model parameter.

This is an original practice case, not a candidate report or a claim about any employer's test. The code and observed outputs below are pinned to scikit-learn 1.7.2 with pandas 3.0.5 and SciPy 1.18.1 in the project environment. The scikit-learn 1.7 ColumnTransformer reference and 1.7 OneHotEncoder reference establish the documented API behavior. The tiny data frame, failure cases, and recommended checks are an engineering exercise. They do not describe a reported company interview. For adjacent practice, Encode Categorical Variables Across Model Families provides the category-side questions before you trace the exact feature matrix here.

The fixture: a three-column input becomes five features

The training frame has age, city, and spend, in that order. Three rows contain ages 10, 20, 30; cities a, b, c; and spend zero. The first branch one-hot encodes city; the second scales age; the remainder passes spend through. Notice the transformer declaration order starts with city, although the raw frame starts with age. The official reference says selected transformer outputs are concatenated, and the remainder is appended after them.

import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder, StandardScaler

train = pd.DataFrame({
    "age": [10, 20, 30],
    "city": ["a", "b", "c"],
    "spend": [0.0, 0.0, 0.0],
})

ct = ColumnTransformer(
    [("city", OneHotEncoder(handle_unknown="ignore", sparse_output=True), ["city"]),
     ("age", StandardScaler(), ["age"])],
    remainder="passthrough",
    sparse_threshold=0.3,
)
X = ct.fit_transform(train)
print(X.shape, type(X).__name__)
print(ct.get_feature_names_out().tolist())
print(ct.output_indices_)

In the pinned run, X.shape was (3, 5) and its type was ndarray. The feature names were city__city_a, city__city_b, city__city_c, age__age, and remainder__spend, in that order. output_indices_ mapped city to slice 0:3, age to 3:4, and remainder to 4:5. These are inspected facts from this fixture, not a promise that all categories or matrix types look like this on another dataset. A model receiving X[:, 0] receives the first city indicator, not raw age.

The fastest interview answer is a column ledger. It separates each branch's input selector, fitted behavior, output width, and output position:

Branch in declaration order	Selected input	Output in this fitted run	Slice in final matrix
city	DataFrame column city	Three indicators for fitted categories a, b, c	0:3
age	DataFrame column age	One standardized numeric feature	3:4
remainder	Fitted, unselected spend	One unchanged numeric feature	4:5

That ledger is the first article-specific asset. It tells you where a coefficient, feature importance value, or failed type conversion came from. If a custom transformer expands one selected field into four columns, the input name alone is insufficient; inspect get_feature_names_out() and output_indices_ after fitting. If a custom transformer lacks a compatible feature-name method, an attempt to get names may fail even when transformation succeeds. The repair is to implement an honest naming contract for that transformer or inspect its output explicitly, not to invent names based on the original frame.

Question 1: what does a missing input column mean?

Remove city from the frame passed to ct.transform. In the pinned run, scikit-learn raised ValueError: columns are missing: {'city'}. The exact error wording is version-dependent, but the diagnosis is stable for this named-column fixture: the fitted city branch expects a source column that the new frame does not supply. Adding an all-null city field merely to silence the error is an application policy, not something ColumnTransformer can decide for you. The value might be truly unavailable, renamed upstream, or accidentally dropped by a join; each warrants a different fix.

Ask for the upstream schema contract. If city is required, fail the request with a clear missing-field error before predicting. If absence is an accepted business state, define and test an imputation or explicit “unknown” representation inside a versioned preprocessing workflow. Do not fit a new transformer silently on each request: that would learn a different category set and change the feature-space mapping. An absent column also differs from a present city column containing a null value. A missing cell reaches the branch; a missing column cannot be selected at all. Discuss those as separate cases, especially when an imputer precedes an encoder.

handle_unknown="ignore" in this fixture solves a different problem. A frame with the city column present but value z can be transformed; the one-hot positions for that value are all zero in the pinned run. The official encoder documentation describes that behavior. It does not mean an absent city field is ignored. It also does not make the unknown category semantically equivalent to a legitimate all-zero reference category in every modeling setup; monitor the frequency of unknowns and choose a policy appropriate to the application.

Question 2: can input order change the output mapping?

I passed the fitted transformer a frame ordered spend, city, age, with a new coupon column appended. The output matched the original X element for element. In this exact case, selectors use DataFrame names, and the fitted remainder is the original spend column. The extra coupon was not automatically added to the feature matrix. The reference explicitly says extra DataFrame columns unseen at fit are excluded from transform output; scikit-learn has also documented reordered named DataFrame columns as supported. The branch order still determines output order.

Do not generalize that observation to every selector. An integer selector refers to a position, not a DataFrame name. Pass an unlabeled NumPy array or use positional selectors and a reorder can point a transformer at the wrong data while preserving the same shape. A remainder estimator has an additional documented requirement that DataFrame columns at fit and transform appear in identical order. This exercise uses remainder="passthrough", so the observed reorder result should not be presented as proof for that different configuration. The interviewer is looking for precisely this distinction: which contract did you fit, and which variation did you actually test?

The coupon example exposes another operational choice. Silently excluding new columns is useful when serving data contains metadata that should not enter a trained model. It can also hide an upstream rollout that expected a new feature to be used. Compare the incoming schema to the fitted schema separately from the transformation call and alert on unexpected additions if the application requires strict compatibility. feature_names_in_ records the all-string input names seen at fit in this fixture; it is a better baseline than assuming the latest payload defines the model's feature space.

Question 3: why did a sparse encoder yield a dense matrix?

OneHotEncoder(sparse_output=True) returns a sparse representation for that branch, but ColumnTransformer decides how to stack all branches. Its documented sparse_threshold default is 0.3: when at least one branch is sparse, the combined output is sparse if the overall density is below the threshold; otherwise it is dense. If every branch is dense, the threshold is ignored. A three-category encoder plus a scaled numeric column and a pass-through column is not automatically sparse enough to keep the entire result sparse.

For the same three-row fixture, sparse_threshold=0.3 produced a dense ndarray; rebuilding and fitting the same transformer with sparse_threshold=0.9 produced a sparse csr_matrix. The feature names and five-column order were the same. The numeric density in this tiny fixture is not a capacity estimate for a production vocabulary. What matters is the mechanism: inspect ct.sparse_output_ and the actual output type after fitting, and size the downstream estimator or serialization path for that representation. This choice is made when the transformer is fitted; a later row with an unknown category does not constitute an automatic refit of the output contract.

Changing sparse_threshold can avoid accidental densification, but “always sparse” is not what a threshold of 1.0 promises in every possible input; all-dense branches remain dense, and density/representation rules still matter. Conversely, setting threshold 0 requests dense stacking even if a branch is sparse, which may multiply memory use for high-cardinality one-hot features. Ask the interviewer for row count, category cardinality, output width, downstream estimator support, and memory budget before prescribing dense conversion. A quick single-row demo tells you shape and type, not peak memory at production scale.

There is also an output-container trap. In the pinned sparse configuration, calling ct.set_output(transform="pandas").transform(train) raised a ValueError explaining that pandas output does not support this sparse encoder output. The encoder API exposes sparse_output=False if you deliberately want dense output; choose it only after considering width and memory. set_output changes the requested output container, not the semantics of a missing column, an unseen category, or the order of fitted features. Report the actual exception and configuration instead of assuming that DataFrame output will automatically preserve sparse memory efficiency.

Question 4: how do you prove feature names still line up?

Run assertions on the fitted artifact, not only on constructor arguments. ct.get_feature_names_out() should have one entry per output column, and each output_indices_ slice should cover the branch width shown in the ledger. In this fixture, the three city names occupy 0:3. If the category set changes after a deliberate refit, the width can change; do not reuse old coefficient-to-name mappings. Store the fitted transformer with the model and regenerate names from that same fitted object when explaining a result. A copied spreadsheet of feature names from an earlier training run can be accurate-looking and wrong.

The default verbose_feature_names_out=True prefixes names with the branch, which prevents a city feature from being confused with a similarly named feature emitted elsewhere. With verbose_feature_names_out=False, scikit-learn requires unprefixed output names to be unique and can raise if they are not. Version 1.7 also supports documented formatting options for output feature names. If a custom branch creates a dynamic number of features, write a test that verifies both width and names on the exact fitted branch. An explanation such as “the fourth coefficient is age” is credible only when it is derived from that branch slice and feature-name array.

Use a compact failure matrix as the second article-specific asset. Keep the cause and the safe response separate:

Input or configuration change	Observed or expected behavior in this fixture	Interview response
Drop named city column	Observed ValueError at transform.	Diagnose schema break; restore or explicitly model absence.
Keep city, replace first value with unseen z	Observed zeroes in that row's three city indicators.	Explain handle_unknown="ignore"; monitor unknown frequency.
Reorder named columns and add coupon	Observed identical five-column output; coupon excluded.	Show name-based selection and fitted remainder; check schema separately.
Raise sparse_threshold from 0.3 to 0.9 and refit	Observed dense-to-CSR change with the same feature names.	Check representation, memory, and downstream support.
Request pandas output from sparse branch	Observed ValueError in 1.7.2.	Decide deliberately whether dense output is affordable.
Refit on a changed category vocabulary	Expected feature names or width may change.	Treat the fitted preprocessor and model as one versioned artifact.

The first five rows were exercised in the local fixture; the last is a documented and testable inference from category-dependent encoding, not an observed deployment incident. An interviewer can ask for a tiny executable test for each row. That is more convincing than saying “pipelines avoid schema problems” while never inspecting the matrix. Keep this topic focused: train/test leakage, cross-validation, and calibration matter for a complete modeling workflow, but they do not answer which five columns this fitted transformer emits or why its output type changed.

Sources and further reading

• scikit-learn 1.7: ColumnTransformer reference — selectors, remainder, concatenation, sparse threshold, feature names, and output slices.
• scikit-learn 1.7: OneHotEncoder reference — unknown categories, sparse output, and generated names.
• scikit-learn 1.7: Pipelines and composite estimators — column-wise transformation guide and DataFrame behavior.
• scikit-learn 1.7: make_column_selector — selecting columns by name pattern or dtype when explicit lists do not fit.
• scikit-learn 1.7: version 1.0 changes — support for reordered named DataFrame columns during transform.