PCA Interview Questions: Scaling, Explained Variance, and Data Leakage

PCA can compress hundreds of columns into a small set of components and still produce a completely wrong interview answer. The usual mistakes happen before and after the eigendecomposition: one feature's units dominate the covariance matrix, “95% explained variance” is treated as predictive proof, or the components are fitted before the evaluation split.

This guide works through a two-feature covariance example, a leakage-safe pipeline, and a high-variance-but-useless counterexample. For related practice, open Detect Data Leakage in Supervised Learning Pipelines and name every statistic that must be learned inside a training fold.

Evidence boundary: Mathematical and implementation claims below use primary texts and current scikit-learn documentation. The numerical fixtures, acceptance checks, and interview wording are original preparation material.

What an interviewer is testing

A strong answer resolves six questions:

1. What objective does PCA optimize, and in which feature space?
2. Should the variables be centered, standardized, transformed, or weighted?
3. What does an explained-variance ratio actually measure?
4. How will the number of components be chosen for the downstream task?
5. Where must PCA be fitted to prevent leakage?
6. What information can PCA discard even when reconstruction looks good?

The article-specific thesis is: PCA preserves directions of high feature variance in the representation supplied to it. It does not know the prediction target, the business cost, or the evaluation boundary. Scaling defines the geometry, explained variance answers a reconstruction question, and split discipline protects the estimate of generalization.

State the objective precisely

Center a data matrix X by subtracting the training mean of each column. PCA finds orthonormal directions along which the projected data have maximum variance. Equivalently, retaining k principal directions gives the best rank-k linear reconstruction under squared error.

For a covariance matrix S, solve:

S v_j = lambda_j v_j,

with eigenvalues ordered lambda_1 >= lambda_2 >= .... The vector v_j is a loading direction. A sample's component score is its centered feature vector projected onto that direction. The explained-variance ratio for component j is:

lambda_j / sum_m lambda_m.

Current scikit-learn PCA uses singular value decomposition and centers input data, but it does not scale each feature automatically. The sign of an eigenvector is arbitrary: multiplying both a loading vector and its scores by -1 describes the same component. An answer that assigns inherent meaning to “positive PC1” without checking the reported sign convention is overinterpreting the output.

PCA also assumes a linear subspace and uses a squared-error geometry. It can represent correlated linear variation efficiently, but it does not untangle every curved manifold, preserve rare categories automatically, or make a sparse representation more interpretable.

Worked question: scaling changes the first component

Suppose a product dataset contains:

• annual account revenue in dollars;
• satisfaction on a 1-to-5 scale.

If revenue has variance near 10,000 and satisfaction has variance near 1, the raw covariance matrix might be:

[[10000, 8], [8, 1]].

Its first principal direction is almost the revenue axis, and its explained-variance ratio is above 99.9%. That result is numerically correct. It does not prove revenue contains nearly all useful information; dollars simply occupy a much larger scale.

After standardizing both variables with training-set means and standard deviations, a simplified correlation matrix could be:

[[1, 0.8], [0.8, 1]].

The eigenvalues are 1.8 and 0.2. The first direction is proportional to (1, 1), and PC1 explains 1.8 / 2 = 90% of standardized variance. Now a one-standard-deviation move in either feature has comparable geometric weight.

Neither representation is universally right. Standardization says relative movement in each feature deserves similar weight. Raw units say a dollar of revenue and a satisfaction point enter the distance exactly as recorded. Log transforms can make multiplicative revenue differences more meaningful; domain weights can encode an explicit cost. The interviewer wants the assumption, not the ritual “always standardize.”

Explained variance is not predictive importance

PCA is unsupervised. It never examines y. A component can explain large feature variance and have no relationship to the target, while a low-variance direction can contain the predictive boundary.

Consider two synthetic features:

• x_noise ranges widely because of a harmless measurement setting;
• x_signal varies only between -0.2 and 0.2, but its sign determines whether a machine will fail.

If PCA retains only the largest-variance direction, it can keep x_noise and discard x_signal. Reconstruction error remains low relative to total feature variance, yet classification collapses. The example is deliberately constructed, but the lesson is general: a 95% variance threshold is a compression heuristic, not a guarantee of target performance.

Select k with evidence matched to the objective:

Evidence	Question answered	What it cannot prove
Cumulative explained variance	How much sample feature variance does the subspace retain?	Predictive value or causal relevance
Reconstruction error	How closely can the original matrix be linearly rebuilt?	Useful class separation
Cross-validated task metric	Does this pipeline generalize for the chosen target and loss?	Robustness under future drift
Loading and score stability	Do components persist across samples?	That labels assigned to components are causal
Latency and memory	Does compression improve the deployed system?	Statistical validity by itself

Plot the spectrum and compare a small, prespecified grid of k. If PCA is used only for visualization, report how much structure the two-dimensional view omits. If interpretability matters, remember that a component blends features and that correlated samples can rotate nearby directions substantially even when the retained subspace is stable.

The leakage-safe pipeline

The wrong workflow is:

1. standardize all rows;
2. fit PCA on all rows;
3. split the transformed matrix;
4. report test or cross-validation performance.

The held-out feature distribution has already influenced the means, scales, loading directions, and chosen variance cutoff. Labels were not used, but this is still data leakage because test information shaped the fitted representation.

The correct holdout workflow is:

1. split raw rows into train and test using the deployment-relevant rule;
2. fit the scaler on training rows;
3. transform training rows;
4. fit PCA on transformed training rows;
5. choose k using training data and an inner validation procedure;
6. fit the downstream estimator on training component scores;
7. apply the frozen scaler and PCA transform to test rows once.

For cross-validation, every fold needs its own fitted scaler and PCA. A scikit-learn Pipeline keeps those steps inside the estimator that each fold fits. If k is tuned, put that choice inside the same cross-validation or nested-validation boundary. A time-dependent problem also needs chronological splits; a perfectly coded random pipeline cannot repair future-to-past leakage.

The same rule applies at deployment. Save the training means, scales, component matrix, order of input columns, missing-value policy, and software version. A request containing a missing or reordered feature should fail clearly rather than silently project the wrong vector.

When PCA is a poor fit

Sparse counts or text

Centering a sparse matrix can make it dense and expensive. scikit-learn documents TruncatedSVD as an alternative that does not center the data and works efficiently with sparse term-count or TF-IDF matrices. Its components and variance interpretation are related to, but not identical with, centered PCA.

Nonlinear structure

A linear plane may cut across a curved manifold. Kernel PCA, manifold methods, or an application-specific representation can capture different geometry, but each adds assumptions and validation work. A prettier embedding is not automatically a better production feature set.

Outliers and heavy tails

Means and squared reconstruction error give extreme observations influence. Investigate data quality, consider robust transformations or robust PCA variants, and test whether conclusions depend on a few rows.

Changing populations

Loadings learned from one period may no longer summarize another. Monitor feature distributions, reconstruction error, component-score distributions, and downstream metrics. Refit on a governed schedule, then compare subspaces and downstream behavior before replacing the transform.

Need for original-feature explanations

PCA can reduce multicollinearity for a predictive model, but coefficients on components are harder to translate into original variables. Back-transforming a linear coefficient vector is possible, yet uncertainty and correlated-feature interpretation remain. If the decision requires feature-level action, regularization or domain feature selection may be easier to defend.

Common wrong answers and precise repairs

Wrong answer	Why it fails	Better response
“PCA standardizes the data.”	Standard PCA centers; common implementations do not automatically equalize scale.	State the centering and scaling steps separately.
“95% variance means 95% accuracy is retained.”	Feature variance is not target performance.	Tune k with a leakage-safe downstream metric.
“Fit PCA once before cross-validation.”	Validation rows influence every fold's representation.	Fit the whole preprocessing pipeline inside each fold.
“PC1 is the most important business feature.”	PC1 is a direction, not one feature or a causal effect.	Inspect loadings and validate against the decision.
“The loading sign has meaning.”	Eigenvector sign is arbitrary.	Interpret relative patterns with a stated convention.
“PCA solves multicollinearity for free.”	It trades original-feature meaning for orthogonal components.	Explain the prediction and interpretation trade-off.

A concise interview answer template

Start with: “PCA preserves high-variance linear directions in the feature representation I choose, so I will define units and the evaluation boundary first.” Then:

1. Define the sample, columns, missing-data rule, and deployment split.
2. Fit centering, optional scaling, and PCA only on training data.
3. State whether k serves reconstruction, visualization, latency, or prediction.
4. Compare cumulative variance with leakage-safe downstream validation.
5. Inspect loadings, score distributions, stability, outliers, and discarded signal.
6. Freeze and version the complete transform for test and production data.
7. Decline PCA when sparse, nonlinear, unstable, or feature-level requirements dominate.

That sequence shows the linear algebra and the judgment needed to keep a clean notebook result from becoming a misleading model claim.

Sources and Further Reading

• scikit-learn: PCA API — centering, solvers, attributes, and implementation behavior.
• scikit-learn: Decomposing signals in components — PCA and related decomposition methods.
• scikit-learn: Common pitfalls and recommended practices — inconsistent preprocessing, pipelines, and data leakage.
• Jolliffe: Principal Component Analysis — standard mathematical treatment of components and variance.
• Tipping and Bishop: Probabilistic Principal Component Analysis — probabilistic formulation and reconstruction view.

Sources checked September 13, 2026. All matrices, feature ranges, and deployment examples are synthetic.