Articles

Spark MLlib Interview Questions: Pipelines, Unseen Categories, and Reliable Evaluation

Prepare for Spark MLlib interviews by tracing fitted pipelines, unseen categories, one-hot vectors, row coverage, cross-validation, and metric choices.

For Spark MLlib interview questions, follow one fitted pipeline all the way to its evaluation rows. Explain what fit() learns, what transform() reuses, how an unseen category is represented, and whether every intended example reaches the metric. A pipeline that finishes without an exception can still change feature meaning or discard difficult cases.

This article uses official Spark 4.1.1 documentation and original category and evaluation exercises. The version is pinned for reproducible reference, not described as the newest release. We found no candidate evidence establishing how often employers ask these exact questions.

Are you discussing spark.ml or the older spark.mllib API?

Clarify the API before answering. “MLlib” can refer to Spark's machine-learning library broadly, while Python's pyspark.ml and pyspark.mllib names identify different interfaces. This article focuses on DataFrame-based pipelines in pyspark.ml.

Official fact: Spark's ML guide describes the DataFrame-based API as the primary API, with the RDD-based API in maintenance mode. That does not mean every existing RDD implementation has disappeared or that migration is automatic. Spark MLlib guide.

If an interviewer says “MLlib pipeline,” ask which interface the code uses, then identify the input schema. For our exercise, it is one row per customer scoring event, with a categorical channel, numeric spend, a binary label during training, and a stable identifier for checking coverage.

Keep this question separate from Spark execution-plan tuning. Shuffle, skew, and caching matter when a job runs poorly, but they cannot repair a changed category mapping or an evaluation set that silently loses rows. First establish correctness, then investigate its computational cost.

What does Pipeline.fit learn, and what does PipelineModel reuse?

Official fact: An Estimator learns from data through fit() and returns a model that acts as a Transformer. A Pipeline chains stages; fitting it produces a PipelineModel whose fitted stages transform later data in order. Spark pipeline documentation.

In our classifier, StringIndexer learns a category-to-index mapping. OneHotEncoder learns category sizes. VectorAssembler combines existing feature columns, and LogisticRegression learns the classification model. The fitted pipeline packages those stages so serving can use the same feature meaning.

Draw two paths during the interview. Training goes through fitting the indexer, fitting the encoder, assembling features, and fitting the classifier. Evaluation goes through the resulting fitted transformations and classifier. It does not learn a new vocabulary just because the incoming DataFrame contains different frequencies.

A common broken answer is pipeline.fit(test).transform(test). This trains a new model on the evaluation data and evaluates that new model, rather than measuring the model trained on the intended training set. The correct operation for that trained artifact is fitted.transform(test).

Another mistake is replacing only the indexer at serving time while keeping the classifier. If a category changes index, the classifier's coefficients now refer to different meanings. Preserve the fitted stages together, and version the input schema with the artifact.

Why can category indices change after a second fit?

Official fact: StringIndexer defaults to frequency-descending order; equal frequencies are broken alphabetically. Its invalid-value policies are error, skip, and keep, with keep reserving index numLabels for unseen or null values. StringIndexer reference.

For this original fixture, training has three web rows and two store rows. The fitted mapping is therefore web → 0, store → 1. A later batch containing one web and four store rows should still use those indices when transformed by the fitted indexer.

If you fit a fresh indexer on that later batch, the mapping becomes store → 0, web → 1. The numbers look valid and the vector shape can remain unchanged, yet their meanings have swapped. This is a semantic error that a simple schema check may miss.

Record the learned labels and inspect them with the model artifact. To make a vocabulary intentionally stable across independently trained models, define that policy explicitly rather than accidentally relying on current frequencies. A different ordering choice changes the mapping contract; it does not excuse fitting preprocessing on held-out data.

Also distinguish feature categories from target labels. A new input channel can use an unknown-feature policy. An unexpected ground-truth class changes the evaluation problem and should not be casually converted into a new valid binary label.

How do unseen categories interact with OneHotEncoder?

Do not answer only “set handleInvalid='keep'.” Name the stage, the fitted category size, and the representation produced downstream.

Official encoder fact: OneHotEncoder drops the last category by default. With its own handleInvalid='keep', it adds another category for invalid numeric indices; if dropLast=True, that additional category is represented by an all-zero vector. OneHotEncoder reference.

Our indexer with keep has known labels web and store, plus an unknown bucket at index 2. Its output metadata advertises that extra bucket. When the encoder is fitted on this indexed column, index 2 is within its category metadata; it is not the encoder's separate out-of-range bucket.

Fitted setup in this exercise	`web`, index 0	`store`, index 1	Unseen `partner`, index 2
Indexer `keep`; encoder `error`, `dropLast=True`	`[1, 0]`	`[0, 1]`	`[0, 0]`
Indexer `keep`; encoder `keep`, `dropLast=True`	`[1, 0, 0]`	`[0, 1, 0]`	`[0, 0, 1]`

In the second setup, a numeric index outside the encoder's fitted range would use its additional invalid category and become all zeros. Thus “unknown strings always become all zeros” is wrong: the answer depends on both stages and their metadata.

Inspect the encoder model's categorySizes and the transformed vectors in a tiny fixture. This matters especially if an intermediate operation rebuilds the indexed column and loses metadata. Do not assume the category count inferred from observed rows is equivalent to the indexer's advertised vocabulary.

An all-zero categorical vector is not automatically an error. It may be the chosen reference category. The task is to establish which category it represents and whether that behavior matches your intended unknown-value policy.

What should a minimal fitted classifier look like?

This original PySpark review snippet assumes a ready Spark session and already separated train and test DataFrames with complete numeric inputs and labels 0 or 1. It illustrates the artifact contract; it was not executed on a Spark cluster for this article.

from pyspark.ml import Pipeline
from pyspark.ml.feature import StringIndexer, OneHotEncoder, VectorAssembler
from pyspark.ml.classification import LogisticRegression

indexer = StringIndexer(
    inputCol="channel", outputCol="channel_idx", handleInvalid="keep")
encoder = OneHotEncoder(
    inputCol="channel_idx", outputCol="channel_vec",
    handleInvalid="keep", dropLast=False)
assembler = VectorAssembler(
    inputCols=["channel_vec", "spend"], outputCol="features")
classifier = LogisticRegression(featuresCol="features", labelCol="label")
pipeline = Pipeline(stages=[indexer, encoder, assembler, classifier])
fitted = pipeline.fit(train)
predictions = fitted.transform(test)

Here dropLast=False makes category membership easier to inspect during the exercise. With three indexer categories and the encoder's extra invalid bucket, this setup has four categorical coordinates: partner maps to [0, 0, 1, 0]. Explain that dimension and the classifier's regularization choices before treating the example as a production specification.

Check one familiar category, one unfamiliar category, one null category, and one missing numeric value. The indexer's keep does not define the assembler's treatment of a null spend. Stage-specific policies remain separate even inside one pipeline.

For a serving check, omit the training label from a small scoring input while preserving required feature columns. Compare its feature vector with the training contract, then load the saved artifact and repeat that transformation in the intended runtime. Successful training alone is not a persistence or compatibility test.

Why can skip improve accuracy while reducing coverage?

Use an invented evaluation population of 100 rows, including 20 positives. Twenty rows contain an unseen channel; among those are 16 positives and four negatives. The remaining 80 rows contain four positives and 76 negatives.

Suppose the classifier predicts negative for every row. On all 100 rows its accuracy is 80%. If an indexer configured with skip removes the unseen-channel rows, the evaluator sees only 80 rows and reports 76/80, or 95% accuracy. Recall is still zero among the four surviving positives. The score improved because the population changed.

Report coverage = scored rows / intended rows alongside quality metrics. In this exercise coverage is 80%. Record counts before and after stages that may reject or filter data, and preserve identifiers so you can inspect the excluded cases.

Do not silently count dropped rows as false negatives either. Their outcomes were not scored under the stated pipeline. Report the exclusion separately; if the deployed workflow has an explicit fallback prediction, evaluate that complete workflow as a different, clearly defined policy.

Ask whether unseen-category frequency differs by customer segment or period. If new acquisition channels contain more positives, dropping them removes cases the business needs to assess. Compare unknown-category and exclusion rates by segment before relying on the aggregate score.

Where should fitting happen during cross-validation?

Our recommendation is to pass the unfitted pipeline as the estimator to model selection so every fold fits its learned preprocessing on that fold's training portion. Fitting the indexer on the entire dataset first gives each validation fold a vocabulary informed by data outside its training portion.

Official tuning fact: Spark's CrossValidator evaluates parameter settings across folds and fits the selected estimator on the full input dataset afterward. foldCol can supply explicit fold assignments. Spark model-selection documentation.

Define those assignments according to the task. If multiple rows belong to one customer, putting related rows in different folds can overstate generalization. For temporal prediction, ordinary randomly mixed folds may not represent the future deployment. Check that each fold is appropriate for your chosen design; a fold identifier alone does not create a valid time-series evaluation.

Keep an external final test outside parameter selection. The chosen model being refitted on all supplied training data is expected; supplying the final holdout to that same selection call would defeat its purpose.

Account for repeated work before tuning a large grid. Learned preprocessing inside folds is required for the stated evaluation contract, even when it adds cost. Cache a stable input only where measurement supports it, rather than moving fitted transformations outside the folds merely to reduce runtime.

Which evaluator matches the decision?

Official fact: BinaryClassificationEvaluator supports areaUnderROC and areaUnderPR, defaults to ROC area, and expects an appropriate score column rather than only a thresholded class decision. BinaryClassificationEvaluator reference.

For an imbalanced task, explain why the selected ranking metric matters and add precision/recall at the operating threshold. Neither area metric describes how many cases the business will act on at a particular threshold. Choose that threshold using validation data, then preserve it for the final test.

Write a compact evaluation record: model artifact, input snapshot, fold policy, intended rows, scored rows, positive count, metric definition, and threshold. This record lets another engineer reproduce the meaning of the score, even if the distributed job's execution details change.

Sources and Further Reading

Research checked October 4, 2026. Category vectors and metric arithmetic are original worked examples checked against version-pinned documentation and local fixtures. No distributed training benchmark or employer interview was conducted.