Articles
Spark RDD Interview Questions: combineByKey, Partitioners, and Mutable Aggregation State
Practice Spark RDD interview questions with combineByKey traces, weighted means, partitioner evidence, mutable-state ownership, and targeted counterexamples.
Strong Spark RDD interview questions test whether your aggregation remains correct when values are split across partitions, partial results arrive in another order, or a callback mutates an object. A result that looks right on one small input can hide a broken merge rule or misleading partitioning metadata.
Explain three contracts: the state you retain, who owns mutable state, and where each key belongs. Keep those contracts separate from performance observations. Fewer partitions or fewer allocations do not establish correctness.
Practice with Word Count and Multiset Intersect Using Only a Mock Spark RDD API. Then use the original mean-aggregation exercise below to trace the callbacks and challenge your assumptions.
Establish the API and execution boundary
Official documentation boundary: The Apache Spark API pages checked on October 6, 2026 identify PySpark 4.2.0. Establish the installed Spark version, language API, and execution environment before transferring implementation details. The documentation version is not evidence of your cluster’s version.
Official API fact: combineByKey transforms keyed values into keyed combiners. The combiner can have a different type from the input value. Its three callbacks create state from one value, add another value to state, and merge two states.
Evidence boundary: The worked data and debugging cases are original practice, not candidate reports. We executed the Python callbacks and partition-aware models locally. PySpark is not installed in this environment, so the RDD calls are source-reviewed examples, not executed Spark jobs. No shuffle, spill, task retry, or cluster performance result is claimed.
Trace all three callbacks on two partitions
Original exercise: Partition P0 contains (a,2), (a,4), and (b,9). Partition P1 contains (a,100), (b,3), and (b,6). Compute each key’s mean. Retain a pair (sum,count) rather than a running mean with no weight.
| Callback | Input contract | Example result |
|---|---|---|
createCombiner(value) | One value becomes a new state. | 2 becomes (2,1). |
mergeValue(state,value) | Existing state absorbs one value. | (2,1) and 4 become (6,2). |
mergeCombiners(left,right) | Two partial states become one state. | (6,2) and (100,1) become (106,3). |
The local states for a are (6,2) in P0 and (100,1) in P1. For b, they are (9,1) and (9,2). Merging yields a’s mean 106/3, about 35.3333, and b’s mean 18/3, or 6.
Original PySpark excerpt: Assume pairs is a pair RDD containing these integer observations. The callbacks themselves were executed locally; the RDD invocation was not.
def create(v):
return (v, 1)
def add_value(state, v):
return (state[0] + v, state[1] + 1)
def merge_states(left, right):
return (left[0] + right[0], left[1] + right[1])
states = pairs.combineByKey(create, add_value, merge_states,
numPartitions=4)
means = states.mapValues(lambda state: state[0] / state[1])
Four output partitions are an exercise choice, not a tuning recommendation. The input contains only two keys; increasing partitions does not manufacture additional independent keys or useful parallel work.
Why averaging local averages fails
Original counterexample: The local means for a are 3 and 100. Their unweighted average is 51.5, which differs from the correct 106/3. One partial mean represents two observations; the other represents one.
A mean alone loses information required for arbitrary merging. Carrying sum and count preserves that information, then divide after the merge. A weighted mean representation can also work if it retains the corresponding weight and uses the correct combination rule.
For another metric, identify its sufficient mergeable state explicitly. A maximum needs the current maximum. A variance needs more than an unweighted mean. An exact distinct count cannot generally merge two counts by addition because the underlying sets may overlap.
Define eligible input too. Are nulls skipped, rejected, or counted? Are negative values permitted? Our exercise accepts integers and contains no nulls. Adding a null policy changes the count contract and should produce new examples, rather than silently borrowing a library’s behavior.
Ask what an absent key means. If no observation exists for a key, this aggregation emits no state for it. Producing every expected key with a default value requires a separate domain of keys and a deliberate missing-result rule.
Make merges insensitive to partition and arrival order
Preparation guidance: For this unordered mean problem, combining partial integer states must produce the same sum and count across valid grouping and merge orders. The state merge is associative and commutative; changing the partition split does not change the intended answer.
Test an uneven split, a single partition, several partitions with empty ones, and reversed merge order. Also insert an unrelated key between two values of the first key. That last case can expose accidental shared mutable state that a single-key example cannot detect.
Floating-point arithmetic adds a separate limit. In ordinary Python floating-point evaluation, (1e16 + -1e16) + 1 produces 1, while 1e16 + (-1e16 + 1) produces 0. An algebraically sound sum can therefore vary numerically with grouping.
Decide whether the application needs an error tolerance, a more stable summation method, or an exact numeric representation with appropriate range and cost. Do not promise identical floating-point answers merely because the mathematical operation is associative.
The RDD programming guide discusses parallel reduction contracts and distributed execution. Use it to establish the execution model; demonstrate your particular aggregation with a counterexample and a state invariant.
Mutable aggregation is allowed, but ownership matters
Official API fact: The combineByKey reference permits mergeValue and mergeCombiners to modify and return their first argument. Mutation is therefore not automatically a bug. Sharing that mutable argument across unrelated states is the problem.
Original broken callback: A captured list is returned each time a combiner is created in the same callback environment.
shared = []
def bad_create(v):
shared.append(v)
return shared
Creating a’s state with 2 and b’s state with 9 now returns the same list object containing both values. Later mutation under one key changes the other key’s apparent state. A local identity check makes the ownership failure visible before discussing cluster scheduling.
For a collection-valued state, allocate a fresh container on creation, then mutate only the state the callback owns. For a mean, the immutable tuple above avoids this aliasing risk and keeps the state small.
Do not describe the captured list as one globally shared driver list accessed by every executor. The programming guide’s closure discussion explains why distributed tasks receive serialized closure state. Our aliasing counterexample concerns repeated creations in one callback environment; it does not assume universal object identity across workers.
Likewise, avoid using external side effects in a combiner as the authoritative business result. Re-evaluation or task retry can repeat work. A write to another system needs its own idempotency and commit contract; correct in-memory merging does not supply one.
Does aggregateByKey reuse one mutable zero everywhere?
Official implementation fact: The current PySpark RDD source implements aggregateByKey by deep-copying the zero value when creating an initial state, then applying the sequence function. The blanket claim “a mutable zero is always the same object for every key” is incorrect for that implementation.
The aggregateByKey API describes the sequence and combiner functions and a neutral zero value. Your zero and functions must still express the correct operation. A non-neutral initial contribution can make the result depend on how many partial states are created.
For our mean, (0,0) is neutral for the sum/count state. Starting each partial state with (10,1) adds synthetic observations and corrupts the answer. More partitions can reveal the corruption, even if the one-partition result initially looked plausible.
Deep copying is not a substitute for reviewing application-owned aliases, unusual object copy behavior, or external resources. Explain the documented implementation first, then inspect the particular state object instead of applying a generic mutable-default warning indiscriminately.
Partition count is not a partitioner contract
Official behavior: mapValues keeps keys unchanged and retains partitioning. map defaults to preservesPartitioning=False. partitionBy establishes keyed partitioning using a partition count and function.
Original ledger: Assume a pair RDD has four partitions with valid keyed partitioning metadata.
| Transformation | What remains true? |
|---|---|
mapValues changes only the aggregate value. | Key placement remains valid and the partitioner is retained. |
Ordinary map returns the same keys with changed values. | Physical partition count remains four, but default mapping does not retain the partitioner declaration. |
map changes keys and incorrectly declares preservation. | Four partitions remain, but records may no longer belong in the partition claimed by their new keys. |
In the PySpark Partitioner implementation, equality checks both partition count and partition function. Equal counts alone are insufficient. Two separately defined functions that happen to compute the same mapping are not automatically evidence of equal partitioner objects under this check.
Treat preservation as a correctness declaration. For a two-way toy partitioner key % 2, changing key 0 to 1 without moving its record leaves it in the wrong partition. If downstream code trusts the false metadata and skips redistribution, an aggregation can retain separate partial results for what is now the same key.
Explain performance with evidence from the actual job
Official source fact: The PySpark implementation locally combines values, applies keyed partitioning, then merges combiners. Local aggregation can reduce transferred state when many values of the same key compress into a small combiner.
Our performance reasoning: A constant-sized sum/count state has different growth from a list retaining every observation. A hot key still concentrates its final aggregate; increasing reducer count does not split that one key automatically. A large state or expensive callback can remain a bottleneck after local combination.
Inspect the actual lineage, partitioner metadata, stage boundaries, shuffle bytes, spill evidence, and task duration distribution. “combineByKey always shuffles” ignores compatible existing placement; “same partition count means no shuffle” ignores the partitioner contract. Check the installed implementation and observed job rather than asserting either slogan.
Original debugging check: If a result doubles after a rerun, distinguish duplicate business observations from repeated task-attempt logs. Two equal values may be legitimate separate events. Deduplicating by value alone changes the metric; a business event identifier and an explicit duplicate policy are needed if the input contract requires deduplication.
Record a tiny input, expected per-key sum and count, callback state after each local merge, and final state after combining. If those invariants pass but a real job fails, investigate serialization, input changes, false placement metadata, and external effects separately. Keep diagnostic logging observational: it should not change the state or decide which events count.
Our local model exercised seeded input reorderings, partition splits, merge permutations, weighted means, independent state identity, the shared-list failure, and false key placement. Those checks support the original callback logic. They do not replace a Spark integration run or prove a performance improvement.