Articles

Spark Adaptive Query Execution Interview Questions: Skew, Coalescing, and Plan Changes

Prepare for Spark Adaptive Query Execution interviews with skew-threshold examples, coalescing decisions, initial versus final plans, and result checks.

Spark Adaptive Query Execution interview questions test whether you can explain a plan change from runtime evidence. AQE can combine small shuffle partitions, split eligible skewed join work, and reconsider a join strategy after statistics become available. A strong answer identifies which boundary changed, which action produced the evidence, and whether the result stayed correct.

Here, the focus is a narrower debugging task: explain why an adaptive plan changed, or why it did not.

What AQE changes, and what you must still prove

Official facts: Apache Spark's SQL performance documentation describes AQE as reoptimizing Spark SQL plans using runtime statistics. It documents post-shuffle coalescing, supported skew handling, and join-strategy conversion. The umbrella setting is spark.sql.adaptive.enabled; AQE has been enabled by default in upstream Spark since 3.2.

An enabled flag does not prove that a particular rule applied. A cluster distribution can change defaults, a plan can be ineligible, and a workload can have no useful adaptive change. Record the actual Spark version and settings. The official latest pages checked for this article resolve to Spark 4.2.0 documentation; use the documentation matching your environment when rehearsing.

Evidence boundary: The labs and partition sizes below are original interview exercises, not candidate-reported employer questions or measured cluster results. The debugging recommendations are engineering inferences. Do not quote the illustrative numbers as evidence that AQE guarantees a speedup.

Read the initial and final plans for the same action

Before execution, Spark has estimates. After eligible stages materialize, it has additional statistics. An initial sort-merge join can therefore differ from the final physical plan without a code change.

The DataFrame.explain reference documents formatted and cost modes. Use formatted output to locate exchanges, joins, and adaptive readers; use cost output to examine estimated statistics. Then connect the plan to a specific execution in the SQL UI.

result = events.join(customers, "customer_id")
result.explain("formatted")

# Small lab only: materialize this exact DataFrame.
rows = result.collect()
result.explain("formatted")

For a small fixture, collecting the exact DataFrame makes that relationship clear. The collect documentation warns that all returned data reaches driver memory. Do not use this lab action on a production-scale result; materialize an appropriate write and inspect that execution instead.

Be careful with result.count(). The count API returns the row count, but its aggregate execution is not automatically the same physical query as writing every joined column. Do not present a plan for one action as proof about another.

In an interview, identify the query execution, not just a screenshot of a plan. Ask whether the display shows an initial or final adaptive plan. Operator names and formatting vary by version, so describe the behavior behind the label.

Why AQE produces fewer shuffle tasks

Coalescing answers a small-partition problem. Suppose an original lab starts with 200 shuffle partitions, but filtering leaves only modest data. Executing hundreds of tiny reduce tasks may spend disproportionate time on scheduling and setup. Combining compatible neighboring partitions can reduce that overhead.

This does not mean the source read used fewer partitions, or that no shuffle happened. Distinguish input-file partitions, initial shuffle partitions, adaptive shuffle-reader partitions, and final output files. They describe different points in execution.

Spark's CoalesceShufflePartitions implementation corroborates the role of partition groups, advisory size, and parallelism preference. Read the source matching your installed release for an implementation-level answer; master is moving evidence, not a version guarantee.

For an interview exercise, imagine sixteen 4 MiB partitions becoming four 16 MiB groups. That arithmetic is a simplified illustration, not a prediction of Spark's grouping. Check the actual reader layout before reporting the final count.

Fewer tasks can also reduce parallelism. If tasks become too large or spill heavily, combining them can hurt latency. The question is whether scheduling overhead was the bottleneck, not whether the task count became smaller.

Why one giant join partition may be split

Skew handling addresses the opposite shape: one eligible join partition carries much more work than its peers. A hot customer identifier can leave most tasks finished while one task continues reading and processing a large share of the shuffle.

The OptimizeSkewedJoin source uses a threshold derived from the larger of an absolute byte threshold and a factor times median partition size. Supported join sides and split construction also matter. Skew optimization can split work and replicate matching work from the other side when needed.

Consider this original calculation with explicitly chosen settings:

Quantity	Illustrative value
Median shuffle partition	20 MiB
Skew factor	5
Absolute threshold	256 MiB
Effective comparison threshold	max(100, 256) = 256 MiB
Large partition	600 MiB

The 600 MiB partition exceeds both comparisons. A 200 MiB partition exceeds five times the median but not the absolute threshold. Calling the latter “ten times bigger” does not make it eligible under these settings.

Do not promise that 600 MiB becomes exactly ten 60 MiB tasks. Shuffle-block boundaries, the other join side, and the implemented rule affect the split. The useful prediction is eligibility; the actual plan establishes what happened.

A slow task is not automatically byte skew. A costly function, remote I/O, garbage collection, or a struggling executor can produce a similar duration chart. Compare partition bytes and task metrics before blaming the key distribution.

Why a sort-merge join becomes a broadcast join

A filter can make a join side smaller than its original estimate. Runtime evidence may then allow a broadcast hash join. The performance documentation notes that an adaptive conversion can avoid later work, but it does not undo every cost already paid before the decision.

Ask which side is being broadcast, whether the join supports that strategy, and whether the actual relation fits safely in the available execution memory. A small row count is not enough: wide values and concurrent work can matter.

An explicit hint and an adaptive decision are different mechanisms. Spark's hint documentation explains strategy precedence and that hints are not guaranteed for unsupported join types. Inspect hints before treating a changed plan as a pure AQE result.

For an original lab, create a large-looking customer relation, then apply a selective filter before joining. Compare its estimated and runtime size. If conversion does not occur, check the effective thresholds and join eligibility before raising memory-related settings.

The follow-up answer should be conditional: “I would compare the build-side statistics with the configured threshold and verify the final operator.” Avoid asserting that every small dimension must become a broadcast join.

Build two fixtures so you do not confuse skew with duplicates

For a tiny correctness fixture, make six events with customer keys [0, 0, 0, 0, 1, 2]. Give the dimension exactly one row per key. An inner join should return six rows.

from pyspark.sql import functions as F

events = spark.createDataFrame(
    [(i, key) for i, key in enumerate([0, 0, 0, 0, 1, 2])],
    ["event_id", "customer_id"],
)
customers = spark.createDataFrame(
    [(0, "A"), (1, "B"), (2, "C")],
    ["customer_id", "segment"],
)
joined = events.join(customers, "customer_id", "inner")

Now duplicate customer 0 in the dimension. The four matching events each produce two joined rows, so total output becomes ten. AQE must preserve that join meaning; partition balancing is not a deduplication rule. The official join reference is the semantic baseline.

This tiny fixture checks cardinality, not skew-threshold activation. Build a separate larger experiment for byte skew. Increase the hot key's payload volume, control broadcast choices, and inspect actual shuffle bytes. A repeated compressible string may not produce the bytes you expect; measure rather than deriving activation from input row count alone.

Keep the dimension unique in the performance experiment unless duplicate output is deliberately the question. Otherwise, a huge join result may be caused by multiplicity, and “turn on skew handling” obscures the real contract problem.

Compare configurations as a controlled experiment

Capture settings before changing them. Spark's configuration documentation explains configuration sources and the environment information shown in the UI. A copied notebook setting is not proof that the relevant execution used it.

For the lab, record the AQE umbrella, coalescing and skew switches, initial shuffle count, advisory partition size, and broadcast settings. Vary one mechanism at a time. If you alter data, executor count, and three thresholds together, you cannot attribute the observed change.

Use separate executions with comparable inputs. Preserve cold versus warm-cache conditions and note competing workloads. You do not need to promise a statistically rigorous benchmark in an interview, but you should avoid reporting one lucky run as a stable improvement.

If estimates are persistently wrong, examine catalog statistics as well. The ANALYZE TABLE reference documents collecting statistics. AQE does not make initial planning evidence irrelevant; a good initial choice can avoid work that an adaptive correction notices only later.

Evidence that closes an AQE debugging answer

Use the Spark Web UI to connect SQL operators with stages and task metrics. For later investigation, the monitoring documentation describes event logs and the history server.

Claim	Evidence to show
Small tasks were combined	Final adaptive reader and task count
Join skew was handled	Split join work plus partition-byte distribution
Join strategy changed	Initial and final operators for the same execution
Performance improved	Comparable elapsed time, spill, and shuffle metrics
Results stayed correct	Counts, key multiplicity, and reconciled aggregates

Also compare output-file count when the job writes data. An improvement in join latency can create inconvenient file sizes downstream. Decide whether the objective is query latency, job throughput, cluster cost, or downstream read efficiency before declaring success.

If AQE changes nothing, say so. An unchanged plan with balanced partitions and accurate estimates can be a healthy result. Your job is to explain the available evidence, not to force a visible adaptive rewrite.

A useful final counterexample is a balanced shuffle whose slowest task spends most of its time inside expensive row processing. Splitting a join partition does not necessarily address that cost. Ask whether the expensive operation happens before or after the adaptive boundary, and whether it can be expressed more efficiently without changing results. A diagnosis should point to the observed work, not to whichever setting you remember first.

The exercises here use batch DataFrame joins. Do not transfer their expectations to a stateful streaming job without checking that execution mode's supported behavior and version. State retention and streaming recovery introduce a different set of constraints.

Five questions for practising plan-based explanations

Question	AQE follow-up
Explain Spark Execution and Optimization	Compare initial and final stages
Reason About Duplicate Data and Scaling in Spark	Separate multiplicity from byte skew
Data Pipeline Reliability, Backfills, and Spark Optimization	Validate a change under repeatable inputs
Compare Spark RDDs, DataFrames, and Spark SQL Benefits	Identify the SQL optimizer boundary
Design MapReduce and Spark jobs	Defend partitions and shuffle decisions

Explain the initial plan, the runtime observation, the permitted rewrite, and the result checks in that order.

Sources and Further Reading

Research checked October 1, 2026. Documentation, operator labels, defaults, and rule eligibility should be checked against the deployed Spark version.