Spark Structured Streaming Interview Questions: Watermarks, Checkpoints, and Recovery

Spark Structured Streaming interview questions often turn on two separate timelines: how event time advances, and when an external write becomes durable. A watermark helps a stateful query decide what information it can retire. A checkpoint helps it recover progress. Neither phrase, by itself, explains whether your custom sink writes a record twice.

This guide uses Apache Spark 4.0.1, the version executed in the local examples. It does not claim that version is the newest release. Official semantics, observed local results, and editorial design advice are labeled separately. These are original technical exercises, not candidate-reported company questions.

Here, focus on predicting a small stream and defending what happens when the query stops at an inconvenient moment.

Define the query before predicting its output

“What happens to late data?” is incomplete without an operator, output mode, delay, and input sequence. The local exercise uses JSON files arriving one at a time, a timestamp column called event_time, five-minute tumbling windows, a ten-minute watermark delay, and append output.

Each file contains two events. After adding a file atomically, the harness calls processAllAvailable() before adding the next one. Spark can run additional empty micro-batches during that processing. Consequently, the three input steps are not identical to three Spark batch IDs.

The session time zone is UTC, with two local worker threads and two shuffle partitions. The output callback collects only this tiny fixture's finalized rows for inspection. Collecting an unbounded production batch to the driver is not the implementation being recommended.

counts = (
    events.withWatermark('event_time', '10 minutes')
    .groupBy(window('event_time', '5 minutes'))
    .count()
)

query = (
    counts.writeStream
    .outputMode('append')
    .foreachBatch(capture_small_test_output)
    .option('checkpointLocation', checkpoint_path)
    .start()
)

The timestamp is a real timestamp column, and withWatermark precedes the aggregation on that same event-time column. Those details matter more than remembering a code fragment with a plausible method name.

Predict six events across three input steps

All events belong to September 10, 2026, in UTC. The IDs are distinct; this exercise counts events rather than deduplicating business identifiers.

Input step	Event times and question
First file	A at 12:01 and B at 12:04. Is the 12:00–12:05 window final?
Second file	C at 12:03 and D at 12:16. Can C still contribute before finalization?
Third file	E at 12:02 and F at 12:17. Can E change the already emitted result?

Before reading the results, separate the largest event time observed from the time shown on your laptop clock. Advancing wall-clock time without newer event timestamps is not the same as advancing this query's event-time evidence.

Observed locally: after the first input step, no output row had been emitted. The reported watermark was 11:54. The early window existed in state, but append mode had not produced a final count for it.

After the second step and its subsequent processing, the watermark was 12:06. Spark emitted the 12:00–12:05 window with count 3, incorporating A, B, and C. The output callback received that row at Spark batch ID 3. The final progress record for that step showed one state row removed.

After the third step, the reported watermark was 12:07. No revised row for the early window appeared; its emitted count remained 3. The newer 12:15–12:20 window had not yet produced an append result.

The distinction between source steps and engine batches is not cosmetic. If an interview answer says “the second batch emits the row,” clarify whether that means the second supplied file or a specific engine batch. Empty batches and progress timing can make an otherwise correct explanation look contradictory.

Explain the watermark guarantee in one direction

Official semantics: the Spark aggregation guide guarantees inclusion for data within the configured event-time delay under the supported watermarking conditions. Data beyond that delay is not guaranteed to be dropped; it may still be aggregated.

The withWatermark API reference connects the delay to event-time progress and stateful processing. Treating it as “discard anything older than the current wall clock minus ten minutes” describes a different rule.

The local example demonstrates why a memorized lateness slogan is insufficient. C has an earlier timestamp than D, but both arrive in the second file and C contributes to the finalized count. E arrives only after the early window has been finalized in this run. The input schedule is part of the evidence.

Editorial advice: when asked about a boundary event, avoid promising an outcome from the delay alone. Name the relevant window, the watermark used by the operator, and whether state is still available. If the interviewer supplies an exact execution schedule, reason from it; otherwise, state the guarantee and the remaining uncertainty.

Do not silently treat a watermark as deduplication either. A second copy of A with the same business ID is a separate question from whether its event time is late. Your system needs an explicit identity and duplicate policy if one logical event must contribute only once.

Choose what the downstream consumer receives

For this aggregation, append output is useful when the consumer wants finalized window rows. The trade-off is that a recent window may remain absent while event-time progress is insufficient to finalize it. “No output yet” does not necessarily mean the source was unread or the query failed.

Update output instead exposes changed result rows, while complete output presents the full result table for each trigger. Complete aggregation mode retains the result rather than using the watermark to discard old aggregate state. These modes describe result delivery, not interchangeable settings that all fit every query and sink.

In an interview, ask what the consumer does with another row for the same window. Does it replace a prior value, add a delta, or insert another fact? Sending updated counts to an append-only consumer can double-count even if the streaming computation itself is correct.

For the fixture, a count of 3 is a total for one window. A downstream job should not interpret it as three additional events every time that same result is retried. Explain the output key and meaning before discussing throughput.

Inject failure before and after the sink commit

The second original experiment uses a separate file stream containing two records, A and B. Its foreachBatch callback writes to a local SQLite database. We restart the same query with the same checkpoint after an intentionally raised exception.

Two failure positions are tested: before the database transaction writes anything, and after the database commit but before the callback completes successfully. These are actual local Spark callback failures, not manually duplicated input rows.

Observed results:

Sink strategy and failure	Rows after restart
Ordinary append; fail before commit	2 rows. The retry performs the first successful write.
Ordinary append; fail after commit	4 rows. A and B are written again.
Transactional batch guard; fail before commit	2 rows. No completed batch was recorded before retry.
Transactional batch guard; fail after commit	2 rows. The repeated batch is recognized.

Every case invoked batch ID 0 again after restart. The checkpoint existed in all four cases. It did not, by itself, prevent the naive sink's duplicated effects after the commit.

The foreachBatch API supplies a batch identifier that can support deduplication. The broader guarantee still depends on your callback and sink. Describe exactly what you made atomic instead of saying “foreachBatch is exactly once.”

Make the deduplication record part of the write

The guarded SQLite fixture stores the batch ID and event rows in the same transaction. When a retry finds the batch already committed, it skips the repeated effects. If the transaction never committed, the retry can write the data and marker together.

This is the essential shape of the original experiment:

begin transaction
  if this query-generation and batch are already committed:
    finish without repeating effects
  else:
    write the batch's effects
    record the batch as committed
commit transaction

The fixture uses a separate database per query, so the batch ID is unambiguous there. A shared production sink needs an appropriate query-generation identity as well. Starting a fresh checkpoint can restart batch numbering; reusing old deduplication keys blindly could suppress legitimate new work.

A marker written before the data in a separate transaction risks recording completion without effects. A marker written afterward in a separate transaction leaves a crash window in which effects exist but the marker does not. The local test avoids both arrangements.

This experiment does not establish a transaction across two independent databases, a database and an email provider, or a cloud service that ignores your idempotency key. If a callback writes to multiple destinations, discuss each destination's replay behavior and the consistency requirement between them.

Decide whether a checkpoint can be reused

The versioned programming guide limits changes across restarts: changing the number or type of input sources is unsupported, and changing grouping keys or aggregate state schemas is not a routine compatible restart. Some filter, projection, and rate-limit changes are allowed subject to their documented conditions.

Original interview scenario: your query groups by a five-minute window, and a new requirement adds customer ID to the grouping key. Do not answer “keep the checkpoint and deploy” without addressing the changed state shape. Explain a migration plan with a new query boundary, replay inputs, output reconciliation, and a cutover condition.

Deleting a checkpoint is not a neutral repair. It changes the recovery starting point and can affect which source data is read again and how sink deduplication identifiers behave. Before choosing a fresh checkpoint, define the replay range and how existing outputs will be reconciled.

Similarly, a restart that succeeds technically does not prove that a changed business definition produces comparable metrics. If the meaning of a count changes, retain a version boundary so downstream users can distinguish old and new results.

Explain what you would monitor

For the local timeline, we inspected reported event-time watermarks, emitted rows, and state-operator counts. A production investigation would also relate input progress, processing duration, state growth, and sink behavior to a concrete symptom.

If recent windows are not appearing, first check whether event-time progress is advancing. If state grows continually, inspect the query's retention conditions and key cardinality. If outputs duplicate after recovery, trace the sink commit and retry rather than changing the watermark.

Keep monitoring evidence tied to a question. A small state size does not prove completeness, and a healthy throughput chart does not prove absence of duplicate business effects. State which invariant the measurement can actually test.

Sources and Further Reading

• Spark 4.0.1 Structured Streaming guide: aggregation, output modes, and restart compatibility.
• PySpark withWatermark reference: event-time delay semantics.
• PySpark foreachBatch reference: callback and batch identity.

Sources checked September 10, 2026. Local tests used Spark 4.0.1, file input, UTC, and SQLite; no Kafka, cloud-load, or cross-system transaction guarantee was tested.