Articles

Dataproc Interview Questions: Failed Spark Jobs, Cluster Lifetimes, and Diagnostic Evidence

Prepare for Dataproc interviews by tracing failed Spark jobs, driver and executor logs, cluster deletion, durable diagnostics, and safe retries.

A Spark job fails, its temporary cluster disappears, and the next engineer sees only “ERROR.” Dataproc interview questions become useful when you can reconstruct which execution failed, distinguish the failure phase, and preserve enough evidence to justify a retry.

Record the failed execution's identity before changing cluster lifetime. A job reference, cluster name, YARN application, and Spark task attempt describe different objects. Recreating a cluster with the same name does not recreate the evidence or the exact environment of the failed execution.

Official facts below come from Google Cloud and Apache Spark documentation. Google currently presents the relevant cluster and serverless documentation under Managed Service for Apache Spark; the examples retain the Dataproc API and CLI terminology. Job names, partitions, times, and retry policies here are original exercises, not a deployed-cloud incident. Start with Explain Spark Execution and Optimization, then add the service identities and evidence boundaries below.

Which identity tells you what actually failed?

Official fact: the Dataproc Job resource includes status and status history, cluster placement, a driver-output URI, and information about associated YARN applications when available. jobUuid identifies a job over time, whereas a user-set job reference can be reused. done=true means completed; inspect the status to distinguish success, failure, or cancellation.

For our original case, a scheduler submits job daily-orders-42 to cluster orders-temp. The job UUID identifies execution J42, and the placement records cluster UUID C7. A YARN application handles its Spark computation. Stage 7 has a failing task for partition 39. Those labels are abbreviated for readability; actual service identifiers should be captured without truncation.

Evidence layer	Original example	Question it answers
Project, region, job reference, job UUID	J42 in the intended region	Which service submission are we investigating?
Cluster placement and UUID	Name `orders-temp`, UUID C7	Which cluster incarnation ran it?
YARN application and attempt	Application associated with J42	Which distributed application owns the containers?
Spark stage, partition, task attempt	Stage 7, partition 39, attempt 0	Which unit failed, and did another attempt recover?
Code and input identities	Package digest and input manifest	What was executed against which data?
Output and publication ledger	Attempt-specific files and manifest	What effects must a replay reconcile?

Do not start with a screenshot of the newest cluster named orders-temp. If that cluster is C8, its logs can be irrelevant to C7's failure. Record the join between the job's placement and the cluster UUID, then use it to constrain the evidence search.

Likewise, one executor loss is not automatically a failed application. Look for the affected task's later attempts and the final application/job outcome. An error from C7 can lead you to change C8 if you search only by the reused cluster name.

How do submission, driver, executor, and output failures differ?

Use this original diagnostic table to choose evidence that can disprove your hypothesis:

Failure phase	Example symptom	First evidence to examine	Why “add workers” is weak
Submission or service setup	Request denied before useful work begins	API response, identity, region, job status details	Capacity does not grant permission
Driver initialization	Python dependency missing before stages begin	Driver output, package location, runtime versions	More executors do not install the dependency
Distributed execution	Partition 39 repeatedly exhausts memory	Executor/container logs and task attempts	Skew may remain concentrated
Output commit	Computation completes but destination rejects writes	Destination response and publication ledger	More writers may repeat the same denial

In case J42, suppose the driver imports the program successfully and stages 1–6 finish. Stage 7 repeatedly fails on the same oversized partition. Completed stages 1–6 and repeated partition-39 failures favor a data-distribution or per-task resource issue over a missing driver package. It still does not establish the fix: inspect partition size, shuffle behavior, memory use, and the actual exception.

Now change the evidence: no stage starts, and driver output contains ModuleNotFoundError. The appropriate experiment changes. Check the submitted dependency artifact and the selected environment before debating shuffle partitions. A local laptop import succeeds in a different environment and therefore cannot settle this question.

If writes fail with an authorization error, identify the runtime service identity that reaches the destination. The human who submitted the job and the identity used by the workload can differ. Verify the intended permission instead of assuming that console access proves the job can read and write every referenced object.

Make the investigation bounded. State the first evidence you will read, the leading hypothesis, and one observation that would reject it. A short evidence-led investigation is easier to explain than an unprioritized list of Spark tuning knobs.

Where are driver and executor logs stored?

Official fact: Google distinguishes service job driver output from Spark driver and executor logs. Their destinations depend on submission path, deployment mode, and logging properties. The job-output guide documents Cloud Storage's driverOutputResourceUri, job-scoped logging, and cluster-scoped yarn-userlogs behavior.

For a Jobs API submission, driver output normally streams to the client and is stored at the recorded output URI. Enabling job driver logging additionally sends that output to Cloud Logging. A Spark driver in cluster deployment mode has different destinations from one in client mode. Read the actual deployment configuration before interpreting an empty log query.

Official fact: executor logs can appear under the cluster's yarn-userlogs or a job-scoped container log, depending on dataproc.logging.stackdriver.job.yarn.container.enable. Jobs submitted directly with spark-submit on a cluster node do not have the same Dataproc Jobs API job ID and driver-output record. Submission and log-destination distinctions

Thus, “there are no job logs” can mean the query uses the wrong resource scope. For the original J42 investigation, record submission route, deploy mode, log ID, time interval, cluster UUID, and application ID before treating missing results as evidence that nothing happened.

Apache Spark's monitoring documentation explains the application UI, event logging, and history-server workflow. Retained event logs can support later task and stage analysis. They serve a different purpose from driver exception text; save both when the investigation requires execution structure and the actual failure message.

Do not imply that an event log alone reconstructs every external effect. An HTTP call from a task or a partly committed output path requires its own evidence. Tie the logs to the output ledger rather than treating a Spark UI as a complete business audit.

What evidence should outlive a temporary cluster?

Official fact: Google documents checkpoint diagnostic collection and one-time snapshot collection. Checkpoint diagnostic data is stored in a Cloud Storage path associated with the cluster UUID. Snapshot collection can target job IDs or YARN application IDs; collection of YARN application logs requires log aggregation. Cluster diagnostics

The following original evidence bundle lets another engineer reconstruct J42 after C7 is gone:

execution.json   project, region, job UUID, cluster UUID, application IDs
environment.json image/runtime versions, package digests, Spark properties
inputs.json      source object versions, partitions, schema and date boundary
driver-output   durable URI plus retained relevant error excerpt
task-evidence   stage, partition, attempt, executor/container log location
outputs.json    attempt paths, accepted partitions, publication manifest
diagnostics     collection operation, object location, retention policy

Capture facts that distinguish executions rather than copying every log line. A mutable package URI is weaker than the package digest or object generation that was actually used. “Yesterday's data” is weaker than a dated input manifest and timezone rule. A filename alone is weaker than a verified durable location and its retention policy.

Official fact: gcloud dataproc clusters diagnose supports selecting relevant jobs and YARN applications. Its CLI reference is the source for the available options. Diagnostic collection produces an artifact; it is not a substitute for examining that artifact's contents and saving its location.

This guide does not run diagnostic commands against a cloud account. For interview practice, explain when you would collect evidence, which scope you would select, and how the next engineer would retrieve it after compute is gone. Existing diagnostic and log retention can expire, so inspect the actual bucket and logging policies rather than promising indefinite preservation.

Does idle deletion mean there is no important work?

Official fact: scheduled deletion can use an idle interval, an absolute time, or a lifetime measured from cluster-creation submission. Jobs running when the cluster is deleted fail. On supported image versions, dataproc:dataproc.cluster-ttl.consider-yarn-activity defaults to true: both pending/running YARN applications and pending/running Jobs API work affect idle calculation. Setting it false makes that calculation consider Jobs API activity alone. Scheduled deletion

Original exercise: a notebook submits a YARN application directly while the Jobs API has no pending jobs. With YARN activity considered, the application matters to the idle timer. A design that checks only the Jobs API queue can reach the wrong conclusion. Verify the image version and property, not just a label saying “temporary.”

An idle threshold is also different from an absolute deletion deadline. Work beginning shortly before a fixed deadline may still be interrupted. For our daily batch, budget cluster startup, package staging, computation, validation, diagnostic collection, and an agreed investigation period. Make the cleanup condition visible to the scheduler and the on-call owner.

Do not solve every failure by extending lifetime forever. The decision is whether continued compute supports a specific investigation or safe completion. Once durable evidence exists and recovery has an owner, keeping an unused cluster may add cost without increasing confidence. This is a preparation recommendation, not a pricing benchmark.

Should you retry on the same cluster, a new cluster, or serverless?

For J42, rerunning identical code against the same skewed partition is likely to repeat the failure. A new cluster can remove some environmental uncertainty, but it does not change that partition's data or the program's algorithm. State which variable the retry changes and why the hypothesis predicts a different outcome.

Original output case: attempt A writes partitions 1–38 before the job fails. Attempt B reruns the full input into the same append-only destination. Duplicate output is now possible even if the new Spark job succeeds. A defensible proposal uses attempt-specific staging, checks expected partitions, and publishes through a defined commit or replacement policy appropriate to the sink.

Do not delete the previous production output as an unexamined retry step. Preserve the last accepted publication while evaluating the new attempt. If the sink supports transactional replacement, explain its boundary; if it does not, design a manifest or pointer protocol whose readers understand incomplete and accepted attempts. A success marker without that reader contract is insufficient.

Official fact: the deployment comparison distinguishes managed serverless execution from clusters with greater infrastructure control. In the serverless model, a batch workload is the execution resource; do not carry cluster/YARN troubleshooting assumptions across unchanged. Service key concepts

Choose from workload constraints: required framework, environment control, shared-cluster needs, and the evidence available for diagnosis. Serverless can simplify infrastructure ownership without eliminating dependency errors, bad inputs, destination permissions, or repeated side effects. Explain those retained responsibilities explicitly.