Databricks Workflow Interview Questions: Repair Runs, Task Dependencies, and Repeated Side Effects

A Databricks task marked failed may already have written its output. Repairing the run can recover the workflow while repeating that write. The strongest answer to Databricks Workflow interview questions therefore separates two decisions: which tasks should the scheduler rerun, and which business effects must be reconciled before execution resumes?

Databricks now documents this orchestration through Lakeflow Jobs. This guide uses current official documentation and an original failure ledger; it does not claim these scenarios were reported in a Databricks hiring round.

Read the run history before proposing a retry

The official repair guide directs users to the run history and task details to identify failures. Start with the specific task, attempt, error, compute context, input boundary, and output location. A red job summary does not identify the failed stage, so it cannot tell you which remedy to apply.

For an interview, distinguish the job definition from one run and from an individual task attempt. The definition describes dependencies and configuration. The run is one execution of that definition, while retries or repairs can create additional task executions associated with recovery.

Our diagnostic rule is to inspect three records together: scheduler status, task logs, and the target system's state. A timeout in the logs does not prove the target rejected a write. A successful target write does not prove that every later operation in the task completed.

Before increasing workers, locate the failure boundary. Did the task fail before starting, while reading, during a transformation, after a write, or while reporting completion? Resource pressure, permissions, bad data, and a lost acknowledgement require different remedies.

Dependencies describe eligibility, not business correctness

Databricks' task-dependency documentation describes Run if conditions. The default All succeeded waits for the dependencies to succeed. All done permits execution after dependencies finish regardless of their outcomes.

That distinction is useful for a cleanup or audit task, but it does not make a failed publication complete. A notification downstream of publication normally needs a successful publication contract, while an audit task may need to run precisely to record the failure.

Other conditions answer different questions. At least one succeeded does not require every upstream task to succeed. None failed also requires at least one dependency to have run. Read the exact predicate rather than interpreting the label as a general success guarantee.

The official documentation also distinguishes Excluded from Upstream failed and Upstream canceled. Excluded dependencies are treated as successful when evaluating conditions; upstream-failed and upstream-canceled states are treated as failed. If every dependency is excluded, the task is excluded as well.

An answer that calls all of these states “skipped” loses useful information. A failure-handler task can be excluded because its condition was not met, while a normal downstream task can be blocked by an upstream failure. Neither state tells you that the task executed its business logic.

Original run ledger: extract, publish, notify, audit

Consider this original multi-task job. E extracts a frozen input batch. P depends on E and publishes records. N depends on P with All succeeded and sends a completion notification. A depends on P with All done and records an audit outcome.

The batch contains three business records, order-41, order-42, and order-43. P writes order-41 and order-42, then loses its acknowledgement and fails before establishing whether the batch is complete. The exact target state must be read; the scheduler cannot settle that uncertainty.

Task	Recorded execution outcome	What it establishes
E	Succeeded	Frozen extraction is available
P	Failed	Attempt did not complete successfully; target effects uncertain
N	Upstream failed	Success-dependent notification did not run
A	Succeeded under All done	An audit ran, possibly recording failure

The important trap is A. A successful audit does not transform P's partial publication into success. It also does not necessarily refresh itself when P is later repaired. Decide whether the audit is an immutable attempt record or a representation of the latest business outcome.

This ledger is specific to the declared dependency graph. A different graph, conditional branch, or run predicate changes the task eligibility. Draw those inputs before making a statement about what will run next.

What does repair rerun?

Databricks officially supports repairing unsuccessful multi-task runs by rerunning unsuccessful tasks and the dependent tasks included in the repair selection rather than repeating every successful task. It also states that unsuccessful tasks use the current job and task settings at repair time.

For the ledger above, inspect the repair selection for P and the unsuccessful dependent N. E should remain available as the original successful extraction for the three-order batch. Do not assume the successful audit sibling A will execute again merely because P is being repaired.

Record the selected tasks rather than saying “repair the job” as though selection were irrelevant. If a business requirement needs a fresh final audit, explicitly arrange an appropriate finalization action or repair plan and explain why it is necessary.

Current settings introduce another boundary. If you change P's notebook, parameters, or compute configuration before repair, the rerun is not necessarily an identical execution of the original configuration. Keep the code version and input identity in the recovery record.

A settings fix can be legitimate, but it can also make E's preserved output incompatible with P's new expectations. For example, changing the expected schema after extraction requires an explicit compatibility check. A green rerun under changed code is not evidence that the original code succeeded.

Why can a failed task repeat a successful effect?

The current official repair guide explicitly warns that repair restarts an unsuccessful task from the beginning and does not make tasks idempotent. A task that wrote partial output can duplicate that output on rerun. This is a documented platform boundary, not a candidate's speculation.

In our ledger, suppose inspection finds order-41 and order-42 already present. Replaying an append of all three records creates a second copy of both existing records. The repaired task might finish successfully while the target silently contains five rows for three business records.

The scheduler's success and the data's correctness are separate assertions. An append counterexample makes that distinction measurable: first attempt writes two rows; repaired attempt appends three; total rows become five. Expected business identities remain three.

Do not solve this by generating a new key from each attempt ID. The same business operation would receive a different identity on every replay, preventing deduplication. Attempt IDs are useful provenance, but the stable business key must survive the recovery path.

Design the idempotency boundary around the effect

For this original batch, define a key that identifies the intended output across attempts. It might include tenant, business record ID, and the version of the operation. A run ID alone may not survive a manual rerun that performs the same business operation.

The target should reject or reconcile conflicting content under the same key. Silently treating a changed payload as an already completed operation can hide a real correction. State whether later versions update the record, produce a new version, or require review.

For a keyed table, a deterministic merge can be part of the design. It is not a complete exactly-once guarantee for the whole workflow. Source duplicates, concurrent writers, version precedence, and external operations still need contracts.

Use the Delta Lake MERGE interview guide for the table-specific boundary. This article focuses on scheduler recovery and the effects that sit outside a single table operation.

For a simple local oracle, represent the target as a map keyed by business identity. Applying the same three records twice should leave three entries with the same payloads. Then deliberately change order-42's payload under the same version and verify that the conflict is detected.

That map exercise demonstrates a logical invariant, not database atomicity. A production implementation needs an atomic target operation or a transaction policy that protects the key against concurrent attempts. A process-local set disappears after restart and cannot prove durable deduplication.

Notification delivery needs its own recovery policy

N introduces an external side effect. A published table and a sent notification may not participate in the same transaction. If N sends a message and crashes before recording completion, its next execution faces the same uncertain-outcome problem as P.

An outbox can record notification intent alongside durable business state. A dispatcher then retries delivery using a stable message identity where the provider supports it. This is our architectural recommendation, not a promise that every downstream API offers idempotent delivery.

A local “sent” flag written after delivery leaves a crash window between the provider accepting the message and the flag being saved. A flag written before delivery creates the opposite window: the process can crash after marking the message but before sending it.

Explain what evidence the provider exposes. If it supports querying by an operation key, reconcile before retrying. If it offers neither deduplication nor status lookup, exact duplicate prevention may be impossible across that boundary; choose and document the delivery policy instead of inventing certainty.

In an interview, distinguish notification intent, provider acceptance, and recipient delivery. A successful API response usually proves only the stage defined by that provider. The workflow's recovery record should name the actual observed stage.

Preserve the input contract through recovery

E's success is useful only if its output remains identifiable and available. Record a batch identity, source interval, data version, and relevant schema assumptions. An extraction that reads “latest data” again during a separate full rerun may produce a different batch.

When a repair uses E's original result, specify how P obtains that result and what retention policy keeps it available. Do not assume every temporary file survives a new compute instance. Recovering compute and preserving business input are different responsibilities.

If late records arrive after E finishes, decide whether they belong to a later batch or require a controlled correction. Replaying the original batch should not silently widen its source interval. Otherwise the operation's stable identity stops identifying one consistent input.

For mutable reference data, pin the relevant version or define an intentional refresh policy. A repaired transform with a newer exchange rate or customer mapping can produce different payloads under the same record key. That is a content conflict to resolve, not merely a duplicate to suppress.

What would prove that the repaired run is correct?

Our recovery acceptance contract requires more than a green task. Verify the three expected business IDs, the expected payload or version for each, no duplicate operations, and the defined notification state. Also verify that the final audit reflects the intended attempt or final outcome.

Keep the scheduler proof separate: selected tasks, resulting states, attempt history, and the settings used. A comparison between the original and repaired settings helps explain why the second attempt behaved differently.

Test the dangerous interruption points in a synthetic fixture: before any write, after two writes, after the final write but before acknowledgement, and after notification acceptance. The expected final business state should remain explicit for every interruption.

Avoid claiming a cloud integration test from a local map or SQLite fixture. Those can validate arithmetic and replay logic, while permission errors, runtime configuration, transaction isolation, and provider behavior require platform-specific verification.

For an interview answer, end with the evidence that would change your plan. If the target contains conflicting versions, pause replay and reconcile. If no effect occurred and the root cause is resolved, rerun the selected tasks under the recorded configuration.

Five questions for dependency and replay practice

Question	Follow-up to rehearse
Implement Task Dependencies and Failure	Distinguish failed tasks from blocked dependents.
Recover Durable Workflows with External Effects and Long Waits	Reconcile external state before repeating effects.
At-Least-Once Job Queue with Timeouts, Retries, a Dead-Letter Queue and Dependencies	Define stable business identity across attempts.
Load Daily JSONL LLM Chat Logs Into a Warehouse: Schema, Validation, Dedup, Idempotency	Pin batch identity, source schema and duplicate policy.
Design idempotent daily loads with deduping	Check the target after interrupted and repeated loads.

Choose one Databricks Data Engineer question and bring two ledgers: the task-state ledger and the business-effect ledger. Your answer should justify both the repair selection and the expected target state after replay.

Sources and Further Reading

• Databricks: troubleshoot failures and repair multi-task jobs
• Databricks: task dependencies, Run if predicates, and excluded states

Research checked October 5, 2026. The DAG, records, interruption points, and acceptance contract are original practice material. No live Databricks job or external notification was executed.