Articles

Lakeflow Declarative Pipeline Interview Questions: Streaming Tables, Expectations, and Recovery

Practice Lakeflow pipeline interviews with streaming tables, expectation actions, checkpoint guarantees, and recovery when source history has expired.

A Lakeflow Declarative Pipeline interview question becomes difficult when the happy path ends. A streaming table ingests new files, an expectation detects a bad row, and someone suggests a full refresh to “clear the error.” That refresh can be harmless only if the source history is still replayable. The valuable answer connects the declaration, checkpoint, quality action, and recovery plan.

This guide uses an original order-events pipeline to work through that chain. For broader lakehouse preparation, see Databricks Interview Questions for Data Engineers. The current product name is Lakeflow Spark Declarative Pipelines; some older material calls the feature Delta Live Tables (DLT). Databricks' name-change explanation is useful when an interviewer or legacy code uses the older term.

Evidence boundary: Official facts about pipeline flows, expectations, processing guarantees, and full refresh come from Databricks documentation. Candidate reports: no verified same-cycle candidate report says a named employer asks this exact order-events scenario. Inference: the pipeline design, failure drill, and interview rubric are original practice examples, not a report of a real customer incident.

The exercise: late events, invalid totals, missing history

An order-event feed arrives in object storage. Each record has event_id, order_id, event_time, amount_usd, and a status. The business wants a cleaned streaming table and a daily revenue view. The feed may resend an event after a network retry. One batch contains a negative amount that violates a new data-quality rule. The upstream bucket retains only thirty days of files, and a worker has failed after processing some recent files.

Your task is to explain how to declare the pipeline, which quality action to choose, and how to recover without duplicating or deleting legitimate history. The initial question should be: Is a negative amount truly invalid, or is it a refund with a missing event type? An expectation encodes a business assertion, so an ambiguous rule can be more dangerous than a failed job.

A strong opening answer: “I would make the source and table contracts explicit, use a managed streaming flow and stable event identity, choose a deliberate action for violations, and inspect the checkpoint plus event log before replaying anything. I would not full-refresh a target whose source no longer holds all historical files.”

What is a streaming table in this pipeline?

Official fact: Databricks Lakeflow flows documentation describes flows as the units that update targets. A streaming table's default flow appends newly processed records across pipeline updates; its managed checkpoint tracks progress. A materialized view is a different target type whose query can be recomputed or incrementally maintained depending on the query and system support. Choose based on semantics, not which decorator looks shorter.

An append-only raw event stream is a plausible streaming-table source. A daily revenue calculation may be a downstream materialized view or another carefully defined transformation. If statuses can reverse an order or late events change a past day, simply appending a daily aggregate may be wrong. State whether the input is immutable events, corrections, or CDC, and define the business key and event-time policy before choosing a target.

A minimal illustrative Python shape in the current API is:

from pyspark import pipelines as dp
from pyspark.sql.functions import col

@dp.table(name="orders_bronze")
def orders_bronze():
    return spark.readStream.format("cloudFiles") \
        .option("cloudFiles.format", "json") \
        .load("/Volumes/catalog/schema/orders_inbound")

@dp.table(name="orders_silver")
@dp.expect_or_drop("has_identity", "event_id IS NOT NULL AND order_id IS NOT NULL")
def orders_silver():
    return spark.readStream.table("orders_bronze").filter(col("status").isNotNull())

This is a design sketch, not deployable configuration for every workspace. It omits schema evolution, permissions, retention, deduplication, and actual refund semantics. Databricks documents the current from pyspark import pipelines as dp form; older dlt code can appear in existing projects. Do not treat a copied example as a substitute for checking the workspace runtime and API support.

Expectations: warn, drop, or fail?

An expectation evaluates a rule on rows in a streaming table or materialized view. Official fact: Databricks' expectations guide distinguishes three actions. expect keeps violating rows while recording quality information; expect_or_drop removes them from the target; expect_or_fail fails the update when a violation occurs. The choice changes the data contract, not merely an alert color.

For missing event_id, dropping may be acceptable only if the raw bronze record and reason remain available for repair and reconciliation. For an impossible amount_usd, failing a customer-facing silver update may be safer than silently excluding revenue. If the negative value is a valid refund, neither drop nor fail should be applied until the schema distinguishes that case. A “warn” path is useful for early observation but allows invalid data downstream; pair it with a clear quarantine or consumer gate when correctness matters.

Do not claim fail action always gives a convenient quality count. Databricks notes that metrics are shown for warn and drop, while a fail stops the update and does not record the same expectation metrics. It provides failure details in the event log and error messages. Also distinguish triggered and continuous pipelines: Databricks documents different propagation behavior for a failed flow. Your monitoring design must use the mode actually deployed.

The interview answer should include a numerator and denominator: number of source events, accepted events, dropped events by reason, and deliberately retained warning events. If five events fail validation, a daily revenue number without a reconciliation count is incomplete. An expectation is a filter or assertion, not a substitute for ownership of excluded records.

What do managed checkpoints guarantee?

Official fact: Databricks' processing guarantees guide says managed streaming tables combine checkpoints and Delta transactional writes to give exactly-once processing within their managed tables across retries. If a micro-batch fails, the system can resume from committed progress rather than blindly append partial output. This is why an ordinary worker restart should first be investigated as a resumable failure, not as a reason to delete state.

That guarantee has boundaries. If the upstream system sends the same business event twice with different source offsets or files, both are new input records to the stream. The pipeline cannot infer they are duplicates unless you define identity and deduplication. Likewise, external side effects such as an API call or a separate non-transactional sink need their own idempotency key. “Exactly once” for a managed Delta target is not a promise that a third-party payment endpoint runs once.

If the exercise includes CDC, Databricks recommends its managed AUTO CDC capabilities over an ad hoc merge loop when appropriate. You still need a stable key and a trustworthy sequence field. Replaying out-of-order changes without a correct ordering rule can produce an incorrect latest state even if the machinery commits each batch once.

A recovery decision tree for the failed worker

First identify the failure class: a transient worker failure, an expectation violation, a schema incompatibility, corrupt checkpoint state, or a source-retention gap. Read the pipeline update status and event log, then compare latest processed source position with what the source still holds. The recovery action follows that evidence.

For a transient execution failure with an intact checkpoint, retry or resume the pipeline update and verify source-to-target counts. For a failed expectation, repair the rule or source data deliberately, then rerun according to the pipeline's mode. For checkpoint corruption, use Databricks' streaming recovery guidance and plan for a controlled recovery path. Do not manually delete checkpoint directories as a first response; that can lose the durable progress record.

For missing historical data, ask whether you have an immutable archive, a replayable Kafka topic, or a verified backfill dataset. A selective backfill flow may be safer than rebuilding everything. The flows guide describes adding separate flows to append historical data to a target. Such a flow still needs duplicate protection at the business-key level and an auditable range. “Backfill yesterday” should mean a precise source window, target, and reconciliation query.

Why can full refresh lose data?

Official fact: Databricks' full refresh for streaming tables describes a full refresh as discarding existing streaming-table data and metadata and restarting the stream from the beginning. The processing guarantees guide warns that a full refresh is safe only when the upstream can still provide the complete history needed to recompute the target.

In our exercise, the upstream bucket retains thirty days, but the revenue table contains a year. A full refresh from the surviving bucket can rebuild only a partial year. If the table contains legitimate older rows, “clearing the error” by full refresh could erase them. Even if the latest dashboard appears correct, past periods could disappear. The safe answer is to locate a durable archive or restore strategy, quantify the replayable range, and get a tested backfill plan before any destructive refresh.

Dependencies matter as well. A full refresh of one target can affect downstream tables or flows. Trace lineage and compare expected record counts and aggregates at each boundary after recovery. Establish a rollback artifact or snapshot where possible; do not rely on “rerun” as a rollback strategy when the upstream retention window has closed.

Diagnose the original bad event without hiding it

Return to the negative amount. Ask whether a refund event is represented by status="refunded", a separate event type, or a negative sale. Check the producer contract and a few raw records. If the rule was wrong, changing it is a schema/business correction. If the event is malformed, preserve the raw bronze row and its source identifier, fail or quarantine it as policy requires, and ask the producer to repair or resend it. Count every discarded event and reconcile against source offsets or files.

Use the event log for pipeline-update status and expectation observations. Distinguish “no new records” from “no records because a flow stopped.” Measure freshness as time since the latest accepted business event and separately as time since the pipeline's last successful update. A pipeline can be green while ingesting a stale feed; a red pipeline can leave the last good snapshot readable. Alerting should reflect both states.

A concise interview test plan

Build a five-record fixture: three valid events, one duplicated event_id, and one negative amount with an explicit refund status. Run the first batch, then resend the duplicate in a later file, fail a worker mid-update, and restart. Verify managed progress, business-key deduplication, expectation behavior, and reconciliation totals separately. Next, expire a source file in a test environment and demonstrate why full refresh cannot reconstruct the original target. This last step tests recovery design rather than mere decorator syntax.

The grading distinction is clear: a weak answer says “DLT will handle it automatically”; a stronger one names precisely what it handles and what your source, business rule, and external sink must handle. Another weak answer says “drop every bad row”; a stronger one can defend the policy for money-bearing records and show where a reviewer can find the excluded data.

Practice the related design questions

Complete practice question	Apply it to this pipeline
Data Pipeline Reliability, Backfills, and Spark Optimization	Choose a replay and reconciliation strategy.
Diagnose data quality and pipeline performance issues	Investigate a Databricks pipeline's quality and runtime signals.
Design a job scheduler ETL pipeline system	Define update retries and alerting.
Explain ETL schema changes and ensure integrity	Handle source drift without hiding bad records.
Reason About Duplicate Data and Scaling in Spark	Separate replay-safe processing from duplicate business events.

For a final rehearsal, explain what happens after one failed batch, after one duplicate business event, and after one full refresh with thirty-day retention. If you can state which one the managed checkpoint solves and which require your own policy, you have answered the interview question. Start with Data Pipeline Reliability, Backfills, and Spark Optimization and defend a recoverable design.