Articles
Delta Lake Change Data Feed Interview Questions: Update Images, Checkpoints, and Retention
Practice Delta Lake Change Data Feed interview questions on update images, inclusive versions, checkpoint recovery, sink deduplication, and retention limits.
A Delta Lake change feed can be read successfully while its consumer produces the wrong result. An update arrives with an old image and a new image; the consumer adds both. A job restarts after writing its sink, then writes the same rows again. A checkpoint survives an outage, but the source history needed to resume has expired.
Good answers to Delta Lake Change Data Feed interview questions explain the contract across those boundaries. Start with what a change means, establish which versions the destination already represents, and then explain what happens when progress and output disagree. This guide develops that reasoning for Data Engineer interview preparation.
Evidence boundary: documented behavior is linked at its first appearance. The ledgers, incident cases, and proposed recovery rules are original practice examples and design analysis. They are not candidate reports, employer question lists, or claims that one sink strategy fits every pipeline.
Establish which change feed you mean
Current Databricks documentation distinguishes automatic CDF from legacy CDF. Automatic CDF requires eligible Unity Catalog Delta tables with row tracking and Databricks Runtime 19.0 or later; only Databricks readers can query automatic Delta CDF. Legacy CDF requires table-level enablement. The two cannot be used simultaneously. State the runtime and feature before answering. Official Databricks CDF documentation.
The following enablement example concerns the explicitly enabled, classic Delta Lake feature:
ALTER TABLE orders
SET TBLPROPERTIES (delta.enableChangeDataFeed = true);
Enabling that feature does not reconstruct changes from before enablement. Also, _change_data is not an interface you should scan directly: some changes can be obtained from the transaction log without dedicated change files. Read through the supported CDF APIs. Official Delta Lake CDF documentation.
An interviewer who asks, “We enabled CDC yesterday; can we replay last month?” is testing this boundary. Ask whether the historical changes were captured elsewhere. A current snapshot can establish current state, but cannot tell you every intermediate update that occurred last month.
Interpret update images against the destination’s purpose
CDF exposes _change_type, _commit_version, and _commit_timestamp. Change types include insert, delete, update_preimage, and update_postimage; the update images represent the row before and after the change. Official Delta Kernel CDF reference.
Use this original ledger. At version 40, A has amount 10 and B has amount 20. Subsequent changes are:
| Source version | Change type | Key | Amount | Meaning |
|---|---|---|---|---|
| 41 | update_preimage | A | 10 | Previous A value |
| 41 | update_postimage | A | 15 | Replacement A value |
| 42 | delete | B | 20 | Remove B |
| 43 | insert | C | 7 | Add C |
Question: what should the destination contain after version 43?
A current-state replica contains A=15 and C=7, totaling 22. It applies the postimage as the replacement row and the delete as a removal. The preimage is useful evidence but is not another current row to insert.
An incrementally maintained sum needs different handling. Starting at 30, subtract the preimage 10, add the postimage 15, subtract the deleted 20, and add the inserted 7. The net change is -8, producing 22. Filtering out every preimage and adding every postimage would overcount the update.
An audit destination may preserve all four change records instead of maintaining either representation. Describe the requested output before choosing filters. “Keep inserts and postimages” is a plausible current-state input policy, not a complete answer for deletes, aggregates, or history.
Follow-up: can _commit_version alone identify a change row? No: this ledger already has two records in version 41. A version identifies the table commit, not a unique row or a business-event sequence. In your design, scope versions to the source table and its lifecycle. Do not invent an ordering between same-version records merely by sorting their commit timestamp.
Make snapshot cutover and inclusive bounds explicit
Delta Lake CDF batch start and end versions are inclusive. Without an explicit starting position, a new CDF stream initially presents the current snapshot as inserts and then processes subsequent changes. Official Delta Lake CDF reading rules.
Question: the destination was loaded from a version-40 snapshot. Where does incremental consumption start?
Start at 41, provided the snapshot truly represents version 40, CDF is available from 41, and loading the snapshot completed correctly. Reading from 40 inclusively can reapply changes already represented by that snapshot. Starting at 42 would miss A’s update.
An illustrative batch read for the ledger is:
changes = (
spark.read.format("delta")
.option("readChangeFeed", "true")
.option("startingVersion", 41)
.option("endingVersion", 43)
.table("orders")
)
For a new streaming query, omit endingVersion, use readStream, and supply a dedicated checkpoint location on the write. These snippets demonstrate the read boundary; they do not implement a complete sink transaction.
After every required change through version 43 has committed successfully, a fresh version-based replay can begin at 44. Be careful with “the maximum version we observed.” Seeing one row from a version does not prove your application committed the complete required output for that version.
For bootstrap, record the actual snapshot version rather than wall-clock time. Otherwise, “the backfill finished at noon” leaves uncertainty about writes that occurred during the backfill. A repeatable source snapshot and a documented version boundary make that gap reviewable.
Separate checkpoint progress from sink acknowledgment
Spark checkpoints store query progress and, where applicable, state. Restart compatibility depends on the query changes being made. A checkpoint is not a generic transaction spanning your database, HTTP endpoint, and source. Official Spark Structured Streaming documentation.
For Delta streaming, initial-position options take effect for a new query; once progress is recorded in an existing checkpoint, those options are ignored. Changing startingVersion while retaining that checkpoint is therefore not a reliable way to rewind a consumer. Official Delta streaming documentation.
Use this original failure matrix when answering a recovery question:
| Crash window | Risk on restart | Required design property |
|---|---|---|
| Before sink commit | Missing output if progress advanced prematurely | Advance progress only after durable acknowledgment |
| After sink commit, before query completion is recorded | Repeated write | Replay must have the same effect as one committed write |
| After some external writes succeed | Partial output plus repeats | Atomic sink transaction or individually deduplicated operations |
Ask what “sink succeeded” means. Receiving an HTTP response is insufficient if it only acknowledges a queued operation, unless the pipeline’s contract deliberately treats that queue as its durable destination. Similarly, two successful table writes do not establish one atomic business transaction across both tables.
The strongest answer identifies the exact crash window and the evidence available afterward. “Restart the job” gives the operator no way to distinguish complete output from a partially applied batch.
Explain idempotence without overclaiming exactly once
Spark documents foreachBatch as providing at-least-once writes by default; application logic can use the batch identifier to deduplicate output. Official Spark foreachBatch semantics.
For supported Delta DataFrameWriter writes, txnAppId and txnVersion identify duplicate transactions. Using a stable application identity and the batch identifier is a documented pattern. With a new checkpoint, batch numbering restarts at zero, so use a new application identity to avoid treating valid new writes as old transactions. Official Databricks Delta streaming sink documentation.
That mechanism does not automatically protect a custom API call or arbitrary MERGE implementation. Nor does it make a batch identifier equivalent to a source commit version. Keep the query epoch, batch identity, and source coverage distinct in your recovery ledger.
For the current-state example, design analysis suggests storing the latest applied source version beside each key and applying only a newer replacement. Preserve equivalent version evidence for deletes: otherwise a delayed older update can resurrect a removed key. This is a proposed sink contract, not a built-in CDF guarantee.
If the input contains multiple candidate replacements for one key within the same version, a version-only comparison cannot choose between them. Resolve that with an explicit application contract or reject the ambiguity; arbitrary row order is not a correctness rule. For an audit log, establish a stable event identity or deduplicate an atomically committed batch instead of assuming the metadata columns uniquely identify every record.
Treat retention and schema boundaries as recovery limits
CDF is transient; a version removed from retained history is no longer readable. A new stream configured to start at an unavailable historical version fails to start. Official Databricks CDF recovery guidance.
Consider an original incident: the consumer committed through version 61, the source is at 100, and the earliest readable change version is 70. The missing interval is 62–69. Starting at 70 hides that gap; it does not repair it.
A current-state replica might be rebuilt from a fresh consistent snapshot, reconciled, and restarted from the next version. An audit system that promises every transition needs another retained source or archive for the missing interval. The latest snapshot cannot reveal overwritten intermediate values. Make the loss explicit before choosing a recovery strategy.
Delta history and underlying data files have separate retention requirements; time travel needs both. Do not promise a replay window based solely on how long log history is configured to remain. Official Databricks table-history documentation.
Schema changes create another boundary. CDF reads spanning a non-additive schema change can fail; supported behavior varies with engine, version, and column mapping. Split reads around the change and verify the runtime’s schema rules rather than asserting every historical read uses today’s schema. Official Databricks CDF schema limitations.
A useful operational answer monitors committed progress, oldest available replay position, lag growth, schema changes, and reconciliation results. Translate retention into an outage budget and alert while recovery is still possible. A healthy process indicator alone does not establish that the destination represents the source.
Prove recovery with a small reconciliation exercise
For this original practice case, replay versions 41–43 twice against a copy of the version-40 destination. After each run, assert the exact key set is {A, C}, the values are 15 and 7, and the sum is 22. Row count alone would miss a wrong amount; a sum alone could hide offsetting mistakes. Compare key-level values as well as aggregates.
Next, interrupt after A changes but before B is removed. Your restart must finish the remaining operations while leaving A correct. Finally, simulate the missing 62–69 interval and require an explicit recovery decision instead of a successful-looking cursor advance. These checks exercise the proposed application contract; they are not a substitute for integration tests on the actual Delta runtime, checkpoint storage, and destination. In an interview, explain which failure each assertion catches and what evidence would distinguish source loss from sink duplication.