Articles
Debezium Snapshot Interview Questions: CDC Cutover, Offsets, and Duplicate Recovery
Practice Debezium snapshot interview questions with a CDC cutover timeline, three offset boundaries, duplicate recovery, and a version-aware sink exercise.
Debezium snapshot interview questions get interesting when the connector restarts successfully but the destination is wrong. A snapshot can finish, Kafka can accept records, and a consumer can advance its checkpoint while your application still double-counts an update or resurrects a deleted row.
A strong answer traces source continuity and destination correctness separately. Start with the CDC interview guide for general capture methods. Here, work through a PostgreSQL snapshot, its transition into streaming, and the failure boundaries behind duplicate recovery.
Evidence boundary: official documentation supports the labeled product behavior below. The timelines, proposed sink design, and interview prompts are original exercises and engineering inferences. We did not verify firsthand candidate reports establishing these exact questions or their frequency. Sources were checked September 21, 2026; Debezium’s stable documentation identified version 3.6. No live Debezium deployment was tested for this article.
1. Where does the snapshot hand off to CDC?
Official behavior: the PostgreSQL connector documentation describes a consistent initial snapshot followed by streaming from the corresponding WAL position. An interrupted initial snapshot starts again; a streaming restart uses the recorded source offset and can reproduce changes after a crash. Snapshot reads use operation r; updates and deletes use u and d.
Original exercise: assume public.accounts contains account 42 with balance=100. Call the snapshot boundary S. While snapshot delivery is underway, the application commits an update to 120 at a later source position U. Both records must eventually be interpreted correctly.
| Stage | Evidence available | Required destination interpretation |
|---|---|---|
| Snapshot boundary S | Consistent row state: account 42, balance 100 | Establish the baseline |
| Concurrent source update U | Account 42 changes to balance 120 | Preserve this subsequent change |
| Snapshot delivery | A read record carries balance 100 | Do not count it as a new payment |
| Streaming catches up | An update record carries balance 120 | Advance the projection to 120 |
S and U are symbolic coordinates, not sample LSN strings. This is the initial-snapshot case; an incremental snapshot has a different collision-handling mechanism, discussed below.
An incomplete design starts a table scan and begins CDC only when the scan finishes. Challenge it with a row updated halfway through the scan. Where is that update retained? A stronger answer names the shared consistency boundary, the retained log, and the destination’s application order. Saying “take a snapshot, then stream” leaves those obligations unexplained.
2. Which of the three offsets do you mean?
Official distinction: Kafka Connect’s development guide separates source stream identifiers and source offsets from the Kafka topic, partition, and offset supplied to sink tasks. Its sink flush contract requires outstanding destination writes to be acknowledged before a safe checkpoint proceeds.
Use the following map when an interviewer says, “The offset was committed.”
| Coordinate | Question it answers | What it does not establish |
|---|---|---|
| Connector source offset | Where should the source reader resume? | Whether a particular sink applied the change |
| Kafka record offset | Where is this delivery in this topic partition? | Whether another delivery represents the same source change |
| Consumer checkpoint | Where will this consumer resume reading Kafka? | Whether business effects and the checkpoint were committed atomically |
Exercise inference: a record at partition 0, offset 501 can represent the same source change as another record later appended at offset 540. Deduplicating on (topic, partition, offset) catches redelivery of record 501. It does not recognize 540 as the same logical change.
The opposite mistake is deduplicating on account ID alone. Account 42 can have many legitimate updates. Define event identity separately from entity identity, and inspect the connector’s actual metadata before proposing a source-event key. Neither a timestamp nor an LSN alone is a universal unique event identifier.
3. What happens when the connector crashes after publishing?
Assume an at-least-once source deployment for this exercise. The connector has durably recorded source position S, publishes update U, then crashes before its source checkpoint advances. The destination is a current-state account projection, not an append-only payment ledger.
| Step | Kafka delivery | Destination after processing |
|---|---|---|
| Snapshot read | Offset 500: account 42, balance 100 | 100 |
| Update U | Offset 501: account 42, balance 120 | 120 |
| Connector crash | Durable source checkpoint remains S | Still 120 |
| U published again after restart | Offset 540: account 42, balance 120 | Must remain 120 |
An unconditional balance += 20 produces 140 on the second delivery. Replacing the stored balance with the supplied after-state produces 120. That solves this adjacent duplicate, but it does not prove that an older state arriving after a newer one is safe.
Now change the failure location: the sink writes 120 and crashes before committing its consumer checkpoint. This time it may read offset 501 again. The visible symptom resembles source replay, but the duplicated Kafka coordinate tells you which boundary to investigate first.
A useful spoken answer is: “I would compare source metadata and Kafka coordinates across both deliveries, then inspect the checkpoint that lagged. I would preserve the replay evidence before changing offsets.” Resetting every checkpoint first destroys information needed to distinguish these failures.
Before restarting an interrupted snapshot, inspect whether its completion was recorded. Do not infer completion from a few rows reaching Kafka. For this exercise, keep the source-offset checkpoint distinct from the snapshot rows already applied downstream: restarting capture does not roll back those destination writes. Your recovery explanation must cover both pieces of state.
4. Why is a primary-key upsert not the whole recovery plan?
Official behavior: the Debezium JDBC sink documents idempotent writes with insert.mode=upsert. Source deletes require explicit delete configuration and suitable primary-key mapping. These settings describe that connector; they are not properties of every custom consumer.
Engineering inference: upsert alone does not reject stale state. Suppose a later update changes the balance to 150, then a delayed replay overwrites it with 120. The primary key remains unique while the value becomes wrong. Deleting the row creates another problem: a stale snapshot read can insert it again.
For a concrete design exercise, add an explicit application contract: every mutation carries a strictly increasing revision per account, including a deletion event. Re-creating the same account ID must continue that sequence. A transactional outbox could carry deletion revisions; an ordinary physical delete does not magically provide a new application revision.
Our simplified sink receives these normalized inputs:
| Input | Revision | Desired result |
|---|---|---|
| Snapshot baseline | 7 | Store balance 100 |
| Update, then its duplicate | 8 | Store balance 120 once |
| Delete | 9 | Retain deleted state at revision 9 |
| Delayed update or snapshot | 8 or 7 | Reject as stale; keep deletion |
Use this rule for the proposed sink:
In one transaction, serialize updates for this entity:
incoming revision < stored revision: ignore stale input
incoming revision = stored revision:
same state: ignore replay
different state: reject a contract violation
incoming revision > stored revision:
persist revision and state, including deleted=true
Commit; only then acknowledge the delivery.
We checked ten cases with a local SQLite model, including a failure before commit and conflicting payloads at the same revision. This validates the exercise logic, not a Debezium configuration or a production sink. Real implementations need atomic comparison and mutation under concurrent workers.
Retaining the deleted revision is essential. If you erase that evidence, revision 7 may look like a new row. Define retention against the oldest replay or rebuild you permit. Also distinguish this sink deletion marker from a Kafka null-value tombstone used for log compaction.
If the source cannot supply such revisions, say so before proposing this algorithm. You may instead preserve ordered application for each key and coordinate snapshot replays with that stream, or build a connector-specific ordering scheme whose behavior you have verified. A wall-clock timestamp is a poor substitute when two changes can share a timestamp or clocks can move backward. During a rebuild, also decide whether the new target starts empty or inherits existing revision state; silently mixing two incompatible revision sequences invalidates the comparison.
This design protects a latest-state projection. Counting every business transition requires durable event identity and an atomic deduplication record alongside the effect. Skipping an older revision may be correct for the projection and wrong for an audit ledger.
5. What changes with an incremental snapshot?
Official design: Debezium’s incremental snapshot explanation describes primary-key chunks, a snapshot window, and buffering. When a streamed change collides with a buffered snapshot row’s key during that window, the buffered read is removed. Streaming continues alongside snapshot work, and progress can resume from completed chunks. The 2021 article explains the mechanism; consult your installed connector’s current documentation for configuration.
Practice prompt: a chunk buffers account 42 at balance 100. Before the window closes, the stream delivers its update to 120. Explain why emitting the buffered 100 afterward would regress the destination, then identify which component suppresses that conflicting read.
Keep two protections separate. The connector’s window reconciles snapshot reads with concurrent source changes. Your sink’s recovery strategy handles downstream retries, external backfills, and any reordering introduced after the connector. One mechanism cannot substitute for the other.
Avoid promising that an incremental snapshot is a single database-wide historical view. For an interview, ask whether the requirement is rebuilding current state or preserving cross-table history at one instant. Those are different acceptance criteria.
6. Can you recover by deleting offsets and snapshotting again?
Official PostgreSQL context: logical decoding documentation explains that replication slots retain resources needed by consumers, including WAL. Slots can consume substantial storage when consumers fall behind. PostgreSQL also documents that a crash can cause recent changes to be sent again.
Operational reasoning: preserve the connector configuration, recorded source position, slot state, and destination checkpoints before choosing a repair. Then classify the failure:
- Source history is still available: determine whether normal resume can close the gap. Measure backlog and sink progress separately.
- Required history is unavailable: a fresh snapshot can rebuild current rows, but cannot reconstruct an update followed by a delete that left no current row.
- The destination is corrupted: rebuild a separate target, catch it up, and reconcile it before switching readers. Replaying into the damaged table may preserve old mistakes.
For the separate-target approach, choose a comparison boundary. Compare source and destination key sets, selected values, and deletions at that boundary; a row-count match alone can hide swapped or stale records. Test one controlled update and delete through the complete path. Decide how writes arriving during reconciliation reach the replacement before switching traffic.
Official limit: Kafka Connect’s exactly-once support depends on connector capabilities and configuration; source support requires distributed workers. A Kafka setting alone does not prove that an external destination or business side effect has the same guarantee. Name the boundary covered by any exactly-once claim.
Debezium snapshot interview questions to practice
These related questions exercise the reasoning above.
| Practice question | Follow-up to rehearse |
|---|---|
| Design relational-to-NoSQL migration pipeline | Mark the snapshot boundary and prove the handoff preserves concurrent updates. |
| Reason About Indexes, Kafka, and Database Locking | Place the durable database write relative to consumer checkpointing. |
| Explain ETL schema changes and ensure integrity | Define reconciliation checks before switching to a rebuilt target. |
| Design Real-Time Analytics Pipeline with Kafka and Flink | Separate source replay, processing recovery, and final sink effects. |
| Prevent Duplicate Request Processing | Make duplicate detection atomic with the effect it protects. |
For these Debezium snapshot interview questions, start with the migration exercise, then move the crash from source publication to sink commit. For another rehearsal, use Data Engineer interview questions and defend the exact state that survives each failure.
Sources and Further Reading
- Debezium PostgreSQL connector: snapshot, streaming, and restart behavior.
- Debezium incremental snapshots: window-based reconciliation design.
- Debezium JDBC sink: upsert and delete behavior.
- Kafka Connect development guide: source offsets and sink checkpoint boundaries.
- Kafka Connect user guide: exactly-once prerequisites.
- PostgreSQL logical decoding concepts: slots, retained resources, and replay.