Articles

AWS Glue Interview Questions: Job Bookmarks, Incremental Loads, and Safe Reruns

Practice AWS Glue interview questions with bookmark state, S3 and JDBC reads, failure scenarios, reset behavior, and output checks for safe incremental reruns.

An AWS Glue job bookmark can help avoid reading previously processed input, but it does not make every destination write safe to repeat. In an interview, explain input selection, bookmark state, and output correctness separately. The most revealing follow-up is often what happens after a write succeeds but before the job records its progress.

This guide uses official AWS documentation and an original three-run exercise. It is not a list of reported Amazon interview questions. The SQL checks were run locally against a small SQLite fixture; no AWS job, bookmark service, or S3 recovery was executed. For a related coding exercise, start with designing an idempotent SQL ETL for late data.

What Does a Glue Job Bookmark Track?

Official facts: Job bookmarks persist processing state for supported Glue operations. Support includes JDBC sources and specified S3 formats; AWS lists JSON, CSV, Avro, XML, Parquet, and ORC for Glue 1.0 and later. S3 selection uses object last-modified times. Resetting or rewinding a bookmark does not remove target files. AWS bookmark overview

That last boundary should shape your answer. A checkpoint can say which source input the next run should consider without describing everything already visible to downstream readers. A job may also have more than one destination, making a single “processed” label too vague to establish correctness.

Ask what the interviewer means by incremental. Are new objects arriving? Are existing objects replaced? Are new database rows appended? Can old business records change? Each pattern requires a different explanation of what counts as new input and how the destination interprets it.

Do not say that enabling a job-level option automatically adds bookmark semantics to arbitrary spark.read calls. Identify the supported Glue reader, its configuration, and the relevant state identity. If those details are missing from a prompt, state the assumption before predicting the next run’s input.

How Do S3 and JDBC Incremental Reads Differ?

Consider an original S3 example: yesterday’s object contains two events, and today the producer rewrites that same object with a correction. An event-date filter and an object-modification filter answer different questions. A file associated with an old business date can still need attention because the source object changed.

Your output policy must then decide whether the reread records replace older values, represent new events, or require reconciliation. A filename alone is usually not enough to answer that business question. Preserve enough source identity to explain where a result came from without treating that identity as a substitute for the event key.

Official JDBC rules: Glue can use selected columns as bookmark keys. User-defined keys must each be strictly increasing or decreasing; gaps are allowed. AWS warns against reusing per-record revision numbers as compound-key fields. Keep the source and its transformation_ctx consistent because that identifier connects the operator to stored state. Using job bookmarks

An original counterexample makes the limitation concrete. Suppose the job has read customer IDs through 500, and a customer with ID 120 changes their address. A design that only seeks larger IDs does not describe a way to capture that update. The problem is the chosen change-detection contract, not a mysterious failure of deduplication.

A strong answer asks whether the source supplies a suitable change sequence or another reliable change feed. It also asks how deletes appear. Neither the absence of an ID from a new batch nor a high maximum ID proves that the destination represents the latest complete source state.

Why Do init, commit, and transformation_ctx Matter?

Official setup: AWS instructs script authors to call job.init near the beginning and job.commit at the end, enable bookmark processing, and provide the relevant transformation_ctx. The context is a state identifier, not merely a descriptive variable name. Bookmark programming guidance

Treat changes to that identity as a state migration. Renaming a reader while refactoring can change how prior progress is found. Repointing an existing context at a different source can be equally dangerous. Before releasing such a change, document the intended starting point and which destination data already corresponds to it.

During an interview, sketch the lifecycle in plain terms: initialize state, read supported input, transform, write and validate the required outputs, then record progress. The sketch is a reasoning aid, not proof of a distributed transaction spanning the destination and bookmark store.

Official diagnostic guidance: AWS’s troubleshooting documentation identifies missing commit calls and missing transformation contexts among causes of unexpected reprocessing. Check those alongside the effective job arguments and source changes before reaching for a reset. Glue troubleshooting

The order of diagnosis matters. Resetting first can destroy a useful reference point and enlarge the amount of input eligible for replay. Capture the job identity, relevant run IDs, reader settings, and observed output state while the failure is still understandable.

Trace Three Runs Through a Failure Window

Use this original practice model. Events are immutable, each has a unique event_id, and amounts are 10, 30, 20, and 10 for IDs 1 through 4. The naive destination appends every row. Assume a failed run leaves its written rows visible but does not advance the saved input checkpoint.

Run and source input	Saved progress and destination result
Run 1 reads events 1 and 2.	Write and commit succeed; progress covers 1–2; destination has 2 rows.
Run 2 reads new events 3 and 4.	Write succeeds, then failure occurs before progress commit; destination has 4 rows.
Run 3 retries events 3 and 4.	Append and commit succeed; progress covers 1–4; destination has 6 rows.

The third run can be successful while the destination is wrong. It contains four distinct events but six physical rows. The naive amount total is 100, although the four unique events total 70. A local SQLite check reproduced these counts; the assumed failure window remains a teaching scenario, not an observed Glue incident.

The first interview question is not “How do I stop the retry?” It is “Which effects survived the failure?” If no output survived, replay may be harmless. If some output survived, the destination needs a strategy that handles repeated input. If a consumer already published an aggregate, repairing raw rows alone may leave that aggregate incorrect.

Also separate attempted rows from accepted business events. Logging “two rows processed” on Run 3 may be accurate as a workload count, yet misleading as evidence of two new events. Monitoring should make those meanings distinguishable.

Pause, Reset, and Rewind Are Different Operations

Official modes: job-bookmark-enable tracks progress, while disable processes the eligible dataset without bookmark filtering. Pause processes incremental input without advancing bookmark state. With pause’s optional run-range settings, both job-bookmark-from and job-bookmark-to must be supplied; the range excludes the input represented by the from boundary and includes the to boundary. AWS job parameters

Official reset interface: ResetJobBookmark requires a job name and accepts an optional run ID. Use the documented reset/rewind mechanism for the intended starting state; do not treat it as a destination rollback. ResetJobBookmark API

For the exercise, resetting input progress while leaving the six appended rows untouched does not repair the total. It only changes what a future run may read. Likewise, pause can be useful for a bounded comparison, but directing its output into the same append-only production target can still add repeats.

Before a backfill, write down the exact source scope, output location, validation rules, and publication decision. If the source has changed since the historical run, a run boundary alone should not be mistaken for a frozen copy of the old data. Reproducibility depends on retaining the required source versions or a trustworthy snapshot.

Repair Option One: Isolated Outputs and Explicit Publication

One editorial design option is to write a replay into an isolated batch or version location. Validate that complete candidate output before making it authoritative. A failed attempt remains separate from the currently published dataset, so a retry need not append another copy into the same visible result.

The key is the publication mechanism. A folder called “complete” does not make a collection of object writes atomic. Define how readers select one validated version, how that selection changes, and what happens if the publisher stops immediately afterward. The storage and catalog tools you use must support the required guarantees.

In the four-event exercise, a full rebuild candidate should contain four rows totaling 70. The previous published version remains available until the new candidate passes checks. If the repair covers only a partition, validate that the candidate contains the complete replacement scope, not merely the latest incremental fragment.

This option is useful when replacement boundaries are clear and readers can follow a versioned publication contract. It is less attractive when consumers independently discover every new file or when a “partition” is not a complete unit of business correctness. Explain those conditions instead of promising that a new prefix solves every retry problem.

Repair Option Two: Enforce Event Identity at the Destination

A second editorial option is a destination that accepts each immutable event key once under a suitable transaction or uniqueness mechanism. In the local exercise, a table with a primary key on event_id and duplicate-key handling retained four rows totaling 70 after repeated input.

That demonstrates the logical condition, not a universal sink implementation. Plain appended files do not acquire a database uniqueness constraint because the ETL script uses a set. Concurrent writers, durable commit behavior, and key scope still need a concrete design.

For immutable events, two different payloads under one event ID should trigger a conflict policy rather than silently discarding whichever arrived second. For mutable entity snapshots, event-key deduplication may be the wrong model: define a business key, version ordering, update rule, and delete semantics instead.

In an interview, compare the options through the failure you are fixing. Versioned publication isolates an entire candidate result; key enforcement protects a specified identity inside a destination. Either can fail if its assumptions do not match the data model. Choose the simpler design that satisfies the actual consumer contract.

What Queries Prove the Repair Worked?

For the original immutable-event fixture, begin with an identity check:

SELECT event_id, COUNT(*) AS copies
FROM events
GROUP BY event_id
HAVING COUNT(*) > 1;

The naive output returns IDs 3 and 4, each with two copies. A corrected event table should return no rows. Then check the population and amount together:

SELECT COUNT(*) AS rows_seen,
       COUNT(DISTINCT event_id) AS unique_events,
       SUM(amount) AS total_amount
FROM events;

For the corrected fixture, expect 4 rows, 4 unique events, and total amount 70. These queries were checked locally. Adapt table names and numeric types to the actual query engine; do not present this fixture as a benchmark of Glue.

Counts alone cannot prove completeness. A missing event and an unrelated extra event can cancel out numerically. Reconcile expected IDs or source manifests, inspect rejected records, and compare the affected business totals. When amounts can change, validate the selected version as well as the key.

Finish the repair narrative by explaining what happens on the next normal scheduled run. A recovery that produces one correct snapshot but leaves progress pointing at the wrong source boundary is unfinished. Preserve both the output verification and the state decision in the run record.

Rehearse the Failure, Not Just the Definition

Practice question	Apply it to the bookmark exercise
Design an idempotent SQL ETL for late data	Distinguish late changes from repeated input.
Design idempotent daily loads with deduping	Define the replacement scope or uniqueness contract.
Data Pipeline Reliability, Backfills, and Spark Optimization	Explain the recovery boundary and next scheduled run.
Reason About Duplicate Data and Scaling in Spark	Separate source repeats from processing duplicates.
Implement an Idempotent Versioned Database Update	Handle mutable records with explicit ordering.

Practice daily loads with deduping by moving the failure point: before the write, after the write, and after publication. For each version, state what survived, what will be replayed, and which check establishes a correct result.