Articles
Apache Hudi Interview Questions: Late Updates, Query Results, and Compaction
Practice Apache Hudi interview questions with tested late-update results, explicit merge modes, snapshot versus read-optimized queries, and compaction evidence.
Apache Hudi interview questions about late updates need more than a definition of Merge-on-Read. Before predicting a query result, identify the record key, ordering field, merge mode, query type, and completed table services. The event that arrives last is not necessarily the newest business event, and two valid queries can temporarily return different states.
This guide uses an original six-event experiment on Hudi 1.2.0 with Spark 4.0.1 and Java 21. Official documentation defines the behavior; the tables below contain our local results. The questions are editorial practice, not a list of verified employer interview rounds.
Which Record Wins When Updates Arrive Late?
Consider one order, keyed by order_id = A, in a fixed orders partition. The source supplies a monotonically increasing business sequence when it creates each update, but delivery to the lake can be delayed. The seq field below represents that source ordering; it is not the arrival counter or a Hudi commit timestamp.
Each row was written in a separate, sequential write. The first write used bulk_insert to establish a base file. The remaining writes used upsert. All rows were complete records, with no deletes, partial updates, or equal ordering values.
| Arrival | Source sequence | Status |
|---|---|---|
| 1 | 10 | created |
| 2 | 30 | paid |
| 3 | 20 | pending |
| 4 | 40 | shipped |
| 5 | 50 | delivered |
| 6 | 45 | in_transit |
Predict the final state before choosing a mode. If you want the highest source sequence to win, the final row should remain delivered. If your contract is that each later committed write replaces the previous record, the final row is in_transit.
Hudi's record-merger documentation distinguishes these policies through EVENT_TIME_ORDERING and COMMIT_TIME_ORDERING. We created two separate tables and explicitly configured the mode for each. We did not change one table's merge mode halfway through the experiment or rely on an inferred default.
That distinction matters in an interview: asking which mode is “better” without describing the source contract is incomplete. An authoritative, correctly ordered change stream can have different requirements from independently delayed events. A wall-clock timestamp is also not automatically a trustworthy ordering key if producers disagree about time.
Read the Results One Commit at a Time
Both tables used the same input rows and the same source-ordering field. Their snapshot queries returned:
| After arrival | Event-time mode | Commit-time mode |
|---|---|---|
| 1 | 10 / created | 10 / created |
| 2 | 30 / paid | 30 / paid |
| 3 | 30 / paid | 20 / pending |
| 4 | 40 / shipped | 40 / shipped |
| 5 | 50 / delivered | 50 / delivered |
| 6 | 50 / delivered | 45 / in_transit |
The third and sixth writes are the useful counterexamples. A job can complete successfully while preserving an older business state under the wrong policy for your application. Looking only for a failed Spark stage would miss that semantic problem.
Here are the central writer options from the executed lab:
options = {
"hoodie.datasource.write.table.type": "MERGE_ON_READ",
"hoodie.datasource.write.recordkey.field": "order_id",
"hoodie.datasource.write.partitionpath.field": "part",
"hoodie.table.ordering.fields": "seq",
"hoodie.write.record.merge.mode": "EVENT_TIME_ORDERING",
"hoodie.compact.inline": "false",
"hoodie.compact.schedule.inline": "false",
}
The second table used COMMIT_TIME_ORDERING in the same option. Each table had its own name and fresh path. The write options also disabled automatic cleaning and metadata-table use to simplify this tiny dataset. The separate compaction procedure subsequently created metadata files. These are lab choices, not a production configuration recommendation.
If the interviewer changes the input to two equal seq values, the displayed result table no longer answers that question. Ask for the tie-breaking contract and verify the relevant merger behavior. Likewise, replacing full rows with partial updates changes the problem. Do not extend a six-row experiment into guarantees it did not test.
Why Do Snapshot and Read-Optimized Queries Disagree?
Hudi's table and query documentation explains that a Merge-on-Read table can combine base files with changes in log files. A snapshot query merges the relevant data to present current table state. A read-optimized query reads base-file data without applying those newer log changes.
In our lab, read-optimized queries returned 10 / created after every write before compaction. Snapshot queries reflected the mode-specific results above. Both outcomes were consistent with the chosen query type and file layout.
The query difference was explicit:
snapshot = spark.read.format("hudi").option(
"hoodie.datasource.query.type", "snapshot"
).load(path)
read_optimized = spark.read.format("hudi").option(
"hoodie.datasource.query.type", "read_optimized"
).load(path)
A useful interview answer therefore asks how the downstream report reads the table. “The dashboard is stale” could mean that it uses read-optimized queries while updates remain in logs. It could also mean an upstream write never completed or the query points to another path. The symptom alone does not prove compaction is broken.
Do not describe read-optimized data as universally a fixed number of minutes behind. Its freshness depends on the relevant base files and completed services. Our experiment controlled those conditions; a production table can have different file groups at different stages.
The comparison also does not establish a performance benchmark. We measured returned records, not latency under load. Choosing a query type requires the application's freshness and resource constraints, not a blanket promise that one is always faster.
What Changes When Compaction Completes?
The official compaction guide describes merging log changes with base data. Scheduling identifies work; completion makes the resulting table state available according to the timeline. A requested action is not evidence that its new base files are ready for readers.
We invoked the documented Spark SQL procedure after the sixth write:
CALL run_compaction(op => 'run', path => '/absolute/lab/table');
The command shown uses a placeholder path; the executed harness supplied each actual local table path. The procedure returned COMPLETED for both tables, with one operation in each. We then queried both views again.
The numbers below are source sequences: 50 means delivered, 45 means in_transit, and 10 means created.
| Mode and stage | Snapshot | Read-optimized |
|---|---|---|
| Event-time, before | 50 | 10 |
| Event-time, after | 50 | 50 |
| Commit-time, before | 45 | 10 |
| Commit-time, after | 45 | 45 |
Compaction did not turn the commit-ordered table into the event-ordered table. It applied the table's merge policy while producing new base data. If the policy is wrong for the business, running the service again is not a policy repair.
Our evidence included the files, not only the query output. Before compaction, each table contained one initial Parquet base file and five data log files for the updated file group. After completion, a new Parquet base file existed. The old base and logs also remained on disk because automatic cleaning was disabled.
That last observation prevents another mistaken conclusion: seeing an old file in a directory does not mean the reader should include it in the current view. Hudi uses table metadata and file-slice semantics. Reading every Parquet file as an unrelated directory dataset bypasses that interpretation and can produce a misleading comparison.
How Do Compaction, Clustering, and Cleaning Differ?
These services address different operational questions. Explain the problem first, then name the service that addresses it.
| Service | Main question | What it does not prove |
|---|---|---|
| Compaction | How do accumulated log changes become base data? | That an incorrect merge policy is corrected |
| Clustering | How should data be reorganized into a more useful file layout? | That every workload will become faster |
| Cleaning | Which older file versions can be removed under the retention policy? | That arbitrary historical reads remain available forever |
Hudi's clustering documentation describes reorganizing data layout, including file sizing and sorting strategies. Our two tables did not run clustering. A small-file investigation would need actual file sizes, scan patterns, and a chosen strategy before recommending it.
The cleaning documentation ties removal of older versions to retention. Cleaning is not synonymous with finishing compaction. A long-running reader or an incremental consumer with a long delay can need retained history, so aggressive retention changes require understanding those consumers.
The local decision to keep old files made the evidence easy to inspect. It would be a poor excuse to leave a production table's storage growth unmanaged. Conversely, manually deleting old-looking files is not a substitute for using the table's supported maintenance behavior.
Diagnose a Compaction or Query Incident with Evidence
Suppose a report still shows created after an operator says compaction ran. Our recommended investigation has three distinct checks. First, reproduce the exact reader path and query type. Compare the affected key through Hudi's snapshot and read-optimized readers rather than through a generic Parquet scan.
Second, inspect the timeline for the relevant action and its state. Was it requested, inflight, or completed? Did the completed operation cover this file group? A log message that scheduling started does not answer either question.
Third, compare the file evidence and the record's ordering values. If the base files are current but the snapshot itself selects the unexpected row, revisit the source sequence, key, and merge policy. If snapshot is correct while read-optimized remains older, examine the service coverage and subsequent writes.
Save the concrete record, configured mode, query options, action instant, and relevant files before changing settings. A repeatable one-key example often communicates the issue better than a large screenshot of the job dashboard.
Our experiment verified normal completed compaction. It did not inject a driver crash, test concurrent writers, or validate a recovery runbook. For an actual failed action, follow the installed version's supported recovery procedure and preserve evidence; do not promise that rerunning an arbitrary command is always safe.
Does an Incremental Query Return Every Change Event?
A latest-state incremental result and a change-data-capture event stream answer different questions. The former can identify records changed over a range without delivering every intermediate transition as a separate business event. If you need to count each order transition, establish the query's semantics and CDC configuration explicitly.
In our example, the final snapshot has one row. It cannot tell you by itself that six writes were attempted, that a delayed pending update arrived, or how many times the order changed state. Those facts came from the input ledger and per-write observations.
This is an important preparation distinction: maintaining current state and preserving an audit event history are separate requirements. We did not enable or test Hudi CDC here, and the six-event ledger should not be presented as the output of an incremental query.
Practice the Underlying Data Decisions
These verified questions exercise adjacent skills; they are not advertised as a Hudi-specific question collection. Tie each answer back to the concrete record and reader contract.
| Practice question | What to explain |
|---|---|
| Implement an Idempotent Versioned Database Update | Why a later arrival should or should not replace a stored version |
| Design an idempotent SQL ETL for late data | How reruns interact with late records and business keys |
| Describe ETL and pipeline challenges | Which evidence distinguishes successful execution from correct data |
| Design batch and streaming ETL architecture | Where freshness and maintenance costs enter the design |
| Reconcile ledgers with SQL/Python and late events | How to compare source events with materialized state |
Return to the versioned-update exercise and change the final input from sequence 45 to 60. Predict both tables, then explain which query will reveal that change before compaction. A strong answer names the assumptions before giving the row.