Articles
Apache Impala Interview Questions: Stale Metadata, Table Statistics, and Query Memory
Prepare Apache Impala interview answers by separating stale metadata, optimizer statistics, per-node memory limits, and admission-control failures.
For Apache Impala interview questions, separate three problems before proposing a fix: whether Impala sees the right table and files, whether optimizer statistics describe the data accurately, and whether the chosen query fits its runtime and admission constraints. A refresh can fix visibility without fixing a poor estimate. Better statistics can improve planning without making every query fit memory.
This article uses official Apache documentation for command behavior and labels its worked cases as original interview practice. The main examples assume an existing HDFS-backed table registered in the Hive Metastore. Commands are source-reviewed examples, not results from an Impala cluster we operated. Kudu, Iceberg, and automated metadata-event handling require their own version and configuration checks.
A fast answer is useful only after you establish which rows the query should return.
What does stale metadata mean in this case?
Metadata describes the table, schema, partitions, and files Impala uses to plan and execute a query. Statistics summarize properties such as row counts and column distributions for the optimizer. A visible file and a current row-count estimate answer different questions.
Official fact: Apache documents REFRESH as reloading table metadata and incrementally reloading file and block metadata. The table must already exist and be known to Impala. For the HDFS-backed case, externally added or changed files are a reason to check whether a targeted refresh is needed.
First ask who made the change and where. A file copy into a table directory, a table created through another engine, an Impala-issued insert, and an HDFS block-location change are different events. Also ask whether metastore event processing is configured to handle the relevant change automatically. Name the writer, table type, and automatic handling before recommending a command.
Avoid treating every surprising count as metadata staleness. The SQL might filter the wrong partition, a join might duplicate rows, or the load might be incomplete. Inspect the intended data and the query predicate before clearing caches. A visibility hypothesis needs a comparison between the files or partitions that should exist and what the engine actually uses.
REFRESH or INVALIDATE METADATA?
Official fact: INVALIDATE METADATA marks cached metadata stale, with loading triggered when a subsequent query needs it. Apache distinguishes this more expensive full reload from the incremental work of REFRESH.
The following original decision map assumes the change has not already been handled automatically. It gives a starting action to verify, not a universal repair script.
| Observation about the external change | Starting action to consider | Evidence needed afterward |
|---|---|---|
| Files added to an existing known HDFS table | Targeted REFRESH db.table | File visibility and result reconciliation. |
| A new metastore table is not known to Impala | Targeted INVALIDATE METADATA db.table | The table becomes discoverable with the expected schema. |
| Cached structural metadata or block locations are stale | Review targeted invalidation for that change | Reloaded structure/locality and the next query's behavior. |
| Results are correct but the plan uses poor estimates | Inspect and update the appropriate statistics | Estimates, join plan, and actual execution evidence. |
A schema or table-discovery change requires more care than appending a known data file. Read the installed version's documentation and establish the scope of the stale object. Do not assume that partition-scoped syntax handles every structural change.
A global invalidation needs a reason that also justifies reloading unrelated tables. It can impose broad reload work and affect unrelated queries. Explain why the smallest justified scope is sufficient, and what would make you expand it.
The distinction is also about timing. REFRESH performs its reload synchronously; invalidation can defer loading until a later query. If the next query is slow after invalidation, consider metadata loading before claiming that the underlying SQL suddenly became slower.
Trace an external load through an exact result check
Consider this synthetic, original fixture. An unpartitioned table demo.sales already contains two files with rows (1, 10), (2, 20), and (3, 40), where the second value is an amount. An external writer adds a third file containing (4, 5) and (5, 25). The five IDs are distinct, all files use the same schema, and the query has no date filter.
The old visible result has three rows and an amount sum of 70. The complete intended result has five rows and a sum of 100. These values come from the fixture, not a production measurement or an optimizer prediction.
Ask which failure you are investigating. If Impala still sees only the original files, a visibility repair is relevant. If it sees all five rows but an estimated row count remains stale, the result is already correct and the remaining investigation concerns planning. If five source rows become seven after a join, investigate the join grain rather than blaming the file catalog.
For the stated manual-refresh case, the diagnostic commands might include:
REFRESH demo.sales;
SELECT COUNT(*) AS row_count, SUM(amount) AS amount_sum
FROM demo.sales;
SHOW TABLE STATS demo.sales;
SHOW COLUMN STATS demo.sales;
These are source-reviewed Impala commands for an illustrative table. The local fixture arithmetic is checked independently; no cluster command output is claimed. In a real system, preserve the query ID, table identity, load completion evidence, and relevant partition scope so that the observations refer to the same data version.
COUNT and SUM are useful checks but cannot prove full equality. Replacing (4, 5) with another row carrying amount 5 preserves COUNT and SUM while losing key 4. Compare key coverage or stronger reconciliation evidence when the load contract requires it. Here the expected IDs are exactly 1 through 5, so the fixture permits a direct key check as well.
Why statistics are a separate maintenance decision
Official fact: Apache's COMPUTE STATS documentation explains how table and column statistics support join planning and memory estimates. It distinguishes full statistics from incremental statistics for partitioned tables and warns against mixing the two maintenance approaches without the required transition steps.
Updating file visibility does not mean you have recomputed optimizer statistics. For the fixture, choose a statistics policy separately after confirming the load. A small unpartitioned teaching table may use COMPUTE STATS demo.sales; a large partitioned production table needs a policy that reflects its partition changes, existing statistics mode, and maintenance cost.
Explain what you expect to change. Statistics may alter estimated cardinalities, join order, or a broadcast-versus-partitioned choice. They do not add a missing physical file to the query result or correct a logically wrong join. Keep the correctness check independent of a performance hypothesis.
A useful before-and-after comparison holds the data version, SQL, and relevant options constant. Save EXPLAIN before maintenance, then inspect EXPLAIN and actual query evidence afterward. If the plan changes, name the change. If it does not, do not report an improvement just because the statistics command completed.
Estimates can remain imperfect with current statistics, especially when a workload's skew or correlated predicates are not adequately represented. Accurate statistics are valuable input, not a guarantee that the optimizer's resource estimate equals the largest runtime allocation.
What does MEM_LIMIT constrain?
Official fact: Apache describes the MEM_LIMIT query option as a per-query limit on each node. It distinguishes that limit from the impalad process memory limit shared by concurrent work. A statement about cluster totals must not obscure the constraint on an individual host.
For an original memory trace, suppose the effective query limit is 2 GiB per participating host, while observed peaks are 1.1, 1.4, and 3.2 GiB. The hottest host exceeds the assumed limit even though the average is 1.9 GiB. This is arithmetic illustrating the scope, not an actual Impala profile or a promise about precisely when an operator fails.
Inspect per-host and per-operator evidence. A skewed key can concentrate work on one host. A replicated build side can consume memory on several hosts. Wide intermediate rows or an unexpectedly large aggregation state can make a selective-looking query expensive. The final result's size is not a reliable measure of the memory needed to produce it.
Spilling is another boundary to explain carefully. Some work can reduce memory consumption through disk spill, but spill support, scratch space, minimum reservations, and configuration matter. Apache also documents failure examples when memory is insufficient. “It will spill, so it cannot fail” is not a safe general conclusion.
Before raising the limit, explain the hottest allocation and measure the headroom available under concurrent load. Explain what the larger allowance would compete with, and whether reducing intermediates, correcting statistics, or changing the query plan addresses the cause more directly.
Separate admission failure from execution failure
Official fact: Admission control manages concurrent work and resource-pool constraints. Queries can wait in a queue or fail under configured admission conditions before doing the work you intended to inspect.
The following original evidence matrix keeps three symptoms distinct:
| Symptom | Evidence to collect | Next question |
|---|---|---|
| Query remains queued | Pool, queue duration, configured limits, competing work | Is this temporary contention or a pool policy mismatch? |
| Query is rejected before execution | Admission error and reservation/limit requirements | Can the pool admit the current plan under its effective limits? |
| Query starts and exceeds memory | Query profile, effective options, hottest host/operator | Which runtime allocation or skew crossed the boundary? |
Do not compare a queued query's wall time with another query's execution time as if they measure the same thing. Separate admission waiting from execution before attributing latency to a join or scan.
Likewise, a cluster-wide pool budget and a per-host query limit are not interchangeable. A query can be inadmissible because of aggregate pool pressure, or fail on one host despite a seemingly modest cluster average. The effective settings and observed error determine which explanation applies.
Answer the follow-up as a diagnosis, not a command list
For the file-load scenario, a concise answer begins with the intended five rows and sum 100, identifies that the writer acted outside Impala, and asks whether the catalog currently sees the third file. Under the stated manual-refresh assumptions, it proposes targeted REFRESH, checks the five expected IDs, and reviews statistics separately.
For a memory follow-up, begin with the effective per-host limit and whether the query was admitted. Trace one operator on the hottest host. If the interviewer changes the table type or says automatic event handling is enabled, revise the metadata hypothesis instead of insisting that every external write needs your manual command.
All five focus on row selection, aggregation, or time-window semantics; none simulates an Impala memory profile.
| Question | What to check before engine tuning |
|---|---|
| Query carpool ride metrics | Establish the unit counted before aggregation. |
| Query departments and top earners | Preserve grouping and tie semantics. |
| Query the Top Ten Customers by Recent Spend | Define the time window and qualifying spend. |
| Query conversion and retention with SQL windows | Separate cohort definitions from physical execution. |
| Query seven-day conversion with windows and dedupe | Check duplicate events before blaming a plan. |
Start with one Data Engineer SQL/Python exercise, write its result invariant, and then explain an Impala visibility or memory follow-up. Keep the sequence explicit: correct data, justified metadata action, statistics evidence, and the actual resource boundary.