Articles

Dataplex Interview Questions: Quality Scans, Sampling, and Failed-Row Evidence

Prepare for Dataplex interview questions on quality scan scope, incremental markers, sampling, pass thresholds, and failed-row evidence.

A Dataplex quality scan can pass while invalid rows remain in the table. A useful interview answer explains which records were eligible, which rules ran, what threshold was applied, and which evidence supports the result. Identify the evaluated rows before interpreting the green status.

Official naming note: Google Cloud's current documentation calls the product Knowledge Catalog, formerly Dataplex Universal Catalog. This article retains the Dataplex interview keyword and focuses on auto data quality scans on BigQuery tables; API identifiers still use Dataplex. Check the product generation before applying examples from older CloudDQ tasks. Current auto data quality documentation.

The cases below are original interview exercises, not candidate reports or claims about an employer's assessment.

What does a data quality scan actually prove?

A quality scan evaluates configured expectations over a selected data population; the worked examples here illustrate that contract without claiming a deployed-cloud test. An interview answer should distinguish the source table, the scan definition, a particular run, and the result you are discussing. A changed definition may not describe the configuration used by an earlier run.

Official API fact: DataQualityResult.passed is true when all evaluated rules pass. Its rule results provide more specific evidence. The result also records the processed row count and scanned data. These fields help identify the population behind a verdict. DataQualityResult reference.

That does not establish that every table row was checked, that every business requirement has a rule, or that downstream totals reconcile. A perfectly executed scan can test the wrong slice. For example, a non-null customer identifier rule says nothing about whether an identifier belongs to the correct customer.

A good opening answer is: “I would read the run's scope and rule definitions before treating its pass as a release signal. Then I would compare its evaluated population with the data the consumer will actually use.” This is an engineering recommendation, not a documented hiring rubric.

Ask the interviewer which requirement matters. Blocking a financial close may justify complete reconciliation of a bounded partition. Monitoring a large exploratory table may tolerate sampling. The scan configuration should follow that decision; a cheaper run is useful only if its remaining blind spots are understood.

How do incremental scope, row filters, and sampling interact?

Official API fact: executionSpec.field is the immutable, unnested Date or Timestamp field used for incremental scanning and should increase monotonically. Omitting it means scanning all data, subject to other selection settings. The field cannot simply be changed on an existing scan. DataScan execution specification.

Official configuration facts: rowFilter selects data rows using a GoogleSQL condition. filter selects which rules run. They are different controls. samplingPercent chooses a percentage of records; omitted, zero, or 100 means no sampling is applied. Setting zero does not mean “scan zero rows.” DataQualitySpec reference.

For incremental scans, sampling applies to the latest increment. BigQuery quality-scan guidance. Explain these controls as population restrictions, rather than inventing a guaranteed physical query execution order.

Here is an original coverage exercise. The agreed marker boundary is October 5 at midnight UTC. The exercise explicitly selects timestamps strictly after that boundary and country US. This comparison is part of the exercise, not a promise about undocumented service watermark boundary behavior.

Record	Recorded timestamp	Country	Customer ID	Eligible before sampling?
A	October 4, 23:00	US	NULL	No: outside the increment
B	October 5, 01:00	US	NULL	Yes: a potential failure
C	October 5, 02:00	CA	NULL	No: outside the row filter
D	October 5, 03:00	US	C-104	Yes: valid ID

If A arrives late but retains its old event timestamp, a scan covering only newer event times can miss it. Calling the scan incremental does not repair that selection gap. A possible design is to use a trustworthy ingestion marker and separately validate corrections to older partitions. Another is to schedule bounded historical rescans. Both are design choices that require a clear late-arrival contract.

Do not assume a ten-percent sample contains exactly one tenth of the eligible rows or necessarily includes B. Check the recorded counts. If a comparison scan sees different countries, marker windows, or service identities, its improved result may reflect a changed population rather than a repaired pipeline.

Can a rule pass with failed rows?

Yes. Official rule fact: a row-level threshold is a minimum passing proportion. A threshold of zero uses the default of one, rather than making the rule automatically permissive. Aggregate checks have different semantics; a SQL assertion fails if its query returns rows. DataQualityRule reference.

Consider this original observed-sample exercise: a non-null rule evaluates 80 rows, 79 pass, and one has a missing customer ID. The observed pass ratio is 79 / 80 = 0.9875. With a threshold of 0.95, the rule passes. With a threshold of 1.0, the same cohort fails.

The first outcome is consistent with the configuration. It does not mean the missing identifier disappeared. Whether one failure is acceptable depends on the business contract. A single missing settlement record may matter more than several incomplete optional profile fields.

This configuration sketch expresses the exercise's intended controls. It is not a complete create request: the source table and other required resource fields are omitted.

{
  "executionSpec": {
    "field": "ingested_at",
    "trigger": {"onDemand": {}}
  },
  "dataQualitySpec": {
    "samplingPercent": 10,
    "rowFilter": "country = 'US'",
    "rules": [{
      "name": "customer-id-present",
      "column": "customer_id",
      "dimension": "COMPLETENESS",
      "threshold": 0.95,
      "nonNullExpectation": {}
    }]
  }
}

Answer the follow-up by separating three decisions: which rows to examine, what makes a row valid, and how many failures the rule permits. Changing sampling from ten to 100 percent increases coverage of the eligible population. Changing the threshold changes the acceptance policy. Neither action widens an incremental window or removes the country filter.

A scan that evaluates zero rows needs an expected-volume check. Before explaining a surprising pass, inspect whether useful records were actually evaluated. Do not describe a zero-row result as evidence that incoming records meet the contract. Record the observed service output and decide whether the pipeline needs a separate expected-volume check.

What does failed-row evidence tell you?

Official result facts: row-level rule results expose evaluated and passed counts, a pass ratio, and failingRowsQuery. That query field is scoped to row-level rules; it is not a universal output for every aggregate or SQL assertion. SQL assertions instead expose their assertion row count. Rule result fields.

In the 80-row exercise, investigate the one missing identifier even when the threshold passes. Preserve the run identifier, rule name, cohort definition, and query alongside the counts. That package lets another engineer distinguish an accepted exception from a rule that never examined the relevant record.

Treat a generated query as a way to investigate failures. It is not necessarily an immutable copy of the rows as they existed at scan time. If the table has changed, executing the query later can produce different evidence. State when it ran and which source version it read.

Suppose row B was repaired between the scan and investigation. The original run reports a missing ID; the later diagnostic query returns none. Those observations can both be true. Re-running the scan against the repaired cohort can verify the repair, while the original record preserves the incident history.

For an aggregate revenue check, ask what evidence would explain the deviation: partition totals, contribution by source, or join fan-out. Expecting an automatic bad-row list for an aggregate rule is the wrong contract. Exported scan-result metadata also should not be mistaken for a stored copy of every failing source record.

Why can custom SQL evaluate the wrong population?

Official SQL fact: ${data()} references the source with its scan preconditions, including incremental filtering, row filters, and sampling. It is case sensitive. An explicit reference to another table serves a different purpose. Data reference parameter.

For an original SQL assertion sketch, the following query returns missing identifiers in the scan population:

SELECT order_id
FROM ${data()}
WHERE customer_id IS NULL

This text belongs inside the supported scan-rule environment; it is not standalone BigQuery SQL before the service expands the parameter. In this exercise, any returned row makes the SQL assertion fail, unlike the earlier non-null rule with a 95-percent acceptance threshold.

Now replace the parameter with a hard-coded whole-table reference. The query might include old US rows and Canadian rows that the scan otherwise excludes. Its answer then describes a different cohort. Before blaming a product bug, compare the two rules' evaluated populations.

Name the intended scope for each referenced table. The fact table should reflect the population being validated. A reference dictionary might deliberately include all currently valid currencies. Copying the same date filter to both could incorrectly remove valid dictionary entries and generate false failures.

How do you distinguish rule failure from execution failure?

A completed run with a false quality verdict differs from a run that could not execute. Official job-state fact: the client reference distinguishes successful completion from FAILED, which indicates an execution error. Inspect the returned state and message before discussing quality results. DataScanJob state reference.

Use this original diagnosis table to choose the next observation:

Observation	What to investigate next
Completed run; non-null rule fails	Counts, missing IDs, selected cohort, upstream mapping
Run fails with an access error	Configured execution identity and access to every referenced table
Quality result exists; result export fails	Export outcome, destination access, and retained run evidence
Rules pass; downstream totals differ	Coverage, absent business rules, grain, and downstream transformations

Official access fact: the default scan uses the Knowledge Catalog service account. Current documentation also supports custom execution identities. For the documented default-service-account case, read access must cover scanned and referenced tables; exporting requires destination permissions. Its row-level access visibility determines which rows it can scan. Execution identity and permissions.

Start by identifying the actual principal. Your personal ability to query a table does not establish that a service-agent run can read it. Conversely, a restricted principal may see a smaller valid-looking population. Expanding permissions should follow the intended access design, not be the automatic answer to every disappointing result.

The result schema records export state separately from rule results. Post-scan export results. If quality succeeded but evidence export failed, preserve the available result and repair the export path. Do not erase the distinction by calling the dataset valid or invalid based only on the export status.

For this original recovery exercise, propose one bounded verification: retain the failed run, identify the specific permission or SQL issue, apply the appropriate fix, and rerun the intended cohort. Reconcile the new counts before promoting a downstream partition. This proposed workflow does not assume an undocumented watermark advancement policy after failures.

Practice explaining the result, not just naming the service

Question	What to practice
Load Daily JSONL LLM Chat Logs Into a Warehouse: Schema, Validation, Dedup, Idempotency	Define validation before accepting an incremental load.
Design a Data Service for Downstream Consumers	Specify freshness and evidence consumers can trust.
Single-Pass Python Cleaning Pipeline for Million-Document Web Crawl Parquet Files	Separate rejected records from processing failures.
Measure Game Monetization and Validate a Reporting Repair	Prove a repair through reconciled reporting outcomes.
Design an Incremental Rolling-Metrics Data Pipeline	Explain late arrivals and bounded historical correction.

Begin with the JSONL warehouse-loading exercise. Explain the cohort you would scan, calculate one threshold result, and name the evidence you would save before letting a downstream consumer use the load.