Kubeflow Pipelines Interview Questions: Artifacts, Evaluation Gates, and Cache Reuse

Kubeflow Pipelines interview questions become interesting when a workflow runs successfully but produces the wrong business outcome. A training task can finish, an evaluation task can report a poor score, and an incorrectly connected promotion task can still publish the model. A cache hit can also save hours while returning results for yesterday's data.

Prepare to explain three contracts: what each artifact identifies, what permits promotion, and what makes a cached execution reusable. Start with Find the Flaws in an End-to-End ML Pipeline Case Study Report, then map its evaluation problems onto an explicit pipeline graph.

Official KFP documentation establishes the platform behavior below. The DAG, thresholds, replay ledger and failure scenarios are original preparation exercises.

Kubeflow Pipelines interview questions: artifacts or parameters?

An artifact represents a model, dataset or other ML object with associated properties and lineage. A parameter carries a small value that another component needs, such as a threshold, snapshot identifier or Boolean decision. Choose by the contract, rather than treating everything as a string containing a storage address.

The official artifact guide distinguishes an artifact's object location, local filesystem path and metadata. In traditional component syntax, Input[Dataset] declares a dataset input and Output[Model] declares a model output. The backend supplies the artifact objects; the component writes its output rather than returning that output object as an ordinary Python value.

For the original workflow, define a dataset artifact for the training snapshot, a model artifact for the candidate, and a metrics artifact for the evaluation record. Use a separate passed: bool output parameter to drive the release branch. A metrics artifact containing auc=0.84 does not automatically become a release condition.

The parameter guide documents Boolean parameters and named multiple outputs. This lets evaluation expose a compact decision while retaining detailed metrics for inspection. Declare both outputs deliberately; a UI chart is useful evidence, but it is not an executable gate.

Why can promotion run before evaluation finishes?

Consider an original pipeline that trains a candidate, evaluates it, and promotes it. Evaluation and promotion both consume the training task's model output. That establishes two dependencies on training. It does not establish a dependency from evaluation to promotion.

Once training finishes, the two downstream tasks can become eligible independently. If promotion changes a production alias before evaluation completes, discovering a poor score afterward cannot undo the already completed side effect. Drawing boxes in a visually pleasing left-to-right order does not impose an execution dependency.

Adding promote_task.after(evaluate_task) introduces ordering. The PipelineTask API documents this dependency method. However, ordering alone still does not express acceptance: an evaluation task can successfully calculate a disappointing score and exit normally. Promotion then runs after a successful computation of a failed business criterion.

Explain the distinction in one sentence: “Evaluation completion answers whether a score was produced; the gate answers whether that score permits release.” A task failure, a rejected candidate and a skipped downstream branch are different outcomes and deserve different monitoring signals.

Build an explicit evaluation gate

For this original exercise, require candidate AUC of at least 0.86, measured on a fixed evaluation snapshot, plus no failed required segment check. The threshold is illustrative; a real release policy needs a metric and acceptance rule tied to the product's costs.

The official control-flow guide uses dsl.If to create runtime conditional execution. A normal Python if inside a pipeline definition is not a substitute for branching on a task's future output: the pipeline function constructs topology before those runtime values exist.

The following is a wiring excerpt for separately implemented, typed components. It is not a standalone training program, and it was not compiled or run against a KFP backend here.

from kfp import dsl

@dsl.pipeline(name="evaluated-promotion")
def pipeline(snapshot_uri: str, snapshot_id: str,
             eval_snapshot_id: str, policy_version: str):
    data = load_snapshot(uri=snapshot_uri, version=snapshot_id)
    trained = train(dataset=data.outputs["dataset"])
    checked = evaluate(
        model=trained.outputs["model"],
        eval_version=eval_snapshot_id,
        policy_version=policy_version,
    )
    with dsl.If(checked.outputs["passed"] == True):
        promote(model=trained.outputs["model"]).set_caching_options(False)

The conditional needs evaluation's Boolean output. Promotion also consumes the exact candidate model artifact that evaluation inspected. Do not have the promotion component independently resolve a mutable latest-model pointer: another run could change that pointer between evaluation and release.

The diagram distinguishes control from artifact flow. Its direct model edge supplies the release payload; it grants no permission to bypass the gate. The rejected branch leaves the currently deployed model unchanged.

What should happen when the gate returns false?

With no failing task, a false condition can skip promotion while the overall run finishes successfully. That is often a legitimate outcome: training and evaluation worked, and the release policy rejected the candidate. Record a separate outcome such as not_promoted, with the decision reason and candidate identity.

Use this original decision table to make the acceptance policy reviewable:

Evaluation evidence	Gate outcome	Release consequence
AUC 0.84, valid snapshot, segment checks pass	Reject	Keep the current model
AUC exactly 0.86, valid evidence, all checks pass	Accept under this inclusive threshold	Candidate becomes eligible for promotion
AUC 0.90, required segment fails	Reject	Aggregate score cannot override the segment rule
Required metric missing or not finite	Invalid evidence	Fail closed; do not substitute zero or true
Candidate and evaluation record refer to different models	Invalid provenance	Stop promotion and investigate identity

“Fail closed” specifies the release effect, not necessarily one universal pipeline status. Your component can raise an error for malformed evidence and return false for a valid but insufficient score. Document that distinction so an operator can separate broken measurement from ordinary rejection.

Do not compare two models evaluated on different snapshots and call the difference an improvement. In an original follow-up, the candidate scores 0.87 on an easier new dataset while the baseline scored 0.85 on last month's dataset. The numbers alone do not justify promotion. Re-evaluate both under the same declared comparison policy.

Why can an unchanged data URI produce a stale cache hit?

Caching concerns the task's declared execution inputs, not everything its code may read from the world. Suppose load_snapshot receives only s3://training/latest.csv. On Monday that address contains snapshot A; on Tuesday someone overwrites it with snapshot B. The URI string is unchanged even though the data is different.

The official caching guide describes reuse for matching inputs and parameters when prior outputs remain available. Inferring that an undeclared change to external bytes is automatically detected would be unsafe. Artifact-based cache identity and exact key construction depend on the backend; this example specifically concerns a mutable URI supplied as an ordinary string parameter.

Expose meaningful immutable identity before the cached task executes. Pass a snapshot version or expected digest as an input, resolve a versioned object, and verify that the fetched bytes match that identity. Merely attaching a digest to output metadata after execution cannot make an earlier cache lookup aware of it.

Original replay	Declared input identity	What to investigate
Monday, snapshot A	latest.csv alone	Initial execution produces dataset A
Tuesday, snapshot B overwrites the same URI	Same URI parameter	A previous output may be reused despite changed external bytes
Tuesday, immutable snapshot B	URI plus snapshot-B input	Relevant input identity changes; verify backend cache behavior
Same model and data, stricter acceptance policy	policy-v2 passed to evaluation	Previous acceptance must not stand in for the revised policy

This ledger is an original diagnostic model, not a claim that every KFP installation uses a key consisting only of those columns. Ask which component specification, parameters, artifacts and custom settings participate in your actual backend's cache decision.

Which tasks should disable caching?

Reuse fits computations whose declared inputs fully describe the work and whose outputs remain usable. It is a poor default for a task whose purpose is an external side effect, such as updating a production alias or sending a release notification. If that task is served from cache, the required action may not happen for the new run.

Use .set_caching_options(False) on the promotion task, then inspect the effective submission settings. The official caching documentation also describes run-level overrides and compiler/default configuration. Do not claim that one line in the DSL prevents every submitter from overriding it.

Disabling caching does not make a task deterministic or retries harmless. A component can still read mutable data, use an unpinned image or make an external request whose outcome changes. Conversely, changing a parameter to a random run ID defeats useful reuse without solving provenance. Prefer meaningful version inputs over arbitrary cache busting.

Check output availability too. A cache record referencing a removed object is not a usable model. In the interview, ask about artifact retention, access permissions and what happens when a reused output cannot be read. Do not invent a universal cache expiration period across open-source KFP and hosted platforms.

How do retries and overlapping runs make promotion unsafe?

An original promotion task changes the alias from model A to model B, but loses the response. A retry cannot assume the update failed. It should inspect durable release state and recognize the same logical promotion, rather than creating duplicate notifications or another inconsistent deployment.

Use an idempotency key tied to the promotion intent, with a durable receipt. Distinguish repeating the same candidate release from a deliberate later rollback to that candidate: those may be different authorized operations even if the model identifier matches.

Now let an older run finish after a newer model C has been approved. Unconditionally assigning the alias to B can regress production. An external registry or release service can enforce a conditional update against an expected alias revision and reject stale intent. This is original system-design reasoning; KFP's graph ordering alone does not serialize independent runs or provide an external transaction.

Keep recovery tasks separate from release authorization. Exit handlers and failure-ignoring tasks can collect diagnostics or clean temporary resources after errors. Their ability to run after failure is not evidence that they should promote a candidate whose evaluation is missing.

What would you inspect before trusting this pipeline?

The official compilation guide explains IR YAML generation and static type checking. Compilation can catch incompatible component interfaces. It cannot prove that dataset contents are correct, that the evaluation is leakage-free, or that a threshold encodes an appropriate release policy.

Inspect the compiled graph for the evaluation-dependent conditional, the model artifact passed to promotion, declared snapshot and policy inputs, and effective cache settings. Then exercise acceptance, rejection, malformed metrics, missing artifacts, response loss and two concurrent release attempts in an appropriate test environment.

The numerical policy cases in this article were checked locally with standard-library Python. That checks the original acceptance examples only; the KFP SDK and backend were not available for native graph compilation or execution. Preserve that boundary when describing your own preparation: a reviewed DAG and a successful production run provide different evidence.

Sources and Further Reading

• Kubeflow: Create, use, pass, and track ML artifacts
• Kubeflow: Pass small amounts of data between components
• Kubeflow: Control flow
• Kubeflow: Use caching
• Kubeflow: Compile a pipeline
• KFP SDK: PipelineTask API

Sources checked October 4, 2026. Examples use KFP v2 concepts; verify SDK, backend and submission behavior in the environment you will discuss.