Articles
MLflow Tracking Interview Questions: Run Identity, Artifact Storage, and Reproducible Comparisons
Prepare for MLflow Tracking interviews with run identity, resume behavior, nested runs, artifact routes, partial logging, and reproducible comparisons.
Two MLflow runs share the name baseline. One has a higher score, but its evaluation split differs and its artifact upload failed. Which model is better? MLflow Tracking interview questions test whether you can connect an execution's identity, its stored evidence, and the contract that makes a comparison meaningful.
A run name is a label, not proof of identity or comparability. Before comparing scores, identify the exact runs, inspect where their artifacts live, and check the evaluation conditions. A recorded score needs an evaluation contract before you can interpret it.
Official facts come from current MLflow documentation. We also executed a local fixture using MLflow 3.3.2, SQLite, and local file artifacts. The saved fixture results identify those version-specific observations; the fixture logs toy scalar values and does not train models or test a remote tracking server. The comparison ledger and recovery advice are original exercises. Start with Test and Run a Reproducible Data Science Pipeline and add the tracking acceptance checks below.
How do experiment, run ID, and run name differ?
Official fact: MLflow Tracking groups runs into experiments and records execution information such as parameters, metrics, tags, and artifacts. The tracking overview describes those objects and the supported storage arrangements. Treat the run ID as the execution identifier; the name is a human-readable label that can be reused.
In our local observation, creating two runs named baseline produced two distinct IDs. A search by name therefore returned a selection problem rather than a unique execution. Persist the intended run ID in the scheduler's execution record before a retry searches for baseline by name.
An experiment name also does not guarantee a shared dataset, environment, or evaluation policy. It organizes work. If different teams log to the same experiment, the comparison contract still needs to be established from their recorded metadata and retained artifacts.
Build an identity ledger like this original example:
| Item | What to retain | Why it matters |
|---|---|---|
| Tracking destination | Explicit tracking URI and relevant workspace/context | Avoid reading a different tracking installation |
| Experiment | Experiment ID and intended grouping | Explain where the run belongs |
| Run | Run ID plus human label | Resume or inspect the correct execution |
| Code and configuration | Commit/package identity and immutable configuration | Reconstruct what produced the evidence |
| Evaluation | Dataset/split manifest and metric definition | Determine which scores are comparable |
| Artifacts | Recorded artifact URI and verified files | Retrieve evidence rather than infer it from a UI row |
Separate a training execution from an operational retry attempt. If the original execution truly continues, resuming its ID can be appropriate. If the retry changes data, seed, code, or the model configuration, a new linked run often makes the comparison clearer. Define the intended execution unit before choosing a retry policy so continuation and a changed experiment remain distinguishable.
What does start_run do when resuming a run?
Official fact: supplying run_id to start_run() attempts to resume that run. Its status becomes RUNNING and its end time is unset. A new-run name or experiment argument does not move or rename the resumed run. See the Python API reference for the precise parameter behavior, including tags and descriptions.
Our MLflow3.3.2 fixture confirmed that the resumed ID was unchanged, its status during the context was RUNNING, and passing run_name="new-name-ignored" left the existing name baseline. This is observed local behavior, not a claim about every older MLflow version or storage plugin.
This original sketch shows the intended identity handoff:
with mlflow.start_run(experiment_id=experiment_id, run_name="baseline") as run:
saved_run_id = run.info.run_id
mlflow.log_param("seed", 42)
# Continue the same execution; persist saved_run_id outside the process.
with mlflow.start_run(run_id=saved_run_id):
mlflow.log_metric("validation_score", score, step=step)
experiment_id, score, and step are supplied by the surrounding application. The code is an identity sketch, not a complete training program. Save the run ID durably before a crash can make the relationship ambiguous, and record which work a continuation is authorized to perform.
Our fixture also tried replacing the existing seed=42 parameter with seed=99. That write was rejected. The correct recovery is not to repeatedly overwrite the seed until logging works. If the training configuration changed, record a distinct execution and preserve the relation to the earlier one.
Practice explaining what happens when a parameter error is caught. In our fixture, we caught the rejected write inside the context, continued, and later observed FINISHED. That status did not mean every attempted operation succeeded. Keep a separate acceptance record for required evidence.
What do nested runs inherit?
Official fact: start_run(nested=True) supports a child run within an active parent context. The official nested-run example shows parent-child association through mlflow.parentRunId.
Our local parent had toy metric toy_score=0.8; the nested child logged toy_score=0.9. The child's parent tag identified the parent, while the parent's metric remained 0.8. The relationship did not copy the child's metric into the parent or automatically produce a cross-validation mean.
Original design exercise: the parent describes a hyperparameter trial and children describe its five folds. Each child records its split identity and fold score. The parent records an explicitly computed aggregate only after the intended folds complete and the aggregation rule is checked.
If fold 4 fails, explain whether the trial is incomplete, retried, or evaluated under a documented missing-fold policy. Averaging the four surviving scores silently changes the experiment. A nested relationship makes lineage visible; it does not decide which statistical summary is defensible.
Also clarify who owns the active run context. A helper that starts or ends runs implicitly can attach evidence to an unintended execution. Pass the intended identity or keep context boundaries explicit, then inspect the actual destination of a logged item.
Why can metrics exist while artifacts are missing?
Official fact: MLflow's backend store holds tracking metadata. Its artifact store holds files such as plots, manifests, and other execution outputs. They are different storage contracts; a successful metric write does not prove that the artifact path is reachable.
In an original remote case, the tracking API accepts a metric but artifact upload receives an access error. The first question is the run's recorded artifact URI. That URI determines which storage path must be investigated. Checking only the tracking URI leaves half the diagnosis unresolved.
Official fact: the tracking-server guide distinguishes proxied artifacts from direct storage access. With a proxy setup, the server accesses the configured artifact destination; with --no-serve-artifacts and a direct artifact root, clients require storage access. Existing experiments retain their recorded artifact location when server configuration changes.
| Original failure observation | Evidence to inspect | Unjustified conclusion to avoid |
|---|---|---|
| Metric logs, artifact upload fails | Run artifact URI, writer identity, route and response | “The whole tracking service is down” |
| New experiments work; old ones fail | Each experiment's recorded artifact location | “Restarting changed every existing run” |
| UI downloads a file; training upload fails | Actual download route versus training writer rights | “One successful read proves write access” |
| Artifact appears under a different run | Active/specified run ID at logging time | “The intended run's artifact is missing from storage” |
The supporting illustration shows a server-mediated arrangement. A direct-artifact deployment has a different client-to-storage path. Describe the arrangement actually configured rather than assuming every file passes through the tracking server.
Do not prescribe a default storage mode from memory. MLflow's documented defaults have changed across releases. For a reproducible local exercise, specify the backend URI and artifact location explicitly and record the installed version. For a remote case, inspect both configuration and the persisted experiment/run locations.
What makes two model runs comparable?
Original comparison exercise: run A records AUC0.84 and F1 0.76 on validation split V3 with threshold0.50. Run B records AUC0.87 and F1 0.80 on test split V4 with threshold0.40. These illustrative scores are not outputs of our local fixture.
B's larger numbers do not establish an improvement under a shared evaluation. The examples differ in split and threshold. AUC uses score rankings rather than that threshold, but different evaluation rows can still change its value. F1 depends on the chosen classification threshold as well as the examples and label definition.
| Comparison requirement | A | B | Acceptance decision |
|---|---|---|---|
| Evaluation rows | V3 | V4 | Reject direct comparison; re-evaluate on agreed rows |
| Positive-label meaning | Declared as class1 | Must be verified | Do not infer from metric name |
| F1 threshold | 0.50 | 0.40 | Compare under a declared threshold-selection policy |
| Preprocessing ownership | Fit on training data | Must be verified | Reject evaluation leakage |
| Code/environment identity | Retained | Must be retained | Reproduce each model's prediction path |
| Artifact availability | Checked manifest | Upload failed | Recover required evidence before accepting result |
Hold evaluation conditions constant where the experiment calls for it. Model configuration may differ intentionally; that is the change under investigation. Requiring every hyperparameter to match would defeat a model comparison. Instead, distinguish the treatment from the conditions needed to interpret its effect.
Official fact: MLflow dataset objects record metadata including source, digest, schema, and profile where supported. Dataset API reference Logging that metadata should not be treated as a guarantee that every original input file has been preserved immutably.
Our preparation recommendation is an evaluation manifest: immutable source versions, row/split identity, target definition, positive label, feature schema, preprocessing version, metric implementation, and threshold policy where relevant. Retain enough input and environment evidence to reproduce the intended calculation. A mutable CSV path and a metric called accuracy are too little to resolve most disagreements.
The local fixture's manifest comparison deliberately changed only split_hash; the acceptance check found that field as the mismatch. This tests manifest comparison logic, not statistical validity or the accuracy of a trained model. Explain both the useful result and its limit.
How do you recover partial logging without mixing executions?
Begin with an original required-evidence ledger: run ID, expected metrics and steps, configuration, model or output files, evaluation manifest, and artifact checksums. For each item, record whether it was accepted and verified. Then resume or create a linked run according to the execution policy established earlier.
In our local test, uploading evaluation.json, downloading it through the client, and comparing its SHA-256 hash succeeded. That verifies a local file roundtrip for this artifact. It does not establish remote proxy authentication, object-store consistency, or an atomic transaction combining all metadata and artifacts.
If a retry logs the metric again, inspect the metric history and the intended step identity. Do not assume that a second logging call is an application-level deduplication protocol. If the artifact is a mutable name, decide whether overwriting it is permitted or whether attempt-specific paths and a verified manifest provide clearer evidence.
Keep execution completion separate from review readiness. In our test, FINISHED coexisted with a caught rejected parameter write. A practical acceptance rule is “required evidence exists under the intended run, its configuration is consistent, and its comparison contract passes.” Check required evidence separately from the run's completion status.
We completed ten local checks covering identity, resume status and name, parameter rejection, nesting, metric separation, artifact hash, final status, and a split mismatch. They make the examples inspectable while leaving remote behavior and model-quality claims to separate experiments.