LangSmith Interview Questions: Datasets, Traces, and Offline vs Online Evaluation

LangSmith interview questions become useful when you can explain what was measured, which executions were compared, and what the evidence permits you to conclude. A trace helps you investigate one operation. A dataset gives you repeatable examples. An experiment records results for an application configuration. Offline evaluation supports controlled comparisons; online evaluation observes behavior on production traffic.

Start with Design an Evaluation Platform for AI Agents Across Prompt and Model Changes, then work through the release decision below. The challenge is explaining why a candidate with a higher overall score can still be unsafe to ship.

Official LangChain documentation supplies the LangSmith behavior described here. The datasets, numerical examples and release policies are original preparation exercises.

LangSmith interview questions: name the entity before the metric

In LangSmith, a run records a unit of work, such as retrieval or a model call. A trace groups the runs belonging to one operation; a thread connects turns in a conversation. These are execution records, rather than interchangeable names for an evaluation dataset. See the official observability concepts.

For evaluation, a dataset contains examples: inputs, optional reference outputs and metadata. An experiment associates application executions and evaluation results with those examples. LangSmith's evaluation concepts describe these entities and the difference between reference-based and reference-free scoring.

Suppose a support assistant receives “Which refund rule applies to my order?” Its retrieval call and answer-generation call belong to the same trace. A reviewed version of that question, with an approved answer and allowed evidence, can become a dataset example. Running two assistant configurations against that example produces two executions to compare; copying a trace screenshot does not create a verified reference answer.

In an interview, clarify the unit of success. Is it one answer, one tool call, or completion of an entire conversation? A correct first turn can be followed by an unauthorized tool action. Averaging turn scores would obscure a conversation-level failure unless you define a separate criterion.

How do you make an offline comparison reproducible?

According to the official dataset-management documentation, adding, changing or deleting examples creates dataset versions. You can fetch a particular version with list_examples using as_of, then pass those examples to evaluation. Human-readable version tags help organization, but record the resolved version when reproducibility matters.

For the original support-assistant exercise, write an experiment manifest containing the dataset version, selected example IDs, application commit, prompt revision, model identifier, retrieval-index snapshot and evaluator revision. These are proposed engineering controls, not a claim that LangSmith automatically freezes every external dependency.

Consider two experiments both named “refund-test.” Baseline A used an old policy corpus; candidate B used yesterday's corpus. If the question asks about today's rule, a difference could reflect retrieval freshness rather than a better prompt. Repeat with the same corpus snapshot before attributing the gain to the prompt change.

The same problem occurs when a team quietly removes difficult examples. Record both the population and exclusions. Compare matched example IDs, explain missing executions, and preserve the original results rather than replacing them with a convenient rerun. Reproducibility means another engineer can reconstruct the comparison, including unsuccessful cases.

Which checks should be code, references or a judge?

Use a deterministic check for a deterministic requirement: required JSON fields, valid tool arguments, or a classification label. Use reviewed references when correctness depends on an expected outcome. Use a calibrated judge for qualities such as whether an answer explains its evidence clearly. The official evaluation-types guide separates these evaluator approaches from when evaluation runs.

Here is an original citation-membership check following the named-argument convention in LangSmith's code-evaluator SDK documentation:

def allowed_citations(outputs, reference_outputs):
    if not reference_outputs or "allowed_ids" not in reference_outputs:
        raise ValueError("Missing reviewed citation policy")

    cited = outputs.get("citation_ids")
    allowed = reference_outputs["allowed_ids"]
    if not isinstance(allowed, list) or not all(
        isinstance(x, str) for x in allowed
    ):
        raise ValueError("Malformed reviewed citation policy")

    valid = (
        isinstance(cited, list)
        and bool(cited)
        and all(isinstance(x, str) for x in cited)
    )
    return {
        "key": "allowed_citations",
        "score": valid and set(cited).issubset(set(allowed)),
    }

This check rejects an empty citation list and citations outside the approved set. It does not prove that the cited text entails the answer, that retrieval found every necessary passage, or that the user was authorized to access those passages. Those require distinct evidence and checks.

The policy exceptions are deliberate: absent or malformed evaluation policy is a setup failure. Counting it as a model failure would mix evaluator health with application quality. This pure-Python logic was checked locally; no hosted LangSmith experiment or LLM judge was executed for this article.

Why can a higher average still fail the release gate?

Use this original hypothetical dataset of 120 examples. Both configurations run on the same example IDs under one scoring rule. “Critical” cases test whether the assistant handles a restricted action correctly; routine cases test ordinary answers.

Dataset slice	Examples	Baseline A passes	Candidate B passes
Routine answers	100	86	92
Critical-action cases	20	20	16
All examples	120	106	108

A passes 88.3% overall; B passes 90.0%. B improves routine answers but fails four critical cases that A passes. Under a predeclared policy requiring all critical cases to pass, B fails the gate despite its higher average. The numbers illustrate a decision rule, not a LangSmith benchmark or recommended universal threshold.

Inspect the paired changes. Were B's critical failures caused by a missing authorization check, a newly permissive prompt, or different retrieved context? The repair depends on that diagnosis. A pooled score identifies neither the cause nor the cost of a severe failure.

For stochastic systems, one execution per example is also limited evidence. Repeat runs under a declared sampling design and inspect variability by example. Repeated outputs from one question do not become independent coverage of many questions. Report uncertainty at the level relevant to the release decision and retain the critical gate separately from the aggregate.

What should a trace prove when an answer is wrong?

Use the trace to test competing explanations. In the original example, the final answer quotes an expired refund policy. Inspect the retrieved document ID and revision, the text supplied to generation, the prompt revision, the output and any tool result. Then determine the earliest point where the execution diverged from the intended contract.

If retrieval returned the current document but the answer cited an older rule from conversation history, reindexing is not a demonstrated fix. If retrieval returned an expired document, changing the generation prompt may conceal the symptom while leaving stale evidence available. Explain which observation distinguishes those hypotheses.

Timing requires similar care. A parent run can include child runs, and parallel work can overlap. Do not sum all displayed durations and label that total user latency. Establish the request boundary and identify the critical path before proposing a performance change.

The trace is evidence of recorded execution. It does not reveal an uninstrumented authorization check or an external write that was omitted from logging. State the instrumentation gap explicitly. A confident explanation should identify what the trace shows and what additional evidence would falsify the proposed cause.

Why do offline and online evaluation scores disagree?

The official online-evaluator guide lets you filter eligible runs and configure a sampling rate. This means an online score can describe a selected subset rather than every user request. Record the filter, sample and completed evaluations before comparing periods or releases.

Consider this original production-observation ledger:

Observation boundary	Count	Interpretation
All requests in the interval	10,000	Traffic population
Requests eligible under the tool-use filter	2,000	Different population from all requests
Runs actually selected for evaluation	198	Observed selection, not an assumed quota
Evaluations successfully completed	180	Eighteen selected runs have no usable score
Completed scores marked failed	18	10.0% of completed evaluations

The failure rate is 18/180 for completed evaluations, not 18/10,000. You cannot infer the overall production failure rate from this selected, incomplete sample without a defensible sampling and missingness analysis. Report the eighteen unscored runs separately; a judge outage is not eighteen passing outputs.

Offline data may intentionally overrepresent difficult cases. Online traffic may contain new intents, different languages or changed tool availability. First compare task mix, application version, evaluator revision and score availability. A disagreement can be informative evidence about coverage rather than proof that one evaluation system is broken.

Online runs ordinarily lack curated reference answers. A format or helpfulness score therefore does not establish factual correctness against ground truth. Define the narrower property measured, and obtain reviewed evidence when the release question requires it.

How does a production failure become a regression case?

First confirm the failure and preserve enough context to reproduce its cause. Review the user's intended outcome, relevant permissions, document revision and expected behavior. Then construct a minimal example with a reviewed reference or acceptance criterion. A thumbs-down is a triage signal; it is not itself an answer key.

LangSmith provides input/output hiding and masking controls. Decide what may be recorded before collecting traces. Check child-run payloads, metadata and attachments separately; hiding one field should not be treated as proof that every other channel is safe.

For this exercise, replace customer identifiers with synthetic values while retaining the permission relationship that caused the failure. If redaction removes the only distinguishing fact, the case no longer tests the original bug. Document that trade-off and avoid copying unnecessary production content into a long-lived dataset.

Assign an example ID, record provenance and reviewer decision, add the case to a new dataset version, and rerun the relevant baseline and candidate. Keep an untouched comparison set for assessing generalization; do not repeatedly tune against every newly observed failure and then call the same collection an unseen test.

The official dataset UI guide supports splits and metadata. Treat overlapping diagnostic slices differently from disjoint train/test partitions. “Critical” and “Spanish” can both describe one example; a test example accidentally included in prompt-development data creates a separate leakage problem.

How do you defend an LLM judge's reliability?

LangSmith's LLM-as-a-judge documentation describes configurable scoring criteria. A model-generated score remains a measurement requiring validation. Specify the rubric, expected evidence, output schema and treatment of uncertainty before using the judge in a release gate.

Build an original calibration exercise with clear passes, clear failures and ambiguous cases reviewed by humans. Inspect disagreements individually. A judge that rewards confident wording may prefer an unsupported answer over a cautious correct one. Test whether changing answer order or adding irrelevant length changes the grade when the substantive evidence stays constant.

Keep judge revisions separate from application revisions. If both change together, a score increase is ambiguous. Report evaluator exceptions, latency and completion rate alongside quality metrics, and retain a human-review route for severe or disputed cases. A release explanation is stronger when it identifies how the measurement could be wrong.

Sources and Further Reading

• LangSmith observability concepts
• LangSmith evaluation concepts
• Evaluation types
• Manage datasets and versions
• Create and manage datasets in the UI
• Define a code evaluator with the SDK
• Define an LLM-as-a-judge evaluator
• Set up online LLM-as-a-judge evaluators
• Prevent logging sensitive data in traces

Research checked October 4, 2026. Confirm SDK signatures and workspace controls against the documentation for your installed version.