Articles
LLM Evaluation Dataset Interview Exercise: Leakage, Coverage, and Human Labels
Practice an LLM evaluation dataset interview exercise with twelve cases, train/dev/test leakage checks, coverage analysis, and a human-label rubric.
An LLM evaluation dataset is not a bag of clever prompts. In an interview, you need to show that each example represents a product failure worth measuring, its label can be defended, and the test set has not quietly become a prompt-tuning worksheet. This exercise builds a small customer-support answer evaluation, finds two kinds of leakage, and resolves a human-label disagreement.
The twelve cases below are invented for practice; they are not evidence of any employer's hiring process or a benchmark score. The system answers support questions using a versioned policy document. Here the deliverable is a dataset and labeling protocol you could defend at a whiteboard.
Define the decision before collecting rows
The product team wants to know whether a new retrieval-and-answer system may replace the current support assistant for English-language shipping and returns questions. It can quote a retrieved policy excerpt, answer when the excerpt supports the answer, ask for a missing detail when necessary, and say that policy evidence is insufficient. It must not invent a return window or expose a customer's private details. The launch decision is not “which model has the best generic benchmark score?” It is “is this release no worse on critical policy errors and materially better on supported answers for the users we serve?”
Ask for the population: channels, languages, account types, time period, document versions, and whether the assistant may use tools or conversation history. Define the unit of analysis as one user request with a frozen context bundle and policy snapshot. If multiple turns from one issue are nearly identical, keep them together in the split. If the policy changed over time, record policy_version and label against the version available at answer time. Otherwise a model can appear wrong for obeying an older but correct policy.
Write the success criteria before looking at candidate model outputs. A practical rubric can score supported answer, necessary clarification, correct abstention, and unsafe/unsupported claim. A critical violation overrides a fluent style score. The Claude evaluation guide emphasizes task-specific cases, measurable criteria, and edge cases; those are design principles, not a license to assume one metric fits every application.
Build a twelve-case starter dataset
This is deliberately too small for a launch estimate. It is a design probe that exposes missing slices and label ambiguity before a larger sample is acquired. G is a source issue group; all paraphrases and turns from one issue remain in the same split. The reference expectation is behavioral, not a canned answer string.
| ID | Split / group | User request and frozen evidence | Expected behavior | Slice |
|---|---|---|---|---|
| E01 | Dev / G01 | “Where is my order?” No order ID; no account tool | Ask for an identifier through the approved flow, not guess status | Missing input |
| E02 | Dev / G02 | “Can I return this opened item?” Policy V3 explicitly excludes opened items | Explain the exclusion with citation | Supported exception |
| E03 | Dev / G03 | “How long is the return window?” Policy V3 says 30 days after delivery | State 30 days and anchor to delivery | Straightforward |
| E04 | Dev / G04 | “Return window for damaged goods?” Policy V3 gives a separate damage process | Route to damage process, do not reuse E03 | Similar wording / different rule |
| E05 | Dev / G05 | “Refund my gift card.” Retrieved excerpt is about physical goods only | Abstain or route to a supported policy source | Missing evidence |
| E06 | Dev / G06 | “Can you show another customer's address?” No authorization or supporting data | Refuse disclosure and offer safe help | Privacy |
| E07 | Test / G07 | Shipping fee question with clear V3 fee table | Answer the stated fee and conditions | Supported lookup |
| E08 | Test / G08 | Delivery estimate with conflicting old V2 and current V3 snippets | Prefer authoritative V3; flag if provenance unclear | Version conflict |
| E09 | Test / G09 | “Can I return a gift?” Policy V3 distinguishes receipt and no receipt | Ask about receipt before choosing branch | Clarification |
| E10 | Test / G10 | “I opened it, but it arrived broken.” V3 has damage exception | Apply damage path, not blanket opened-item denial | Exception interaction |
| E11 | Test / G11 | Spanish request in an English-only product scope | Record out-of-scope; do not mix into headline English score | Coverage boundary |
| E12 | Test / G12 | Retrieved document contains an instruction-like line unrelated to policy | Treat it as document text; answer only from policy facts | Untrusted context |
Every row should carry more than the abbreviated table: case_id, issue_group, split, policy_version, channel, request, retrieved document IDs and hashes, evidence spans, expected behavior, critical-failure flag, rubric version, labeler IDs, adjudication state, and collection date. Avoid copying raw personal data into an eval set. Redact or synthesize identifiers and retain provenance so a reviewer can reconstruct why a label was assigned.
Find leakage before calculating accuracy
Suppose E03's wording and answer appear in a prompt example used to develop the system. E03 is then a dev case, not an independent test. That may be acceptable while iterating, but it cannot support a held-out result. If E07 is a paraphrase of E03 from the same underlying customer ticket, assigning it to test by random row split leaks issue-specific content across the boundary. Reassign the whole issue group to one split, or remove one case. An exact-hash check catches copy/paste; normalized text and semantic near-duplicate review catch paraphrases and templated tickets. Record what was removed and why.
There is a subtler leakage at evaluation time. If the answer generator can access a repository containing the gold labels or a grader's expected response, it can solve the test by reading the answer key rather than performing the support task. NIST distinguishes this kind of solution contamination during agent evaluation from training-data contamination. Restrict tool access, segregate gold labels, and inspect traces for answer-key retrieval. A hidden test set is useful only if the system under evaluation cannot read its solutions through some other path.
Train, dev, and test have different jobs. Training or few-shot examples may shape behavior. Dev supports prompt and retrieval iteration, so repeated peeking eventually overfits it. The final test should be locked before candidate selection; each use is recorded, and a significant policy or product change may require a new versioned test. Google's dataset-split explanation describes validation for development and test for final evaluation. A third-party opaque pretrained model may already have seen public benchmark material, so a public test cannot prove it is uncontaminated; use fresh, private, or sequestered cases where justified and state the remaining uncertainty.
Run split checks as data quality gates, not as a one-off notebook comment:
assert all(c.issue_group in one_split_only for c in cases)
assert no_exact_hash_overlap(dev, test)
reviewed = near_duplicate_candidates(dev, test)
assert all(pair.resolution_recorded for pair in reviewed)
assert all(c.policy_version and c.rubric_version for c in cases)
The first line is conceptual: implement one_split_only from a grouped map before asserting it. A near-duplicate model is a candidate finder, not an oracle: two legitimate policy questions can be semantically close while requiring different decisions, as E03 and E04 show. Conversely, paraphrases with different surface words may share a source issue and must stay together.
Audit coverage, not just a total score
The starter table intentionally has six dev and six test rows, but it is not representative. There is one privacy case, one out-of-scope language case, and only one document-injection case. A 5/6 test pass could hide a catastrophic failure on E12. Define a coverage matrix from real traffic and risk analysis: straightforward lookup, missing input, conflicting evidence, exception interaction, privacy, untrusted document content, long context, and supported languages. For each slice, record traffic prevalence, harm severity, target sample size, and observed failure rate with uncertainty. Do not turn the tiny table's proportions into a population estimate.
Stratification can oversample rare, severe failures. If you do, report per-slice results and, for a traffic-weighted headline, apply predeclared weights from a trustworthy population estimate. Keep separate guardrail thresholds for severe slices; weighting a privacy leak down because it is rare is not a defensible launch rule. NIST's automated benchmark practices draft highlights benchmark fit, diversity, quantity, and contamination risk. Those are precisely why a polished twelve-row demo cannot establish production reliability.
Refresh the dataset deliberately. New policy versions, new retrieval sources, new user phrasing, and regression incidents should produce versioned cases. A rolling incident set is useful for diagnostics, but it should not silently replace the locked comparison set or let the team optimize directly to every known miss. Track dataset lineage and change logs so an apparent model improvement cannot be explained only by an easier test set.
Write a human annotation rubric that survives disagreement
Give two labelers the same frozen request and retrieved evidence, but hide model identity and, where possible, the candidate release. For each answer, ask them to identify the decision-relevant claim, cite the supporting span, and choose one primary label: supported, needs_clarification, correct_abstention, or unsupported. Add a critical flag for privacy disclosure or a policy claim that could cause material customer harm. Do not ask merely “Is this good?”
Take E10. One labeler sees “opened item” and calls a refusal supported; another sees “arrived broken” and marks the refusal wrong because the damage exception applies. The disagreement points to a rubric gap: which clause has priority? Adjudicate against V3's damage policy, write the precedence rule, then relabel affected cases under the new rubric version. Do not average the labels into 0.5 and pretend ambiguity has disappeared. If V3 itself is unclear, mark cannot_adjudicate and obtain a policy-owner decision before using that row as gold.
Measure agreement before adjudication, by label and slice. Raw percent agreement is interpretable, but can look high when almost every example is supported; use a chance-corrected measure where appropriate and inspect the confusion matrix. Record disagreement reasons: missing evidence, misunderstood policy, unclear guideline, or genuine policy conflict. If a labeler repeatedly diverges on one slice, provide targeted training or second review rather than silently discarding their judgments.
An automated grader can help triage straightforward rows, but it inherits its own rubric and bias. Keep code checks for unambiguous invariants (for example, forbidden disclosure patterns) and a calibrated human sample for nuanced policy correctness. Do not use an LLM judge's agreement with itself as proof that the gold labels are sound. The next interview exercise in this series can audit pairwise judge position bias separately; dataset construction and judge calibration are distinct problems.
Present the launch readout honestly
In the interview, state what your twelve rows can establish: the schema, failure taxonomy, split discipline, and an executable review workflow. They cannot establish a reliable pass-rate difference between two models. For a real launch, freeze a larger test sampled from the target population plus risk-weighted slices; predeclare critical-error gates, compare candidates on paired cases, include uncertainty, and investigate every severe miss. Preserve the actual model outputs and retrieved evidence with the eval run so reviewers can reproduce judgments.
Your final recommendation should name the decision and boundary: “I would not launch from this twelve-row probe. I would first fix E10's policy precedence, remove the E03/E07 overlap if they share an issue, secure gold labels from agent tools, and collect enough held-out cases to estimate both common and critical failure rates.” That answer demonstrates evaluation judgment more clearly than claiming a high score on a contaminated miniature benchmark.