SFT Data Interview Questions 2026: Curation, Deduplication, Quality Filters, and Evaluation

SFT data interviews are about making training data useful, safe, measurable, and reproducible. Prepare to discuss where examples come from, how they are filtered and deduplicated, how annotators or synthetic generators are evaluated, and how data choices affect model behavior. This is not a checklist of secret prompts. It is a framework for answering data-quality questions with evidence and for spotting leakage or harmful shortcuts.

Use the linked drills as evidence-building exercises: solve first, then compare the explanation and record the failure mode you missed.

Quick answer

Use one answer framework: objective, source, quality risks, selection method, evaluation, and production feedback. Strong candidates distinguish duplicate removal from contamination control, agreement from correctness, and dataset quality from downstream model quality. They also state what their filters will miss.

The SFT data pipeline to draw in an interview

Sketch the path from raw source to released dataset: acquisition, provenance, parsing, normalization, policy and privacy checks, exact and near-duplicate detection, labeling or generation, quality scoring, mixture selection, split construction, model training, evaluation, and error-driven iteration. Attach a version to every transformation and keep rejected-example statistics so a filter change is auditable.

Interviewers often probe the boundaries. Who can license or consent to the source? Can the same conversation cross train and evaluation splits? Does a quality model favor one language or writing style? Can a synthetic generator reproduce benchmark answers? A pipeline diagram is useful only if it shows where these questions are measured and who owns the decision.

Start with the objective and unit of quality

Define what the supervised fine-tuning data is meant to improve: instruction following, tool use, style, domain knowledge, refusal behavior, coding, or something else. Then define the example unit and the quality dimensions: correctness, relevance, completeness, safety, diversity, consistency, and provenance. A filter is only good relative to the failure it prevents.

Avoid reducing quality to a single score. A dataset can have high agreement and still teach the wrong behavior, or high task success and poor coverage of edge cases. Slice by task, source, language, difficulty, length, and risk. Store enough metadata to reproduce the selection and understand which examples drove a change.

Curation and annotation

Curation begins with inclusion and exclusion rules that a reviewer can apply consistently. For human data, describe instructions, calibration examples, disagreement handling, adjudication, and reviewer privacy. For synthetic data, explain generation constraints, filtering, model or prompt versions, and human spot checks. A small high-quality sample can guide the pipeline more effectively than an unexplained volume target.

When comparing labels, report the question the metric answers. Agreement can identify ambiguity, but it does not prove correctness. Use gold examples or expert review where the cost of an error is high, and inspect disagreement rather than deleting it automatically. Document label definitions so a future team can tell whether a shift is a data change or a rubric change.

Deduplication and contamination

Deduplication should operate at multiple levels: exact text, normalized text, near-duplicate structure, prompt or answer templates, and source or conversation overlap. Explain the trade-off between removing repeated examples and preserving legitimate paraphrases. At scale, use fingerprints or embeddings as candidate generation, then verify with a threshold appropriate to the data.

Contamination requires a broader view. Training data can overlap with evaluation sets, public benchmarks, or future test material. Track source and time, use held-out or newly created evaluation data, and inspect suspiciously high performance. Do not claim that one string-matching pass proves the absence of leakage; state what the method can and cannot detect.

Quality filters and safety

Filters can check format, language, length, toxicity, privacy, correctness, policy, code execution, or instruction structure. Make the order explicit because early filters can change the distribution seen by later ones. Measure false positives and false negatives on a reviewed sample. A filter that removes difficult but valuable examples may improve a simple metric while weakening generalization.

Safety is a data and process property, not only a classifier. Protect personal information, minimize retention, separate sensitive sources, restrict access, and record why an example was included. For high-risk domains, define escalation and expert review. Explain how you would update a filter without making historical training runs irreproducible.

Evaluation that connects data to behavior

Use a fixed evaluation suite for iteration, but do not rely on one benchmark. Include task success, instruction following, factuality or groundedness where relevant, robustness, safety, and regression slices. Compare a baseline and an ablation so you can distinguish the effect of more data from the effect of a changed filter or mixture.

Track dataset and pipeline versions alongside model, code, configuration, and evaluation versions. Investigate outliers manually. When a metric improves, ask which examples or slices moved and whether the change represents the target users. This is the difference between optimizing a number and learning from evidence.

How to practice data interviews

For each prompt, answer in this order: objective, data source, quality risks, method, evaluation, and operational follow-up. Draw the pipeline and mark where metadata, review, and rollback live. Then challenge your own design with a rare language, a duplicated template, a privacy issue, a mislabeled example, or a leaked benchmark.

Interview questions and strong answer signals

These prompts are practice questions, not predictions of an exact interview. A strong answer defines the problem, states assumptions, makes trade-offs visible, and explains how correctness or impact would be measured.

How would you deduplicate instruction data?

Define the duplicate unit, normalize safely, use exact fingerprints for obvious matches, generate near-duplicate candidates, and validate the threshold on reviewed pairs. Track provenance and measure what diversity is lost. Do not remove paraphrases blindly when they represent useful coverage.

How do you evaluate a quality filter?

Create a reviewed sample with positive and negative cases, measure precision and recall by slice, inspect false positives and negatives, and assess downstream model behavior. Version the filter and keep a rollback path.

How would you detect benchmark contamination?

Track source and time, compare against known evaluation data, use overlap and similarity checks, inspect suspicious performance, and evaluate on fresh or private data. State the detection limits clearly.

What makes annotation reliable?

Clear definitions, calibration examples, reviewer training, disagreement analysis, adjudication, privacy controls, and periodic audits. Agreement is a signal to inspect, not a substitute for correctness.

How would you handle synthetic SFT data?

Version generator model and prompts, constrain the task, filter outputs, sample human review, measure failure categories, and compare against human or baseline data. Monitor artifacts introduced by the generator.

Tell me about a data-quality failure.

Explain the symptom, how you localized the data issue, the impact, the correction, and the process change. Include how you prevented silent recurrence and communicated uncertainty.

How would you build train, validation, and test splits for conversational SFT data?

Group related turns, users, source documents, and templates before splitting so near-identical context does not cross boundaries. Use time-aware splits when deployment data changes over time, reserve a private test set, and run contamination checks after every major curation change.

How would you choose a mixture across tasks and sources?

Start from target behaviors and minimum coverage requirements. Compare source quality and marginal contribution with ablations instead of scaling every source by volume. Monitor whether one high-frequency template dominates gradients or suppresses rare but important capabilities.

What would you log for a rejected example?

Store source and pipeline version, rejection reason, filter score or rule, and enough privacy-safe metadata to reproduce aggregate counts. Do not retain sensitive raw text merely to make debugging convenient; define access, retention, and sampling rules.

How do you evaluate a learned quality model?

Build a reviewed set that includes hard negatives and diverse slices, measure ranking and operating-point metrics, inspect calibration, and test downstream model impact. Audit whether the quality model rewards superficial length, formatting, or model-generated style.

When should disagreement be preserved rather than adjudicated away?

Preserve it when the task is genuinely subjective, policy is underspecified, or multiple responses are acceptable. Record the distribution or rationale, then decide whether training needs a consensus label, preference data, abstention, or a clarified rubric.

How would you find a harmful data shortcut?

Inspect correlations between labels and source artifacts, run counterfactual or template-stripping tests, compare performance by source, and train ablations without suspicious features. Confirm with fresh evaluation data before claiming the shortcut is removed.

Seven-day preparation plan

Use the plan as a compact loop. Spend more time reviewing mistakes than collecting new topics, and adjust the emphasis to the current job posting.

Day	Focus	What to do
Day 1	Focused practice	Define objectives, labels, and quality dimensions.
Day 2	Focused practice	Curation, annotation, agreement, and adjudication.
Day 3	Focused practice	Exact and near-duplicate detection.
Day 4	Focused practice	Filters, privacy, safety, and provenance.
Day 5	Focused practice	Evaluation slices, ablations, and contamination.
Day 6	Focused practice	Run a data-pipeline interview mock.
Day 7	Focused practice	Review metrics, failure cases, and clear definitions.

Frequently asked questions

Is there one best deduplication threshold?

No. It depends on the data, risk, and the cost of removing legitimate variation. Validate it on reviewed examples.

Should all synthetic data be removed?

Not automatically. Evaluate its failure modes and contribution against the target behavior and human data.

What is data contamination?

Overlap between training material and evaluation or future test material that can make performance look better than generalization.

How do I discuss safety without being vague?

Name a risk, control, measurement, escalation path, and retention or access decision.

Final takeaway

Treat an SFT dataset as a versioned product, not a folder of accepted examples. A strong interview answer shows how each curation decision changes coverage, risk, and model behavior, then names the evaluation that could disprove the decision. That combination of data operations and experimental discipline is more valuable than reciting filter names.

Sources and Further Reading

• OpenAI Evals Guide - Public evaluation concepts and workflow.
• Anthropic Agent Evals - Public discussion of evaluation design and failure analysis.
• Automated Data Curation for Robust Language Model Fine-Tuning - Research on systematic curation and data-quality selection.
• AutoSelection: Automatic Data Selection for Language Model Fine-Tuning - Research on fixed-pool data recipe search across filtering, mixing, and deduplication.

Research note: Public facts and documentation were checked on September 3, 2026. Hiring stages, pricing, platform behavior, and product details can change; the reader's current invitation and official pages are the source of truth.