Anthropic Research Engineer Interview Guide 2026: Coding, ML Systems, Research Judgment, and Mission

An Anthropic Research Engineer interview is easier to prepare for when you can defend a complete experiment: the question, implementation, baseline, measurement, failure and next decision. “The average improved” is only the beginning of that explanation. You also need to know what changed, who was harmed and what the evidence cannot establish.

This guide works through an original paired evaluation where aggregate success rises from 81% to 84% while a critical group falls from 90% to 75%. It includes runnable analysis, uncertainty estimates, a decision memo and a constrained follow-up plan. The outcomes are synthetic teaching data; no Claude model or other LLM was evaluated to produce them.

Ground your preparation in the actual role

Anthropic's careers page describes substantial overlap between engineering and research. For technical interviews it names live coding tools such as Colab and CodeSignal, permits looking things up, and emphasizes familiarity with syntax and standard libraries. It also describes discussions of experience and motivation. This does not establish one fixed research-engineer loop across teams.

Confirm the role's balance of implementation, ML systems, evaluation and research discussion. Prepare one project where you can explain your own contribution down to a failed assumption or debugging trace. Choose a modest experiment you understand over a broad claim you cannot reproduce.

The candidate AI guidance, last updated July 10, 2025 when reviewed, allows preparation and refinement while requiring take-home and live interview work without AI unless explicitly permitted. Documentation lookup and AI assistance are different permissions. Follow the instructions for the actual session and describe assistance honestly when it is allowed.

For additional practice, the RAG evaluation and debugging question offers a transferable exercise in stage-level metrics and failure diagnosis. Its company tag is Plaid; it is not evidence of an Anthropic interview question. Use it to rehearse the same discipline on a retrieval pipeline.

Define a falsifiable research question

Our fictional hypothesis is that a new context-selection rule improves answer success without materially worsening tasks that depend on an exception in a policy document. Define success before reading results: a task passes only when the final answer satisfies its task-specific factual requirements. The critical slice contains exception-dependent tasks. The common slice contains the other tasks.

For this teaching example, set the decision rule in advance: an overall improvement is necessary, and the critical slice must not lose more than five percentage points. This is an editorial threshold chosen for the exercise, not Anthropic policy or a general safety standard. The baseline remains the comparison point, not an assertion that its own 90% critical-slice performance is acceptable for deployment.

Both versions run on the same task IDs with the same reference answers and environment snapshot. Pairing matters because easy tasks may pass under both versions and difficult tasks may fail under both. The useful comparison is the change on each task. Comparing two independent proportions would discard that relationship.

Read the paired outcomes before the summary score

The synthetic fixture contains 900 common tasks and 100 critical tasks. Among common tasks, 690 pass both versions, 30 pass only the baseline, 75 pass only the variant, and 105 fail both. Thus baseline success is 720/900, or 80%, while the variant reaches 765/900, or 85%.

For critical tasks, 70 pass both, 20 pass only the baseline, five pass only the variant, and five fail both. The baseline reaches 90/100 and the variant 75/100. Across the whole fixture the variant gains 45 common successes but loses 15 critical successes, producing a net gain of 30 among 1,000 tasks.

This is not a change in slice mix: both versions see exactly the same 900/100 allocation. The aggregate improvement and the critical regression are simultaneously true. A researcher should report both rather than trying to explain one away with the other.

The per-task difference is +1 for a variant-only success, -1 for a baseline-only success and zero for agreement. Average those differences to obtain the paired change. Estimate its standard error from the sample variance of the differences divided by the number of independent tasks. The code reports a normal-approximation 95% interval using 1.96 standard errors.

Complete analysis code

import math

def paired_report(rows):
    groups, seen = {'all': []}, set()
    for task, group, baseline, variant in rows:
        if type(task) is not str or not task or task in seen:
            raise ValueError('task IDs must be unique nonempty strings')
        if group not in ('common', 'critical'):
            raise ValueError('unknown slice')
        if type(baseline) is not bool or type(variant) is not bool:
            raise ValueError('outcomes must be booleans')
        seen.add(task)
        pair = (int(baseline), int(variant))
        groups['all'].append(pair)
        groups.setdefault(group, []).append(pair)
    if not seen:
        raise ValueError('empty evaluation')
    result = {}
    for group, pairs in groups.items():
        n = len(pairs)
        differences = [v-b for b,v in pairs]
        delta = sum(differences)/n
        if n > 1:
            se = math.sqrt(sum((x-delta)**2 for x in differences)/(n*(n-1)))
            interval = (delta-1.96*se, delta+1.96*se)
        else:
            interval = None
        result[group] = dict(n=n, baseline=sum(b for b,v in pairs)/n,
            variant=sum(v for b,v in pairs)/n, delta=delta,
            approximate_95_interval=interval)
    return result

def fixture():
    rows=[]
    # Synthetic paired outcomes: both pass, baseline only, variant only, neither.
    for group,counts in [('common',[690,30,75,105]),('critical',[70,20,5,5])]:
        for (b,v),count in zip([(True,True),(True,False),(False,True),(False,False)],counts):
            for _ in range(count): rows.append((str(len(rows)),group,b,v))
    return rows

print(paired_report(fixture()))

Each record has a unique nonempty task ID, a known slice and two Boolean outcomes. Duplicate IDs reject the evaluation instead of silently inflating the sample size. The function also rejects an empty evaluation and numeric stand-ins for Boolean outcomes. For a single task it reports no interval because the sample variance cannot be estimated from one observation.

The implementation takes O(n) time and O(n) retained memory. It stores task IDs for exact duplicate detection and outcome pairs for grouped calculations. A streaming aggregation could retain counts and squared differences, but exact duplicate detection would still need state or an external uniqueness guarantee. Reducing the metric state alone does not make the whole evaluator constant space.

Interpret the uncertainty and make a decision

The overall estimated improvement is 3 percentage points, with an approximate interval of +0.77 to +5.23 points. The common slice improves by five points, with an interval of about +2.79 to +7.21. The critical slice loses 15 points, with an interval of about -24.40 to -5.60. These calculations were executed locally against the synthetic fixture.

Under the stated assumptions, even the upper end of the critical-slice interval is worse than the allowed five-point regression. The decision is hold the variant and investigate. Passing an aggregate gate cannot override a separately declared critical-slice condition. Nor does a hold decision prove the baseline is safe enough for every use.

These intervals assume independent tasks and are an approximation. If many prompts are variants of the same document or user scenario, task IDs alone do not create independence. Resample or analyze at the independent document or scenario level. If a slice was chosen after scanning many possible groups, report that exploratory status and confirm the finding on new data.

Repeated stochastic trials answer another question: how stable is a system on the same task? They do not turn one task into many independent tasks. Keep task-level and trial-level variation distinct. The fixture contains one Boolean outcome per version per task; it does not estimate model randomness, judge variability or a production population's true success rate.

Local checks verified the four principal fixture results, six hundred comparisons across one hundred shuffled orderings, four invalid-input rejections and the single-task interval behavior. That validates the analysis behavior on these cases. The checks do not validate the synthetic labels as representative of real users.

Investigate the regression with a discriminating ablation

The leading hypothesis is that the new selector drops exception clauses while improving relevance on common tasks. A competing explanation is a changed answer grader that penalizes a different writing style. A third is stale source material. These hypotheses imply different observations, so do not spend the next budget on a larger undirected benchmark run.

First inspect matched baseline/variant traces for the 20 critical tasks that changed from pass to fail. Check retrieved documents, selected context, final answer and grading evidence. Preserve failures where the baseline already failed too; restricting analysis only to changed cases would hide the broader error distribution.

Then run a two-by-two intervention on a diagnostic set: old versus new context selector, and old versus new generation settings, with the grader and source snapshot fixed. If the generator did not change, replace that axis with full versus shortened context. State exactly which variable is being isolated. A table of four labels without controlled inputs is not an ablation.

An illustrative follow-up budget is 200 independent critical tasks, two selector settings and three stochastic trials each: 1,200 model calls. Reserve another 300 calls for replication or unexpected failures rather than consuming the entire budget on the first preferred explanation. These are proposed calls, not executed model experiments. Estimate cost from actual token use and current provider pricing before running them.

Keep development cases separate from a held-out confirmation set. Fix the selector using the diagnostic evidence, lock the implementation and then evaluate the confirmation set once under the declared decision rule. Tuning repeatedly on the same final set makes it a development set, however carefully its file is named.

Build an evaluation harness you can debug

Record the dataset version, task ID, source snapshot, model identifier, prompt version, tool configuration, sampling settings, code revision and grader version. Store restricted traces only where their data policy permits. A result without its configuration cannot be reproduced reliably, even when the numerical aggregation is correct.

Distinguish a failed task from a broken evaluation run. A container startup failure or unavailable dependency might prevent the system from attempting the task. Do not silently remove those records from the denominator to improve the score. Report harness failures separately and define when a rerun is permitted. Retain the original failure and the rerun relationship.

Anthropic's evaluation engineering article distinguishes tasks, trials, graders, transcripts and final environment outcomes. It also discusses code-based, model-based and human grading. The useful preparation lesson is to inspect what the system actually achieved alongside its transcript. A model saying that a change was completed does not establish that the expected state exists.

For an LLM grader, create a calibration set with independently reviewed examples, including borderline cases. Test sensitivity to answer order, verbosity and plausible unsupported claims. Keep deterministic checks for machine-readable contracts. Disagreement between a human reviewer and a model judge is evidence to investigate; it is not automatically proof that either side is right.

Explain the result as a research decision

A concise answer might be: “The variant gains three points overall, but loses fifteen on the predeclared exception-dependent slice. The paired interval for that slice is approximately minus twenty-four to minus six points, so it breaches our five-point regression limit. I would hold it, inspect changed-task traces, isolate context selection and confirm a repair on fresh independent cases.”

That answer contains a hypothesis, evidence, a decision and the next discriminating test. It also leaves room for challenge. If the interviewer says tasks share documents, revise the uncertainty analysis at the document level. If the critical labels are unreliable, audit them before treating a narrow numerical interval as trustworthy. Clear research judgment includes changing the conclusion when its assumptions change.

For a mission discussion, connect your values to an actual decision from your own work. Explain who could be affected, what evidence you had, what you escalated and which uncertainty remained. The synthetic experiment can demonstrate your reasoning in practice; it should not become a fabricated personal research accomplishment.

A seven-day preparation plan with concrete outputs

Day one: read the role and confirm the interview format, then select one genuine project. Day two: implement the paired evaluator from a blank file and reproduce the three slice results. Day three: challenge duplicate IDs, missing outcomes and correlated tasks. Day four: write the two-by-two ablation and a budget that includes replication.

Day five: rehearse a pipeline-debugging case and identify the first divergent artifact. Day six: explain your real project at two-minute and ten-minute depth, including a negative result. Day seven: run a timed coding-and-research mock, check permitted tools and review the weakest explanation. Each day should produce something another person can inspect.

Frequently asked questions

Are these Anthropic interview questions?

No. The experiment, code, synthetic outcomes and decision threshold are original preparation material. Official hiring sources establish general context; your invitation establishes the actual sessions.

Can I use Claude during the interview?

The reviewed candidate guidance requires live interviews and take-homes without AI unless explicitly allowed. It permits preparation. Confirm the rules for the specific assessment.

Why use paired analysis?

Both versions run on the same tasks, so each task's change is the observation of interest. The method preserves that pairing and avoids treating related results as independent samples.

Does a positive overall interval justify shipping?

Not under this exercise's rule. The critical-slice regression breaches its separate limit. The decision must account for all predeclared requirements and the reliability of the data.

What if I have limited compute?

Choose the smallest controlled comparison that separates plausible explanations, reserve replication budget and keep a held-out confirmation set. More runs of an uncontrolled experiment do not resolve confounding.

Sources and further reading

• Anthropic careers and hiring guidance
• Anthropic candidate AI-use guidance
• Anthropic: Demystifying evals for AI agents

Checked October 9, 2026. The Python analysis and arithmetic were executed locally; model experiments, deployment decisions and the example budget are fictional or proposed as labeled.