Articles

Coding Agent Evaluation Interview Questions: Tests, Sandboxes, and Benchmark Leakage

Prepare for coding agent evaluation interviews with independent tests, sandbox boundaries, failure denominators, fair budgets, and benchmark leakage checks.

A coding agent reports “all tests passed.” Its patch also deletes an assertion, catches every exception, and returns a constant that satisfies the visible example. Did it solve the task? Coding agent evaluation interview questions begin with that distinction: the agent's account of success is an observation to verify, not the evaluation contract.

A defensible answer evaluates a patch against an independent oracle, in a reproducible environment, under a stated resource budget. It also separates functional failures from infrastructure failures and investigates whether the agent could see solutions or hidden evaluation material.

No candidate reports are used as technical evidence.

What is the unit of a coding-agent evaluation?

Define a task as a particular repository revision, issue description, allowed files and tools, environment, and success criteria. A patch is an artifact from one attempt at that task. The model is only one component: prompting, retrieval, terminal tools, retry policy, and human intervention can change the outcome.

Record the base commit, dependency lock or environment digest, task version, agent configuration, tool policy, resource limits, and patch hash. Store the actual trace and test reports. A model name without those fields is an incomplete experimental description.

Official benchmark example: SWE-bench's evaluation guide describes applying generated patches to repositories and running tests in a Docker environment. It also exposes per-instance reports and logs. That is evidence for its documented evaluation workflow, not proof that every passing patch meets every unstated product requirement. Evaluation guide.

For an interview, begin with the release decision. Are you choosing a model for autonomous issue fixing, recommending patches for human review, or measuring a tool's debugging assistance? Those decisions have different tolerances for regressions, unsafe actions and user effort. Keep correctness and authorized behavior as separate dimensions.

Why do you need both target tests and regression tests?

A target test reproduces the behavior the issue asks to repair. Run it on the frozen base and confirm that it fails for the expected reason. Otherwise, a passing result after the patch cannot demonstrate that the issue was fixed.

A regression test exercises previously valid behavior that must remain valid. It passes before the patch and should pass afterward. Existing failures outside the task need a documented baseline disposition; do not silently redefine them as agent mistakes or exclude them only for the favored system.

Consider an original parser exercise. The function accepts a nonnegative integer or a string containing nonnegative decimal digits, with surrounding whitespace permitted. It rejects booleans, negative values, floats and malformed strings. Zero is valid. A baseline implementation wrongly rejects zero because it checks truthiness before validation.

Patch	New zero test	Existing valid/invalid tests	Decision
Unchanged baseline	Fails	Pass	Issue unresolved
Remove truthiness rejection; retain validation	Passes	Pass	Accept on these tests
Return zero for every input	Passes	Fail	Reject shortcut

The constant return satisfies the visible zero example but breaks positive values and invalid-input rejection. This is why a screenshot of one green test is weak evidence.

def core(raw):
    if isinstance(raw, bool):
        raise ValueError("boolean is not a quantity")
    if isinstance(raw, int) and raw >= 0:
        return raw
    if isinstance(raw, str):
        s = raw.strip()
        if s and all("0" <= ch <= "9" for ch in s):
            return int(s)
    raise ValueError("invalid quantity")

def baseline(raw):
    if not raw:
        raise ValueError("missing quantity")
    return core(raw)

def correct_fix(raw):
    return core(raw)

def shortcut(raw):
    return 0

The local fixture executed these known, harmless definitions against zero, positive values, padded digit strings and invalid inputs. The baseline failed the zero contract; the correct fix preserved all checked behavior; the shortcut failed the regression contract. This is a controlled example, not an execution of an external agent or a security test.

How do you keep the evaluator independent of the patch?

Run scoring in a fresh evaluator-controlled workspace reconstructed from the frozen base. Apply the allowed patch there, then inject or mount trusted tests according to the task contract. Check whether test files, configuration, collection rules, or import paths were changed in ways that could suppress the oracle.

Allowing an agent to add helpful development tests can be reasonable. Those additions do not become the only ground truth. A patch that changes a test may also be legitimate when requirements changed, so inspect intent and authorized scope rather than banning every test modification without context.

Ensure tests actually ran. A zero-test collection, a skipped suite, an early process exit, and a reported zero exit code deserve different records. Track expected case IDs, collected counts and individual outcomes. Capture reports through evaluator-owned channels instead of trusting text the candidate process printed.

Primary research: EvalPlus studies how stronger test sets reveal code-generation errors missed by weaker evaluations. Its 2023 paper supports scrutiny of the test oracle; it does not imply that adding arbitrary tests guarantees semantic correctness. EvalPlus paper.

Add meaningful boundary and property checks derived from the specification. For the parser, valid padded strings should equal their unpadded forms, while a float should not silently become an integer. Check the oracle itself against known correct and incorrect implementations. An incorrect hidden test can punish the right patch.

What does the sandbox need to protect?

Separate reproducibility from containment. A pinned environment improves repeatability. A security boundary limits what generated code or tool actions can access. An environment can achieve one goal while failing the other.

Official security facts: Docker's documentation identifies daemon access, host mounts, capabilities and kernel configuration as important security considerations. A container with powerful host access can undermine the intended isolation. Docker security.

Evaluation design recommendation: use disposable workspaces, narrowly scoped filesystem access, bounded CPU/memory/process counts, timeouts, and an explicit egress policy. Keep production credentials and unrelated repositories out. Prepare dependencies before execution where possible so a task does not need unrestricted downloads during scoring.

Do not mount the container control socket or a broad host directory just to make the harness convenient. Different threat models may require a VM or additional isolation. Saying “Docker” is not a complete containment design, and no single list of settings proves a sandbox secure.

Repository text and tool output can contain instructions aimed at the agent. They are task data, not permission to read secrets, change the evaluation contract, or contact external systems. Include representative malicious-instruction fixtures and measure whether the agent respects the tool policy. This is an action-safety evaluation distinct from the patch's functional tests.

How should timeouts and missing attempts affect the score?

Use an explicit outcome ledger. In a synthetic run, 100 tasks were scheduled, 80 received a patch, 75 were graded, and 60 were resolved. The remaining graded 15 were unresolved; five submitted attempts failed infrastructure setup; 20 had no submission.

Reported denominator	Calculation	What it describes
All scheduled tasks	60/100 =60%	End-to-end task completion under the protocol
Submitted patches	60/80 =75%	Resolution among attempts with artifacts
Graded patches	60/75 =80%	Conditional correctness after grading succeeded

All three numbers can be arithmetically correct. Only labeling them makes the report interpretable. Publishing 80% without the other counts conceals operational failures. The script checks the ledger arithmetic, not a real benchmark score.

Distinguish a patch that hangs because it introduced an infinite loop from a worker that never started. Retain logs, rerun infrastructure failures under a predefined rule, and report ambiguous cases. Apply the same exclusion and retry rules to every system, with the original scheduled denominator still visible.

If a task is defective, adjudicate it using evidence independent of the favored result. Report a sensitivity analysis with and without the defect rather than removing difficult tasks after inspecting which model lost.

Is pass@k the same as reliability in one attempt?

No. “At least one successful patch in k attempts” measures the opportunity to find a success after spending a larger budget. It does not tell you how a user selects the successful patch without access to hidden ground truth.

Declare whether the agent can inspect development-test failures and retry, how many tool calls and wall-clock seconds it receives, and whether a human chooses the final artifact. Hidden-test-guided selection changes the protocol. Report one-attempt reliability separately from any best-of-many result, along with cost and selection method.

Repeated outcomes from the same repository or related issues are dependent. Compare systems on the same tasks and retain paired outcomes; examine uncertainty and clustering where relevant. A tiny improvement within one noisy run is not sufficient evidence for a general superiority claim.

Where can benchmark leakage enter?

Training contamination is one route. At evaluation time, an agent may retrieve the merged fix, a later repository snapshot, a copied solution, hidden tests, or earlier traces that reveal the answer. The exact base revision does not prevent leakage if unrestricted search supplies the future pull request.

Primary benchmark context: the original SWE-bench paper frames repository issue resolution as the task. Public issue and repository provenance make revision and access boundaries consequential. Its historical dataset description is not a statement that every modern evaluation is uncontaminated. Original paper.

For a task-solving evaluation, define what documentation and history are allowed. Restrict retrieval to the agreed snapshot when the goal requires it, and log retrieved material. Use fresh, permission-cleared tasks or temporal holdouts where appropriate; split related fixes carefully so near-duplicates do not cross partitions.

Freshness alone is not a contamination guarantee. A renamed issue with the same code and answer can remain a near-duplicate. Record evidence of suspected exposure and avoid claiming that every unexpectedly good result proves memorization. Compare performance on public and genuinely held-out tasks under the same budget.

What would a credible release report contain?

Include the task population, environment identity, agent/tool configuration, budgets, independent tests, scheduled/submitted/graded/resolved counts, failure categories, regression severity and safety violations. Preserve patch hashes and traces so a reviewer can reconstruct a disputed result.

Separate “passes this benchmark” from “safe to merge without review.” Tests sample requirements; human review may still detect architectural or authorization problems. An agent that solves more tasks but takes unsafe actions can be inappropriate for a proposed deployment.

In an interview, explain the parser shortcut and the denominator ledger before proposing a giant platform. They expose the central evaluation contract.

Practice patch and evaluation follow-ups

Question	Skill to demonstrate
Design an Agent Harness and Evaluation System	Define trusted scoring and the release decision.
Design an Evaluation Platform for AI Agents Across Prompt and Model Changes	Compare complete configurations under one protocol.
Review AI-Generated LRU and LFU Cache Changes	Find regressions that plausible patches conceal.
Debug ML pipeline and build text parser	Reproduce failure before validating a minimal fix.
Evaluate Subjective and Nondeterministic Agent Outputs	Explain where deterministic tests stop being enough.

Open the harness exercise and specify what the agent can change, what the evaluator controls, and how a missing artifact enters the result.