Long-Context Evaluation Interview Questions: Position Bias and Evidence Retrieval

A model accepts a long document packet and answers a question correctly when the key fact is near the beginning. Move that same fact into the middle, and the answer changes. Did the model run out of context, fail to retrieve evidence, or ignore evidence it actually received? Long-context evaluation interview questions require a controlled experiment before that diagnosis.

The central answer is to separate accepted input length, evidence inclusion, and evidence use. Compare the same task across placements, inspect the final serialized request, and score answers together with their supporting evidence.

This article uses original evaluation fixtures, not measured model accuracy. Primary-paper findings are dated and attributed. No candidate reports establish an employer's assessment format, and no model endpoint was run for the examples.

Does a large context window prove reliable understanding?

No. A context limit describes what an interface or implementation can accept under its rules. Usable context depends on whether the system can perform the required task with the evidence at that length. A single advertised capacity does not establish retrieval, multihop reasoning, contradiction handling, or citation accuracy.

Primary research: Lost in the Middle, first submitted in 2023, examined multidocument question answering and key-value retrieval. The tested models' performance depended on where relevant information appeared, with important middle-position weaknesses. That motivates position-controlled testing; it is not a present-day scorecard for every model. Paper.

State what the application needs. An assistant finding one account code has a different task from comparing policy versions across documents. A repository agent joining a call site with a distant definition has another. Context length should be evaluated against these behaviors rather than against the ability to echo a hidden phrase alone.

Record the model version, serving settings, tokenizer, prompt template, output budget, retrieval configuration and date. Repeat comparisons when any of these change. Do not attribute a deployment regression to the underlying model before checking the surrounding pipeline.

How do you isolate evidence position from other variables?

Create paired examples with the same question, evidence content, distractors and answer. Move the relevant block to different positions while preserving the block multiset. Keep instructions and answer format fixed. Use stable source identifiers so moving a document does not change its citation identity.

This original pilot has 12 blocks. Eleven contain facts about unrelated cases; one says that case K47 has a 14-day return window. Place that block at zero-based indices 0, 6 and 11. Each variant should have exactly one relevant block and the same gold answer.

def make_variants():
    distractors = [f"Case X{i:02d}: window {i+1:02d} days."
                   for i in range(11)]
    fact = "Case K47: window 14 days."
    width = 48
    distractors = [s.ljust(width) for s in distractors]
    fact = fact.ljust(width)
    variants = []
    for position in (0, 6, 11):
        blocks = distractors.copy()
        blocks.insert(position, fact)
        variants.append(blocks)
    assert len({len("\n".join(v)) for v in variants}) == 1
    assert all(sorted(v) == sorted(variants[0]) for v in variants)
    return variants

The local checks verified equal character length, identical block content, unique evidence and invariant gold answers. A deterministic lookup extracted the fact at every placement. That establishes fixture integrity; it says nothing about an LLM's position robustness.

Character equality also does not guarantee equal token counts under every tokenizer or message wrapper. Measure actual serialized tokens for the evaluated system, including separators and metadata. Adjust padding only after that measurement, and preserve the intended semantic content. This small pilot is a generator sanity check, not itself a long-context benchmark.

What should the evaluation matrix vary?

Change one dimension at a time for diagnosis, then include combinations that reflect deployment. Length and evidence position are a starting pair. Distractor similarity, number of required facts, evidence distance, conflicting versions, and missing evidence can expose different failures.

Dimension	Controlled comparison	Failure it can expose
Position	Same evidence early, middle, late	Dependence on location
Length	Same task with additional distractors	Degradation with context size
Evidence count	One fact versus a required pair	Incomplete multihop use
Similarity	Unrelated versus plausible decoys	Confusion among near matches
Availability	Full evidence versus withheld link	Unsupported guessing or poor abstention

Include naturalistic tasks alongside synthetic probes. A synthetic fixture makes one mechanism easier to isolate; a representative document packet tests whether that mechanism matters in actual work. Neither substitutes for the other.

Primary research: RULER's 2024 benchmark extends simple needle retrieval with varied needles, multihop tracing and aggregation. It supports testing more than one-string lookup. Published results concern the models and settings in that study, not a universal current effective-context threshold. RULER.

Do not generate all variations from one reusable template and treat them as independent tasks. Group variants by their underlying question and document set when comparing systems. Otherwise, hundreds of correlated permutations can create misleading confidence.

What does a multihop policy case look like?

Use this original, fictional packet. The question asks for the current return window for case A17 under active policy version v2. The task contract states that superseded policies must not govern the answer.

Source ID	Evidence
D01	Case A17 has customer tier Silver.
D02	Active policy v2 gives Silver customers 14 days.
D03	Superseded policy v1 gave Silver customers 7 days.

The grounded answer is 14 days, supported by D01 and D02. D01 connects the case to the tier; D02 connects the tier to the active policy. D03 is a plausible decoy with the right tier and the wrong version.

Now move D01 and D02 independently, including distant placements. Keep the sources' text and identifiers unchanged. A model that handles one needle may miss one link, prefer the nearest decoy, or quote the right number with the wrong source.

Withhold D01: the tier is unknown, so the answer should abstain under this contract. Withhold D02: only the superseded rule remains, so the current value is unsupported. This is a deliberate missing-evidence task, not permission to silently fall back to v1.

A real policy may specify a fallback, precedence hierarchy or effective-date interval instead. Put that rule in the task and gold annotation. Avoid expecting the model to infer an unstated “newer always wins” rule from document order alone.

How do you separate retrieval failure from evidence-use failure?

Inspect the final assembled input. A retrieval hit in an intermediate log does not prove that the passage survived reranking, permission filtering, deduplication, compression and truncation. Track source spans across each stage and check what was actually delivered.

If the required passage is absent, investigate retrieval and assembly first. If it is present and the answer is wrong, examine selection, instruction following, reasoning and version resolution. If the answer is correct but its citation does not support it, score grounding separately.

These categories narrow investigation; they do not prove one root cause. An apparently present passage might be malformed or contradictory. A “retrieval failure” could originate in a parsing or permission bug upstream of the retriever. Preserve enough evidence to reproduce the classification.

Compare three conditions on the same task: full supplied context, the production retrieved context, and an oracle packet containing only the required evidence plus fixed instructions. Strong oracle performance with weak retrieved-context performance suggests a pipeline problem. Weak oracle performance suggests that shorter context alone may not solve the task. Interpret these as diagnostic interventions, not perfect causal identification of every internal mechanism.

What should be scored besides the final answer?

For the policy exercise, separate answer correctness, support completeness, citation validity and abstention. A strict task success requires all applicable conditions, but component scores explain the failure. Do not award full success for 14 days with a citation to D03.

Evaluate exact answers with normalized units and explicitly accepted formats. For longer explanations, define the propositions that must be correct and which sources support them. If a model judge is used, calibrate it against blinded human labels and inspect disagreements, order effects and grading stability.

Measure unsupported answering when evidence is missing and unnecessary abstention when evidence is complete. A system can avoid hallucination by refusing everything while being unusable. Report both behaviors rather than celebrating one isolated safety percentage.

Primary research: LongBench, introduced in 2023, evaluates multiple long-context task categories in English and Chinese. Its multitask scope is evidence for looking beyond one retrieval probe. The paper's task mix and historical lengths do not prescribe the mix or target language for your application. LongBench.

Choose representative tasks and their weights before viewing system results. Report task-specific outcomes as well as any overall summary; a model's strong summarization score can conceal poor evidence joining in a weighted aggregate.

How do budgets and truncation change the interpretation?

Reserve space for instructions, wrappers and output rather than filling the entire declared context with source text. Document whether the serving interface truncates, rejects or preprocesses oversized input. Verify that the received task still contains both the question and evidence.

Truncation from the front can accidentally turn a middle-position example into a beginning-position example. Retrieval compression may preserve a number while dropping the source version that makes it valid. Save span identifiers and the transformed text so these errors are visible.

When comparing full-context and retrieval-based systems, report both fixed-budget and quality-versus-cost comparisons. A system given a longer input or more generation attempts has a different resource allowance. Include latency, tokens, retries and failure rates; accepted requests alone exclude rejected or timed-out work.

Also log cache and memory behavior where the application reuses context across turns. A stale conversation state can inject an older policy even when the current document packet is correct. Reset or explicitly version state during a controlled test, then evaluate the realistic stateful workflow separately.

What statistical comparison is credible?

For each underlying task, compare systems on matching placement variants and aggregate at the task level. Retain the paired outcomes. Estimate uncertainty with a method that respects shared documents, questions and repeated runs; do not pretend every permutation is independent.

Report cell counts, not just percentages. If one placement contains only ten tasks, one answer changes the score by ten percentage points. Predefine the important comparisons or adjust claims for multiple exploratory comparisons. A visually appealing heatmap without sample sizes can overstate the evidence.

Do not turn this article's fixture checks into a fictional score table. No model was evaluated here. A useful interview answer explains how to collect the observations, what they would establish, and what uncertainty would remain.

How would you propose a mitigation?

Match the change to the diagnosed failure. Missing evidence may call for parsing, indexing, permission or retrieval repairs. Conflicting evidence may require preserving version metadata and explicit precedence. Poor evidence use may justify task-specific prompting, model comparison, structured joins or shorter verified context.

Moving important passages to an edge is a hypothesis to test, not a universal remedy. It can change other evidence positions and does not solve missing facts. Evaluate the mitigation on untouched tasks, including middle-position, decoy and abstention cases, before expanding deployment.

The concise interview answer is: “I will hold the task and evidence constant while varying placement, verify the final input, and score correctness with support and abstention. Then I will compare representative tasks under equal budgets before choosing a mitigation.”

Practice controlled evaluation and retrieval

Question	Follow-up to rehearse
Evaluate a Retrieval-Augmented Generation System	Separate evidence inclusion from grounded answering.
Design and Evaluate a Retrieval-Augmented Generation Pipeline	Preserve source offsets, versions and evaluation splits.
Training LLM Agents for Long-Context, Long-Horizon Tasks: Environments and Rewards	Distinguish long-input use from multistep completion.
Optimize attention for long sequences	Connect resource constraints to a quality evaluation.
Design Retrieval for a Coding Agent: What to Index, Handling Noise, and Metrics	Attribute retrieval failures before changing the generator.

Open the RAG evaluation task and describe one paired position test and one missing-evidence test with explicit success criteria.

Sources and Further Reading

• Lost in the Middle: How Language Models Use Long Contexts,2023
• RULER: What's the Real Context Size of Your Long-Context Language Models?,2024
• LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding,2023