Articles
Reranker Interview Exercises: Candidate Recall, Cross-Encoder Scores, and Latency
Practice reranker interview exercises with a three-query candidate fixture, cross-encoder score trace, recall calculations, and latency trade-offs.
A reranker cannot rescue a passage that the first-stage retriever never supplied. That is the decisive observation in many RAG reranker interviews. Before tuning a cross-encoder or celebrating a better top result, label the candidate set, calculate its recall, inspect what the reranker moved, and measure the latency of the whole request path.
This article is an original three-query interview exercise with invented passages, scores, and a planning budget—not a benchmark of any production model. Here the goal is to diagnose two different failures using the same candidate and scoring table: a relevant document missing before reranking, and a present document ranked below a plausible but stale distractor.
The interview prompt: one pipeline, three symptoms
A support assistant retrieves four passages per query, scores those four with a cross-encoder, and passes the top two into answer generation. Some answers omit a necessary exception; others cite an outdated policy. The team proposes increasing the candidate cutoff to eight and upgrading the reranker. You have a 100 ms illustrative pre-generation budget, but no real production traces yet. Explain what you would measure before changing anything, calculate the toy fixture below, and choose an experiment.
Ask which corpus and query population matter, whether passages have relevance grades or only binary labels, whether multiple passages are jointly needed, how document version and access controls are enforced, and whether the final answer needs one supporting hit or complete evidence. A correct score for a forbidden tenant document is not a success. Define the unit of evaluation as a (query, authorized passage, corpus version) pair; freeze that unit while comparing first-stage retrieval and rerankers.
In a two-stage system, the first stage trades some precision for fast candidate recall. A cross-encoder consumes the query and candidate together and produces a score for that pair, then sorts only the supplied candidates. Sentence Transformers' cross-encoder documentation describes this retrieve-then-rerank arrangement. The score is useful for ordering under a particular model; it is not a universal probability of truth. Cohere's reranking guidance likewise cautions against treating score ratios as literal relevance ratios.
Work the labeled candidate fixture
Each query has its own four first-stage candidates in original rank order. The “Relevant?” column is an adjudicated toy label against the current authorized corpus. The cross-encoder values are invented. For Q3, D10 is a stale policy passage that sounds relevant but must not answer the current question.
| Query | Candidate rank / ID | Passage summary | Relevant? | Toy cross-encoder score |
|---|---|---|---|---|
| Q1: damaged shipment refund | 1 / D1 | General refund timeline | No | 0.12 |
| Q1 | 2 / D2 | Current damage-refund procedure | Yes | 0.94 |
| Q1 | 3 / D3 | Return-window overview | No | 0.68 |
| Q1 | 4 / D4 | Shipping label instructions | No | 0.22 |
| Q2: retention policy | 1 / D6 | Audit log retention (different product) | No | 0.15 |
| Q2 | 2 / D7 | Current requested-product retention policy | Yes | 0.89 |
| Q2 | 3 / D8 | Data deletion overview | No | 0.52 |
| Q2 | 4 / D9 | Security glossary | No | 0.11 |
| Q3: current return window | 1 / D10 | Outdated V2 return-window page | No | 0.90 |
| Q3 | 2 / D11 | Current V3 standard window | Yes | 0.77 |
| Q3 | 3 / D12 | Current V3 exception | Yes | 0.72 |
| Q3 | 4 / D13 | Warehouse handling note | No | 0.20 |
There is one more relevant passage, D5, for Q1: a damage exception that the first-stage retriever ranked fifth and therefore did not supply. Q1's gold set is {D2, D5}; Q2's is {D7}; Q3's is {D11, D12}. This lets you separate candidate coverage from sorting quality. D5 is not an omitted table row by accident—it is the retrieval miss the interviewer wants you to catch.
Candidate recall at four is Q1 1/2, Q2 1/1, and Q3 2/2. Across all five query-passage relevance labels, micro recall is (1 + 1 + 2) / (2 + 1 + 2) = 4/5 = 80%. Macro recall across the three queries is (0.5 + 1 + 1) / 3 ≈ 83.3%. Do not mix these denominators. For a product that needs all supporting passages to answer, Q1 has answer-support coverage failure despite a seemingly respectable micro average.
Now sort each query's four candidates by the toy cross-encoder score and keep two. Q1 yields D2 and D3; D5 remains impossible to recover. Q2 yields D7 and D8, so the relevant D7 moves from original rank two to first. Q3 yields D10 and D11; D12 is pushed out while stale D10 wins. Final relevant-passage recall at two is (1 + 1 + 1) / 5 = 3/5 = 60%. This does not prove reranking made the whole system worse: it compares a four-candidate pool with a two-passage context budget. To assess rerank benefit fairly, compare against taking the original top two under the same budget, then inspect per-query answer outcomes.
The original top two already contain D2, D7, and D11—also three of five positives. On this tiny fixture, the reranker does not improve binary recall@2. It does improve the position of D7, but it fails to reject stale D10 and cannot recover D5. A top-line “reranker lifts a hit to rank one” is therefore incomplete. Evaluate relevance, version freshness, and the ability to supply all necessary evidence, not just first-hit rank.
Diagnose the stage that owns each failure
Q1 is primarily a candidate-generation problem. Check whether D5 is indexed, authorized, fresh, and findable under the exact query. If D5 is rank five, raising K to eight might include it, but that is only one option. A better keyword synonym, hybrid lexical+dense retrieval, corrected chunk boundary, or metadata filter may bring it into the existing four-candidate budget. If the relevant passage is not in the index or is wrongly access-filtered, no K increase helps. Inspect retrieval logs and candidate IDs before asking the cross-encoder team to retrain.
Q2 is the clean reranker success shape: the relevant passage was available but originally second. The cross-encoder's pairwise assessment puts it first. Verify that improvement over a larger labeled query set and multiple slices: acronyms, product names, short queries, long questions, and ambiguous terms. Rank movement on one handpicked query is a debugging clue, not a launch result.
Q3 combines a reranker failure with a data-governance issue. D10 is semantically similar but outdated. If a reliable version filter can exclude superseded documents before scoring, apply it at retrieval; do not expect the model to infer policy validity from prose alone. Preserve document version, effective date, and authorization metadata in the candidate payload. If versions conflict but the correct source is unclear, the system should disclose uncertainty or abstain instead of choosing the most fluent passage. Reranker training can include stale-policy hard negatives, but freshness filtering is a separate, deterministic control.
Microsoft's Azure semantic ranking overview makes the same architectural limit explicit for its product: its semantic ranker reranks an existing top-result set rather than searching the entire corpus again. The exact Azure top-result limit is product-specific; do not copy that number into a different cross-encoder stack. The general interview invariant is that downstream ordering cannot raise recall beyond the set it receives.
Make the latency trade-off explicit
The following numbers are a synthetic planning calculation, not measured p95s. Suppose first-stage retrieval takes 35 ms, reranking has 12 ms fixed overhead plus 4 ms per candidate, and context assembly takes 25 ms. With K=4, the arithmetic budget is 35 + (12 + 4×4) + 25 = 88 ms. With K=8, it is 35 + (12 + 4×8) + 25 = 104 ms, past a nominal 100 ms pre-generation target. The extra four candidate scores cost 16 ms in this toy model.
Do not add real component p95 values and call the sum end-to-end p95: tail events can correlate or overlap, and batching changes the curve. Instrument request-level wall time plus stage spans, queue wait, batching, tokenization, model inference, and cache behavior. Report p50/p95/p99 and throughput at a realistic concurrency level. Include candidate count and passage length in traces; K=8 with long passages may behave differently from eight short snippets. Record quality against the same timeout and fallback policy that production will use.
If K=8 raises Q1 coverage but breaches latency, try targeted changes. Improve first-stage recall@4; use hybrid retrieval or query rewriting only on a diagnosed slice; shorten or preselect passages without dropping decisive evidence; batch cross-encoder pairs; choose a smaller model; or apply adaptive K when first-stage confidence and risk justify it. Cache only where query, corpus version, tenant permissions, and freshness allow it. Never trade away access control or silently fall back to a stale result to hit a timing number.
The business decision is a quality-latency frontier, not one maximum score. A reranker with slightly higher nDCG but materially worse p95 and no better answer support may not be worth the cost. Conversely, a higher-latency path could be justified for a high-stakes query category if product policy permits it. Predeclare which slice uses which budget so the team cannot retroactively excuse every slow or wrong answer.
Build an evaluation that cannot hide retrieval misses
Collect representative queries with judged supporting passages; include unanswerable queries, multi-passage answers, near-duplicate versions, permission boundaries, and hard negatives. Freeze corpus snapshots and labels for controlled experiments. Measure candidate recall@K and “all required evidence present” before reranking. Then measure reranked nDCG@2, MRR if first relevant hit matters, precision@2, and support coverage for the actual context size. Finally evaluate generated answers for correctness and citation support. No single ranking metric substitutes for downstream answer quality.
When using a reranker evaluator, ensure it does not secretly insert the gold positive into every candidate list. Sentence Transformers' evaluator documentation exposes an always_rerank_positives setting; a setting that includes positives can be useful for testing score discrimination but cannot estimate end-to-end retrieval-plus-rerank recall. For a realistic pipeline test, preserve exactly what the first-stage retriever returned and count missing positives as misses.
Run paired experiments on the same queries and corpus version: baseline top two, cross-encoder top two at K=4, and cross-encoder top two at K=8. Break down gains and regressions by slice, not only average. Inspect cases where the relevant passage is absent, present but low-scored, present but filtered as stale, or scored correctly yet lost to a final context-length truncation. A trace should let you say which stage excluded D5 or D12.
For an interview answer, show the Q1 calculation first. Then explain Q2 and Q3 as counterexamples to “always increase K” and “a higher score means better evidence.” Finish with a concrete experiment: retrieve K=4 and K=8 on a labeled set; compare candidate recall and final answer support under a measured latency distribution; try a targeted retrieval change for the Q1-like slice before paying for eight cross-encoder calls everywhere.