Articles
Learning-to-Rank Interview Questions: Pairwise Losses, Position Bias, and Offline Metrics
Practice learning-to-rank interviews with pairwise losses, position bias, worked NDCG calculations, query grouping, and offline evaluation limits.
Learning-to-rank interview questions usually become interesting when three things stop agreeing: the training loss, the observed clicks, and the metric reported offline. A pairwise loss can improve while an important top-ranked result gets worse. Clicks can favor a less relevant document because users see it more often. A higher NDCG can depend on an easier candidate set rather than a better ranker.
For machine learning candidates, a strong answer keeps those distinctions explicit. Define the query group, explain the preference signal, choose a loss, and evaluate the resulting order under realistic exposure and candidate constraints. Use Rank songs by pairwise user preferences to practice explaining the objective before naming an algorithm.
Evidence boundary: Algorithm descriptions use the original RankNet research and official XGBoost documentation. Position-bias discussion draws on primary counterfactual learning-to-rank research.
What exactly is the model ranking?
Start with the unit of competition. In search, documents compete within a query's candidate set. In a feed, items may compete within a user's request. A relevance label of three for one query does not necessarily mean that document should outrank every document labeled two for another query.
A training example therefore needs a group identifier, item features available at prediction time, and a defined target. State whether that target is human relevance, a click, a purchase, or a longer-term outcome. They encode different preferences and different observation biases.
Pointwise methods predict a label for each query-item pair. Pairwise methods learn relative preferences between two items in a group. Listwise methods use a list-level objective or distribution. These categories describe how supervision is organized; they do not by themselves settle which method fits the product.
Official XGBoost learning-to-rank documentation makes query grouping central to its ranking interface. Keep each query group intact and follow the library's expected ordering and group metadata. A ranking loss receiving incorrect groups is optimizing the wrong comparisons, even if training completes normally.
How does a pairwise logistic loss work?
Suppose item A should rank above item B within the same query. The model assigns scores s_A and s_B. A simple pairwise logistic model estimates the preference probability from their difference:
P(A above B) = sigmoid(s_A − s_B)
Loss = log(1 + exp(−(s_A − s_B)))
This is the unit-scale, certain-preference case. The original RankNet paper develops probabilistic pairwise training using gradient descent. The score difference, rather than either score's absolute value, determines this pair's predicted preference.
In a synthetic example, equal scores give probability 0.5 and loss about 0.693. If A's score exceeds B's by two, the probability is about 0.881 and loss about 0.127. Reversing that difference gives probability about 0.119 and loss about 2.127. The loss penalizes confidently ordering the pair incorrectly.
Add the same constant to both scores and the difference does not change. That illustrates why a ranking score is not automatically a calibrated probability of clicking or buying. Scaling differences does change the loss, even when it preserves the displayed order.
Explain how pairs are selected. A group with many items can produce quadratically many pairs, and pairs with tied labels may need different handling from strict preferences. Sampling reduces cost but changes which comparisons dominate training. Record whether queries contribute equally or large groups receive more influence through more pairs.
For numerical implementation, use a stable softplus or equivalent library operation rather than directly exponentiating an arbitrarily large negative difference. You should be able to explain both the mathematical objective and how a naive implementation can overflow.
Why does reducing pairwise errors not guarantee better NDCG?
A swap near the top can matter much more than a swap at the bottom. A plain count of pairwise inversions does not encode that position-dependent cost. It also does not necessarily reflect the size of the relevance difference.
XGBoost's official ranking guide describes LambdaMART objectives that weight pairwise training signals using ranking-metric changes. The distinction between rank:pairwise and rank:ndcg matters: naming both “pairwise ranking” can hide how the objective prioritizes errors. Check the configured objective rather than assuming the default matches your intended metric.
Here is an explicit synthetic NDCG example. Define gain as 2^relevance − 1, discount as 1 / log2(position + 1), and evaluate the first three positions. For relevance grades [3, 2, 0], the ideal discounted cumulative gain is approximately 7 + 3 / log2(3) = 8.893.
| Ranked relevance grades | NDCG at 3 under this definition |
|---|---|
[3, 2, 0] | 1.000 |
[2, 3, 0] | 0.834 |
[3, 0, 2] | 0.956 |
The last two lists each contain one inversion relative to the ideal ordering, yet their metric penalties differ substantially. State the gain function and cutoff before quoting a value: linear gain and exponential gain answer differently weighted relevance questions.
This does not prove one training objective always wins. Label quality, pair sampling, model capacity, optimization, and the actual business target still matter. Use a metric-aware objective as a hypothesis to evaluate, not a substitute for measuring the relevant operating point.
How does position bias distort click labels?
A click is an interaction between relevance and exposure. A user cannot click an item they never examine. Treating every unclicked result as equally irrelevant lets the previous ranking policy influence the next model's target.
For an illustrative examination model, suppose P(click) = P(examine) × P(click | examine). A top-position document has examination probability 1.0 and conditional click probability 0.4, producing expected CTR 0.4. A lower-position document has examination probability 0.2 and conditional click probability 0.8, producing expected CTR 0.16. Raw clicks favor the item with lower conditional appeal.
This is a deliberately simplified model, not an empirical law. Real users can show trust bias, snippet effects, dependent browsing, and other behavior that the multiplication model does not capture. Separate assumptions from measurements when interpreting the example.
Joachims, Swaminathan, and Schnabel's primary research studies learning from biased feedback using propensity weighting. Its relevance here is that the observation mechanism must be addressed explicitly. More logged clicks do not automatically remove a systematic exposure bias.
Merely adding displayed position as an ordinary serving feature does not establish debiasing. The position is an output of the ranking decision you are trying to make. A model may use position within an explicit click or examination model, but you must explain how it recovers the desired relevance signal and what features are available when scoring candidates.
What can inverse propensity weighting fix, and what can it miss?
Under an appropriate observation model, inverse propensity weighting gives more weight to feedback that was less likely to be observed. If examination probability is 0.2, an observed event may receive weight five. That compensates for unequal opportunity under the model's assumptions.
It also creates variance. A very small propensity produces a very large weight, allowing a few examples to dominate. Clipping weights can improve stability but introduces a bias-variance trade-off. Report the clipping policy and sensitivity instead of describing clipped estimates as automatically unbiased.
You need credible propensity estimates and support: relevant actions must have some chance of appearing in the logged data. A deterministic policy that never shows an item in a context cannot reveal that item's outcome there through reweighting alone. Randomized exposure or carefully designed interventions can improve identification, subject to product constraints.
XGBoost documents an optional position-debiasing approach called Unbiased LambdaMART and currently marks the feature experimental. That is a specific implementation with assumptions and configuration, not a universal switch that makes every click dataset unbiased. Validate the feature and label pipeline, and check whether the method addresses the bias present in your logs.
When evidence is insufficient, a useful interview answer says what additional data is needed: displayed candidates, positions, logging policy, exposure probabilities where available, and a clear outcome window. It does not invent an examination curve from the observed click rates alone.
What does NDCG measure, and what does it leave out?
NDCG divides a ranked list's discounted gain by the ideal gain for the same evaluation problem. The normalization makes scores easier to compare across queries with different relevance totals, but only after label, gain, candidate, and cutoff conventions are defined.
Explain all-zero relevance queries. Their ideal gain is zero, so the mathematical ratio is undefined without a convention. Libraries can choose different treatments; XGBoost documents its behavior and alternative metric forms in its parameter reference. Report the convention because changing how these queries enter the average can change the headline result.
Also identify the averaging unit. A macro average gives each query equal weight. A traffic-weighted average emphasizes common requests. Neither should be silently substituted for the other. Segment head and tail queries if the product needs both, and include sample sizes.
NDCG does not directly measure latency, diversity, safety, fulfillment, or satisfaction after a click. It also depends on the relevance judgments available. Unjudged items are not necessarily irrelevant, and a candidate set built from one retrieval system can favor that system's coverage.
A ranker evaluated only on retrieved candidates cannot recover relevant items missing from retrieval. Measure candidate recall separately. When comparing rankers, use a controlled candidate protocol; when comparing end-to-end systems, include retrieval differences rather than attributing every gain to ordering.
How would you design the offline comparison?
Freeze the evaluation definition before tuning. Specify the request population, query grouping, relevance labels, candidate eligibility, and cutoff. Split by query identity or another appropriate grouping when repeated queries would leak information, and use time boundaries when the deployment question concerns future traffic.
Keep features valid as of the request. Future clicks, final item popularity, or labels generated after a user's session cannot become historical serving features. A model can achieve impressive ranking metrics by learning these leaks rather than relevance.
Evaluate both a simple baseline and the candidate model. Inspect a paired per-query metric difference, then estimate uncertainty with a resampling unit that respects dependence. If many requests belong to the same user or session, treating them as independent can understate uncertainty.
Examine regressions, not just the mean. Read a sample of queries where the top result changed, especially high-value and low-frequency intents. A metric improvement can coexist with a harmful repeated failure pattern that a small aggregate average obscures.
Finally, connect offline progress to a launch hypothesis. Define a product outcome, latency guardrail, and rollback condition for an online experiment. A lower pairwise loss and higher offline NDCG are useful evidence, but neither alone establishes that users prefer the resulting experience.