Search Relevance Interview Case: Better Clicks, Worse Results

A search experiment raises click-through rate from 20% to 24%, while successful tasks fall from 12% to 10%. Would you ship it? A strong search relevance interview case answer starts by separating the action the interface encourages from the outcome the user wanted. More clicking can mean better discovery, more attractive snippets, or more work to find an answer.

The useful response is a diagnosis: define the population and denominator, inspect relevance independently of clicks, and identify an experiment that could distinguish competing explanations.

This is an original, synthetic interview case. Research findings and official documentation are identified where used. No candidate reports establish a particular employer's interview format here.

What exactly improved, and what got worse?

Suppose a shopping search experiment changes result ordering and snippet text together. Each arm records 10,000 eligible search sessions. Control has 2,000 sessions with at least one result click and 1,200 sessions with a completed purchase attributable under a predefined rule. Treatment has 2,400 clicked sessions and 1,000 successful sessions.

Here, CTR means clicked sessions divided by eligible search sessions, not total clicks divided by document impressions. Task success means successful sessions divided by that same eligible population. CTR increases four percentage points, or 20% relative. Success decreases two percentage points, or about 16.7% relative. These are descriptive differences; the aggregates alone do not supply a valid significance test when users have repeated sessions.

The product objective matters. A documentation search might define success as finding a valid answer without opening a result. A purchase is inappropriate for that product. A shopping search might need contribution margin, cancellations, and eventual returns in addition to checkout. Establish the decision the metric supports before debating algorithms.

Ask whether eligibility is measured before treatment can affect it. If treatment causes users to search again, per-search results describe a changing set of search sessions. Also report outcomes per assigned user, including users who never search. That preserves a view of the randomized population rather than quietly conditioning on treatment-induced behavior.

Why are clicks an imperfect relevance label?

Primary research: Joachims and colleagues' 2005 clickthrough study used eye tracking and relevance judgments to examine implicit feedback. Its findings support treating clicks as informative but biased observations, rather than direct absolute relevance labels. This is historical mechanism evidence, not a claim about every present-day interface. Primary paper.

A first-position result receives more opportunity to be seen. A compelling title can attract a click despite an unsuitable destination. Conversely, a useful answer displayed directly in the results can reduce clicking. A result that appears relevant but disappoints may lead to a quick return and a second click. Counting those actions as independent successes rewards the effort the user had to expend.

For this case, three hypotheses deserve different tests:

Hypothesis	Observation that would help distinguish it
Attractive snippets increase low-quality clicks	More quick returns and fewer successful outcomes on the same query/result pairs
The ranker promotes superficially appealing documents	Lower blinded relevance grades at prominent positions
The query population changed	Stable or declining within-query-type performance alongside different traffic weights

A quick return is itself a proxy. Someone may obtain a phone number immediately, then leave satisfied. Inspect sampled journeys and task outcomes instead of assigning a universal dwell-time cutoff. State the uncertainty: the table gives plausible mechanisms, not conclusions already established by the dashboard.

How would you test relevance without using CTR as ground truth?

Official documentation: Elasticsearch's ranking evaluation interface uses representative queries and query-specific document judgments to compute retrieval metrics, including discounted cumulative gain. Its guidance makes the information need part of the judgment contract. That supports a judged evaluation set; it does not make one metric a complete product objective. Ranking evaluation documentation.

Construct a small evaluation pool from documents retrieved by both systems. Give assessors the query and enough context to understand the intended task. Hide the originating system and its displayed rank, randomize presentation, and define grades with examples. For a specific product query, grade 3 might mean the exact available item, grade 1 a related but incompatible item, and grade 0 an irrelevant item.

Include tail queries, misspellings, ambiguous queries, and important low-volume tasks. A set consisting only of popular easy queries can conceal the harm under investigation. Record assessor disagreement and adjudicate a sample; disagreement may reveal an unclear information need rather than careless judging.

Do not automatically label unjudged results irrelevant. That can penalize a system for finding new useful documents outside the assessment pool. Report judgment coverage, expand the pool, and keep the missing-label convention explicit. Freeze the evaluation split before iterative tuning so it can still test generalization.

Can NDCG expose a rank-order regression?

Consider three judged documents with grades 3, 2, 0. System A returns those grades in that order. System B returns 0, 3, 2. Using exponential gain and logarithmic discount, define:

DCG@3 = sum((2**grade - 1) / log2(rank + 1))

Rank starts at one. The ideal ordering defines IDCG for this judged candidate set; NDCG is DCG divided by IDCG. Specify these conventions because implementations can use different gain settings and judgment policies.

Rank	System A grade	System B grade
1	3	0
2	2	3
3	0	2

A has DCG approximately 8.893 and NDCG 1.000. B has DCG approximately 5.917 and NDCG 0.665. The irrelevant first result wastes the most valuable position even though both lists contain the same documents. If B generates more clicks, that observation does not overturn the judgment-based ranking loss.

from math import log2

def dcg(grades):
    return sum((2**g - 1) / log2(i + 2)
               for i, g in enumerate(grades))

def ndcg(grades):
    ideal = dcg(sorted(grades, reverse=True))
    return dcg(grades) / ideal if ideal else None

assert ndcg([3, 2, 0]) == 1.0
assert round(ndcg([0, 3, 2]), 3) == 0.665
assert ndcg([0, 0, 0]) is None

The final assertion makes an all-zero judgment case undefined under this exercise's convention. Decide whether production excludes such queries or uses another documented rule. Also distinguish ranking quality within retrieved candidates from candidate recall. Neither system can rank a relevant document it failed to retrieve.

How can overall CTR rise when every query cohort declines?

Now examine a second, separate synthetic dataset. Navigational queries have relatively low CTR because some answers are visible immediately. Exploratory queries encourage more clicks. Traffic shifts toward exploratory queries during a campaign.

Query cohort	Before share	Before CTR	After share	After CTR
Navigational	80%	10%	20%	9%
Exploratory	20%	50%	80%	45%

Before pooled CTR is 0.8×0.10 + 0.2×0.50 = 18%. After pooled CTR is 0.2×0.09 + 0.8×0.45 = 37.8%. Both cohort rates declined. Their weights changed enough to reverse the aggregate conclusion.

Using the original 80/20 mixture, the after rates imply 0.8×0.09 + 0.2×0.45 = 16.2%. This standardized summary answers how the observed cohort rates compare under a fixed mixture. It is not automatically a causal treatment effect: cohorts must be comparable, and unmeasured differences can remain inside them.

Distinguish an external marketing shift from one induced by the ranking change. Standardizing away a treatment-induced change can remove part of the effect you wanted to measure. Keep the assigned-population result, the raw session result, and the fixed-mix diagnostic separate, with clearly named estimands.

What data would you request before blaming the model?

Request one row per eligible search session with assignment, stable user identifier, timestamps, query type, result IDs and positions, rendering version, click events, and the eventual outcome. Avoid unnecessary raw personal information in a shared analysis extract. Preserve the metric definition and instrumentation version alongside the data.

Audit missing impressions, duplicate click retries, delayed purchases, and sample-ratio mismatch. A logging change that sends two events for one interaction can imitate a ranking improvement. Compare exposure counts and event joins before calculating increasingly sophisticated estimates.

Inspect latency and zero-result rates by query cohort. A treatment can raise CTR among surviving successful requests while timing out on difficult requests omitted from the denominator. A session remains eligible even when the request fails if that is the agreed product contract.

Build uncertainty estimates at the randomization unit. User-level assignment with multiple sessions requires preserving that dependence in aggregation or resampling. Do not treat 2,400 clicked sessions as 2,400 independent people. Predefine important slices and avoid turning dozens of exploratory comparisons into unsupported subgroup discoveries.

Would propensity weighting solve the click bias?

Primary research: the 2016 paper Unbiased Learning-to-Rank with Biased Feedback develops a propensity-weighted framework for biased implicit feedback. Its correction depends on the exposure process being modeled or estimated appropriately. Paper.

Interview inference: known randomized position swaps can help estimate examination effects under a stated model. Logged rank alone does not reveal a reliable propensity. Very small exposure probabilities produce unstable weights; unsupported positions cannot be recovered by division. State overlap assumptions, estimator validation, and variance controls before proposing weighting as the answer.

Even a good examination correction does not convert a misleading snippet click into completed task success. Match the method to the bias it addresses. A click model and an outcome model can serve different purposes.

What experiment and launch decision would you propose?

Separate ranking and snippet changes with a factorial design if traffic and interference constraints allow it. A smaller ranking-only experiment with fixed snippets may be easier to interpret. Randomize at a stable unit, maintain consistent assignment, define exposure, and verify logging before ramping.

Choose task success as the primary outcome for this shopping case, with CTR as a diagnostic. Predefine safety margins for cancellations, latency, and important query slices. Use a mature attribution window, practical effect thresholds, and an analysis plan appropriate to the assignment structure. A low p-value on CTR does not compensate for violating the success criterion.

My recommendation for the supplied numbers is to pause expansion and investigate the success decline. That is a case-specific judgment, not a universal rule that every conflicting metric means immediate rollback. Severe user harm warrants faster action; an immature purchase window may instead require waiting for complete labels while limiting exposure.

An interview-ready answer is: “Clicks increased, but the user outcome declined. I will verify denominators and logging, compare blinded relevance and fixed query slices, then isolate ranking from presentation. I will ship only if the predefined task outcome and guardrails support that decision.”

Practice the follow-up reasoning

Question	Follow-up to rehearse
Evaluate Product-Ranking Algorithm with Precision and Recall Metrics	Separate relevance judgments from online outcomes.
Design and analyze an A/B test	Keep repeated sessions aligned with user-level assignment.
Diagnose Low CTR in an Advertising Campaign Funnel	Distinguish aggregate facts from causal hypotheses.
Plan and analyze a ranking A/B test	Specify carryover, interference and rollout criteria.
Evaluate new-product notification feature	Resolve clicks-versus-outcome conflicts explicitly.

Open the product-ranking exercise and explain one measurement check, one relevance check, and one experiment before suggesting a new model.

Sources and Further Reading

• Joachims and colleagues: Accurately interpreting clickthrough data as implicit feedback,2005
• Joachims, Swaminathan and Schnabel: Unbiased Learning-to-Rank with Biased Feedback,2016
• Elasticsearch: Ranking evaluation and relevance judgments