Articles
Hybrid Search Interview Questions: BM25, Embeddings, Fusion, and Evaluation
Practice hybrid search interview questions on BM25, embeddings, RRF, candidate recall, reranking, permissions, and evaluation with worked ranking examples.
A search system finds a product when someone types its exact model number, but misses it when they describe what it does. After adding embeddings, the description works—and the exact model number starts returning similar products instead. Hybrid search interview questions begin where that trade-off becomes a design decision.
The central answer: combine lexical and semantic retrieval when their candidate sets contain useful differences. Then evaluate candidate coverage separately from the order of the final results. A sophisticated fusion function cannot rank a document that neither retriever found.
This guide uses official documentation for implementation facts, a research paper for evidence about fusion methods, and clearly labeled synthetic exercises for interview practice. It makes no claims about company interview frequency or candidate-reported questions. To rehearse the decision aloud, start with Choose Tokenization and Dense or Sparse Retrieval.
What does BM25 contribute when embeddings already capture meaning?
Official fact: Elasticsearch describes BM25 as a lexical ranking method using term frequency, inverse document frequency, and document-length adjustment. Its vector retrieval instead scores similarity between vector representations. These are different relevance signals, not two measurements on a shared numerical scale. Elastic ranking documentation
Interview reasoning: imagine an equipment catalog containing AX-410 and AX-410B. A query for the first identifier has an exact constraint. A query for “a compact pump that tolerates salt water” expresses a need that may use different words from the product description. Both deserve evaluation, but their failure modes differ.
BM25 is not an exact-match guarantee. Tokenization might split the identifier, remove punctuation, or normalize two distinct values into similar tokens. An exact identifier field or explicit filter may be necessary. Likewise, a dense model can retrieve exact identifiers successfully; the risk is that this behavior is insufficiently reliable for the product's requirements.
Explain what you would inspect before adding infrastructure: analyzed tokens, searchable fields, representative nearest neighbors, missing document text, and corpus coverage. If the correct product was never indexed, changing fusion weights addresses the wrong stage.
A useful follow-up is whether synonyms alone would solve the vocabulary mismatch. They may be enough for a small, stable domain. Hybrid retrieval becomes more convincing when the semantic branch adds relevant documents that a maintained lexical baseline still misses.
Why is adding BM25 and cosine scores a weak default?
Consider a synthetic scoring exercise. Document A has BM25 score 12 and cosine similarity 0.30. Document B has BM25 score 2 and similarity 0.90. Adding the raw values gives A 12.30 and B 2.90. Calling this an equal-weight blend hides the numerical dominance of one score scale.
A weighted score combination can still be reasonable. State how scores are transformed, how missing branch scores are handled, which judgments tune the weight, and what happens when the query distribution changes. A 50–50 coefficient is not evidence of equal practical influence.
Per-query min-max normalization introduces its own edge cases. What happens when all returned scores are equal? Does one outlier compress every other score? A candidate outside a branch's cutoff has no observed score in that list; assigning zero should be a documented convention, not an accidental consequence of a join.
Research evidence: Bruch, Gai, and Ingber's study compares convex score combination with reciprocal rank fusion. In their experimental settings, the learned convex combination outperformed RRF, and RRF was sensitive to its parameters. That supports testing alternatives; it does not establish one universal winner for every corpus. An Analysis of Fusion Functions for Hybrid Retrieval
How do you calculate reciprocal rank fusion?
Official fact: RRF sums reciprocal rank contributions from the lists containing a document. Rank starts at one. Elasticsearch exposes a ranking constant and a per-retriever candidate window; its documented default ranking constant is 60. Elastic RRF reference
For an unweighted two-list example:
RRF(document) = sum over returned lists of 1 / (c + rank)
Absent from a list: contribute zero
Use c = 2 solely for this synthetic hand calculation. The lexical list is A, B, C. The semantic list is D, B, A.
Ranks below are lexical / semantic; a dash means absent. For A, the calculation is 1/3 + 1/5 = 0.5333.
| Doc | Ranks | Score |
|---|---|---|
| A | 1 / 3 | 0.5333 |
| B | 2 / 2 | 0.5000 |
| C | 3 / — | 0.2000 |
| D | — / 1 | 0.3333 |
The fused order is A, B, D, C. D wins the semantic branch but loses to documents supported by both lists. This is the behavior to explain, rather than merely recalling the formula.
Now truncate each input to two documents. A loses its semantic contribution, so B moves above A. A and D tie at 1/3. The candidate window has changed the answer even though neither branch changed its underlying ordering. Choose a deterministic tie-breaker, such as stable document ID, for reproducible tests.
RRF discards score margins. A narrow first-place win and an overwhelming first-place win contribute the same rank term. That makes it convenient when score scales differ, while also discarding potentially useful confidence information.
Is a high RRF score evidence that the answer is relevant?
No. Official documentation: OpenSearch cautions against treating an RRF threshold as a relevance threshold or comparing these scores across queries. The score depends on rank positions, the number of contributing clauses, their weights, and the ranking constant. OpenSearch RRF interpretation
Interview inference: if every available document is irrelevant, one can still rank first in both lists. Consensus between retrievers does not establish that the corpus contains an answer.
For a no-answer decision, define a separately evaluated policy. It might use a relevance classifier, a validated reranker threshold, explicit evidence requirements, or a request for clarification. Measure false acceptance and false rejection on answerable and unanswerable queries. Do not turn a fusion implementation detail into a probability.
The same distinction matters during an outage. If one branch disappears, a document loses that branch's contribution. A fixed threshold can suddenly reject results for operational reasons. Log which branches returned successfully and evaluate fallback quality as its own condition.
What can reranking fix, and what cannot it fix?
Official fact: multistage retrieval applies a more expensive reranker to a smaller set of candidates produced earlier in the pipeline. Elastic ranking and reranking
The following original coverage exercise makes the limit concrete. Suppose five documents are judged relevant to a query. The lexical branch retrieves two of them; the semantic branch retrieves three. One relevant document appears in both branches.
Relevant documents in union = 2 + 3 - 1 = 4
Candidate recall = 4 / 5 = 0.80
A reranker operating only on that union cannot recover the fifth relevant document. It can improve the placement of the four available ones. If the final display allows at least four results, its maximum recall is still 0.80; if it displays only three, the maximum falls to 0.60.
Do not confuse candidate overlap with relevant-candidate overlap. Two branches might return many identical irrelevant documents. Inspect the added relevant documents per branch, not just the size of the union.
Also define the evaluation unit. Five overlapping chunks from one document do not necessarily represent five independent answers. If the product shows documents, aggregate or deduplicate at that level and judge document-level relevance. If a RAG system needs passages, track passage relevance and source diversity separately.
How would you evaluate a hybrid-search proposal?
Proposed interview experiment: construct paired queries that express the same information need differently. One uses the exact identifier or terminology; the other uses a natural-language description. Keep the intended relevant documents consistent where appropriate, and include ambiguous queries where the descriptions genuinely change the intent.
| Query slice | Example | Failure to investigate |
|---|---|---|
| Exact identifier | AX-410 installation manual | Analyzer or identifier confusion |
| Paraphrase | How do I mount the compact saltwater pump? | Vocabulary mismatch or missing context |
| Mixed constraint | AX-410 outdoor mounting requirements | Semantic matches that ignore the identifier |
| No answer | AX-999 service history in a catalog without it | Confident retrieval of an unrelated neighbor |
| Restricted content | An otherwise relevant private service memo | Unauthorized candidate exposure |
These examples are invented, not benchmark observations. Label relevance against a fixed corpus snapshot and distinguish unjudged documents from known irrelevant ones. Include candidates from multiple systems in the judgment pool so the baseline does not define every possible relevant answer.
Compare lexical only, dense only, their union plus RRF, a tuned score blend, and the strongest justified reranking variant. Tune on a development set, then report a held-out result. Near-duplicate query paraphrases should not leak across splits and make generalization look easier than it is.
Measure candidate recall before reranking. Measure nDCG at the final display cutoff for graded relevance, or another metric that matches the actual task. Report slice results alongside the aggregate, with uncertainty appropriate to the sample size. A gain on paraphrases can conceal a regression on exact identifiers.
Keep latency, request cost, and candidate budgets visible. Comparing a 20-candidate baseline with a 500-candidate hybrid system without reporting the extra work confounds architecture with budget. A useful conclusion states the additional relevant coverage bought by the additional cost.
Which production details belong in the answer?
Design recommendations: apply consistent authorization and eligibility constraints to both branches, and recheck access before content is exposed. A ranking score must never override a permission failure. Filtering after a small top-k retrieval can leave too few eligible results; measure filtered-query recall and inspect the engine's supported filtering behavior.
Use stable document and chunk IDs across indexes. Otherwise, fusion can count two representations of the same passage as different results. Record source version, embedding model version, and indexing timestamps to diagnose cases where lexical retrieval sees a new revision but the vector branch returns an older one.
Give each branch a deadline within the overall request budget. Define whether a timeout produces a labeled lexical-only fallback, a retry within budget, or an error. Track timeout rates, partial responses, and tail latency rather than averaging away the slow queries.
For debugging, retain branch ranks, candidate counts, filter decisions, fusion settings, and the final ordering in appropriately access-controlled traces. That lets you distinguish “never retrieved,” “removed as ineligible,” and “retrieved but ranked too low.” Each points to a different intervention.