Articles
Negative Sampling Interview Questions: In-Batch Negatives, False Negatives, and Bias
Practice negative sampling interview questions covering in-batch negatives, false negatives, popularity bias, hard mining, and evaluation.
Negative sampling makes an otherwise expensive retrieval objective trainable. It can also teach the model the wrong boundary. When an interviewer asks about in-batch negatives, the strongest answer does more than say “every other item in the batch is negative.” It asks how the batch was formed, which unlabeled pairs are actually relevant, and how the sampled training task differs from full-catalog serving.
This guide builds that answer around a two-tower recommendation system. It traces one batch through the contrastive loss, identifies false negatives and popularity bias, and specifies diagnostics before recommending harder or more numerous negatives.
Start with Design a production RAG system if you want a related retrieval exercise with different document, permission, and evaluation constraints.
Evidence boundary: The method claims below come from primary word2vec, YouTube recommendation, CLIP, sampling-correction, and debiased-contrastive research. The batch example, debugging checklist, and interview answer structure are original preparation material.
What an interviewer is testing
A useful answer resolves six urgent questions:
- What event creates a positive pair, and what relevant behavior is missing from the label?
- From which distribution are negatives drawn?
- Does the loss correct for that sampling distribution?
- Which sampled negatives may actually be relevant, unobserved positives?
- How do batch construction and distributed training change the negative pool?
- Which offline slices and online outcomes would expose a biased sampler?
The article-specific thesis is: a negative sampler defines the classification problem the model learns. In-batch sampling is computationally efficient, but its batch frequency, false-negative rate, and exposure mechanism must match or be corrected toward the serving objective.
Separate three ideas called negative sampling
The term covers related but distinct techniques.
In word2vec, negative sampling replaces an expensive vocabulary-wide objective with binary discrimination between observed word-context pairs and samples from a noise distribution. Mikolov and coauthors introduced the objective as a simpler alternative to hierarchical softmax; the historical paper does not imply that every modern recommender uses the same distribution or loss.
In sampled softmax, a small class set approximates a very large softmax. The implementation may adjust logits for the proposal probability so the sampled objective estimates the intended full-class objective more faithfully. TensorFlow's candidate-sampling reference also defines an “accidental hit” when a sampled class matches a target class.
In contrastive dual encoders, the other paired items in a minibatch often become in-batch negatives. CLIP's symmetric image-text objective is a prominent primary example: matched batch pairs are positives, while nonmatching positions provide contrast. Recommendation and search systems use similar score matrices, but their unlabeled pairs can have different semantics.
An interview answer should name the exact setting rather than transfer a rule from one of these objectives to another.
Walk through one in-batch loss
Suppose a batch contains four user-query anchors and four clicked items:
| Anchor | Labeled positive | Other items treated as negatives |
|---|---|---|
| running shoes | Trail Runner | Road Runner, Hiking Boot, Dress Shoe |
| marathon shoes | Road Runner | Trail Runner, Hiking Boot, Dress Shoe |
| hiking footwear | Hiking Boot | Trail Runner, Road Runner, Dress Shoe |
| formal shoes | Dress Shoe | Trail Runner, Road Runner, Hiking Boot |
The user tower produces anchor embeddings q_i; the item tower produces v_j. A score matrix contains all dot products, often divided by temperature tau:
s_ij = q_i dot v_j / tau.
For row i, the diagonal j=i is the labeled positive. A row-wise cross-entropy pushes that score above the other batch items. A symmetric objective may also classify the matching anchor for each item column.
This is efficient because one encoder pass for B pairs creates roughly B(B-1) off-diagonal comparisons. Cross-device gathering can enlarge the pool further. But the mathematical shortcut creates a data question: are the off-diagonal pairs valid negatives for the intended notion of relevance?
In this batch, Trail Runner may be relevant to “marathon shoes” even though it was clicked by another user. Treating it as negative pushes a potentially useful pair apart. That is a false negative under semantic or preference relevance, even though it is not the recorded positive for that row.
Unclicked does not mean irrelevant
Implicit-feedback labels mix preference with exposure. A user cannot click an item they never saw, and may ignore a relevant item because of position, price, availability, fatigue, or an already-satisfied need. Sampling from all unclicked catalog items silently labels many unknowns as negatives.
The false-negative definition must match the product target:
- If the target is “the exact next clicked item,” an alternative relevant item is still not the recorded class.
- If the target is “any item the user would find useful,” the same alternative should not be repelled strongly.
- If the target is purchase, returns and delayed conversions may revise the positive label.
- If items are substitutes, two highly relevant products can compete for one observed click.
Chuang and coauthors' NeurIPS work formalizes sampling bias in contrastive learning when random negatives can share the anchor's true class. Their correction is research evidence that false negatives can matter under their studied settings; it is not a universal production recipe for recommendation data.
Batch frequency creates popularity bias
When batches are formed from logged positive interactions, popular items appear more often. They consequently serve as negatives for more anchors. That changes the sampled class distribution away from uniform catalog sampling.
The effect is not simply “popular items get higher scores” or “lower scores.” They receive more positive updates because they are clicked more and more negative competition because they appear in more batches. The resulting bias depends on the loss, proposal correction, data, and item frequency.
Google's 2019 sampling-bias-correction paper specifically studies two-tower recommendation with in-batch negatives under a power-law item distribution. It proposes estimating item frequency and correcting the batch loss. A later Google paper mixes batch and uniform negatives to address selection bias in an app recommender. These are primary system case studies, not proof that one mixture works for every catalog.
In an interview, write the observed sampler as q(item). If the intended objective assumes another distribution, explain whether you will correct logits using q, importance-weight the loss, change the sampler, or deliberately optimize a different objective. A proposal correction can reduce one bias while increasing variance; it does not fix mislabeled relevance.
Hard negatives help only when they are truly negative
Random negatives soon become easy: their scores are already low and gradients contribute little. Mining items with high model scores can focus learning on confusable alternatives. Sources may include approximate-nearest-neighbor retrieval, a previous model, the current batch, or items exposed but skipped.
Hardness and correctness are separate axes. Nearest neighbors are exactly where false negatives, near duplicates, alternate language versions, and acceptable substitutes concentrate. A sampler that chooses the top-scoring unlabeled item can punish the model for learning semantic similarity.
Use a bounded policy:
- Exclude known positives across a suitable history window, not only the current label.
- Deduplicate item families, variants, and repeated content when the product considers them equivalent.
- Filter by availability, locale, permissions, and timestamp as they existed at the training event.
- Draw from a difficulty band instead of only the single hardest item.
- Mix hard, random, and exposure-aware negatives so training covers confusable and broad catalog regions.
- Audit sampled pairs manually and with any available multi-positive labels.
Do not remove every semantically similar item. That can make the task too easy and erase the ranking boundary among genuine substitutes. The policy follows the definition of acceptable relevance.
A concrete batch audit
Assume a 256-pair batch creates 255 candidate negatives per anchor. For each anchor, record sampler source, item frequency, similarity score, whether the item was shown, whether it was clicked within a later window, and whether it shares a product family with a positive.
Then compare:
| Slice | Diagnostic | What it can reveal |
|---|---|---|
| Head vs tail items | negative appearances per positive appearance | frequency imbalance in training pressure |
| Exposed vs never exposed | later positive rate | unknown items mislabeled from lack of exposure |
| Hardest similarity decile | duplicate or relevant-pair rate | false negatives concentrated by mining |
| New vs established items | retrieval coverage and gradient counts | cold-start exclusion |
| Same family vs unrelated | loss and manual relevance | substitute or variant conflicts |
| Local vs cross-device batch | effective negative count and composition | distributed batch semantics |
These diagnostics estimate label and sampler behavior. They do not by themselves establish causality. A controlled training ablation can compare samplers under the same model, data cutoff, compute budget, and evaluation suite.
Evaluate against full-catalog serving
Sampled training loss can improve while retrieval quality declines. Evaluate with a candidate set that resembles serving, preferably the full eligible catalog or a documented approximation. Report recall at K, precision at K where labels permit it, mean reciprocal rank or NDCG when graded relevance is meaningful, catalog coverage, head-tail exposure, freshness, and latency.
Create counterfactual or human-judged sets for pairs that logs cannot label. Slice substitutes, duplicate items, multilingual content, rare interests, and cold-start items. If evaluation uses the same negative sampler as training, both can reward the same bias.
Online metrics should match the surface: successful long-term engagement, qualified consumption, purchase after returns, query reformulation, hides, or other guardrails. A historical YouTube architecture paper describes separate candidate-generation and ranking stages and practical deep-learning lessons. It is useful system evidence, but it does not document current YouTube architecture or an interview requirement.
Common wrong answers and precise repairs
| Wrong answer | Why it fails | Better response |
|---|---|---|
| “Every other item in the batch is a true negative.” | Batch position supplies a label, not semantic certainty. | Estimate false negatives using history, families, judgments, and later outcomes. |
| “More negatives always improve quality.” | Extra samples can add easy duplicates, bias, and compute cost. | Ablate count, composition, temperature, and effective batch size. |
| “Use only the hardest negatives.” | Top neighbors concentrate false negatives and label noise. | Filter known positives and mix difficulty bands. |
| “Uniform catalog sampling is unbiased.” | It may mismatch exposure and produce mostly trivial negatives. | Define the target distribution and correct or choose the proposal deliberately. |
| “A low sampled loss proves retrieval is good.” | Training candidates differ from the serving catalog. | Evaluate full-catalog retrieval and product slices. |
| “Cross-device batches only increase efficiency.” | They also change negative count, frequency, and duplicate risk. | Log the global pool and keep distributed semantics reproducible. |
A concise interview answer template
Begin: “The negative sampler is part of the learning objective, so I will define the positive event, proposal distribution, and serving target.” Then:
- Specify the anchor, positive, eligible catalog, and event-time filters.
- Write the in-batch score matrix and identify off-diagonal assumptions.
- Separate sampling bias, false negatives, accidental target hits, and exposure bias.
- Propose a mixed sampler and any proposal-probability correction.
- Audit head-tail pressure, later positives, item families, and hard-negative quality.
- Evaluate on serving-like candidates and business guardrails.
- Launch only after an ablation shows the gain is not confined to the sampled loss.
That sequence connects loss mechanics to the product outcome without pretending unknown feedback is a clean negative label.
Sources and Further Reading
- Google Research: Distributed Representations of Words and Phrases — primary word2vec negative-sampling paper.
- Google Research: Deep Neural Networks for YouTube Recommendations — historical two-stage recommendation architecture and system lessons.
- Google Research: Sampling-Bias-Corrected Neural Modeling — in-batch frequency bias and correction in two-tower retrieval.
- Google Research: Mixed Negative Sampling — mixture of batch and uniform negatives in an app recommendation case study.
- NeurIPS: Debiased Contrastive Learning — false-negative sampling bias in contrastive learning.
- OpenAI: CLIP — primary project page and paper for batch contrastive image-text training.
- TensorFlow:
compute_accidental_hits— accidental target hits in candidate sampling.
Sources checked September 12, 2026. The four-pair batch and all diagnostic tables are original examples.