Articles

Word2Vec Interview Questions: CBOW, Skip-Gram, and Negative Sampling

Practice Word2Vec interview questions with CBOW and skip-gram pairs, two embedding tables, a negative-sampling loss calculation, and rare-word checks.

“CBOW predicts the center word; skip-gram predicts its context” is a useful opening answer. A stronger Word2Vec interview answer can construct the training examples, identify the two sets of parameters, and explain exactly what a negative-sampling update optimizes.

The central idea: Keep architecture, training objective, and exported representation separate. They are related choices, but knowing one does not determine the others. Trace a small sentence through those choices before discussing analogies or nearest neighbors.

Evidence boundary: The original Word2Vec papers, primary research derivations, and current Gensim documentation were checked on October 4, 2026. The corpus, vectors, and numerical calculations below are original teaching fixtures. They are not a pretrained model, a semantic-quality benchmark, or candidate reports about interview frequency.

For a broader comparison prompt, practice From Early NLP Embeddings to Modern LLM Embeddings, then use this guide to make the Word2Vec portion precise.

How do CBOW and skip-gram turn a sentence into examples?

Primary research fact: The original Efficient Estimation paper describes continuous bag-of-words, or CBOW, as predicting a word from its surrounding context, and skip-gram as predicting surrounding words from the current word. Classic CBOW combines context embeddings without preserving their order in that combined representation.

Use this original fixture:

Sentence: cats chase mice
Window radius: 1
No subsampling, vocabulary pruning, or sentence-boundary crossing

The fixed radius is a teaching choice. It means the context contains adjacent positions, excluding the center. At the sentence edges, fewer context words are available.

Center position	CBOW example: context → target	Skip-gram examples: center → target
`cats`	`[chase] → cats`	`cats → chase`
`chase`	`[cats, mice] → chase`	`chase → cats`; `chase → mice`
`mice`	`[chase] → mice`	`mice → chase`

There are three CBOW examples and four ordered skip-gram pairs in this fixture. “Ordered” matters: (cats, chase) and (chase, cats) access different input and output rows even though the same two words appear.

For the middle CBOW example, a mean-based context representation is h = (v_cats + v_mice) / 2. Exchanging the two context positions leaves that mean unchanged. The combined vector loses the original order within that context bag; a task requiring that order needs an additional representation.

A useful clarification question is: “Does the window parameter mean a fixed radius here, and are we predicting one target per context bag or one target per center-context pair?” That prevents an otherwise correct answer from counting the wrong examples.

Why are there two embeddings for each vocabulary item?

Primary derivation: Xin Rong's parameter-learning note distinguishes input vectors from output vectors. They are separate learned parameters. One is not automatically the transpose, copy, or normalized version of the other.

For our notation, store both tables as vocabulary-by-dimension matrices:

E_in:  V × d     row v_w represents word w as input
E_out: V × d     row u_c represents word c as output
Pair score: s(w,c) = v_w · u_c

Some derivations arrange the output matrix as d × V; that changes indexing notation, not the existence of separate parameters. State your layout before describing which rows receive updates.

In a skip-gram pair chase → mice, the input row for chase and output row for mice participate. Reversing the pair uses the input row for mice and output row for chase. In CBOW, context input rows contribute to a combined vector used to score the center target.

Original footprint calculation: With V = 50,000 and d = 100, two dense float32 tables contain 10 million scalars, or 40 million bytes, about 38.1 MiB. That excludes vocabulary metadata, auxiliary sampling structures, and any additional training state. Saving one table can reduce the vector payload while discarding information needed to resume the original training process.

Official implementation boundary: Gensim's KeyedVectors documentation distinguishes storing and querying vectors from retaining a complete trainable Word2Vec model. Always say which representation you exported and what operations the artifact supports.

What does negative sampling optimize?

Start with the normalized softmax alternative for one skip-gram pair:

p(c | w) = exp(v_w · u_c) / sum_j exp(v_w · u_j)

The denominator involves the output vocabulary. Primary derivation: Goldberg and Levy's explanation treats skip-gram negative sampling, or SGNS, as a different objective: distinguishing observed word-context pairs from sampled noise pairs.

For one observed pair (w,c) and k sampled noise contexts n_i, minimize:

L = -log sigmoid(v_w · u_c)
    -sum_i log sigmoid(-v_w · u_n_i)

This uses binary labels for each selected pair. It does not produce a vocabulary-wide distribution whose values must sum to one. “Negative” labels sampling provenance; it does not certify that two words are semantically incompatible or never co-occur anywhere.

Primary research distinction: Chris Dyer's note on NCE and negative sampling explains why the two methods should not be treated as interchangeable estimators. Do not describe SGNS as an unbiased estimate of the original full-softmax gradient. It replaces the objective with a useful binary classification problem for learning representations.

Architecture and output objective remain separate choices. CBOW versus skip-gram specifies how inputs and targets are formed. Full softmax, hierarchical softmax, or negative sampling specifies how output training is performed. Saying “skip-gram” alone does not specify the loss.

Calculate one positive pair and two noise pairs

Consider the observed pair chase → mice and two illustrative noise draws, cloud and stone. This extends the fixture's output vocabulary for the loss example; the three-word sentence alone is not its entire vocabulary.

Assume all three pair logits are zero at the moment of calculation. This does not require all parameters to be zero. For example, nonzero orthogonal input and output vectors can have zero dot products.

For sigmoid(0) = 0.5, each of the three binary terms contributes -ln(0.5) = ln(2). Therefore the original summed-loss calculation is L = 3 ln(2) ≈ 2.07944. A mean reduction over the three terms would instead give approximately 0.69315. Declare the reduction before comparing implementations.

For a binary term with logit s and label y, the loss derivative with respect to s is sigmoid(s) - y:

Pair in the original fixture	Binary label	Logit derivative at zero	Direction for an independent score update
`chase → mice`	1	-0.5	Gradient descent increases the score
`chase → cloud`	0	+0.5	Gradient descent decreases the score
`chase → stone`	0	+0.5	Gradient descent decreases the score

The input-vector gradient combines the three output-vector contributions. Each selected output row also receives a contribution involving the input vector. With a mean reduction, scale these derivatives accordingly. Shared parameters mean the full vector update is richer than three independent scalar adjustments.

This example catches several implementation errors: reversing the negative term's sign, forgetting the positive term, confusing sum with mean, or updating a single shared word table while claiming to implement the two-table model.

Use a numerically stable binary-loss formulation rather than computing a sigmoid followed by a logarithm at extreme logits. In a small diagnostic fixture, compare an analytic derivative with a finite-difference approximation and check the expected signs before attempting a large training run.

How do the noise distribution and window change the model?

Primary research fact: The Distributed Representations paper describes negative sampling with a smoothed unigram noise distribution and subsampling of frequent words. A common noise choice is proportional to word frequency raised to the 3/4 power. It is not equivalent to uniformly choosing an unrelated word.

Original distribution calculation: If two vocabulary frequencies have ratio 16:1, exponent 3/4 changes their unnormalized noise-weight ratio to 8:1. The frequent token remains more likely to be drawn. If those were the only two tokens, their normalized probabilities would be 8/9 and 1/9. A real vocabulary requires normalization across all included tokens.

Changing the sampling distribution changes which output rows receive negative updates and how often. Increasing k changes the training workload and relative contribution of sampled terms under the chosen reduction. It should be evaluated as a training choice, not described as a universal quality improvement.

A sampled context can also be plausible or observed elsewhere. Check the implementation's handling of repeated draws and collisions with the positive target. Repeated sampled rows require accumulated gradient contributions; casually deduplicating them changes the objective being computed for that draw.

Implementation fact: Gensim's Word2Vec API exposes architecture, negative-sample count, noise exponent, vocabulary pruning, and window settings. Dynamic windows and frequency subsampling mean the simple four-pair fixture should not be assumed to describe every real training pass.

What changes for rare words, unseen words, and evaluation?

The first check is vocabulary coverage. In our tiny corpus, every word occurs once. A minimum-count setting larger than one can remove all three words. That failure occurs before any meaningful comparison of CBOW and skip-gram quality.

For a rare included word, limited training evidence constrains the learned representation. Do not promise that choosing skip-gram automatically solves rarity. For a token absent from a plain word-level vocabulary, there is no trained row to retrieve. Specify an unknown-token policy, a preprocessing decision, or a different model with appropriate subword behavior; none is a free property of basic Word2Vec.

Original evaluation recommendation: Keep training diagnostics separate from representation usefulness. Inspect finite loss and update signs, then measure coverage and the downstream task you intend to support. For sentence similarity, define how word vectors become sentence representations and test cases where the same words appear with different meaning or order.

Nearest neighbors can reveal obvious anomalies, but one attractive analogy is not a complete evaluation. Static word vectors map a vocabulary item to its stored representation without conditioning that lookup on the current sentence. If sense-specific representations matter, name that limitation instead of hiding it inside an averaging step.

The Gensim Word2Vec tutorial discusses training and reproducibility considerations. Record preprocessing, vocabulary settings, architecture, objective, dimension, window, seed, and worker configuration. A seed alone does not settle scheduling differences in multithreaded training.

What should a complete interview answer sound like?

Use the same example throughout: “With a fixed radius-one window, cats chase mice produces three CBOW contexts or four ordered skip-gram pairs. I keep separate input and output embeddings. Under SGNS, an observed pair and sampled noise pairs use binary logistic terms, so the resulting scores are not normalized vocabulary probabilities. My zero-logit, two-noise summed loss is about 2.079. I would then check vocabulary coverage, sampling behavior, and downstream performance.”

An interviewer can now change one assumption and inspect your reasoning. What happens with a wider window? What if the output table is tied to the input table? What if you average instead of sum the binary terms? Answer the change directly rather than repeating the model definition.