Cross-Entropy and KL Divergence Interview Questions: Direction, Support, and Stable Computation

Cross-entropy and KL divergence interview questions are easiest to answer by fixing three things first: which distribution generates the expectation, which distribution is evaluated, and where either distribution has zero probability. Only then should you choose a stable implementation or framework loss.

Cross-entropy measures the expected negative log probability assigned by a model. KL divergence subtracts the source distribution’s entropy from that quantity. KL is directional, can be infinite, and is not a symmetric distance. These distinctions matter in classification, distillation, and language-model training.

For machine learning interviews, practice the math alongside implementation. The examples below are original calculations, not benchmark results or claims about an employer’s question frequency.

Cross-entropy versus KL: write the ordered pair

For discrete distributions P and Q over the same outcomes, using natural logarithms:

H(P)       = -sum_i P_i * log(P_i)
H(P, Q)    = -sum_i P_i * log(Q_i)
KL(P || Q) =  sum_i P_i * log(P_i / Q_i)

The expectation is taken under P. Q supplies the probabilities whose quality is being assessed. On appropriate support, H(P,Q) = H(P) + KL(P||Q). The SciPy entropy reference documents these entropy and divergence relationships.

Entropy is uncertainty in the source distribution. Cross-entropy includes that uncertainty plus the penalty for using another distribution. KL isolates the extra discrepancy relative to the source. If P is fixed while training Q, minimizing cross-entropy and minimizing this forward KL have the same minimizers because H(P) is constant.

That last condition is important. If you also optimize P, or compare objectives that reverse the arguments, you cannot discard the entropy term casually. Say which quantities are fixed, which are learned, and which parameters receive gradients.

Using base-2 logs expresses the quantities in bits; natural logs express them in nats. A different log base rescales the value. It does not reverse KL or change which support mismatch produces infinity.

Worked probability ledger: direction changes the answer

Take the original two-category example P=(0.75,0.25) and Q=(0.50,0.50). Both are normalized and strictly positive, so all terms are finite.

Quantity	Calculation result	Interpretation
H(P)	0.562335 nats	Source uncertainty
H(P,Q)	0.693147 nats	Expected loss when scoring Q under P
`KL(P		Q)`
`KL(Q		P)`

The ledger was checked with NumPy 2.5.3 and SciPy 1.18.1. The numbers are synthetic, not fitted model results. 0.693147 - 0.562335 equals the forward KL up to rounding, but it does not equal the reverse KL.

If an interviewer asks whether KL can be used as a distance, answer carefully. It is nonnegative for valid distributions and equals zero when the distributions agree, but generally lacks symmetry and the triangle inequality. Do not infer geometric-distance properties from its name or nonnegativity.

The common “mode-covering versus mode-seeking” explanation is a useful intuition in particular approximation problems, not a theorem about every optimization setup. First establish the direction and allowable model family. Which outcomes carry the expectation explains the penalty more precisely than repeating those slogans.

Zero support: infinity is sometimes the correct result

For a single KL term, the support conventions are:

• If P_i > 0 and Q_i = 0, the term is infinite.
• If P_i = 0 and Q_i > 0, its contribution is zero.
• If both are zero, its contribution is zero under the usual extension.

The SciPy rel_entr reference specifies these elementwise cases. Summing the terms gives the distribution-level KL after the input contract has been checked.

Zero source mass contributes nothing. Positive source mass assigned zero model probability creates an infinite penalty; replacing that zero changes the modeled distribution.

For example, if P=(1,0) and Q=(0,1), forward KL is infinite. A function returning a large finite value after clipping is evaluating a modified objective. That can be an intentional modeling choice, but it is not an exact computation of the original divergence.

Avoid computing 0 * log(0) directly and hoping multiplication removes the problem. The intermediate log can be negative infinity, and the product can become NaN. Use an operation that implements the intended limiting convention, or mask valid contributions before evaluating the logarithm.

Distinguish a genuine support zero from numerical underflow. A finite logit can produce a probability rounded to zero after exponentiation, although its log probability remains finite. In that case, computing directly in log space preserves information that a naive softmax-then-log path discarded.

Stable computation starts with logits

A logit is an unnormalized model score. For a hard target class k, softmax cross-entropy can be written as logsumexp(z) - z_k, without first materializing tiny probabilities. Subtracting the largest logit inside the exponential calculation controls overflow.

Consider logits [1000,1001] with target index 1. Direct exponentiation overflows in common floating-point implementations. The stable expression gives approximately 0.3132616875 nats:

import numpy as np
from scipy.special import logsumexp

z = np.array([1000.0, 1001.0])
target = 1
loss = logsumexp(z) - z[target]
assert np.isclose(loss, 0.3132616875182)

SciPy documents the stable summation operation in its logsumexp reference. Our fixture checks the stated example; it does not claim that every arrangement of infinite or invalid logits becomes meaningful.

For normalized soft targets P, use -sum(P_i * log_softmax(z)_i). Hard-label loss is the special case where all target mass is on one class. Replacing a soft target with its argmax discards uncertainty and changes the objective.

One implementation check is shift invariance. Adding the same finite constant to every logit should not change the softmax distribution or cross-entropy, within numerical tolerance. A loss that changes materially under that shift likely has an unstable normalization or incorrect formula.

PyTorch argument order is not mathematical KL notation

PyTorch’s loss interfaces use model output as input and reference values as target. For KLDivLoss, the usual input is log model probability log(Q) and the default target is probability P. That computes KL(P||Q), not KL(Q||P).

The official KLDivLoss documentation also distinguishes log_target=True from the default target representation. A variable named p does not settle direction; write the expectation and inspect the actual representations.

For ordinary classification, CrossEntropyLoss expects unnormalized logits. Passing softmax probabilities to it makes those probabilities act as new logits. Applying another softmax internally changes the distribution you intended to score.

A useful code review question is: “Which tensor is log probability, which is probability, and which dimension contains classes?” Shape compatibility alone is not semantic compatibility. A transposed tensor can be accepted by an operation while representing the wrong distribution axis.

These framework contracts were checked in official documentation. PyTorch was not installed in the local verification environment, so the article’s numerical execution evidence comes from NumPy and SciPy, not a claimed PyTorch training run.

Reduction and masks can change the reported loss

For a two-dimensional [batch, classes] KL tensor, summing across classes and averaging across batch items differs from averaging across every element. If all examples have the same class count, the latter adds a class-count divisor.

PyTorch’s KLDivLoss reference identifies batchmean as the reduction aligned with batch-average KL. Its default mean averages individual elements. With additional sequence or spatial axes, define your intended denominator explicitly rather than assuming a reduction name expresses a per-token objective.

Suppose a batch contains two sequences with two and eight eligible tokens. Averaging all ten eligible token losses weights the longer sequence four times as much. Averaging two sequence-level means gives each sequence equal weight. Neither is universally correct; they answer different training and reporting questions.

For padded tokens, excluded positions must not contribute to the numerator or accidentally remain in the denominator. Class weighting introduces another normalization decision. Before comparing losses from two implementations, align eligible positions, target representation, weights, and reduction.

For KL between normalized distributions, a materially negative result is a debugging clue. Check invalid inputs, missing normalization, reversed representations, and reduction code before assuming the mathematical property failed. Tiny negatives near zero can instead reflect floating-point rounding; use a tolerance appropriate to the computation.

Validate distributions before trusting a library result

An interview implementation should check finite values, nonnegative probabilities, compatible support and shapes, and normalization along the intended axis. Decide whether invalid input should raise an error or be normalized deliberately. These policies should be visible to callers.

SciPy’s entropy function normalizes input arrays when necessary. That convenience can conceal a mistaken caller contract: counts and already normalized probabilities are not interchangeable descriptions even if the final calculation succeeds.

PyTorch’s cross-entropy documentation warns that soft-target probability constraints are not enforced automatically. An accepted tensor is not evidence that it sums to one or represents a valid distribution.

Build a small test set before attempting a large training loop:

1. Identical positive distributions give KL zero within tolerance.
2. The probability ledger above verifies direction and the entropy identity.
3. Positive P mass at zero Q support returns infinity, not silently clipped output.
4. Extreme finite logits produce a finite stable cross-entropy.
5. Shared-logit shifts preserve the loss; masks and reductions use the declared denominator.

When a result fails, retain the unreduced terms long enough to locate the offending class or token. A single scalar can hide both a support error and an unrelated normalization mistake.

What should the cross-entropy gradient look like?

For normalized soft targets and softmax probabilities on one example, the derivative with respect to each logit is Q_i-P_i. With a hard label, subtract one from the target class probability and leave the others as their probabilities. Averaging examples introduces the corresponding batch divisor; masking introduces the denominator you specified.

The components sum to zero because both distributions sum to one. This agrees with shared-logit shift invariance: moving every score together should not change the distribution. A gradient whose components materially fail this check may reveal invalid targets or a normalization-axis error.

Use central finite differences on small, moderate logits to test a handwritten backward pass, then separately test the extreme-logit forward path. Do not infer that a stable forward loss guarantees a correct gradient. Saturated probabilities, reduction factors, and accidentally detached intermediate values can hide different backward mistakes. For distillation, also state whether the teacher target is fixed; allowing gradients through both distributions creates a different optimization problem from training only the student against a frozen reference.

Finish with direction, support, and denominator

A strong answer is not “KL is like cross-entropy.” It is: “I score Q under P, check support, and compute in log space. With P fixed, cross-entropy differs from forward KL by constant entropy. I then align the framework’s argument order and reduction with that objective.”

Start with the two-category ledger, then add masks and extreme logits only after the direction is correct.

Sources and Further Reading

• SciPy: entropy
• SciPy: elementwise relative entropy
• SciPy: logsumexp
• PyTorch: CrossEntropyLoss
• PyTorch: KLDivLoss

Research note: Primary documentation checked October 3–4, 2026. Probability, zero-support, and extreme-logit fixtures ran with NumPy 2.5.3 and SciPy 1.18.1. PyTorch contracts are documentation-verified, not locally executed. Numbers use natural logarithms and are expressed in nats.