Dropout Interview Questions: Scaling, Train/Eval Modes, and Reproducible Predictions

You set the seed, call model.eval(), and run the same input twice. The predictions still differ. Is dropout broken, did evaluation mode fail, or is the random-number stream doing exactly what your code asked it to do?

Dropout interview questions become easier when you separate three contracts: what the mask preserves in expectation, when the layer is active, and what makes a stochastic run reproducible. Memorizing “dropout prevents overfitting” does not resolve a functional call that ignores evaluation mode or a test that resets the generator before every forward pass.

This guide uses current PyTorch documentation and small, original calculations. The standard-library fixtures are reproducible without PyTorch; the PyTorch snippets illustrate documented behavior rather than a benchmark across devices. For graph and gradient questions beyond dropout, continue with PyTorch autograd interview questions.

What does dropout actually preserve?

For ordinary inverted dropout, let p be the probability of dropping an activation, not the probability of keeping it. During training, a retained value is multiplied by 1 / (1 - p). A dropped value becomes zero. PyTorch documents this convention for nn.Dropout; evaluation makes the layer an identity operation. PyTorch Dropout

For 0 <= p < 1, write one output as:

M ~ Bernoulli(1 - p)
Y = M * X / (1 - p)
E[Y | X] = X
Var(Y | X) = X² * p / (1 - p)

The expectation is conditional on a fixed input activation. It is not a promise that every masked vector has the same sum, norm, or prediction as the unmasked vector. The variance grows as the drop probability increases. Scaling corrects the mean; it does not remove the noise.

Take X = [1, 2, 3, 4], p = 0.5, and mask [1, 0, 1, 0]. The output is [2, 0, 6, 0]. Its sum is eight, although the input sum is ten. That individual draw is perfectly consistent with mean preservation. An answer claiming that dropout preserves each sample's total has confused an expectation with an invariant.

A worked calculation that exposes the nonlinear trap

Consider one activation equal to two. At p = 0.5, its dropped output is either zero or four, each with probability one half. Therefore E[Y] = 2, but E[Y²] = 8. Squaring the mean gives only four.

This is the central reason not to describe evaluation as an exact average of all randomly thinned networks. A later nonlinear operation need not commute with expectation. Even if a layer's activation mean is preserved, the final prediction distribution requires its own analysis. The original dropout paper motivates an efficient approximation to combining thinned networks; it does not make arbitrary nonlinear predictions identical. Dropout paper

For an exact, finite check, enumerate all sixteen masks of the four-element vector:

from itertools import product
from fractions import Fraction

x = [1, 2, 3, 4]
outputs = [
    [2 * value * keep for value, keep in zip(x, mask)]
    for mask in product([0, 1], repeat=len(x))
]
means = [Fraction(sum(row[j] for row in outputs), 16)
         for j in range(len(x))]
variances = [
    sum((Fraction(row[j]) - means[j]) ** 2 for row in outputs) / 16
    for j in range(len(x))
]
assert means == x
assert variances == [1, 4, 9, 16]

This tests the mathematical contract, not PyTorch's RNG algorithm. It avoids a flaky assertion based on a small random sample. For another drop probability, assign masks their correct probabilities rather than treating every mask as equally likely.

An interviewer may ask what happens at p = 0, or at the endpoint p = 1. Zero is the identity even in training. At one, all activations are dropped; the formula with division by zero is not a valid way to calculate that endpoint. State the domain before deriving the expression.

Does no_grad disable dropout?

No. Model mode and gradient recording are different controls. model.train() or model.eval() changes the behavior of mode-sensitive modules. torch.no_grad() controls autograd recording. PyTorch explicitly treats evaluation mode as orthogonal to gradient modes. Autograd notes

For an ordinary nn.Dropout in a correctly registered model, reason through this matrix. “Graph enabled” below assumes an input requiring gradients and operations that otherwise participate in autograd.

Mode and contract
Training, gradients enabled: random masking and scaling; gradients can flow through retained activations.
Training inside no_grad: random masking and scaling; no graph is recorded for those operations.
Evaluation, gradients enabled: identity dropout behavior; evaluation does not itself disable autograd.
Evaluation inside no_grad: identity dropout behavior and no recorded graph.

This distinction explains two common debugging mistakes. A validation loop using only no_grad() can still apply dropout. A model using only eval() can still build graphs and retain unnecessary tensors. If both contracts are required, set both explicitly.

Do not broaden that statement to “everything is deterministic in evaluation.” Other stochastic code, preprocessing, custom layers, and device operations have separate behavior. Start by isolating the smallest forward path that disagrees.

Repair a functional dropout call

Here is a plausible implementation with a subtle defect:

import torch.nn as nn
import torch.nn.functional as F

class Head(nn.Module):
    def forward(self, x):
        return F.dropout(x, p=0.5)

Calling head.eval() does not automatically alter the argument passed into this standalone function. The documented default for F.dropout is training=True. The call therefore remains stochastic. Functional dropout reference

Two repairs express the intended ownership:

class FunctionalHead(nn.Module):
    def forward(self, x):
        return F.dropout(x, p=0.5, training=self.training)

class ModuleHead(nn.Module):
    def __init__(self):
        super().__init__()
        self.dropout = nn.Dropout(p=0.5)

    def forward(self, x):
        return self.dropout(x)

The first passes the enclosing module's flag explicitly. The second registers a module whose mode follows the parent. Creating a fresh dropout module inside forward is not the same repair: its lifecycle and mode are not owned by the existing parent in the intended way.

A useful evaluation test asserts that the dropout-only head returns the input. A useful training test checks shape, allowable output values, and statistical behavior over enough samples. Merely asserting that two training calls differ can fail by chance, especially for tiny tensors. An all-zero input cannot reveal masking at all.

Why does a seed not make consecutive calls identical?

A seed establishes a generator's starting state. A random operation consumes state. The next operation normally samples from the advanced stream rather than replaying the first mask.

You can demonstrate the distinction without making claims about PyTorch's implementation:

import random

rng = random.Random(17)
state = rng.getstate()
first = [rng.random() for _ in range(8)]
second = [rng.random() for _ in range(8)]
rng.setstate(state)
replayed = [rng.random() for _ in range(8)]
assert replayed == first
assert second != first

This fixture tests Python's generator-state contract. It is not evidence that Python and PyTorch produce interchangeable sequences. In a training job, adding an earlier random operation can change later masks even when the initial seed is unchanged.

Resetting a seed before every minibatch can make debugging look stable while repeatedly imposing correlated randomness. Prefer a documented initialization policy and controlled replay when replay is actually the test. Do not silently change the training procedure to satisfy an assertion.

PyTorch also warns that complete reproducibility is not guaranteed across releases, platforms, or CPU and GPU execution. Report the environment, device, relevant determinism settings, operation order, and generator state when investigating a mismatch. A seed by itself is an incomplete reproduction record. Reproducibility notes

How should you debug changing evaluation predictions?

Start with a fixed tensor and a single forward call, not the whole data pipeline. Confirm the model mode and inspect the exact dropout implementation. Look for functional calls missing training=self.training, custom random operations, and submodules deliberately returned to training mode.

Next separate input differences from forward differences. Compare the actual tensors entering the model, including augmentation, padding, dtype, and device. Identical filenames or example IDs do not establish identical input values. Run the smallest dropout-only module in evaluation to test its identity contract.

Then compare numerical tolerance, not only bitwise equality. If dropout is inactive but another operation produces small device-dependent differences, the explanation belongs to that operation. Record the first layer where outputs diverge rather than blaming the last stochastic-looking component.

For checkpoint replay, weights alone do not capture a training trajectory. Optimizer state, data order, relevant RNG states, and environment can matter. Describe exactly which states your experiment restores and which it does not. “The checkpoint loaded” is not proof of identical next-step predictions.

The strongest debugging answer names a discriminating experiment: bypass preprocessing, replace the suspicious call with an identity, or explicitly pass the training flag. Each experiment should distinguish two explanations, not merely produce another screenshot of different outputs.

What changes for Monte Carlo dropout?

Monte Carlo dropout intentionally samples predictions with selected dropout layers active during inference. That is a different inference policy, not evidence that ordinary evaluation should be random.

Do not casually call model.train() on the whole model to enable it. Other mode-sensitive modules can change too. A narrower implementation begins in evaluation and deliberately re-enables the known dropout modules required by the method, while keeping gradient recording disabled if gradients are unnecessary.

Explain the limitations before interpreting the sample spread. The variation reflects the chosen model and stochastic mechanism; it is not automatically a calibrated confidence interval, a complete measure of uncertainty, or proof that a shifted input is safe. Choosing the number of samples also creates a latency-versus-estimation trade-off.

In an interview, distinguish the ordinary deterministic evaluation contract from the explicitly stochastic uncertainty experiment. Then explain how you would validate the uncertainty estimates against held-out outcomes. More stochastic forward passes cannot compensate for a badly chosen evaluation dataset.

Five questions that test the contracts in practice

Practice question	Evidence to produce
Explain Neural-Network Regularization and Dropout	Derive mean preservation and explain why nonlinear predictions need not be preserved.
Implement common neural network layers in PyTorch	Own the module lifecycle and test train/eval behavior explicitly.
Debug and fix a PyTorch Transformer training loop	Separate mode, gradient recording, and input-pipeline faults.
Handle cold start, dropout, and training stability	Explain the noise introduced by a drop probability rather than recommending one blindly.
Bias-Variance Tradeoff and Vanishing Gradients in Feedforward Networks	Connect regularization to generalization without claiming dropout fixes every training failure.

A concise answer that survives follow-ups

“During training, ordinary inverted dropout drops an activation with probability p and scales retained values by 1/(1-p), preserving the activation's conditional mean. Evaluation turns a correctly mode-controlled dropout layer into an identity. Gradient recording is independent. Reproducibility requires control of generator state and execution conditions, not just a seed.”

Then volunteer one counterexample: functional dropout defaults to training, or mean-preserving activations do not imply mean-preserving nonlinear predictions. A concrete counterexample demonstrates that you understand the boundaries of the definition.

Sources and Further Reading

• PyTorch: Dropout
• PyTorch: functional dropout
• PyTorch: autograd and evaluation modes
• PyTorch: reproducibility
• Srivastava and colleagues: Dropout, JMLR

Sources checked October 4, 2026. Calculations and debugging exercises are original educational examples.