PyTorch Autograd Interview Questions: Graph Lifetime, In-Place Operations, and Detached Tensors

Autograd interview questions become easier when you stop treating a PyTorch tensor as just a number. A tensor may also be a leaf, a view of shared storage, or a node whose backward computation needs an earlier value. The fastest way to debug a training loop is to name the graph edge that should carry the gradient, then check whether your code recorded, severed, reused, or modified it.

This guide uses one small, invented scalar exercise rather than a broad catalog of PyTorch trivia. The arithmetic can be checked by hand; the PyTorch behavior is grounded in the official autograd notes. The code patterns below are instructional examples, not a report of a local PyTorch execution: PyTorch was unavailable in the writing environment.

Interview prompt: derive the gradient before touching the debugger

Suppose an interviewer gives you a single trainable scalar w, a constant input x = 3, and loss (w*x - 1)^2. Start at w = 2. What is the loss, what should w.grad be after one backward call, and what changes if a teammate inserts detach() between the prediction and loss? Explain your answer without appealing to a framework printout.

import torch

w = torch.tensor(2.0, requires_grad=True)
x = torch.tensor(3.0)
prediction = w * x - 1
loss = prediction.square()
loss.backward()
print(loss.item(), w.grad.item())

The expected values are 25.0 and 30.0. At the start, prediction = 2*3 - 1 = 5, so loss = 5² = 25. By the chain rule, d(loss)/dw = 2(w*x - 1)*x = 2*5*3 = 30. A finite-difference calculation around w = 2 also approaches 30; it is a sanity check, not a substitute for understanding what autograd recorded. If the answer is zero or None, identify the missing edge before changing the learning rate.

The original w is a leaf tensor with requires_grad=True. prediction and loss are results of operations and therefore non-leaf tensors. After backward, the leaf's .grad is populated by default; inspecting an intermediate's .grad requires retain_grad() if you actually need it. “I printed prediction.grad and saw None” alone is not proof that backpropagation failed. PyTorch's autograd notes distinguish leaf gradient accumulation from the graph's intermediate operations.

One good interview response states the invariant first: if the optimizer is meant to update w, the scalar loss must remain differentiably connected to that exact leaf, and the backward path must use values that have not been invalidly changed. Then test that invariant at the smallest failing point.

Question 1: When does the graph exist, and what does backward consume?

PyTorch records operations as the forward pass executes. The graph is dynamic: each new training iteration normally builds a new path for its new inputs and branches. Calling backward() traverses the recorded path in reverse and accumulates into leaf gradients. It is not a permanent static graph that survives every optimizer step unchanged. This matters for loops with data-dependent branches: a changed branch can create a different graph on the next iteration without a special recompilation step in ordinary eager execution.

Try this variation as a diagnostic thought experiment:

w = torch.tensor(2.0, requires_grad=True)
loss = (w * 3 - 1).square()
loss.backward()
loss.backward()  # same forward graph, second traversal

The second call normally fails because backward has freed saved intermediates needed to compute gradients. The torch.autograd.backward reference documents that graph retention is off by default. retain_graph=True can be necessary when an algorithm genuinely needs more than one backward traversal of the same graph, but it also keeps graph data alive and can increase memory use. In an ordinary training loop, the fix is usually to run a new forward pass, not to sprinkle retain_graph=True over an accidental reuse.

Be precise about the boundary: “the graph is freed” is shorthand, not a claim that all tensors or Python variables disappear. The values you kept may remain accessible, while the saved tensors required for a second backward have been released. An interviewer may ask why a logging call like loss.item() works after backward while a second backward does not; the logging call reads a scalar value and does not replay the graph.

Gradient accumulation is a separate issue. If you do two new forward/backward passes without clearing w.grad, the gradients add on the leaf. In the toy case, two identical passes would accumulate 30 + 30 = 60. That can be deliberate for gradient accumulation across microbatches, but it is a bug if the loop assumes a fresh gradient every step. Use optimizer.zero_grad(set_to_none=True) or equivalent at the intended accumulation boundary, not automatically between microbatches you mean to combine. Check whether a zero value or None is expected before using it as a failure signal.

Question 2: What exactly does detach() cut?

Now insert this line before constructing the loss:

prediction = w * x - 1
detached_prediction = prediction.detach()
loss = detached_prediction.square()
loss.backward()

Here the loss has no differentiable path to w. In this particular snippet there is no other input requiring gradients, so loss.backward() raises an error about a tensor that does not require grad; it does not quietly produce the original 30. If another trainable branch were also in the loss, backward could succeed for that branch while w.grad stayed None. That distinction is useful in a real model: successful completion of backward() does not prove every intended parameter was reached.

The detach() reference says the returned tensor does not require gradients and shares storage with the original. Shared storage is an often-missed detail. detach() is a gradient boundary, not necessarily a data copy. If you need both an independent value and no autograd history, use prediction.detach().clone() and explain why the copy matters. In-place edits through a shared detached tensor can still alter data used by the original computation or violate a version check.

Sometimes a detach is intentional. A teacher prediction used as a fixed target, a stop-gradient branch in a self-supervised objective, or a recurrent hidden state deliberately truncated between windows may need one. But annotate which branch is frozen. If you detach the value merely to make an error go away, you may have converted a visible training failure into a silently untrained component.

Another interview trap is converting the loss to a Python scalar and then trying to train on it. loss.item() is appropriate for metrics and logging; it cannot itself carry a gradient. Keep the tensor-valued loss for backward(), and send only a detached scalar to logs. Similarly, moving a tensor through a non-differentiable external library is not a transparent autograd operation; restoring it as a new tensor does not recreate the missing edge.

Question 3: Why can an in-place operation fail only at backward?

Consider a new graph with an intermediate z:

w = torch.tensor(2.0, requires_grad=True)
z = w * 3
loss = z.square()
z.add_(1)       # mutation after loss was constructed
loss.backward() # backward needs the earlier z

The forward loss was formed from z = 6. Its derivative with respect to w should be 2*z*3 = 36, based on that earlier value. The in-place add_ changes z after square() has recorded what backward needs. PyTorch tracks version counters for saved tensors and raises when a required value has been changed in place. The autograd in-place operation guidance explains this safety check. The problem is not that every underscore method is forbidden; it is that this mutation invalidates a specific saved value on a specific path.

Do not confuse this with changing a leaf requiring gradients directly. An in-place edit to w in grad mode may be rejected immediately for a different reason. In an interview, point to the exact tensor and timing. Ask: Which backward formula needs the pre-mutation value? Was the mutation on a view or an alias? Was the value modified after the forward operation that saved it? The stack trace often identifies a backward function and a version mismatch, but the useful answer is the data dependency, not merely “remove all in-place operations.”

A clean fix is to avoid mutating z if the earlier z is needed for loss. If the next computation truly needs a changed value, create a separate out-of-place tensor and keep the original path intact. If an optimizer updates parameters, do that at the intended step boundary under a no-grad context; do not mutate parameters between the forward and backward of the same loss. Memory savings from in-place operations should be measured, not assumed: autograd already reuses buffers, and a mutation that breaks correctness is not an optimization.

The shared-storage caveat from detach() makes the diagnosis less obvious. A detached alias may not show an autograd edge, yet writing through it can modify storage used by the tracked tensor. The safe mental model has two axes: gradient connectivity and storage aliasing. Detaching changes the first; cloning changes the second. A candidate who can state both axes usually avoids the misleading “detach makes the tensor safe to edit” answer.

Question 4: Why do eval(), no_grad(), and inference mode differ?

Suppose a teammate writes model.eval() and expects w.grad to remain empty. That expectation is wrong. Evaluation mode changes the behavior of modules such as dropout and batch normalization; it is not an autograd switch. PyTorch's autograd notes make this distinction explicit. You may still build a graph in eval mode if grad mode is on and inputs require grad.

torch.no_grad() disables recording for ordinary operations performed in its scope. It is appropriate for parameter updates and many evaluation paths where you do not need derivatives. torch.inference_mode() is a stronger optimization with additional restrictions on using inference-created tensors in later autograd-recorded computations. Do not swap it in mechanically if the outputs will re-enter a training graph. Conversely, do not turn off autograd for a validation routine that intentionally computes input gradients, saliency, or a gradient-based metric. State what the pass must calculate, then choose the mode.

This distinction also helps localize memory pressure. If evaluation consumes unexpected memory, check whether the pass is recording a graph and retaining output tensors. If training gradients disappear, check for an over-broad no_grad() context before blaming model.eval(). A context manager can encompass more code than intended; move the boundary to the smallest correct scope and rerun the scalar fixture.

Debugging answer: trace one parameter, one loss, one step

A practical interview answer follows the path of one parameter instead of printing a thousand tensors. First, verify that the intended leaf parameter has requires_grad=True and is in the optimizer. Second, inspect the scalar loss immediately before backward: is it a tensor with a graph (grad_fn) rather than a detached value? Third, inspect the graph breakpoints introduced by detach(), .item(), no_grad(), or a non-differentiable conversion. Fourth, check whether a saved tensor was mutated or a graph was reused after backward. Fifth, check the accumulation boundary and the order zero_grad → forward → loss → backward → step for the simple case.

For the toy exercise, report loss = 25, expected w.grad = 30, and the chain-rule calculation. For the detach variation, explain why backward cannot reach w; for the reused-graph variation, rebuild the forward pass; for the in-place variation, preserve the earlier z. These are three different repairs. A candidate who reaches for retain_graph=True in all three cases has not identified the cause.

If a larger model disagrees with the toy result, reduce it to a deterministic batch and one parameter. Log the input, loss, whether .grad is None or finite, and the point where the edge disappears. Use PyTorch's anomaly detection as a targeted debugging aid when the backward error is hard to locate, then turn it off for normal performance work. Compare an analytic derivative or finite-difference probe only on a small smooth function; finite differences near nondifferentiable points or at unsuitable step sizes can mislead.

Sources and Further Reading

• PyTorch: Autograd mechanics, saved tensors, grad modes, and in-place operations
• PyTorch: Tensor.detach reference
• PyTorch: torch.autograd.backward reference