Articles
GAN Interview Questions: Alternating Updates, Mode Collapse, and Misleading Loss Curves
Prepare for GAN interviews with alternating-update checks, detach boundaries, a scalar gradient exercise, and mode-coverage tests that challenge loss curves.
A GAN training loop has two optimizers, but that does not make its updates correct. The discriminator may accidentally train the generator, the generator may receive no gradient, or attractive samples may hide a model that reproduces only one part of the data distribution. To answer GAN interview questions, trace one update and separate optimization evidence from distribution evidence.
Primary research: the original GAN paper defines a generator and discriminator trained through an adversarial objective. Official implementation reference: PyTorch's DCGAN tutorial demonstrates alternating discriminator and generator steps. The exercises here are original teaching cases, not a reported employer question list. No corroborated GAN-specific candidate-report set establishes their interview frequency.
Start with Implement and derive backprop from scratch if you need to rehearse gradient flow before explaining the two-player loop.
What changes during a discriminator step?
First state the objective convention. In this guide, the discriminator predicts the probability that an input is real, and both networks minimize their own losses. For ordinary binary cross-entropy, a discriminator step penalizes mistakes on real examples labeled one and generated examples labeled zero. Other GAN families use different objectives; identify the family before interpreting its losses.
Official example: the PyTorch DCGAN loop generates fake samples, passes a detached version to the discriminator for its update, then performs a fresh discriminator forward pass for the generator update. The fresh pass uses the discriminator parameters updated in the preceding phase.
For an original update ledger, assume separate parameter sets, separate optimizers, and no shared layers:
| Phase | Input to discriminator | Gradient destination | Optimizer stepped |
|---|---|---|---|
| Discriminator, real | Real batch | Discriminator parameters | Discriminator |
| Discriminator, fake | Detached generated batch | Discriminator parameters | Discriminator |
| Generator | Connected generated batch | Generator through discriminator operations | Generator |
The first two rows can contribute to one combined discriminator loss and one step. They do not require two optimizer steps. Report whether you sum or average those contributions; that choice changes the scale of the reported loss and may affect the effective update magnitude.
Clear the relevant accumulated gradients before computing the next update. Taking a step is not the same as clearing gradients. If a generator gradient left over from an earlier phase remains in memory, observing a nonempty gradient after the discriminator phase does not prove that phase created it. Inspect a clean baseline or compare newly computed values.
Why does detach matter, and when is no_grad wrong?
Official API behavior: Tensor.detach returns a tensor disconnected from the current autograd graph. It shares storage with the original tensor, so detaching is not a promise of an independent copy. Avoid mutating the detached sample in place when the connected sample is still needed.
For the discriminator's fake-example loss, disconnecting the generated batch prevents that loss from backpropagating into the generator. You still need gradients through the discriminator to train it. Wrapping the entire discriminator computation in no_grad would defeat that requirement.
During the generator phase, you need the reverse dependency: derivatives must pass through the discriminator's operations to the generated input and onward into the generator. Detaching the fake samples here cuts that path. Placing the discriminator forward pass inside no_grad also cuts it, even if you later call the generator optimizer.
Freezing discriminator parameters for this phase can avoid computing unused parameter gradients while preserving the derivative with respect to its input. PyTorch's autograd mechanics distinguishes parameter gradient requirements from recording operations. eval() is a separate decision: it changes behavior of modules such as dropout and batch normalization, rather than serving as a general gradient switch.
Explain buffers as well as parameters. A discriminator in training mode can update batch-normalization running statistics even when its trainable parameters are frozen. If your test asserts that the entire discriminator state never changes, explicitly control those buffers or use a model without them. That is a stronger contract than saying its optimizer did not step.
Original scalar exercise: who receives this gradient?
Consider a deliberately small model: generator output x = g*z, discriminator probability p = sigmoid(d*x), and generator loss L_G = -log(p). There are no biases, buffers, or shared parameters. Set g = 1, d = 2, and z = 1.
The probability is approximately 0.8808. The derivative of the generator loss with respect to the discriminator logit is p - 1, approximately -0.1192. Applying the chain rule gives:
dL_G/dg = (p - 1) * d * z ≈ -0.2384
With a generator learning rate of 0.1, one plain gradient-descent update moves g to approximately 1.02384. In this case, increasing the generated input makes the discriminator assign it a higher real probability. This arithmetic demonstrates a connected generator path; it does not demonstrate successful distribution learning.
Now replace the generator objective with a discriminator fake-example objective, L_D_fake = -log(1-p). Its derivative with respect to the logit is p, giving a connected derivative with respect to g of approximately 1.7616. If that derivative reaches the generator during a discriminator phase, the graph boundary is wrong for the intended alternating update.
If the generated input is treated as a fixed detached value, changing g does not change that discriminator-phase loss through the recorded path. The intended generator gradient from that phase is absent. Distinguish this graph statement from differentiating the original mathematical expression with x still defined as g*z.
For practice, compare the analytic generator derivative with a central finite difference using a small perturbation. Then repeat with the input held fixed. These are bounded scalar checks, not a substitute for testing your framework implementation. A full training test should additionally compare parameter snapshots before and after each optimizer step.
How would you debug an alternating loop that fails?
Use one batch and one update pair before changing architecture or learning rates. Log which optimizer owns each parameter, clear gradients, and save parameter snapshots. After the discriminator phase, discriminator parameters should change under a nonzero update; generator parameters should remain unchanged. After the generator phase, check the opposite optimizer ownership.
Inspect the generator gradient itself. A zero change may reflect a disconnected graph, a zero learning rate, a zero input, saturated gradients, or an optimizer that does not contain the intended parameter. A parameter snapshot alone cannot distinguish these causes. Collect gradient norms, optimizer membership, and the loss inputs together.
If a backward pass fails because the graph was already consumed or a needed tensor changed in place, do not immediately add retain_graph=True. Draw the two phases and determine which forward computations each loss needs. A fresh discriminator forward pass after its step usually expresses the intended dependency more clearly than retaining an obsolete result.
Also check shapes and labels. A probability loss expects appropriate probability inputs; a logits-based loss expects logits. Applying a sigmoid before a logits-based binary cross-entropy loss changes the objective. Real and fake labels must match the declared convention, and broadcasting should not silently create an unintended comparison matrix.
These are debugging recommendations inferred from the update contract. They are not claims that one prescribed architecture or one optimizer ratio works for every GAN.
Why can balanced loss curves be misleading?
For this binary-cross-entropy convention, a discriminator that outputs 0.5 everywhere gives about 0.693 for each real and fake contribution. Their sum is about 1.386; their average is about 0.693. The non-saturating generator loss is also about 0.693. A GAN loss chart showing these numbers must disclose its reduction before anyone compares curves.
Those values can occur at the ideal indistinguishability described in the original research, but they are also compatible with an untrained constant discriminator. They do not prove that the generator matches the data. Conversely, a small discriminator loss may mean it easily separates poor generated samples, rather than that the overall system improved.
Loss trajectories depend on both changing players. A generator can improve its samples while its measured loss worsens because the discriminator improved faster. Interpret trends with sample checks and distribution measurements collected at the same checkpoints. Avoid treating adversarial loss like a supervised validation loss with one fixed target model.
The non-saturating generator objective used here also differs from directly minimizing the original minimax generator expression. Name the objective you use when discussing weak gradients. Borrowing a numerical threshold from a Wasserstein critic or another loss family would not answer this binary-cross-entropy case.
Original four-mode case: is this generator collapsing?
Suppose the reference distribution contains four equally likely, well-separated two-dimensional clusters. Draw 400 generated samples and assign each to its nearest reference center, while separately recording its distance. This is an original diagnostic exercise with deliberately simple geometry.
| Generator | Counts near the four centers | Occupied modes | What the counts establish |
|---|---|---|---|
| A | 100, 100, 100, 100 | 4 | Balanced nearest-center assignments in this sample |
| B | 400, 0, 0, 0 | 1 | Severe missing-mode behavior |
| C | 100, 100, 100, 100 | 4 | Coverage counts alone cannot establish closeness |
Generator B can produce sharp, plausible examples from one cluster while ignoring the other three. Generator C may place points far from every cluster, yet nearest-center assignment still gives balanced counts. Add a distance threshold and count outliers so that an unrealistic point is not credited as a successful mode match.
This case separates fidelity from coverage. Fixed latent inputs help compare the same examples across checkpoints, but they cannot estimate the entire distribution. Use additional fresh latent draws, enough samples for the intended comparison, and held-out real data. Report uncertainty when a small generated sample misses a rare mode by chance.
For image tasks, the FID research introduces a feature-distribution comparison. A metric result still depends on the feature representation, preprocessing, reference data, and sample protocol. Compare models under the same procedure and inspect generated failures. A single score does not identify which semantic categories disappeared or prove freedom from memorization.
What makes a strong GAN interview answer?
A strong answer names the objective, traces the connected and detached paths, identifies the optimizer that steps, and proposes a test that could reject its explanation. Then it evaluates what the generator produces. Correct parameter ownership and good-looking samples address different failure modes.
If asked to improve a collapsing model, begin by verifying the loop and documenting the missing modes. Propose one controlled change, such as an update schedule or objective suited to the observed problem, and compare against the same evaluation protocol. State what outcome would contradict your diagnosis. Avoid listing techniques without connecting them to evidence.
| Practice question | Transfer to GAN reasoning |
|---|---|
| Explain ML and statistical modeling | State modeling assumptions and evaluation limits. |
| Explain LLM fine-tuning and generative models | Compare generative objectives without equating their losses. |
| Implement and derive backprop from scratch | Trace the derivative to the intended parameter. |
| Debug and fix a PyTorch Transformer training loop | Check graph, shape, loss, and optimizer contracts. |
| Prevent Vanishing Gradients in Deep Networks | Explain a weak gradient using the actual objective. |
Continue with Implement and derive backprop from scratch, then explain the scalar GAN exercise aloud without reading the derivative. Your answer should make clear what changed, why it changed, and what remains unproven about the generated distribution.