Articles

Distributed Training Interview Questions: Parallelism, AllReduce, Checkpoints, and Failures

Prepare for distributed training interviews with parallelism, AllReduce, checkpoints, scaling failures, stragglers, and recovery debugging.

Your training job takes 2.1 seconds per step on 8 GPUs. After scaling to 64 GPUs, step time rises to 2.8 seconds, one rank regularly stalls, and the first checkpoint after a restart produces a different loss curve. Where do you begin?

That scenario captures what strong distributed training interview questions test. Knowing the definitions of data parallelism and AllReduce is useful. Senior candidates must also reason about memory, communication, topology, stragglers, recoverability, and numerical correctness as one system.

Then practice debugging and scaling a PyTorch training loop and designing an LLM fine-tuning platform. Those prompts turn the concepts below into interview-ready answers.

Quick Answer: Treat Training as a Distributed System

A strong answer follows four resources through one training step.

Compute: Which kernels run, how much useful work each accelerator receives, and where devices wait.

Memory: Which rank owns parameters, gradients, optimizer states, activations, and temporary communication buffers.

Communication: Which collective moves each tensor, how much data crosses the fabric, and whether transfers overlap with computation.

State: What must survive a failure so the job resumes at the intended global step with equivalent optimizer, scheduler, data, and random-number state.

Before proposing a parallelism strategy, clarify model size, sequence length, precision, accelerator memory, node topology, interconnect bandwidth, target global batch, failure rate, checkpoint storage, and the metric that matters: samples or tokens per second, model FLOP utilization, time to train, or cost to target quality.

A Framework for Answering Distributed Training Questions

Use this sequence in design and debugging rounds: establish a baseline, identify the bottleneck, map ownership, map communication, protect correctness.

Begin with a single GPU or single node profile. Break step time into input, forward, backward, collective communication, optimizer, and checkpoint work. Then draw rank groups and tensor shapes. Finally, state how you will validate throughput, loss equivalence, restart behavior, and failure recovery. This prevents a familiar interview mistake: recommending more GPUs before proving what limits the current job.

Parallelism Interview Questions

1. How do data, tensor, pipeline, and context parallelism differ?

Data parallelism replicates the model and splits the batch. Each rank computes local gradients, then synchronizes them. It is simple when the model and optimizer state fit on every worker, but communication grows with gradient volume.

Tensor parallelism partitions operations within a layer. It helps when a layer does not fit on one device, but introduces frequent communication and usually benefits from fast links within a node. Pipeline parallelism places groups of layers on different stages and sends activations between them; microbatches improve utilization, while pipeline bubbles and stage imbalance reduce it.

Context or sequence parallelism partitions work along the sequence dimension, which is useful for long contexts. Expert parallelism distributes Mixture-of-Experts experts and often relies on all-to-all traffic. These methods can be composed. The Megatron-LM scaling paper evaluates combined tensor, pipeline, and data parallelism rather than treating them as mutually exclusive choices.

2. When should you use DDP, FSDP, or ZeRO?

Use DistributedDataParallel when each replica can hold the model, gradients, and optimizer state. DDP is a strong baseline because its ownership and communication are comparatively easy to reason about.

Use sharded data parallelism when replicated state becomes the memory bottleneck. PyTorch FSDP shards module parameters across data-parallel workers, while the ZeRO paper describes partitioning optimizer states, gradients, and parameters to remove redundancy. Sharding saves memory but adds all-gather and reduce-scatter traffic, lifecycle complexity, and checkpoint considerations.

3. How do microbatch size, gradient accumulation, and global batch relate?

For ordinary data parallel training, a useful starting equation is:

global_batch = microbatch_per_rank x data_parallel_ranks x accumulation_steps

Tensor and pipeline ranks do not necessarily multiply independent data samples, so use the data-parallel world size in that equation. If the global batch changes, learning-rate and scheduler behavior may also need to change. A candidate who scales the worker count without tracking the effective batch can mistake an optimization change for a systems improvement.

AllReduce and Collective Communication Questions

4. What does AllReduce do during data-parallel training?

Every rank contributes a tensor, the values are reduced, and every rank receives the result. For gradient synchronization, the reduction is usually a sum followed by the framework's chosen averaging semantics. NVIDIA's NCCL collective documentation also notes that reduce-scatter followed by all-gather is equivalent to AllReduce.

Know what other collectives are for. All-gather reconstructs sharded values on every participant. Reduce-scatter combines values while leaving each rank with one shard. Broadcast sends data from one root. All-to-all exchanges distinct chunks among ranks and is common in expert routing.

5. Why can AllReduce dominate step time?

The job may expose communication because gradients are large, links are slow, ranks span a poor topology, buckets become ready late, or one straggler delays everyone. PyTorch DDP groups gradients into buckets so reduction can potentially overlap with backward computation. If the critical-path buckets become ready only near the end, that overlap disappears.

A good investigation measures bytes per collective, duration by rank, overlap with kernels, topology, and achieved link bandwidth. First compare one GPU, one node, and multiple nodes. Then test bucket sizing, gradient accumulation, lower-precision communication where numerically acceptable, and placement that keeps communication-heavy groups on faster links.

6. How do you debug a distributed hang?

Find the last collective entered by every rank. A mismatch in collective order, tensor count, shape, or dtype can hang or corrupt a job; NCCL requires participating ranks to call a collective consistently. Also check that one rank did not fail earlier in data loading, CUDA execution, or control flow while the others waited.

Capture per-rank logs with timestamps, enable focused NCCL diagnostics, and verify network interfaces and GPU-to-NIC paths. NVIDIA's current NCCL troubleshooting guide recommends narrowing runtime, topology, networking, and performance problems before changing tuning knobs.

Checkpoint and Recovery Questions

7. What belongs in a training checkpoint?

A restartable checkpoint needs more than weights. Save model and optimizer state, global step or consumed-token count, scheduler state, mixed-precision scaler when used, random-number generator state, data or sampler progress, and the configuration and code/data versions needed to interpret the state.

PyTorch's checkpoint tutorial explicitly calls out model and optimizer state for resuming training. The rest closes common reproducibility gaps. Validate recovery by restarting a short controlled run and comparing the next batches, learning rate, gradients, and loss trajectory.

8. How should a sharded checkpoint work?

Do not gather a very large model onto rank 0 simply to save it. Let ranks write shards in parallel, include metadata describing tensor ownership, and commit a checkpoint only after all required pieces are durable. Readers should ignore incomplete generations.

Plan for a different world size at restore time. PyTorch Distributed Checkpoint supports parallel save/load and load-time resharding; its documentation explains that the target model's sharding information guides the load. Measure checkpoint pause time, asynchronous staging memory, storage throughput, restore time, and failure behavior.

9. How often should you checkpoint?

This is an expected-cost decision. Frequent checkpoints reduce lost compute after a failure but consume bandwidth, storage, and possibly training time. Longer intervals do the reverse. Start from observed job failure frequency, checkpoint and restore duration, job cost, and the maximum acceptable rollback window.

For long jobs, combine durable global checkpoints with faster local or incremental mechanisms when the platform supports them. Never claim asynchronous saving is free: copying state can increase host or device memory pressure and compete with training I/O.

Failure and Performance Debugging Questions

10. More GPUs made training slower. What do you check?

Confirm the comparison is fair: same model, precision, sequence distribution, global batch policy, data pipeline, warmup, and number of measured steps. Report strong-scaling efficiency, not only aggregate throughput.

Then split the regression into compute, input, communication, optimizer, and idle time. Check whether per-GPU work became too small, collective traffic crossed nodes, pipeline bubbles grew, a data reader fell behind, or checkpoint and evaluation cadence changed. More hardware cannot repair a workload that has too little parallel work or too much synchronization.

11. One rank is slower than the others. How do you isolate it?

Compare step timelines by rank. A slow rank can come from thermal throttling, hardware errors, CPU contention, uneven input lengths, data skew, network retransmits, topology, or a synchronization point that makes an earlier delay look like an AllReduce problem.

Swap the workload across hosts, compare hardware telemetry, replay a fixed synthetic batch, and run communication benchmarks independently of the model. The goal is to separate model/data skew from host, accelerator, and fabric faults.

12. The job resumes, but the loss diverges. What failed?

Check optimizer, scheduler, scaler, RNG, sampler, and global-step state before blaming floating-point nondeterminism. Verify that every shard came from the same checkpoint generation and that the resumed world size preserves the intended global batch and sample order.

Run a short equivalence test: save at step N, execute steps N+1 through N+k, restore, and replay them. Compare learning rate, sample identifiers, loss, gradient norms, parameter checksums, and overflow decisions. Exact bitwise equality may not be realistic across every topology, but unexplained training-quality drift is not an acceptable recovery policy.

A Complete Distributed Training Incident Walkthrough

Return to the opening case. Scaling from 8 to 64 GPUs makes steps slower, rank 37 stalls, and a restarted run drifts. Treat those as three hypotheses, not one mysterious NCCL failure.

First, profile the 8-GPU baseline and the 64-GPU run with identical workload semantics. The larger run crosses nodes, per-rank microbatches shrink, and gradient reductions move onto a slower fabric. One host also shows periodic input stalls. Communication is exposed because the delayed rank prevents its peers from completing each collective.

Second, audit recovery independently. The checkpoint contains model and optimizer shards but omits scheduler, sampler, and RNG state. The job restarts at the right filename but with a repeated data window and the wrong learning-rate position.

incident_plan:
  compare: [single_gpu, single_node, multi_node]
  trace_by_rank: [input, forward, backward, collectives, optimizer]
  verify: [global_batch, sample_ids, learning_rate, gradient_norm]
  isolate_straggler: [synthetic_batch, host_swap, nccl_test, telemetry]
  recovery_test: save_N_then_replay_N_plus_1_to_N_plus_k
  success:
    - improved_tokens_per_second_and_scaling_efficiency
    - no_persistent_outlier_rank
    - equivalent_resume_within_defined_tolerance

The likely fix combines larger per-rank work or accumulation, better rank placement, a repaired data path on the outlier host, and a complete versioned checkpoint. The interview signal is not guessing the final knob. It is designing experiments that isolate independent failure domains.

How Interviewers Score Distributed Training Answers

Correctness: You explain tensor ownership, group membership, collectives, and effective batch without hand-waving.

Trade-offs: You connect each parallelism choice to memory, communication frequency, utilization, and operational complexity.

Measurement: You use per-rank traces, tail behavior, scaling efficiency, achieved bandwidth, and quality-equivalence checks.

Reliability: You treat checkpoints, partial writes, resharding, retries, and reproducibility as first-class design work.

Judgment: You establish a baseline and isolate bottlenecks before proposing a larger cluster or a framework migration.

A Focused 5-Day Preparation Plan

Day	Practice output
1: Parallelism	Draw data, tensor, pipeline, context, and expert groups for one cluster.
2: Collectives	Trace tensor shapes through AllReduce, all-gather, reduce-scatter, and all-to-all.
3: Performance	Diagnose a poor 8-to-64-GPU scaling result from a rank timeline.
4: Recovery	Specify a versioned, atomic, reshardable checkpoint and replay test.
5: Full mock	Answer one design question and one failure incident aloud in 45 minutes.

Speak in hypotheses, measurements, and decision criteria instead of reciting framework features.

Frequently Asked Questions

Is AllReduce the same as averaging gradients?

AllReduce is the collective that reduces values and returns the result to every rank. Whether the final gradient is a sum or average depends on the reduction operation and framework semantics. In interviews, state those semantics explicitly before reasoning about learning rate or batch scaling.

Does FSDP always train faster than DDP?

No. FSDP can fit larger models by sharding state, but it adds parameter all-gathers, gradient reduce-scatters, prefetch choices, and checkpoint complexity. If a model fits comfortably with DDP, replication may be simpler or faster. Profile the real model and topology.

What is the most important distributed training metric?

There is no single metric. Track tokens or samples per second, scaling efficiency, time to target quality, accelerator utilization, communication exposure, memory headroom, failure loss, and total cost. A throughput improvement that changes optimization semantics or hurts convergence is not a valid win.

What causes most distributed training hangs?

Common causes include mismatched collective order or tensor metadata, a rank failing before peers reach the same operation, network or topology faults, and divergent control flow. Find the last progress point on every rank before tuning timeouts or NCCL environment variables.

Practice the Failure Path, Not Just the Happy Path

The best distributed training answers explain not only how a model uses 64 GPUs, but also what happens when rank 37 slows down, a checkpoint is partially written, the world size changes, or resumed loss no longer matches expectations.

Turn one real scaling or recovery incident into a concise story for behavioral and leadership interview practice: the evidence you gathered, the trade-off you chose, and how you prevented recurrence.