AI Infrastructure Engineer Interview Guide 2026: GPU Clusters, Distributed Training, Serving, and Reliability

“We have eight GPUs” is not a capacity plan. Four might be the wrong type, a job might need two tightly connected nodes, and a checkpoint might saturate the same storage path that serves interactive requests. A useful AI infrastructure interview answer turns those constraints into a placement, a recovery path and an explicit decision about what waits.

This guide follows a mixed workload through admission, GPU allocation, distributed training, serving and failure recovery. The cluster, timings, quotas and workload figures are fictional teaching inputs. They are not a company's architecture, a hardware benchmark or private interview material. Start from the actual posting to choose which part deserves the most depth.

For a diagnostic drill, use Debug Oscillating GPU Utilization During Training. That Tower Data Scientist practice record asks how to distinguish input stalls, CPU preprocessing, synchronization, communication and kernels. It is adjacent infrastructure practice, not a prediction of your interview.

Turn the job description into a preparation map

AI infrastructure can mean training platforms, inference, GPU control planes, data movement or reliability. Do not assume one interview sequence or a universal cloud requirement. Ask about coding, design, debugging, presentations and permitted tools, then match the evidence you prepare to the team's work.

Posting emphasis	Decision to practice	Reviewable artifact
GPU scheduler	Placement, quotas, fairness and topology	Resource ledger and rejected placement
Distributed training	Consistent state and worker recovery	Checkpoint manifest and failure timeline
Model serving	Admission, batching, memory and deadlines	Per-model capacity and degradation policy
Platform/SRE	User-impacting symptoms and safe changes	Diagnostic sequence, rollback and operator ownership
Data pipeline	CPU preprocessing and storage bottlenecks	Per-step trace and controlled experiment

A job being scheduled is different from a job meeting its deadline. A serving replica being alive is different from requests reaching their first token on time. Keep those distinctions in your opening requirements and telemetry.

Original case: allocate eight GPUs without hiding the constraints

There are four GPU nodes, A through D. Each has two GPUs with a nominal 80-GiB memory budget, 32 CPU cores and 128 GiB of host RAM. Reserve 8 GiB on each GPU for runtime buffers and headroom, leaving 72 GiB for the declared workload footprint. A separate CPU node P has 16 cores and 64 GiB RAM.

Assume the training job can use two nodes only if both GPUs on each are available and the required inter-node network is healthy. Inside-node connectivity and cross-node performance have already been validated for this fictional job. Do not infer that any four GPU devices have equivalent communication cost.

Workload	GPU requirement	Per-worker CPU/RAM	Constraint and target
Interactive serving	2 GPUs, 48 GiB each	4 cores / 12 GiB	Reserve node A; 80 requests/s demand
Training	4 GPUs, 64 GiB each	8 cores / 32 GiB	Two complete nodes; finish within 45 min
Batch evaluation	2 GPUs, 24 GiB each	4 cores / 16 GiB	Preemptible; 60-min deadline
CPU evaluation	No GPU	8 cores / 16 GiB	5-min work estimate; 10-min deadline

The declared GPU quotas are serving 2, training 4 and batch 2. They are admission limits, not ownership of arbitrary idle memory. This case uses whole GPUs without time sharing. Put serving on A, training on B and C, batch on D, and CPU evaluation on P.

Training consumes 16 cores and 64 GiB host RAM on each of B and C, below their node budgets. Serving uses 8 cores and 24 GiB on A; batch uses 8 cores and 32 GiB on D. P fits its CPU-only task. The reserved serving node is a policy choice that sacrifices packing efficiency to simplify isolation.

Why three plausible placements fail

One 64-GiB training worker fits a 72-GiB usable device, but two such workers do not share that budget: 128 exceeds 72. Host RAM cannot be silently substituted for GPU memory without an explicit offload design and new performance evidence.

Placing training on B0, B1, C0 and D0 violates this case's two-complete-node requirement, even though the total is four GPUs. Putting batch on an already assigned training device violates exclusive allocation. Moving serving to a spare device outside A violates the reservation policy, unless that policy is deliberately revised.

Translate the policy into a real scheduler carefully

Kubernetes documents vendor drivers and device plugins exposing schedulable GPU resources. GPU limits can supply the request; if requests and limits are both specified, they must match. Node labels and selectors can distinguish accelerator types. A minimal resource declaration is useful vocabulary:

# Fragment for one container, not a complete training platform.
resources:
  limits:
    nvidia.com/gpu: 1

That fragment does not implement the case's gang admission, two-node topology, per-job memory verification or reserved serving pool. The control plane must validate the entire placement against a current inventory and reserve it atomically, or release partial allocations if admission fails. Otherwise several jobs can each hold resources while none has enough to start.

Record an allocation generation so a delayed worker cannot attach to a reassigned device. Define job states such as queued, reserved, starting, running, checkpointing and recovering, with an owner for transitions. A quota or priority declaration is incomplete without explaining how users see why their job waits.

For fairness, distinguish guaranteed capacity from opportunistic batch work. Give the latter a maximum waiting policy or a separate service target; “interactive always wins” can starve batch indefinitely. Admission must also check data access and tenant authorization. Resource availability does not establish permission to load a dataset or model.

Training deadline: account for checkpoint time

The training case needs 2,400 useful steps at 0.5 seconds per step with four workers. Startup takes 60 seconds. Every 300 useful steps, including the final step, a coordinated checkpoint pauses training. The checkpoint has 32 GiB across the workers, an effective aggregate write rate of 2 GiB/s and 4 seconds of coordination overhead.

One checkpoint costs 32/2 + 4 = 20 seconds. There are eight checkpoints, so the no-failure plan is 60 + 2,400 × 0.5 + 8 × 20 = 1,420 seconds, or 23 minutes 40 seconds. That fits the 45-minute deadline under the supplied assumptions.

from fractions import Fraction

useful = 2400 * Fraction(1, 2)
checkpoint = 32 / Fraction(2) + 4
baseline = 60 + useful + (2400 // 300) * checkpoint
lost_work = (1300 - 1200) * Fraction(1, 2)
one_failure = baseline + 10 + 90 + 20 + lost_work
assert checkpoint == 20
assert baseline == 1420
assert one_failure == 1590 and one_failure < 45 * 60

The checkpoint pause is 20/(150 + 20), about 11.76% of each useful-step-plus-checkpoint cycle, excluding startup. If effective storage bandwidth halves, the same write becomes 36 seconds. This is why checkpoint duration belongs in both capacity planning and incident diagnosis.

Recovery checkpointing is different from activation checkpointing. The former preserves training state across interruptions; the latter trades recomputation for GPU memory during backward passes. Do not answer a recovery-point question by naming a memory-saving technique.

Node failure: restore one complete checkpoint for the worker group

Suppose node C fails after useful step 1,300. The latest complete checkpoint is step 1,200, so 100 steps, or 50 seconds, need recomputation. Healthy workers on B cannot simply continue from 1,300 while replacements resume 1,200; the state must be coordinated.

Pause and checkpoint the opportunistic batch work on D, with a supplied 10-second drain cost. Restart the four-worker group on B and D in 90 seconds. Reading the retained 32-GiB checkpoint plus coordination takes another 20 seconds under the same effective-bandwidth assumption. The extra cost is 10 + 90 + 20 + 50 = 170 seconds.

The one-failure estimate becomes 1,590 seconds, or 26 minutes 30 seconds. It still fits 45 minutes. This is a single-event estimate with no further failure, no extra requeue delay and unchanged step performance on B/D. Name those assumptions instead of calling the result an availability guarantee.

Batch work can resume after training releases D. A conservative bound allows 2 minutes of batch restart and a full 20 minutes of remaining useful work: 26:30 + 2:00 + 20:00 = 48:30, below its 60-minute deadline. Do not depend on retained partial progress until its checkpoint is validated.

PyTorch's torchrun documentation describes worker-group restarts after failures, with bounded restart attempts. Application code still needs to save and load the state required for continuity. Resuming from a snapshot does not automatically make a changed world size or data ordering equivalent to the original run.

A checkpoint is more than a directory that exists

For this teaching design, publish a manifest only after every required rank's files are complete and verified. Include run identity, step, model/optimizer state, scheduler state, data position, random-state metadata, configuration and format version. Store expected lengths and checksums, and separate a temporary write from a checkpoint eligible for recovery.

If a newer step-1,300 directory contains only three of four ranks, choose the complete compatible step-1,200 checkpoint. Reject a wrong run or incompatible format rather than filling missing state with defaults. The local manifest mock checks declared completeness and compatibility; it does not read files or compute checksums.

PyTorch DCP supports parallel multi-rank saving and loading, including resharding, and produces multiple files. Its documentation warns that saved state dictionaries have no cross-version backward-compatibility guarantee. Pin the relevant environment and test restore. For asynchronous saves, distinguish staging completion from upload/save completion before advertising a checkpoint as recoverable.

Serving headroom: memory fit is only one condition

The supplied serving profile supports 60 requests/s per replica for a specified model and request mix. Apply an 80% planning factor: two replicas give 2 × 60 × 0.8 = 96 requests/s, above the assumed demand of 80. These are fictional profile inputs, not measurements made for this guide.

Now suppose shared storage/network interference reduces the same profile to 40 per replica. Planned capacity becomes 64 requests/s, leaving 16 requests/s beyond the chosen budget. Adding a longer request queue does not restore the missing service capacity; it spends the first-token deadline instead.

If one serving replica fails, the surviving planned budget is 48 requests/s. At demand 80, 32 requests/s, or 40%, require rejection, deferral or an explicitly acceptable fallback until capacity returns. This two-replica plan therefore has throughput headroom but cannot meet the full supplied demand after losing one replica.

Define any fallback before the incident. A smaller approved model changes quality and output behavior; silently switching models is not an equivalent response. Use per-model/config telemetry and tenant limits, and communicate degradation. Separate training priority from serving admission so a recovery operation does not unexpectedly consume reserved interactive resources.

Diagnose interference from the timeline

For the linked utilization drill, align per-worker step time, data-loader wait, CPU preprocessing, collectives, checkpoint writes and storage throughput. Synchronized dips at checkpoint boundaries suggest a different investigation from one straggling worker with an empty input queue.

Run one controlled change at a time: reuse a fixed in-memory input batch, vary preprocessing workers, suspend competing storage writes, or inspect collective timing. Keep model/config and correctness checks fixed while comparing throughput. More CPU loader workers can create contention rather than eliminate it.

For a network calculation, the case's four workers each request 120 MiB/s of data against a supplied safe shared-path budget of 350 MiB/s. Their 480 MiB/s demand does not fit. Reducing each to 80 gives 320; adding a 20-MiB/s serving load and 10-MiB/s overhead reaches 350 exactly, leaving no maintenance headroom. These are separate illustrative traffic assumptions, not converted checkpoint bandwidth figures.

PyTorch CUDA operations are asynchronous by default. Use appropriate synchronization or CUDA events when measuring completed device work. A host timer around kernel submission can report a reassuring number while missing the execution it was meant to measure.

A seven-day plan with named outputs

Day	Main exercise	Finished output
1	Resource-bound coding and concurrency	Admission invariant, rejected input and memory bound
2	Place the mixed workload	Per-node ledger, quota checks and three infeasible alternatives
3	Checkpoint interruption and restore	Complete-manifest criteria and recovery-time calculation
4	Serving overload and replica loss	Per-model budget, degraded response and rollback owner
5	Investigate oscillating utilization	Aligned timeline, next signal and one controlled experiment
6	Rehearse a full platform design	Job-state transitions, failure matrix and operational owners
7	Compare with the invitation	Weakest answer, corrected assumption and next mock

Local verification executed 40 checks covering exclusive placement, topology, quotas, GPU-memory arithmetic, CPU feasibility, checkpoint costs, recovery deadlines, serving headroom and declared-manifest fallback. No GPU job, CUDA kernel, Kubernetes allocation, actual training or durable-storage recovery was executed.

Bring one real incident or migration you can discuss with permitted evidence. Explain impact, containment, diagnosis, recovery and what changed afterward. A useful preparation artifact lets another engineer challenge an assumption without needing to trust a confident conclusion.

Frequently asked questions

Do I need to know a specific cloud or GPU product?

Use the target posting to choose the platform depth. Explain resource discovery, topology, placement, isolation and recovery before translating those concepts to a particular vendor.

Does an 80-GiB GPU automatically fit an 80-GiB workload?

No. Account for runtime and allocator overhead, activations or KV cache, temporary buffers and safety headroom. The example reserves 8 GiB and admits workloads against 72 GiB; those are fictional planning inputs.

Can Kubernetes GPU counts alone enforce this training placement?

No. Device resources and node selection are part of placement, but the example also needs atomic multi-worker admission, topology, quotas and serving reservations. The tiny manifest does not implement that policy.

Is activation checkpointing the same as recovery checkpointing?

No. Activation checkpointing trades recomputation for GPU memory during backward passes. Recovery checkpoints retain training state so a job can resume after interruption.

Were the GPU and recovery results measured on a real cluster?

No. Local checks verify synthetic placement, arithmetic and declared checkpoint-manifest selection. The guide performs no CUDA, Kubernetes, actual training or durable-storage execution.

Sources and scope

Reviewed October 8, 2026. Technical documentation grounds resource and recovery concepts, not a universal interview loop. All cluster, deadline, bandwidth, memory and serving figures are supplied fictional inputs; local checks verify their reasoning only.

• Kubernetes GPU scheduling — device resources, limits and node selection.
• PyTorch DCP, version 2.14 — multi-rank checkpoints, completion and compatibility cautions.
• PyTorch torchrun — worker failures and group restarts; main documentation can evolve.
• PyTorch CUDA semantics, version 2.14 — asynchronous operations and correct device timing.