NVIDIA SRE Interview Preparation: Diagnose GPU Cloud and Kubernetes Failures

NVIDIA SRE preparation should connect Linux and Kubernetes symptoms to the GPU workload that is failing. A pending pod, a healthy node, and low average GPU utilization can all coexist with unusable capacity. Your answer needs a diagnostic chain that separates scheduling, device health, workload behavior, and the authority to remediate.

Role evidence, not a guaranteed interview loop: a current NVIDIA Principal SRE posting for AI Platform Runtime emphasizes infrastructure reliability across cloud and Kubernetes environments, with GPU infrastructure among relevant skills. It is a particular principal-level role, not a specification for every NVIDIA SRE opening. Confirm your team's scope and assessment format with the recruiter.

Begin with Design a GPU Telemetry Collection and Analytics Platform. Explain which signals distinguish device failure from poor utilization before choosing dashboards. The worked incident below is original practice, not a reported NVIDIA interview question.

Establish the workload and failure boundary

Clarify whether the service runs inference, independent batch tasks, or distributed training. A failed inference replica may have a different recovery path from a training job whose workers must synchronize. Ask about checkpointing, tenant isolation, scarce capacity, and whether a job needs several GPUs on the same host.

Define the symptom using an observable fact. “Training is broken” might mean a job never scheduled, a container exited, collective communication stalled, or throughput fell while progress continued. Each case leads to different evidence. Separate the time the customer noticed the problem from the earliest relevant node or device event.

For this fictional incident, assume eight nodes with eight dedicated GPUs each, no partitioned GPUs or time-sharing configuration, and a new job requiring eight GPUs on one node. Seven healthy nodes each have seven GPUs assigned. The eighth node has no active job, but its device plugin advertises only seven healthy allocatable GPUs after a previous workload failure.

There are fourteen apparently available GPUs: seven single-device slots on the healthy nodes and seven on the affected node. Nevertheless, no node offers eight. A fleet total cannot satisfy this placement requirement. This is fragmentation plus lost device capacity, not proof that the scheduler ignored idle hardware.

Diagnose Pending before debugging CUDA

Start with the pending pod's events and resource request, then inspect eligible nodes. Look for insufficient device resources, node affinity, taints, topology constraints, and other placement conditions. Use the exact namespace, pod, and node identities from the incident rather than inspecting an unrelated workload with a similar name.

Illustrative read-only commands, after authorization and context verification, include:

kubectl config current-context
kubectl describe pod training-job-0 -n research
kubectl get nodes
kubectl describe node gpu-node-8

These commands are a starting point, not a universal runbook. A pod that has never started cannot yet supply an application-level CUDA failure from its own running process. Conversely, once the container runs, a scheduling explanation alone does not resolve a later driver or application error.

The Kubernetes GPU scheduling documentation describes the device-resource request and limit rules. Inspect the advertised resource name and effective pod demand rather than assuming that every GPU configuration uses the same allocation model. Our exercise deliberately excludes sharing and partitioning to keep the count interpretable.

Explain the next discriminating observation. If one eligible healthy node has eight free advertised devices yet the pod remains pending, revisit placement constraints and scheduler events. If no node has eight, the immediate scheduling outcome is consistent with the stated requirements; investigate why capacity disappeared or whether the workload can legitimately change shape.

Distinguish capacity, allocatable, and available

In the exercise, the affected node may still report capacity of eight while allocatable falls to seven. The device-plugin documentation explains that an unhealthy device reduces allocatable resources without changing capacity. It also warns that a pod already assigned that device may fail; the scheduler does not repair its running workload automatically.

That distinction matters twice. First, sixty-four physical GPUs are not sixty-four schedulable healthy GPUs. Second, allocatable resources are not all free: existing pod assignments consume them. A dashboard that subtracts average utilization from physical capacity can therefore make both errors at once.

For each eligible node, reconcile advertised allocatable devices with assigned requests and the job's placement requirement. Correlate the plugin's device identity with host telemetry carefully; a GPU index in a container is not enough to establish a stable physical-device identity across restarts or configurations.

Do not broaden a healthy node status into a healthy accelerator claim. Node readiness summarizes particular node conditions; it is not a complete GPU functional test. Likewise, an application log mentioning a device does not establish that the hardware is defective. Driver, runtime, permissions, workload, and hardware hypotheses still need separation.

Build a scheduler-to-device evidence ledger

Use a small ledger to prevent unrelated symptoms from becoming one invented root cause. The following observations are fictional and intentionally incomplete.

Signal	What it supports and what to check next
New pod requires eight GPUs on one node; events show insufficient devices	Placement cannot use fourteen fragmented free devices. Check eligible-node counts and constraints.
Affected node: capacity eight, allocatable seven; NodeReady is true	One device is unavailable to new allocation despite node readiness. Correlate plugin and host records.
Earlier workload failed on that node; host log includes Xid 79	Investigate the GPU-disconnection symptom using the applicable catalog and platform runbook; do not declare the cause from the code alone.
Other jobs show lower throughput but no device-health event	Investigate data loading, storage, communication, and application changes separately rather than resetting every node.

NVIDIA's Xid catalog classifies Xid 79 as a GPU falling off the bus. That describes a failure symptom, not a complete causal diagnosis or permission to reset production. Match guidance to the hardware, driver, error context, and current catalog.

Collect the timeline, device identity, relevant logs, workload impact, and recent changes before an action destroys evidence. A driver rollout near the incident is a hypothesis, not causality. Compare affected and unaffected nodes with the same workload and configuration where possible.

Separate low utilization from a healthy workload

If a running job is slow, trace where its time goes. GPU utilization can fall because input batches arrive late, host-to-device transfers stall, network collectives wait, a storage path degrades, or the workload is too small to keep the device busy. Increasing GPU allocation may not improve any of those bottlenecks.

Compare useful progress, step-time distribution, input wait, host CPU and memory pressure, storage latency, and communication behavior. Choose signals appropriate to the workload rather than promising that one metric identifies the cause. A distributed job can lose throughput when one worker slows even though the other workers' devices are healthy.

Use a controlled comparison: the same workload and input on a known-good node, or the previous software version under an equivalent load. Say which variables remain different. A tiny diagnostic job that passes establishes only its tested behavior; it does not prove that a long, memory-intensive training run will remain stable.

Keep the scope proportional. If the failing workload alone is affected, begin with its dependencies and changes. If many unrelated workloads on one host fail, prioritize host and device evidence. If a whole cluster degrades, investigate shared storage, networking, control-plane behavior, and fleet changes before treating it as eight independent GPU faults.

Choose an authorized action with a blast-radius estimate

Propose an action only after naming its objective: stop new placement, protect running work, preserve evidence, or restore a verified device. Cordon, drain, workload restart, driver restart, and GPU reset have different effects. They are not interchangeable synonyms for “recover the node.”

NVIDIA's GPU node triage guidance provides error-specific operational direction. In an interview exercise, explain how you would apply an approved platform runbook and involve the responsible owner. Do not claim that a catalog entry gives you unrestricted remediation authority.

Before eviction, check active jobs, checkpoint recency, disruption policies, and replacement capacity. Losing one process can invalidate progress across a distributed job. Before returning the node, check the device advertisement, workload-level health, and a representative canary. “The command completed” is an action receipt, not a recovery result.

In our fragmented-capacity case, possible choices include waiting for a healthy eight-device node, rescheduling lower-priority work under an approved policy, or restoring the affected device after validated remediation. Changing the job to seven devices is acceptable only if its algorithm and requirements permit that change. Do not alter the customer's compute contract just to make the pod schedule.

Design automation around decision gates

NVIDIA's NVSentinel engineering article discusses health monitoring and automated remediation workflows. It is useful engineering context, not evidence that a particular interview requires that project or that automation is safe for every workload.

An original automation proposal should state the evidence threshold, action owner, exclusions, and verification condition. A single transient signal might trigger observation, while a confirmed failure can trigger isolation according to policy. Add safeguards against repeated remediation loops and mass eviction caused by a broken health signal.

Keep the remediation state distinct from the workload state. A node can pass a device check while its customer's job remains failed. Record both outcomes and notify the correct workload owner. If the canary fails, leave the node isolated and escalate with preserved evidence rather than cycling through resets until a dashboard turns green.

Give a customer-readable incident update

Summarize impact, evidence, action, and uncertainty. For the fictional case: “The new eight-GPU job is pending because no eligible node currently offers eight healthy free devices. One node advertises seven after an earlier device failure. We are preserving its logs and evaluating the approved recovery path; we have not confirmed the underlying cause.”

Add the next checkpoint, such as completion of device triage or availability of another eligible node, instead of inventing a restoration time. Separate what restores scheduling from what proves application correctness. If a distributed job lost progress, include the checkpoint and restart implications rather than reporting only infrastructure health.

Rehearse the incident twice: once as an engineer tracing evidence, then as the person updating a customer who needs a decision. The technical detail should remain consistent while the level of explanation changes. That is more useful preparation than memorizing commands without an impact model.

Practice five GPU-cloud reliability questions

The first three records are NVIDIA-tagged. Snowflake and OpenAI provide transferable control-plane and GPU-scheduling exercises, not NVIDIA prompt predictions. Some records target infrastructure design rather than an SRE-specific interview.

Question	Reliability reasoning to rehearse
Design a GPU Telemetry Collection and Analytics Platform	Correlate stable device identity, health signals, and workload impact.
Design a Dockerized GPU test pipeline	Define what a representative test proves across runtime and driver boundaries.
Design first-time Kubernetes deployment in new cloud	Surface resource advertisement, placement, and operational dependencies.
Design a Control Plane That Manages Cloud Clusters and Monitors Host Health	Prevent a faulty health signal from causing unsafe fleet-wide action.
Design the GPU Job Scheduler for a Text-to-Video Generation Service	Distinguish aggregate capacity from placement, fairness, and recovery constraints.

Return to the GPU telemetry design and trace one alert all the way to an authorized action and a workload-level recovery check.

Sources and Further Reading

• NVIDIA: Principal Site Reliability Engineer, AI Platform Runtime: current role context, not a universal hiring process.
• Kubernetes: Device Plugins: resource advertisement and unhealthy devices.
• Kubernetes: Schedule GPUs: GPU resource configuration.
• NVIDIA: Analyzing Xid Catalog: error classification.
• NVIDIA: GPU Node Triage: operational triage guidance.
• NVIDIA: Automate Kubernetes AI Cluster Health with NVSentinel: automation context.

Evidence checked October 6, 2026. Node counts, observations, placement constraints, and customer updates are original practice examples.