ONNX and TensorRT Interview Questions: Export, Dynamic Shapes, and Numerical Parity

ONNX and TensorRT interview questions usually ask you to explain why a model that worked in training behaves differently after deployment. The useful debugging path is to compare the same input at successive boundaries: the original framework, the exported ONNX model, the TensorRT engine, and the application's final postprocessing.

Export success, engine-build success, and acceptable task accuracy are different outcomes. Dynamic shapes must fit the engine's supported profiles, and reduced precision must be evaluated against a meaningful numerical and task-level tolerance.

This guide uses an image model and a variable-batch deployment as worked examples. Examples describe a validation process rather than promise a particular speedup.

What roles do ONNX and TensorRT play?

ONNX represents models using a defined graph and operator format. TensorRT builds and runs optimized inference engines on supported NVIDIA platforms. Exporting a model to ONNX is one way to supply a graph to a deployment workflow, but the graph still has to use supported semantics and shapes.

The original framework may contain Python control flow, custom operations, training behavior, or preprocessing outside the model graph. An export captures the supported computation represented by the export path; it does not automatically package every part of the application.

In an interview, define the boundary of “the model.” Does it include normalization, resizing, tokenization, non-maximum suppression, or label mapping? Two pipelines can run the same neural network weights while producing different application outputs because these surrounding steps differ.

Keep versions in the deployment record: framework, exporter, ONNX opset, runtime, TensorRT, CUDA, and target hardware as relevant. Compatibility is a concrete matrix, not a general guarantee attached to a file extension.

What should you verify before export?

Put the source model in its intended inference configuration. Disable training-only behavior where appropriate and use representative inputs. Save the exact input tensor and reference outputs so later comparisons do not depend on accidentally different preprocessing.

Check input names, shapes, dtypes, layouts, and value ranges. For an image model, NCHW versus NHWC, RGB versus BGR, and values scaled to 0–1 versus 0–255 are common boundaries to inspect. These examples are not TensorRT-specific bugs; they are pipeline-contract mismatches.

Also decide which dimensions are truly dynamic. A model tested only at one image size may contain assumptions that do not generalize to arbitrary sizes even if an export configuration names a dynamic axis.

Use a small set of ordinary and boundary inputs. Include minimum and maximum supported sizes, unusual aspect ratios where relevant, and values near downstream decision thresholds. A single random tensor is a smoke test, not a complete accuracy evaluation.

How do you locate a numerical mismatch?

Compare adjacent stages with the same saved inputs. First compare framework outputs with ONNX runtime outputs. If they already differ materially, building a TensorRT engine adds another layer before the original mismatch is understood.

If framework and ONNX agree but TensorRT differs, investigate the engine conversion, supported operations, shape handling, and precision policy. If raw outputs agree but final labels or boxes differ, inspect postprocessing and thresholding.

Record more than one scalar summary. Maximum absolute error, relative error with a sensible treatment near zero, and distributions across outputs can reveal different problems. Also inspect the inputs and outputs associated with the largest discrepancies.

A small floating-point difference can matter if it crosses a decision boundary. Conversely, a large relative error around a value extremely close to zero may have little task impact. Choose tolerances based on the operation and application, not an arbitrary universal constant.

What is a useful parity test?

Begin with a clear input-output contract and versioned reference cases. For each case, compare tensor shape, dtype, finite values, and numerical differences. Then compare the task output, such as class agreement or detection metrics, on a representative evaluation set.

Case: saved image tensor, fixed preprocessing
Check 1: output names, shapes, and dtypes
Check 2: no unexpected NaN or Inf
Check 3: absolute and relative numerical differences
Check 4: final labels or boxes after identical postprocessing
Check 5: dataset-level task metric

If differences first appear in a particular intermediate region, use supported graph-debugging tools to compare that region. Be aware that exposing intermediate tensors can affect optimization, so use the result to investigate semantics rather than automatically equating the debug graph's timing with production timing.

Keep a failure artifact with the exact input and configuration. “Accuracy looked worse on one machine” is difficult to reproduce; a saved input with a first-divergence report is actionable.

How do TensorRT dynamic-shape profiles work?

Runtime dimensions are declared in the network and constrained through optimization profiles. A profile specifies minimum, optimum, and maximum dimensions for the relevant inputs. The optimum shape guides optimization; the minimum and maximum bound the supported range.

Suppose batch size varies from one to eight while channels and image size stay fixed. An illustrative profile might use batch one as minimum, batch four as optimum, and batch eight as maximum. This does not mean batch sixteen is valid or that every supported batch has identical performance.

At runtime, select an appropriate profile and set the actual input dimensions using the supported API. Resolve output shapes before allocating or using output buffers where required. Shape tensors and data-dependent outputs can introduce additional rules.

Do not make the profile arbitrarily broad just to avoid thinking about production traffic. Wide ranges can affect tactic availability, memory, and performance. Use observed workload shapes to choose useful profiles, then test the boundaries and common cases.

Why does an engine fail only for one input size?

Check whether the input is inside the selected profile and whether all related dimensions satisfy the model's relationships. A height that is numerically within a range may still violate an operation's divisibility or reshape requirement.

Inspect actual buffer sizes and shape updates. Reusing an output allocation from a smaller input without respecting the new output shape is an application bug, not necessarily a numerical issue in the model.

Symptom	First checks
Shape rejected immediately	Selected profile and runtime dimensions
Reshape or operator failure	Dimension relationships and export assumptions
Wrong output for larger batch	Buffer capacity, binding addresses, and output shapes
Large memory jump	Maximum shape, workspaces, and profile choices
Only odd image sizes fail	Padding, stride, reshape, or model-specific constraints

Create a small shape matrix for testing. Include profile endpoints and representative interior points, not only the optimum shape used during a benchmark demonstration.

Why can reduced precision change accuracy?

Reduced precision changes representable range and resolution. Some computations tolerate that well; others amplify small differences or overflow. Accumulations, large magnitudes, and sensitive nonlinear operations deserve attention.

For example, FP16 has a narrower finite range than FP32. Overflow can introduce infinities, which can propagate to invalid outputs. A model that behaves well on typical inputs may still fail on a rare high-magnitude case.

Start from a reliable reference path, then introduce the intended precision policy and compare. If a region is sensitive, use the supported method for retaining or introducing higher precision there, or reconsider the model and preprocessing range. The APIs depend on TensorRT version and typing mode.

Quantization adds another contract involving scales, calibration or quantization-aware information, and supported operators. Do not treat every INT8 deployment as the same workflow. Evaluate the actual exported graph and engine configuration.

How can preprocessing create a false accuracy alarm?

Imagine the framework path normalizes images after converting to floating point, while the deployment path applies integer arithmetic first. Or one path resizes with a different interpolation rule and changes the input distribution. The engine may be faithfully executing a different input.

Compare the tensors immediately before model execution. Check shape, channel order, dtype, minimum and maximum, and a few known pixel locations. Save both tensors when they differ. This often narrows the problem faster than inspecting every layer.

Postprocessing deserves the same discipline. Box coordinate conventions, class index offsets, score activation, confidence thresholds, and NMS settings can turn similar raw predictions into different final results.

For a deployment interview, explain that parity is an end-to-end contract with inspectable intermediate boundaries. Neural-network arithmetic is only one part of that contract.

How should you benchmark inference fairly?

Define what the measurement includes. Kernel execution alone, host-to-device transfer plus inference, and full request latency with preprocessing and postprocessing answer different questions. Report the chosen boundary clearly.

Warm up the engine and separate initialization or first-run effects from steady-state inference. Account for asynchronous GPU execution using appropriate timing and synchronization. Timing only the host enqueue call does not necessarily measure completed device work.

Measure the workload you plan to serve. Batch size, input shapes, concurrency, precision, and hardware all affect the result. Report latency distributions or relevant percentiles alongside throughput when the application has a response-time requirement.

An engine that is faster at a large batch may be worse for a low-latency single-request service. A benchmark should support a deployment decision, not merely produce the largest samples-per-second number.

Are TensorRT engines portable artifacts?

Serialized engine compatibility depends on the build and runtime environment and on explicitly supported compatibility features. Do not assume an engine built on one GPU and TensorRT version can run unchanged everywhere.

Record the environment used to build the engine, including relevant hardware and software versions. Decide whether deployment rebuilds from an exported model or distributes prebuilt engines for a controlled target set. Keep the source model and export configuration so rebuilding is reproducible.

Treat compatibility errors separately from numerical parity. Successfully loading an engine is only the beginning of runtime validation. Re-run representative correctness and performance checks on the actual deployment target.

A practical interview exercise

Take a small inference model and define three supported input shapes. Save reference inputs and outputs, export to ONNX, compare the ONNX path, and then compare the TensorRT path at the intended precision. Include one near-threshold input and one maximum-shape input.

When a mismatch appears, identify the first divergent boundary before changing tolerances. When timing improves, confirm that the same task-quality criteria still pass. Finally document the shape profile and engine environment so another person can reproduce the result.

This exercise gives you a complete answer to “How would you deploy and validate this model?” It includes export, shape constraints, numerical evidence, task behavior, and an honest performance measurement.

What belongs in a minimal deployment bug report?

Include one saved input, the preprocessing settings, expected raw output, actual raw output, and the first stage where they disagree. Record framework, exporter, runtime, TensorRT, driver, and hardware versions along with precision and optimization profiles. Keep the reproducer small enough to run without the production application.

For a dynamic-shape failure, state the exact failing dimensions rather than saying that large batches fail. For a numerical mismatch, show the chosen tolerance and whether the difference changes a downstream decision. This evidence makes the problem actionable and prevents a team from tuning thresholds to conceal an export or preprocessing error.

Sources and Further Reading

• NVIDIA: working with dynamic shapes
• NVIDIA: dynamic-shape core concepts
• NVIDIA: accuracy considerations
• NVIDIA: performance benchmarking
• ONNX model concepts

Research checked September 24, 2026. TensorRT API and compatibility details vary by release; match profiles, precision configuration, and runtime setup to the installed version.