Vision Transformer Interview Questions: Patch Embeddings, Token Counts, and Input Resolution

Vision Transformer interview questions often expose a simple mistake: a candidate remembers that images become tokens but cannot trace the shape, spatial ordering, or compute consequences of changing resolution. A strong answer follows one image from pixels through patch projection and positional information before discussing attention.

For a patch size of 16, changing a square input from 224 to 384 pixels changes the patch count from 196 to 576. With one classification token, sequence length changes from 197 to 577. That is not merely a preprocessing adjustment; it affects positional embeddings, attention work, and potentially model compatibility.

Use Explain Vision Encoders and LLM Bottlenecks for broader practice. Here the focus is narrower: prove the patch layout, calculate the new sequence, and distinguish mathematical ViT behavior from a specific implementation's input contract.

Derive the patch embedding shape

The foundational Vision Transformer paper describes splitting an image into patches, projecting them into a token representation, and adding positional information before transformer processing. It is a historical architecture source, not a claim that every current vision backbone uses an identical token scheme.

Assume a batch in channels-first layout: (B, C, H, W). For nonoverlapping square patches of side P, require that both spatial dimensions be divisible by P, unless a separate padding or cropping policy is specified. The patch count is (H/P) × (W/P) and each flattened patch contains C × P × P values.

A learned projection maps that flattened vector to dimension D. Its input width and output width are different quantities. For RGB patches of side 16, the input width is 768 because 3 × 16 × 16 = 768. An embedding width of 768 is a design choice, not an equality required by the architecture.

After projection, the patch representation has shape (B, N, D). In the original-style classification setup, prepend one learned class token to obtain (B, N+1, D). If a variant adds other tokens, excludes a class token, or pools differently, derive its sequence length from that actual configuration rather than reuse N+1 automatically.

Be clear about batch layout. Some APIs consume channels-last arrays or internally convert them. A correct-looking product of dimensions cannot rescue an incorrect axis interpretation. Ask for the input layout and expected output shape before implementing a reshape.

Why a reshape alone can silently scramble patches

In a channels-first image, neighboring flattened memory values do not necessarily belong to the patch grouping you want. Splitting dimensions is only part of extraction. You must reorder the grid axes and within-patch axes before collapsing them into patch tokens.

The following original NumPy fixture uses a small coordinate-coded image so that every value can be traced. It compares a vectorized extraction with an independent slice-based oracle, then checks an algebraically equivalent local projection. It does not execute a PyTorch model or benchmark a GPU.

import numpy as np

B, C, H, W, P, D = 2, 3, 4, 6, 2, 5
x = np.arange(B*C*H*W).reshape(B, C, H, W)
gh, gw = H//P, W//P
patches = x.reshape(B, C, gh, P, gw, P)
patches = patches.transpose(0, 2, 4, 1, 3, 5)
patches = patches.reshape(B, gh*gw, C*P*P)
oracle = np.stack([
    x[:, :, r:r+P, c:c+P].reshape(B, C*P*P)
    for r in range(0, H, P)
    for c in range(0, W, P)
], axis=1)
assert patches.shape == (2, 6, 12)
assert np.array_equal(patches, oracle)
weights = np.arange(C*P*P*D).reshape(C*P*P, D)/100
tokens = patches @ weights
kernel = weights.T.reshape(D, C, P, P)
local = np.stack([
    np.einsum("bcpq,dcpq->bd", x[:, :, r:r+P, c:c+P], kernel)
    for r in range(0, H, P)
    for c in range(0, W, P)
], axis=1)
assert tokens.shape == (2, 6, 5)
assert np.allclose(tokens, local)
wrong = x.reshape(B, gh*gw, C*P*P)
assert not np.array_equal(wrong, oracle)

The final assertion is essential: both the correct and wrong arrays have shape (2, 6, 12). A shape-only test would pass the scrambled version. Coordinate checks establish which pixels belong to each patch and which patch appears at each token index.

The projection comparison also makes a useful interview connection. A convolution with kernel size and stride equal to the patch size can implement the same linear patch projection, with matching weight layout and bias. Torchvision's ViT implementation uses a convolutional patch projection in its standard path. Do not generalize that one path to every convolutional-stem variant.

Calculate resolution changes without exaggerating latency

For a 224-by-224 image and patch side 16, the grid is 14 by 14. There are 196 patches and 197 tokens after adding the class token. At 384 by 384, the grid is 24 by 24: 576 patches and 577 tokens.

The patch count ratio is 576/196, approximately 2.94. The ratio of full attention score-pair counts is 577²/197², approximately 8.58. These are different quantities. Saying “resolution increased by 1.71, so compute increased by 1.71” misses the spatial area and pairwise attention effects.

Still, an attention-pair ratio is not an end-to-end latency prediction. Projections and feed-forward layers have different scaling terms, kernels can avoid materializing the full attention matrix, and hardware utilization changes with shape. Explain the mathematical workload first, then measure latency and peak memory for the actual implementation.

Input and patch side	Patch grid	Patches	Tokens with one class token	What changed
224 × 224, P=16	14 × 14	196	197	Baseline.
384 × 384, P=16	24 × 24	576	577	More tokens; positional grid must match.
224 × 320, P=16	14 × 20	280	281	Rectangular grid; do not infer a square from token count.
224 × 224, P=32	7 × 7	49	50	Different patch projection input width and spatial granularity.

Halving patch side approximately quadruples spatial token count for a fixed divisible image and increases the leading quadratic attention term by roughly sixteen. The class token makes exact small-sequence ratios differ. More importantly, changing patch size also changes the projection's input width, so a checkpoint cannot be assumed to transfer by adjusting a scalar configuration alone.

Separate positional-grid adaptation from accepting a new image

Learned absolute positional embeddings associate a position vector with each patch-grid location. A checkpoint trained with a 14-by-14 grid does not automatically provide a 24-by-24 grid. Merely making the sequence lengths equal through arbitrary slicing or padding loses the spatial relationship the checkpoint encoded.

A common adaptation reshapes patch positions into their two-dimensional grid, interpolates that grid, and flattens it back. Keep the class-token position separate: it is not a pixel location to interpolate with the spatial grid. Preserve the extraction order when reconnecting positions to tokens.

Torchvision's source helper provides checkpoint positional-embedding interpolation for a square grid, while the model itself checks configured image dimensions. These are separate operations. Calling a helper does not mean any differently sized image will be accepted by an unchanged model.

Hugging Face's ViT documentation exposes an interpolate_pos_encoding option for relevant forward calls. That is implementation-specific support, not a universal ViT flag. Also inspect the image processor: if it resizes every input back to 224, you have not actually tested 384-resolution inference merely because the source image was larger.

For a rectangular grid, retain both grid dimensions. The square root of 280 is not a valid route to reconstructing a 14-by-20 grid. Track height and width through preprocessing and use an adaptation that explicitly supports that shape. A helper restricted to square grids should reject or be replaced, not coerced into silently guessing dimensions.

Debug a higher-resolution migration in a defensible order

First capture the tensor shape after preprocessing. Confirm layout, dimensions, value normalization, and whether resize or crop was applied. This separates “the model rejected 384” from “the processor never supplied 384.” Record the intended policy rather than rely on library defaults.

Next inspect patch output and sequence length. At 384 with side 16, expect 576 patch tokens before the class token in this setup. If you see 196, revisit preprocessing. If you see a different count, check patch stride, padding, convolutional stems, and special tokens before declaring the model broken.

Then inspect positional compatibility. An error adding tensors of lengths 577 and 197 points to a sequence mismatch. A successful addition is not enough: confirm the interpolated positions correspond to the patch order and that class-token handling remained separate.

Finally evaluate operational behavior and quality. Measure representative batch sizes, peak memory, latency, and task metrics. Include difficult examples where extra resolution might matter as well as cases where it does not. Higher resolution is a proposed trade-off, not an automatic accuracy improvement.

If memory fails, lowering batch size can help diagnose the cause but is not a complete deployment decision. Compare smaller resolution, larger patch size, an alternative architecture, or more efficient attention against the accuracy requirement. Changing patch size may require new compatible weights or training, so do not list it as a free runtime toggle.

Handle nondivisible inputs as a policy decision

Suppose width is 230 and patch side is 16. Simple integer division yields 14 patches with six pixels left over. A strided convolution may discard an edge under a particular configuration; another implementation may reject the input. Neither outcome should be assumed to match the product's intended image semantics.

Choose an explicit policy: resize, crop, or pad. Padding changes the grid and introduces artificial pixels; cropping can remove information; resizing changes scale and possibly aspect ratio. Discuss which trade-off the task tolerates. If using padding, understand whether and how the model accounts for padded content rather than assume a text-style attention mask applies.

Also consider train–evaluation consistency. A new normalization, interpolation mode, or crop policy can cause degradation even when token counts are correct. Keep preprocessing changes visible in the migration ledger and test them independently of positional adaptation where possible.

The small NumPy fixture deliberately uses divisible dimensions and no bias, normalization, or transformer layers. Its assertions verify layout and projection equivalence within that scope. They do not establish checkpoint accuracy, full-model numerical equality, or serving performance.

Practise shape reasoning with these five questions

Question	What to verify in your answer
Explain Vision Encoders and LLM Bottlenecks	Separate vision token count from downstream multimodal bottlenecks.
Compare convolutions and transformers	Explain local projection and global interaction without false absolutes.
Debug a Broken Transformer	Trace shapes and ordering before changing the architecture.
Self-Attention: Implementation, Complexity, and Efficient Variants	Distinguish pair-count scaling from measured memory and latency.
Analyze vision model failures	Test whether resolution changes address the observed failure.

Start with Debug a Broken Transformer, then explain the 224-to-384 migration without notes. A strong answer contains the shapes, patch-order test, positional-grid treatment, implementation restriction, and measured deployment trade-off. Remembered architecture names cannot substitute for that chain of evidence.

Sources and Further Reading

• Dosovitskiy et al.: An Image Is Worth 16×16 Words
• Torchvision: Vision Transformer models
• Torchvision: Vision Transformer implementation and interpolation helper
• Hugging Face Transformers: ViT documentation