vLLM Interview Questions 2026: PagedAttention, Continuous Batching, KV Cache, and Production Serving

vLLM interview questions test whether you can connect transformer inference mechanics to production systems. A strong answer starts with the KV cache as a capacity constraint, explains how PagedAttention maps logical token blocks onto physical GPU blocks, and then shows how the scheduler uses that memory to admit, batch, preempt, and finish requests. Feature names alone are not enough; interviewers want the trade-off, the metric, and the failure mode.

The linked records are practice material, not predictions of an exact vLLM or ML infrastructure interview.

Quick answer: what should you know for a vLLM interview?

This guide is for ML infrastructure engineers, machine learning engineers, and software engineers who may be asked to reason about vLLM in a systems interview. It covers the serving mechanisms and operational decisions behind vLLM; it is not an installation guide or a version-specific flag reference.

Prepare one connected explanation:

1. Prefill processes prompt tokens and creates per-layer keys and values. Decode generates tokens iteratively while reading the stored KV cache.
2. KV-cache demand grows with cached tokens, layers, KV heads, head dimension, and bytes per element. It often limits concurrency before raw compute does.
3. PagedAttention stores a sequence's KV state in fixed-size blocks that can occupy non-contiguous physical memory. A block table preserves the logical order.
4. Continuous batching updates the running batch between model iterations as requests arrive and finish. It is a scheduling policy, not another name for PagedAttention.
5. Prefix caching reuses completed KV blocks when another request has the same prefix. It can reduce prefill work, but it does not make long decode phases free.
6. Production answers must cover admission control, fairness, cancellation, overload, distributed execution, and metrics such as TTFT, inter-token latency, queue time, cache usage, and preemptions.

The unifying idea is block lifecycle. Follow a KV block from allocation to reuse, eviction, or reclamation, and most vLLM follow-ups become easier to reason about.

Interview question 1: derive the KV-cache memory pressure

An autoregressive transformer reuses the keys and values from earlier tokens. For a simple decoder-only model, a useful estimate for one sequence is:

KV bytes ~= cached tokens x layers x 2 x KV heads x head dimension x bytes per element

The factor of two represents keys and values. With grouped-query or multi-query attention, the number of KV heads can be smaller than the number of query heads, so do not blindly substitute the total attention-head count. Multiply by concurrent sequences, beams, or samples to estimate worker-level pressure, then leave room for weights, temporary activations, communication buffers, CUDA graphs, and runtime workspace.

A strong interview answer states what the formula omits. Real layouts may add block metadata, alignment, padding, quantization scales, model-specific attention groups, and fragmentation in a partially filled final block. The estimate is an admission and capacity tool, not a promise that every GPU byte is available to the cache.

Also separate prefill from decode. Prefill processes many prompt tokens in parallel and is often compute-heavy. Decode usually advances each active request by a small number of tokens per iteration while repeatedly reading weights and KV state, making it sensitive to memory bandwidth and scheduling gaps.

Interview question 2: explain PagedAttention from first principles

Traditional contiguous allocation creates a prediction problem: the server does not know how many output tokens a request will actually produce. Reserving the maximum wastes memory; reallocating and copying as the sequence grows is expensive. The 2023 PagedAttention paper reported that only 20.4% to 38.2% of KV-cache memory held actual token states in the baseline systems it studied. Treat those figures as historical experimental results, not a universal property of current servers.

PagedAttention borrows the indirection idea from virtual memory:

Layer	What it represents	Interview consequence
Logical KV block	A consecutive token range within one sequence	The sequence remains ordered even when storage is scattered.
Physical KV block	A fixed-size region in the worker's cache pool	Blocks can be allocated on demand from available GPU memory.
Block table	Logical-block to physical-block mapping	The attention kernel gathers the right K/V data without requiring physical contiguity.
Reference count	Number of active users of a shared block	Shared prefixes or parallel samples can reuse memory safely.

When a sequence needs more space, the manager assigns another physical block and extends its mapping. When the sequence finishes or is cancelled, its unshared blocks return to the free pool. Only the last block needs to be partially filled, so internal waste is bounded by the block size rather than by the request's maximum output length.

Do not say paging removes all costs. A smaller block can reduce tail waste and improve sharing opportunities, while increasing metadata and lookup overhead. A larger block can improve memory access efficiency while wasting more space at sequence ends. The paper measured 20% to 26% higher attention-kernel latency from block-table access in its microbenchmark, while reporting 2x to 4x higher end-to-end serving throughput than its tested baselines at similar latency because better memory use allowed more concurrent work.

Interview question 3: distinguish dynamic and continuous batching

Ordinary dynamic batching collects compatible requests, launches a batch, and waits for that batch to finish. That works well when requests have similar, fixed execution costs. LLM generation is different: one request may stop after ten output tokens while another continues for hundreds.

Continuous batching changes membership between decode iterations. Finished or cancelled sequences leave, newly admitted sequences enter, and the scheduler rebuilds the next iteration's token work. This prevents a short request from leaving an unusable slot behind until the longest member of the original batch completes.

PagedAttention and continuous batching solve different problems:

• PagedAttention makes variable-length KV state easier to place and reclaim.
• Continuous batching decides which requests and tokens run in the next iteration.
• Admission control decides whether enough safe capacity exists to accept more work.
• Scheduling policy decides how to balance throughput, deadlines, priorities, and fairness.

The vLLM V1 tuning documentation says chunked prefill is enabled by default when possible. Decode requests are prioritized, and remaining token budget can be used for pending prefill work; a large prefill can be split into chunks. Explain the benefit carefully: interleaving compute-heavy prefill with memory-heavy decode can improve utilization and protect inter-token latency, but the right max_num_batched_tokens still depends on the workload and hardware.

Interview question 4: when does prefix caching help?

Automatic Prefix Caching reuses KV blocks from an earlier request when the new request has the same completed prefix blocks. The current vLLM design uses block hashes that include the parent prefix, the block's token IDs, and relevant extras such as adapter or multimodal identity. Full blocks are cacheable; a partial tail does not produce the same reusable unit.

Good workloads include repeated long documents, stable system prompts, few-shot templates, and later turns that repeat most of a conversation history. A hit skips part of prefill computation. It does not reduce the cost of generating new output tokens, so a decode-heavy request with a short or unique prefix may see little benefit.

Correctness and isolation belong in the answer. The cache key must separate model versions, tokenization behavior, adapters, multimodal inputs, and any other state that changes K/V values. In a multi-tenant service, cross-tenant reuse can create a timing side channel. Current vLLM security guidance supports a request cache_salt so only requests with the same secret-scoped salt can share prefix blocks. The trade-off is lower reuse across isolation boundaries.

Interview question 5: design the production request path

Start with requirements: model and tokenizer versions, request-rate and sequence-length distributions, streaming behavior, TTFT and inter-token objectives, availability, tenant isolation, hardware, and cost. Then trace one request:

gateway -> authentication and quota -> validation and token limits -> model-version router -> admission queue -> vLLM scheduler -> GPU worker -> streamed response

Keep the model artifact and serving configuration immutable during a request. Propagate cancellation when the client disconnects so abandoned sequences release scheduler and cache capacity. Reject or shed load before KV exhaustion instead of accepting requests that can only time out after consuming work.

Use one GPU when the model and target workload fit. vLLM supports tensor parallelism and pipeline parallelism when a model needs multiple GPUs or nodes. Parallelism can free memory per device or make a larger model possible, but it adds synchronization and communication. Choose it from measured memory and latency constraints, not as a default badge of scale.

For multiple replicas, route to warm workers that hold the requested model version. Account for load time, cache warmth, model placement, draining, canary traffic, and retry boundaries for streaming responses. A retry after partial output can duplicate or contradict text unless the API defines resumable semantics.

Interview question 6: debug latency and cache failures with evidence

vLLM exposes production metrics through the OpenAI-compatible server's /metrics endpoint. Use them as a diagnostic tree rather than a dashboard checklist.

Symptom	First signals to compare	Likely hypotheses	Next controlled check
TTFT rises while ITL is stable	Queue time, waiting requests, prompt tokens, prefill time	Admission backlog, long prefills, cold replica, routing imbalance	Split by model/version and prompt length; compare queue and prefill percentiles.
ITL rises for active streams	Running requests, iteration tokens, decode time, GPU utilization	Oversized active batch, decode contention, communication overhead	Reduce concurrency or token budget on one canary and compare ITL plus throughput.
Preemptions increase	KV-cache usage, request lengths, running/waiting counts	Cache over-admission, long contexts, insufficient headroom	Lower max_num_seqs or batched-token budget; verify preemptions fall before adding hardware.
Prefix-hit ratio falls	Prefix queries/hits, prompt hashes, model and adapter versions	Unique prompts, changed template, cache-key fragmentation, isolation salt scope	Replay a known shared-prefix workload and compare computed prefill tokens.
One replica is saturated	Queue age and cache usage by replica, routing distribution	Hot tenant, skewed model version, stale readiness, uneven hardware	Drain or rebalance one replica; confirm tail latency moves with the load.

Suppose p99 TTFT rises after a prompt-length increase, KV-cache utilization stays near its ceiling, and preemptions climb. Do not start by changing the attention kernel. First split prompt and output distributions by tenant and model version. Check whether the scheduler admitted more cached tokens than the workload budget assumed. Compare queue time, prefill time, prefix-cache hits, and active-sequence counts before and after the change.

A safe response might lower admission concurrency, cap prompt or output length by product tier, increase cache headroom, or add a compatible replica. vLLM's tuning guide also lists increasing GPU memory utilization or parallelism as possible ways to provide more KV capacity, but each has a cost: less safety margin, more synchronization, or higher latency. Change one variable on a canary and verify TTFT, ITL, throughput, rejection rate, preemptions, and quality before broad rollout.

A strong whiteboard answer in eight steps

1. Define the request mix, model version, streaming contract, and latency objectives.
2. Estimate KV bytes per token and total safe cache capacity.
3. Explain logical blocks, physical blocks, the block table, and tail fragmentation.
4. Describe continuous batching at iteration boundaries and how requests enter or leave.
5. State admission, fairness, cancellation, and overload policies.
6. Add prefix caching only for workloads with measurable prefix reuse; define cache-key correctness and tenant isolation.
7. Choose single-GPU or distributed execution from fit and benchmark evidence.
8. Close with metrics, a canary experiment, and the condition that triggers rollback.

This sequence is specific enough to show vLLM knowledge while remaining useful when an interviewer changes the model, GPU type, traffic shape, or SLO.

Common vLLM interview mistakes

• Saying PagedAttention moves KV data to disk. Its central idea is non-contiguous block-based storage; offload or swapping is a separate policy.
• Claiming continuous batching eliminates queueing. It improves slot reuse, but admission pressure, fairness, and long requests still create waits.
• Treating high GPU utilization as the only objective. A healthy service balances throughput with TTFT, ITL, tail latency, rejection, and cost.
• Assuming prefix caching accelerates decode. It avoids repeated prefix prefill work; new output tokens still require decoding.
• Quoting the 2023 paper's throughput results as a current guarantee. Hardware, models, kernels, configurations, and workloads determine present performance.
• Adding tensor parallelism without discussing communication, topology, or whether the model already fits on one GPU.

Final preparation strategy

Prepare one memory calculation, one block-table sketch, one scheduling trace with requests entering and leaving, and one incident diagnosis. For each, name the user-visible metric and the production trade-off. That is more persuasive than memorizing a list of vLLM flags.

Then review whether your answer connected KV-cache capacity to admission, continuous batching, prefix reuse, cancellation, observability, and rollback.

Sources and Further Reading

• vLLM: Optimization and Tuning
• vLLM: Automatic Prefix Caching
• vLLM: Production Metrics
• vLLM: Parallelism and Scaling
• vLLM: Security and prefix-cache salting
• Efficient Memory Management for Large Language Model Serving with PagedAttention, SOSP 2023
• Original vLLM project announcement

Research note: This guide was checked on September 5, 2026. vLLM configuration names, defaults, metrics, model support, and security guidance can change by release; verify the documentation for the version used in your environment.