Articles
LLM Latency Interview Questions: TTFT, Inter-Token Delay, and Tail Latency
Practice LLM latency interview questions on TTFT, token gaps, tail latency, percentiles, load testing, and tracing with worked request-timing examples.
An LLM starts replying quickly, then freezes halfway through a sentence. Another system waits longer before showing anything but delivers the answer smoothly. Which one is faster? In an interview, the useful answer starts by naming the experience you are measuring.
LLM latency has several boundaries: time to first token measures the initial wait, inter-token delay describes progress after generation starts, and end-to-end latency measures completion. Tail latency asks how these behave for the slower requests, not just the average user. Optimizing one does not prove that the others improved.
This guide distinguishes official metric definitions from proposed diagnostic reasoning. All numerical traces are synthetic interview exercises, not provider benchmarks or candidate reports. For a practical starting point, rehearse Design a high-concurrency LLM inference service while keeping the measurement boundaries explicit.
What exactly should TTFT measure?
Start with the observer. A browser can measure from submission to the first visible answer text. An API client can measure from sending a request to receiving the first content event. A model server can measure a narrower interval inside its own processing pipeline. These clocks answer different questions.
Official example: the current vLLM per-request metrics documentation defines time_to_first_token_ms from scheduling to the first generated output token and exposes queue time separately. Its field name alone therefore does not justify treating that value as the user's entire initial wait. vLLM per-request metrics
Interview reasoning: define a client-observed metric such as “submission to first nonempty answer-text delta.” Do not stop the clock at response headers, a connection acknowledgment, an empty stream event, or a loading animation. If tool activity or reasoning is displayed separately, give it a separate milestone rather than silently changing the answer-text definition.
A practical trace includes request creation, gateway receipt, retrieval start and finish, model submission, queue exit, first generated token, first received content, and first rendered content. Keep durations on monotonic clocks within each process. Correlate spans across hosts; subtracting unsynchronized wall-clock timestamps can produce misleading durations.
The interview follow-up is often what you would optimize if client TTFT is high but model generation starts promptly. Inspect retrieval, request preparation, network transit, proxy buffering, and UI rendering before changing the model.
How do TPOT and inter-token delay differ?
Official definition: Google Cloud's inference guide calculates time per output token as (request latency − TTFT) / (output tokens − 1). It distinguishes this request-level average from individual inter-token intervals, which can themselves have a distribution. Google Cloud inference metrics
For a clean hand calculation, define completion as the arrival of the last content token, excluding later usage metadata or transport teardown. Otherwise, the formula includes that extra completion overhead. State the convention before comparing systems.
Here is an original synthetic trace, measured in milliseconds from client submission. Assume each event contains exactly one output token and no buffering changes the spacing.
| Event | Time |
|---|---|
| Submit | 0 |
| Token 1 | 300 |
| Token 2 | 340 |
| Token 3 | 380 |
| Token 4 | 580 |
| Token 5 | 620 |
| Token 6 | 660 |
TTFT is 300 ms. Completion latency is 660 ms. The five token gaps are 40, 40, 200, 40, and 40 ms. Their mean, and the request's TPOT under our definition, is 72 ms:
TPOT = (660 - 300) / (6 - 1)
= 360 / 5
= 72 ms
Dividing completion latency by six gives 110 ms per token, a different quantity that includes the initial wait. Do not label both numbers “decode speed.” For a single-token answer, there are no between-token intervals; report TPOT as undefined rather than dividing by zero or inventing a zero-latency observation.
Can two streams have the same TPOT but feel different?
Yes. Compare the previous gaps with a second synthetic stream whose five gaps are 72, 72, 72, 72, and 72 ms. Give both streams the same 300 ms TTFT. Both finish at 660 ms and have 72 ms TPOT. Only the first contains a 200 ms pause.
The conclusion is narrow: a mean loses information about the shape of the stream. It does not establish that a 200 ms pause violates every product's target. A voice assistant, code-completion tool, and long-form report generator have different interaction requirements.
Choose a smoothness measure that matches the experience: a high percentile of individual gaps, the largest gap per request, or the proportion of requests containing a gap above a defined threshold. Label the denominator. Pooling every token gap weights long responses more heavily; taking one maximum per request gives each request one observation.
A client cannot always observe model-token timing directly. One network chunk can contain multiple tokens, or a token's bytes can span transport reads. Record stream-event or rendered-text gaps under those names unless you have reliable token-level instrumentation. Re-tokenizing received chunks independently does not reconstruct the server's generation timestamps.
Why can throughput improve while latency gets worse?
Design inference: batching can increase useful work completed per device while a request waits longer for its turn. Aggregate output tokens per second measures service capacity; it is not the same as the speed of one user's response.
Consider a proposed change that raises total token throughput by accepting more simultaneous work. Test its queue wait, initial response delay, and stream gaps under a fixed arrival process. A higher throughput number accompanied by more timeouts may be a poor outcome for an interactive endpoint.
Separate prompt processing from ongoing decoding in the trace. Long prompts, long outputs, shared-prefix hits, and memory pressure can create different bottlenecks. Do not infer the limiting resource from GPU utilization alone; a high utilization percentage does not identify queueing, compute, memory movement, or communication as the dominant cause.
Official instrumentation context: vLLM exposes server metrics for first-token latency, inter-token latency, and end-to-end request latency. Check the deployed version's definitions and units before building a dashboard that mixes them with client timings. vLLM metrics design
The defensible interview answer is an experiment: hold workload and quality constraints fixed, vary one scheduling or batching control, and compare capacity with user-facing latency and failure rates.
Why should you never average replica p95 values?
Official fact: Prometheus explains that precomputed summary quantiles cannot be aggregated into a service-wide quantile. Histograms preserve distribution information that can be aggregated before estimating a percentile. Prometheus histograms and summaries
Use this synthetic counterexample. Replica A serves 990 requests, each taking 100 ms. Replica B serves 10 requests, each taking 10,000 ms. Their p95 values are 100 and 10,000 ms. Averaging them produces 5,050 ms.
Across all 1,000 requests, the nearest-rank p95 is the 950th sorted observation: 100 ms. Even weighting the two replica percentiles by request count produces 199 ms, which is still not the global p95. You need the observations or an appropriate aggregate distribution, not a better averaging formula.
That global p95 also conceals the slow replica's users. Report useful segments and failures alongside the overall percentile. Percentile conventions and interpolation matter on small samples; declare the method for a hand calculation and avoid presenting a tiny sample's p99 as a stable operational estimate.
Likewise, adding retrieval p95, queue p95, and generation p95 does not generally produce end-to-end p95. Those slow stages may belong to different requests. Measure whole-request durations and use correlated traces to investigate their slow members.
How would you run a credible latency benchmark?
Proposed protocol: write down the request population before running the test. Include input-length and output-length distributions, concurrency or arrival rate, model and serving configuration, streaming mode, cache conditions, and the quality task. A benchmark dominated by short cached prompts should not be presented as the latency of long uncached conversations.
| Test condition | Question it answers |
|---|---|
| Warm, low offered load | What is the baseline service time? |
| Increasing offered load | Where do queues and tail delays grow? |
| Bursty arrivals | Can admission and scheduling absorb spikes? |
| Long prompts mixed with short requests | Do large jobs delay interactive requests? |
| Cold or cache-miss traffic | How much does favorable cache state hide? |
Distinguish an open arrival model, which submits work according to a schedule, from a closed model, which waits for completion before sending replacement work. A closed client can reduce its offered load as the server slows, hiding the overload condition you intended to test.
Record attempted, admitted, successful, rejected, timed-out, and canceled requests. A dashboard that drops timeouts can show a better p95 while more people receive no answer. Keep latency for successful requests clearly labeled and report deadline success across the full intended population.
Use enough warm-up to characterize steady behavior, but report cold-start behavior separately when it affects users. Preserve individual request traces or suitable distributions for diagnosis. Repeat comparable runs to identify variability, and check that the load generator is not the bottleneck.
A useful capacity result is the highest tested arrival rate that satisfies the chosen latency, quality, and error constraints. It is more actionable than a token-throughput maximum measured after the interactive experience has already deteriorated.
What changes would the evidence justify?
Tie each proposed change to an observed interval. If queue wait dominates only during bursts, investigate admission control, scheduling, and warm capacity. If prompt processing dominates for long inputs, test reducing unnecessary context while checking answer quality. If decoding stalls correlate with competing work, investigate scheduling interference and resource pressure.
If the server emits smoothly but the client receives bursts, examine proxies and transport buffering. If receipt is smooth but rendering stalls, profile the client. Streaming can expose progress sooner, but it does not by itself reduce the amount of model computation or guarantee earlier completion.
Treat quality and cost as constraints, not footnotes. A shorter answer may finish sooner because it omitted required content. A smaller model may reduce service time while failing the task. A cache may improve a benchmark whose repeated prefixes do not resemble production traffic. Verify the intended work still gets done.
For cancellation, follow the signal through the request lifecycle. Closing the browser's stream does not prove that backend generation stopped. Measure cancellation-to-stop time and residual work where instrumentation permits. Retries can multiply load, so include attempt count and an overall deadline in the design.
End the diagnosis with a falsifiable prediction: “If queueing causes the TTFT tail, increased warm capacity should reduce queue wait under this arrival pattern without changing low-load generation time.” That is a stronger answer than listing every available inference optimization.