Articles
Performance Testing Interview Questions: Workload Models, Percentiles, and Bottlenecks
Prepare for performance testing interviews with workload models, percentile pitfalls, failure denominators, and evidence-led bottleneck experiments.
A performance test reports a 90-millisecond p99 and zero server errors. The release still cannot handle a launch. The generator sent fewer requests as the service slowed, checkout failures were excluded from latency statistics, and the report averaged percentiles from unequal worker samples.
Good performance testing interview answers make those measurement choices explicit. Define the work the system must accept, the population each metric describes, and the experiment that would distinguish competing bottleneck explanations. A graph is evidence only when you can explain how it was produced.
Evidence boundary: Tool and monitoring facts below come from Grafana, Prometheus, and Google's SRE material. The traffic ledger, percentile example, and diagnostic scenarios are original exercises, not candidate reports or measured production results. Practice the reasoning with Validate and Assess Performance Benchmarks.
What should you clarify before choosing a load tool?
Start with the decision the test must support. Is the team validating a forecast launch rate, comparing two implementations, finding a saturation point, or checking deterioration over several hours? Those decisions require different workloads and stopping rules.
A workload model describes which operations arrive, how often they arrive, their data characteristics, and how they depend on earlier operations. It should also state whether arrivals continue when responses slow. “One thousand users” leaves most of that model unspecified.
Original interview approach: Ask for the critical user journey, expected traffic shape, request mix, payload sizes, data cardinality, and acceptance criteria. Name the environment differences that limit interpretation. A tiny test database with one hot record may exercise a different path from a production-sized dataset with many tenants.
Choose a tool after identifying what it must simulate and observe. The strongest answer is not a list of products; it is a testable workload contract.
How do you turn user journeys into request rates?
Original worked example: A service expects 120 new journeys per second. Assume 70% browse a catalog once, 20% search once, and 10% attempt checkout. Each checkout journey performs one cart read and one checkout write. Ignore retries initially and say so.
| Journey | Journey arrivals per second | Requests per journey | Resulting request rate |
|---|---|---|---|
| Browse | 84 | One catalog read | 84 requests/s |
| Search | 24 | One search request | 24 requests/s |
| Checkout | 12 | One cart read, one checkout write | 24 requests/s |
| Total | 120 | Mixed | 132 requests/s |
The distinction matters: 120 journeys per second is not 120 HTTP requests per second. If checkout writes retry, the offered request mix changes again. Track original operations and retry attempts separately so increased request throughput is not mistaken for increased useful work.
Now vary the data. Catalog traffic might repeatedly request popular items while search uses a wider key range. Checkout can involve authorization and database writes that reads do not. A single endpoint hit in a tight loop cannot stand in for this mix without a documented reason.
Explain session dependencies too. If checkout requires a cart identifier from an earlier response, use valid identifiers and realistic state transitions. Replaying the same identifier indefinitely may test conflict handling or caching instead of ordinary checkout capacity.
When should arrivals be independent of response time?
Official model distinction: Grafana describes closed models as starting a virtual user's next iteration after its previous iteration completes. Open models decouple iteration arrivals from completion. A closed model can reduce offered traffic as responses slow, concealing the pressure expected from independent arrivals. Grafana's open and closed models
Interview reasoning: Neither model is universally correct. A fixed population of operators who wait before their next action can justify a closed model. New external events that keep arriving despite server delay call for a different model.
In the launch scenario, suppose new shoppers continue arriving at the planned rate. A fixed worker loop that waits for every response may send less work precisely when the system becomes unhealthy. You would need to explain that limitation before using its result as launch-capacity evidence.
Add think time deliberately when it represents user behavior. Do not insert an arbitrary sleep to make the graph look smoother. Describe whether time between operations belongs inside a journey and whether it affects the arrival process.
Keep the model visible in the report. Two tests with the same peak concurrency can create very different pressure if their arrival and waiting rules differ.
What does p99 mean for this particular report?
A percentile locates a value within an ordered population. A latency p99 describes the upper tail of the measured observations under a specified calculation method. It does not state how many operations were never sent, how many timed out outside the dataset, or which user journeys were represented.
Original numerical check: Use the nearest-rank definition for this exercise. Worker A records 99 requests, all taking 10 milliseconds. Worker B records one request taking 1,000 milliseconds. Their individual p99 values are 10 and 1,000 milliseconds. Averaging them gives 505 milliseconds.
Combine the 100 observations instead. The nearest-rank p99 is the 99th sorted observation: 10 milliseconds. The one slowest observation remains 1,000 milliseconds. The average of worker percentiles did not recover the percentile of the combined population.
Different quantile estimators can interpolate and produce a different numerical p99 on a small sample. That does not make averaging worker percentiles valid. State the estimator and sample count before comparing numbers from different systems.
A useful follow-up is whether this result supports a strong tail-latency conclusion. With only 100 observations, one sample materially affects the far tail. Gather enough representative data and repeat the experiment before drawing a production reliability conclusion.
How should you aggregate latency and account for failures?
Official metrics fact: Prometheus explains that precomputed summary quantiles generally cannot be aggregated across instances. Histograms retain bucket counts that can support aggregation before a quantile estimate is calculated, with accuracy depending on the representation and bucket choices. Prometheus histograms and summaries
Original reporting method: Separate endpoint populations before combining them. A fast catalog read and a slow checkout write answer different user questions. A global percentile can improve merely because the test sent more cheap reads.
Record the denominator beside every success or failure rate. Distinguish scheduled journeys, started journeys, sent requests, completed requests, and successful business operations. If a timeout is omitted from successful-request latency, state that exclusion and report it separately.
Consider a comparison where p99 falls but timeouts rise. The slowest requests may have disappeared from the completed-success sample. That is not enough evidence for an improvement.
Report useful throughput together with latency and errors. A checkout request returning quickly with the wrong result should not count as successful capacity. Include lightweight correctness checks so performance changes do not quietly bypass work.
Is the load generator the bottleneck?
Official tool behavior: In k6 arrival-rate scenarios, unavailable virtual users can cause scheduled iterations not to start. Grafana's documentation explains that this may reflect insufficient allocation or longer iterations as the system under test slows. The dropped count alone does not identify one cause. Grafana dropped iterations
Interview reasoning: Compare intended load with delivered load before interpreting service capacity. Monitor generator CPU, network limits, connection behavior, and available execution capacity. A quiet server cannot prove spare capacity if the generator never delivered the intended workload.
A useful controlled comparison distributes the same intended workload across more generator capacity while holding the service and data model constant. If delivered load rises without changing the service, generator limits become a stronger explanation. Confirm that the distribution did not also change connection reuse, source-address constraints, or request mix.
Avoid “fixing” every dropped iteration by raising worker limits. If the service is stalled, additional outstanding work may deepen queues rather than restore useful throughput. Choose a stop condition before reaching that point.
How do you turn a suspected bottleneck into an experiment?
Official monitoring framework: Google's SRE material emphasizes latency, traffic, errors, and saturation as complementary service signals. It also distinguishes successful-request latency from failed-request latency. Monitoring distributed systems
Original hypothesis matrix: Use those signals to narrow the cause, then propose a change that could disprove it.
| Observation | Hypothesis to investigate | Discriminating experiment |
|---|---|---|
| Queue delay rises while execution time stays similar | Admission exceeds processing capacity | Reduce arrivals and observe queue recovery |
| Database wait dominates request traces | Database contention or slow access path | Compare a targeted query or data-layout change under the same workload |
| One tenant degrades disproportionately | Skew or tenant-specific contention | Hold total traffic constant while changing tenant distribution |
| Generator misses planned starts | Generator capacity or prolonged occupied workers | Compare generator resources and iteration duration together |
High CPU is a signal, not a complete root cause. It could be useful processing or avoidable work. Likewise, low average CPU does not exclude a saturated single thread, a lock, or a constrained dependency.
Explain what else changes in your proposed experiment. Doubling application replicas might also double database connections and worsen contention. An experiment is stronger when it isolates a mechanism and records possible confounders.
How should a defensible test sequence work?
Original test plan: Begin with a small correctness run. Verify request mix, identifiers, expected responses, and measurement labels before applying significant load. Then establish a baseline at an agreed representative level.
Increase load in controlled steps when the goal is to study capacity. Hold each step long enough for the behavior relevant to your question to emerge, and record warm-up separately. Choose durations from system behavior rather than a memorized interview recipe.
If the concern is gradual leakage or accumulated backlog, a brief peak test is insufficient. Design a longer observation period and track the state that should remain bounded. If the concern is a sudden burst, include its rise and recovery instead of reporting only the steady plateau.
Repeat meaningful comparisons under comparable conditions. Record build, configuration, dataset, cache state, generator setup, and background activity. If a result is unexpectedly good, first check whether the new version skipped required work.
Define stop conditions around error growth, queue depth, resource pressure, or downstream impact. A test should answer the intended question without turning an uncontrolled overload into the only source of evidence.
What should you say when the interviewer asks for a conclusion?
Use a bounded conclusion that names the workload and measurement limits.
Original model answer: “Under the stated 120-journey-per-second mix, I would compare intended starts, delivered requests, successful operations, and per-endpoint latency distributions. I would verify generator headroom and investigate queue delay before claiming a service bottleneck. I would not average worker p99 values or exclude failed operations from the outcome report.”
A follow-up may change the workload to more writes, colder caches, or larger tenants. Explain which prior conclusion still applies and which requires another test. Capacity belongs to a workload and environment, not to a version number alone.