Articles
Coordinated Omission Interview Questions: Why Your Load Test Hides Tail Latency
Explain coordinated omission in load tests: distinguish closed and scheduled workloads, read tail latency honestly, and design evidence that exposes stalls.
A load test can report a calm p99 while users are waiting through a one-second freeze. The usual culprit is not a bad percentile formula. It is that the load generator stops offering work while it waits for the slow response it just observed. The requests that would have arrived during the pause are absent from the sample, so the measurement coordinates with the system's failure.
That blind spot is called coordinated omission. It is a favorite systems-interview topic because it exposes whether you can separate a latency number from the workload that produced it. A strong answer starts with offered-load timing, explains what was not measured, and proposes an experiment that makes the missing demand visible. Practice the adjacent reasoning with Design tests to measure latency impact.
Evidence boundary: Official and primary-source facts about HdrHistogram's recorded-value correction and the scheduled-rate wrk2 approach are attributed to their linked project documentation and the original Mechanical Sympathy discussion, checked September 2026. Candidate-reported evidence is not used to predict an employer's interview loop. Inference includes the timelines, architecture choices, and interview exercises in this article.
What is coordinated omission in one answer?
Coordinated omission occurs when a measurement process waits for a response before issuing or recording the next measurement, even though the real workload would have continued arriving. During a pause, the client records one slow response and then records nothing for all the work it would otherwise have attempted. A latency histogram built from those observations under-represents the stall's impact.
The name is useful because the test client and service become coordinated: the service is slow, so the client stops creating observations. This can happen in a closed-loop benchmark, a health checker that schedules its next probe after completion, a queue consumer that fetches only after finishing, or an application dashboard that samples only completed requests.
It does not mean every closed-loop test is wrong. A closed loop can accurately model a fixed number of users who truly wait for a page before taking the next action. It becomes misleading when you present that result as the latency under an external schedule, an arrival-rate SLO, or a growing queue of independent requests. The right question is: what does the production demand do while a request is stalled?
Walk through the missing observations
Assume production normally receives one independent request every 10 ms. A test generator sends a request, waits for it to finish, then waits 10 ms before sending the next one. Most responses take 5 ms. At one point, a dependency pauses the service for 1,005 ms.
The response-driven generator records one request near 1,005 ms. It does not issue roughly 100 requests that a fixed 100-requests-per-second schedule would have attempted during that time. If the service queues those requests after the pause, many users experience a latency that includes waiting behind the backlog. The closed-loop histogram has one large value and many ordinary values; its high percentile can look much better than the scheduled workload's percentile.
Do not claim that all 100 omitted requests had exactly 1,005 ms latency. Their observed latency depends on admission control, queue capacity, timeouts, retries, load balancers, and recovery rate. The key fact is narrower: the original test did not observe their latency at all, so it cannot support an arrival-rate claim.
What is the difference between closed and open workloads?
A closed workload keeps a bounded number of in-flight users or workers. A worker commonly sends a request, waits, thinks, and sends the next. Throughput falls when the system slows because the workers are blocked. This is appropriate for some interactive journeys and capacity questions.
An open or scheduled workload attempts arrivals according to an external clock or arrival process. A constant schedule may try one request every 10 ms regardless of whether earlier requests completed. Backlog, rejection, and timeout are outcomes to record, not reasons to silently stop the clock. Real traffic may be neither perfectly open nor perfectly closed, so express the model: cron jobs may be scheduled, a mobile app user may wait, and a message broker may apply backpressure.
| Question | Response-driven loop | Scheduled-rate generator |
|---|---|---|
| When is the next attempt chosen? | After a completion | From an independent schedule |
| What happens during a stall? | Fewer attempts are made | Attempts, queueing, rejection, or timeouts are observed |
| What can it model well? | Fixed concurrent users | External arrival-rate pressure |
| Main reporting risk | Understates external-demand tail latency | Can overload unrealistically if the schedule is wrong |
The interview move is not to declare open load superior. It is to map each model to the product's actual arrival behavior and state the conclusion's scope.
How do histograms and correction help?
HdrHistogram provides facilities for correcting a recorded value when you know an expected interval between observations. Conceptually, a 1,005 ms recorded latency under a 10 ms interval implies that the system could have affected observations at 995, 985, and so on, in addition to the one sample. Adding those synthetic values produces a distribution that better represents the stall's potential impact under that fixed schedule.
That is a correction to the recorded values, not a time machine. It cannot recover whether the service would have rejected requests, whether clients would have abandoned them, or how retries would interact. It depends on a defensible expected interval. If your production traffic is bursty or rate-limited, using a flat 10 ms interval may be less representative than measuring the real arrival process or running a scheduled generator.
The original HdrHistogram guidance is especially useful in an interview because it distinguishes recording a latency from reporting it. You should store enough information to say whether values are uncorrected completion latency, corrected-for-expected-interval latency, queue waiting time, or end-to-end user time. Combining them into one unnamed “p99” creates confusion rather than insight.
Design a load test that exposes the tail
Start with the service boundary and an arrival hypothesis. Define whether the schedule is constant, bursty, or trace-derived; the target rate; concurrency cap; request timeout; retry behavior; and which status codes count as outcomes. Instrument the generator separately from the service. Record scheduled start time, actual send time, connection start, first byte, completion, response code, timeout, and cancellation. If a request cannot be sent because a client-side queue is full, record that too.
Next choose a deliberate fault. A controlled 500 ms dependency delay, constrained CPU, or a temporary blocked worker pool is easier to interpret than an accidental noisy run. First validate the fault in a small environment. Then run both a response-driven and scheduled-rate experiment at the same nominal healthy rate. Compare offered attempts, completions, errors, in-flight requests, server queue depth, and percentile distributions—not only requests per second.
The expected observation is a divergence. The response-driven run often lowers its own offered load during the pause. The scheduled run can reveal queue growth, rejections, or a longer tail. If both look identical, investigate whether the fault actually propagated, the schedule was not independent, or an admission-control layer was hiding the difference.
Which metrics should you put beside p99?
Report attempted arrival rate, accepted rate, completion rate, error rate by class, timeout rate, and in-flight or queue depth. Add latency percentiles with an explicit start and end: client scheduled time to completion, actual send to completion, or server receive to completion. These measure different things. A queue may keep server processing latency low while user-perceived latency climbs.
Look at time series as well as a final histogram. A one-minute aggregate can conceal a ten-second pause that violates the product SLO. Plot offered rate, completion rate, p50/p99/p99.9, queue age, and saturation around the incident. Use a histogram that retains an adequate range and significant figures for the tail you care about; a percentile is only as reliable as the values that entered it.
Also report the client limit. A generator that has one connection or a saturated CPU can become the bottleneck. Verify its event-loop lag, file descriptors, sockets, and outbound bandwidth. Otherwise you may mistake generator backpressure for service recovery.
How should you answer a system-design follow-up?
If an interviewer says, “Our p99 looks fine but users complain about freezes,” ask whether the test was arrival-rate controlled, how next requests were scheduled, and what happens to production work during a slow response. Ask for raw histogram values, queue depth, timeout/retry behavior, and the latency boundary. Then propose one controlled stall and a paired response-driven versus scheduled test.
If the product has a fixed worker pool, explain that a closed-loop test can still be relevant for throughput under that cap. Add a queue-facing test for jobs that arrive independently. If a gateway rejects excess work, count rejections and recoveries rather than interpreting fewer completions as lower demand. If the product retries, decide whether the SLO is per attempt or per logical user operation; otherwise retries can both amplify load and hide failures in a completion-only metric.
Avoid promising a universal multiplier from one corrected p99 to another. The shape of the tail belongs to the service and workload. Your conclusion should be: “This measurement cannot justify the current latency claim until we observe the expected arrival model and label the metric boundary.”
A compact interview exercise
You own a payments API. A test tool runs 50 virtual users; each sends the next request only after the prior one completes. The tool reports p99 of 80 ms. During the test, the database pauses for 800 ms once. Explain why 80 ms may be a poor statement about a checkout request arriving every 20 ms from independent clients.
First, draw the event timeline. The virtual users are blocked during the pause, so their offered rate drops. Independent arrivals may instead wait in a queue, fail a deadline, or be rejected. Then specify a revised experiment: a scheduled 50-per-second workload with idempotency keys, a bounded client queue, recorded attempted and accepted rates, and a 900 ms request deadline. Compare end-to-end latency and failure outcomes to server processing time. This answer shows you understand tail latency as a system behavior, not a dashboard field.
Practice with connected questions
| Question | Practice focus | Why it helps |
|---|---|---|
| Design tests to measure latency impact | Measurement setup | Makes workload and metric boundaries explicit |
| Analyze end-to-end request latency | Latency decomposition | Separates queueing from processing |
| Design a low-latency ML inference API | SLO trade-offs | Connects tail behavior to serving design |
| Debug a cache incident end-to-end | Incident evidence | Practices correlating a stall with service signals |
| Design a low-latency trading system | Tail sensitivity | Forces admission, queue, and deadline decisions |
Frequently asked questions
Does a low p99 always mean coordinated omission occurred?
No. A low p99 can be real for the measured workload. Coordinated omission is a concern when the observation process pauses with the service and the result is used to represent demand that would continue independently. Inspect how attempts are scheduled before diagnosing the number.
Can a fixed concurrency test ever reveal a stall?
Yes. It records the slow requests that were sent. It may reveal throughput collapse as workers wait. It generally cannot show the latency of external requests it never attempted during the pause, so state its scope precisely.
Is histogram correction enough?
It can improve interpretation when the expected interval is known and the recorded values are appropriate for correction. It cannot reproduce queue limits, retries, user abandonment, or admission control. Prefer a workload that actually exercises the arrival pattern when that behavior matters.
Should I report corrected or uncorrected latency?
Report the one that answers the stated question and label it. For diagnosis, showing both can reveal the measurement gap. Do not silently substitute a corrected number for a service SLO whose boundary and arrival model were never defined.
Coordinated omission is a reminder to test the clock as carefully as the code. Define who keeps sending work during a pause, capture what happens to that work, and give every percentile a workload and timing boundary before using it to make a reliability decision.