Articles
Queuing Theory Interview Questions: Utilization, Little's Law, and Tail Latency
Practice queuing theory interview questions with Little’s law, M/M/1 capacity calculations, service variability, and carefully bounded tail-latency claims.
A service can process 100 requests per second. At 80 requests per second, an idealized queue has a mean time in the system of 50 milliseconds. At 99 requests per second, that mean becomes one second. The service capacity did not change; the available headroom did.
Queuing theory interview questions ask you to explain this behavior without confusing an average with a latency percentile. A complete answer identifies the queue boundary, checks units and stability, and states exactly which distributional assumptions support the calculation.
Evidence boundary: MIT teaching materials support the queueing identities and models cited below. Preparation recommendations are editorial inference, not candidate reports or measured production results.
What system are you measuring?
Draw a boundary before using a formula. It could include only requests waiting for a worker, or it could include waiting plus requests currently being served. It might also include a downstream connection pool or an entire customer-visible request lifecycle.
Define:
lambda: the relevant average arrival or throughput rate.L: average number of requests inside the chosen boundary.W: average time spent inside that same boundary.LqandWq: corresponding quantities for waiting only.
Little’s law relates matched averages as L = lambda × W. MIT’s 2026 queueing lecture discusses stationary analysis and the importance of the system being considered. The identity does not require the Poisson arrivals or exponential service assumptions of M/M/1.
An original measurement example makes the boundary problem concrete. Suppose admitted throughput is 80 requests per second, with average waiting time 40 ms and average service time 10 ms. Then W = 0.050 seconds and L = 4. Waiting alone gives Lq = 80 × 0.040 = 3.2.
Multiplying requests per second by milliseconds without converting units gives a number one thousand times too large. Using total in-flight requests with waiting-only time also mixes boundaries.
If requests are rejected before entry, offered traffic is not admitted throughput. If requests cancel after entry, include their departures and time in the system consistently. Counting only successful completions while occupancy includes failed attempts creates a different mismatch.
Can you verify Little’s law with a finite timeline?
Use this original deterministic trace, starting empty at time zero. Three jobs arrive at times 0, 1, and 2 seconds. A single first-come, first-served worker takes two seconds per job.
| Job | Arrival | Service start | Departure | Time in system |
|---|---|---|---|---|
| A | 0 | 0 | 2 | 2 seconds |
| B | 1 | 2 | 4 | 3 seconds |
| C | 2 | 4 | 6 | 4 seconds |
Over the complete six-second interval, the occupancy area is nine request-seconds. You can calculate it from the number present between events, or simply sum the three complete sojourn times: 2 + 3 + 4 = 9.
Average occupancy is 9 / 6 = 1.5. The completion rate over the same interval is 3 / 6 = 0.5 requests per second, and mean time in system is 9 / 3 = 3 seconds. Their product is also 1.5.
This finite identity works because the chosen interval starts and ends empty and includes the complete time of every counted job. It is not a claim that these three arrivals form a stationary process.
Editorial inference: Use the trace to debug monitoring definitions. A dashboard sampled only at departures may miss occupancy between events. A rolling window that begins with old jobs or ends with unfinished jobs needs boundary accounting before the simple calculation can be interpreted.
If a metric disagrees with Little’s law, investigate population, windows, censoring, and units first. You may have discovered a measurement bug rather than an exotic queue.
Why does M/M/1 waiting grow near capacity?
The original request-rate exercise assumes a stationary, first-come, first-served M/M/1 queue: Poisson arrivals at rate lambda, independent exponential service times at rate mu, one server, and no finite buffer limit. Its infinite-buffer steady-state calculation requires rho = lambda / mu < 1.
The mean time in system is W = 1 / (mu − lambda), and the mean waiting time is Wq = W − 1 / mu. MIT’s M/M/1 lecture notes derive the stationary occupancy and delay results.
Fix mu = 100 requests/second, giving mean service time 10 ms:
| Arrival rate | Utilization | Mean waiting | Mean time in system | Mean requests in system |
|---|---|---|---|---|
| 80/s | 0.80 | 40 ms | 50 ms | 4 |
| 95/s | 0.95 | 190 ms | 200 ms | 19 |
| 99/s | 0.99 | 990 ms | 1000 ms | 99 |
The denominator is the remaining service-rate margin. As that margin approaches zero, the model’s mean waiting grows sharply. A utilization target that sounds efficient can leave little tolerance for random arrivals or slow service.
At lambda ≥ mu, these stationary finite-mean formulas do not supply a valid answer. A finite-buffer system can instead reject requests and reach a different stationary behavior; that requires modeling admission and loss, not pretending the infinite-buffer formula still applies.
Editorial inference: Use the table to explain the direction of a capacity risk, then validate the real arrival and service distributions. A database-dependent worker does not necessarily maintain the same service rate as concurrency increases. Locking, CPU contention, or downstream limits can change the effective capacity.
Can mean latency tell you p99 latency?
No distribution-free conversion exists from a mean to p99. Many latency distributions can share a mean while having very different tails. Little’s law connects averages; it does not identify a percentile.
Under the specific stationary FCFS M/M/1 assumptions, the sojourn-time distribution is exponential with rate mu − lambda. For that model, the original p99 calculation is:
p99 = ln(100) / (mu − lambda).
At 80 arrivals per second and capacity 100, this gives about 230.26 ms. At 95 arrivals per second, it gives about 921.03 ms. These are model-derived values, not production forecasts or measured benchmarks.
One way to see the result is to condition on the number of jobs an arrival sees. The stationary geometric distribution over that number mixes the exponential-service sums into the stated sojourn distribution. Changing arrival or service assumptions can invalidate this derivation.
For an interview follow-up, suppose the service occasionally stalls for several seconds. The mean service time might remain near 10 ms, but the exponential assumption no longer describes its tail. You need observed distributions or a justified alternative model.
Do not add the p99 of waiting to the p99 of service and call the sum the request p99. Percentiles of sums depend on the joint behavior of the components. Trace complete request durations or model their dependence explicitly.
A useful answer separates three statements: the mean follows the assumed model; the percentile additionally uses its distribution; deployment needs measurements that verify both assumptions and the target boundary.
What changes when service times have greater variance?
Even with the same arrival rate and mean service time, service variability changes average waiting. For an M/G/1 queue with Poisson arrivals, independent identically distributed service times, FCFS service, stability, and finite second moment:
Wq = lambda × E[S²] / (2 × (1 − rho)).
MIT’s M/G/1 discussion explains the role of service-time variance in the Pollaczek–Khinchine result.
Here is an original comparison using arrival rate 80/s and mean service time 0.01 seconds. Utilization remains 0.8 in both cases.
| Service assumption | Second moment | Mean waiting |
|---|---|---|
| Deterministic 10 ms | 0.0001 seconds² | 20 ms |
| Exponential with mean 10 ms | 0.0002 seconds² | 40 ms |
The second moment equals variance plus squared mean. The exponential case has twice the second moment, producing twice the modeled mean waiting here.
A long request occupies the worker and delays jobs arriving behind it. That intuition explains why reducing variance can improve waiting even when average service time is unchanged.
Editorial inference: Investigate slow dependency calls, large payloads, and heterogeneous job types before assuming average service optimization is enough. Separating long jobs or changing scheduling may help some objectives but can also change fairness and the applicable model. A mean-wait formula does not establish the full tail distribution.
The arrival process matters too. A burst of correlated requests violates a simple Poisson description. Daily average rate hides peak intervals and synchronization from retries or scheduled clients. Examine the time scale relevant to the queue’s buildup and recovery.
How do retries, buffers, and multiple workers alter the decision?
Retries create additional attempts. If a logical request makes several attempts, the queue receives attempts, while the user may measure latency across the entire retry lifecycle. Keep both populations visible.
For an original incident scenario, timeouts trigger immediate retries while old attempts continue consuming workers. User traffic stays constant, but offered attempt rate rises. Increasing a timeout alone does not remove the work already in flight. Bounded retries, backoff, cancellation behavior, and admission rules require a coordinated design.
A larger buffer can absorb a burst, but it does not increase service capacity. It may convert immediate rejection into longer waiting. If queued work expires before execution, admission should consider deadlines rather than merely free queue slots.
With several workers, lambda / (c × mu) can describe a nominal utilization under equal independent server capacity. It does not justify replacing M/M/c with a single server of rate c × mu for all latency calculations. Shared downstream resources and uneven routing complicate the picture further.
A shared queue can pool idle capacity; separate queues can isolate workloads. Neither choice is universally best. Ask about fairness, head-of-line blocking, worker specialization, and whether queues share a bottleneck anyway.
Editorial inference: Propose a measurement-driven test: collect admitted attempt rate, queue depth over time, service-time distribution, cancellation outcomes, and complete latency by traffic slice. Increase load in controlled steps and watch whether service capacity remains stable. Separate steady-state observations from warmup or recovery periods.