Articles
Grafana Tempo Interview Questions: Trace Search, Sampling, and Missing Spans
Explain Tempo TraceQL span conditions, sampling, direct lookup versus search, and missing-span diagnosis with worked traces and version-aware evidence checks.
A trace opens when you paste its ID, but your search query cannot find it. Another trace shows the checkout span and no payment span. Neither symptom, by itself, proves that Grafana Tempo dropped data.
Grafana Tempo interview questions are easier when you identify the span you expected, the query semantics you used, and the last pipeline boundary where you have evidence. This guide works through a TraceQL truth table and a missing-span investigation without assuming that search absence equals ingestion failure.
Evidence boundary: Official facts come from Grafana Tempo and OpenTelemetry documentation. The traces, ledgers, and interview answers below are original practice exercises and engineering inference, not candidate reports. The current Tempo documentation identifies v3.1.x; its ingestion architecture differs from familiar 2.x explanations. Match component names, metrics, and limits to the deployed version.
For a related implementation exercise, use Design a Kafka Consumer Service That Groups Trace Events by Trace ID; the practice table below explains its connection to this topic.
What does a trace contain, and what can be missing?
A trace groups spans that share a trace identifier. Spans describe individual operations, with identifiers, timestamps, attributes, and relationships. A service name is useful context, but it is not a unique span identifier or proof that every operation in that service was recorded.
Engineering inference: When someone says “the trace is missing,” ask whether they mean an entire trace cannot be found, one span is absent, parent-child relationships look broken, or the search returned the wrong set. Each symptom suggests a different investigation.
Original exercise: an HTTP request reaches checkout, which invokes a payment operation. The checkout span is present, but the expected payment span is absent. Possible explanations include missing instrumentation, a span that was never ended and exported, sampling, export failures, a different trace ID caused by propagation problems, or a query that excludes the span. Do not choose among them without evidence.
Record the expected operation, trace ID, approximate time, service and environment attributes, and the ingestion destination. If the downstream operation has another trace ID, you have a propagation or modeling question rather than proof that Tempo deleted its child.
Asynchronous work needs an explicit relationship model. A later operation might use a link rather than appear as a direct child in the same trace. Ask how the instrumentation represents that work before treating every detached operation as a failure.
A screenshot of an incomplete waterfall is a starting observation. A useful interview answer adds the identifiers and a plan for checking where the expected span stopped appearing.
Does this TraceQL query require one span or two?
Official fact: Within one TraceQL brace expression, all conditions must match the same span. Logical operators between spansets can instead require matching sets within the same trace. Grafana's TraceQL query construction guide explains the distinction.
These original queries ask different questions:
{ resource.service.name = "checkout" && duration > 500ms }
{ resource.service.name = "checkout" } && { duration > 500ms }
The first asks for a checkout span whose duration exceeds 500 milliseconds. The second asks for a trace that contains a checkout span and also contains a span exceeding that duration. They need not be the same span.
| Original trace fixture | One-brace query | Two-spanset query | Reason |
|---|---|---|---|
| Checkout 80 ms; payments 700 ms | No match | Match | Different spans satisfy the service and duration conditions |
| Checkout 700 ms; payments 80 ms | Match | Match | The checkout span satisfies both conditions |
| Catalog 80 ms; payments 700 ms | No match | No match | No checkout span exists |
| Checkout 80 ms; payments 90 ms | No match | No match | No span exceeds 500 ms |
These are semantic fixtures, not a claim that the 80 ms checkout span is the synchronous parent enclosing the 700 ms payment span. The exercise needs only two spans in one trace; their precise structural relationship is a separate condition.
Engineering inference: Restate the business question in span terms before writing TraceQL. “Slow checkout operations” and “traces involving checkout that contain any slow operation” are different searches. A correct query for one can be misleading for the other.
Attributes also have scope. A resource service name and a span attribute are not interchangeable because their labels sound similar. Inspect the stored attribute and type in an actual trace. Test a single predicate first, then add the second condition to isolate where the match disappears.
If the interviewer asks for direct parent-child behavior, explain that the logical conjunction above does not establish that relationship. Choose supported structural operators only after defining the intended topology and checking the installed version's semantics.
Why can direct trace lookup succeed while search fails?
Direct lookup starts from an identifier. Search evaluates predicates over a time range and other configured query context. Opening an ID therefore does not prove that every search expression should match that trace.
Engineering inference: Compare the query text, selected time range, tenant, datasource, and actual stored attributes. Check whether the trace falls outside the window, whether the service attribute uses a different value, and whether the predicate requires the same span rather than separate spans.
Original exercise: you search for checkout spans longer than 500 ms. The trace contains checkout at 80 ms and payments at 700 ms. Direct lookup works, while the one-brace query returns nothing. The truth table explains the result without an ingestion incident.
Another case involves a span timestamp outside the selected search window. Verify the instrumented timestamps and the search range rather than simply widening it forever. Clock problems can make telemetry hard to find and can also distort duration or ordering analysis.
Distinguish an empty successful result from a query error or a deliberately partial result. Preserve the response and status. A failed query is not evidence that there are zero matching traces.
Official fact: Grafana's missing-trace troubleshooting guide separates ingestion, sampling and limits, and querying. Its current architecture includes distributor ingestion, Kafka, live-store recent data, and block-builder output to storage. Use that version-specific guide rather than copying a 2.x ingester checklist unchanged.
The practical interview answer is: “I would first show that the trace satisfies the exact predicate and falls within the intended query context. Then I would inspect the query path's error or freshness evidence. I would not diagnose ingestion loss solely from search absence.”
How does sampling change what you can infer?
Official fact: OpenTelemetry distinguishes head sampling, which decides early, from tail sampling, which evaluates received trace information later. Sampling and context propagation affect what reaches downstream systems. See the OpenTelemetry sampling documentation.
Engineering inference: Tempo cannot search spans that were never exported to it. Raising the backend's retention does not recover an SDK decision that discarded an earlier trace. A tail sampler also cannot reconstruct spans already removed by upstream head sampling.
Original exercise: an application handles 1,000 requests, but a policy retains a subset before export. A 10% probability does not guarantee exactly 100 retained requests in every batch. Use observed counts when reconciling a real run, and distinguish requests, traces, and spans: one request can generate many spans.
If you keep all error traces and a smaller fraction of successful traces, the stored set is intentionally biased. An error share calculated from stored traces is not automatically the production request error rate. Use an appropriate request metric or a justified sampling-aware estimator for that question.
A follow-up asks whether tail sampling guarantees complete error traces. The proposed answer should name decision timing, buffering, routing of related spans, late arrivals, and upstream drops. “Tail sampling sees the error” is not enough if the relevant spans never arrive or arrive after a decision.
For consistent distributed sampling, inspect how the parent decision and trace context are propagated. Independently sampling every service can create confusing partial views. Do not “fix” that by temporarily recording every production request without considering the deployment's data and volume constraints; use an authorized, controlled fixture first.
Trace the last known boundary for a missing span
The following original evidence ledger deliberately avoids subtracting unrelated counters. Keep the same trace/span identity, tenant, and time interval when comparing stages.
| Stage | Evidence that narrows the case | What absence does not prove |
|---|---|---|
| Instrumentation | Expected operation creates and ends the named span | Backend failure if the span was never created |
| SDK/exporter | The specific span is eligible and submitted for export | Durable backend availability from an export attempt alone |
| Collector or Alloy | Receiver and exporter observations agree for the test span | That every received span was successfully forwarded |
| Tempo ingestion | Accepted-span or bounded diagnostic evidence for that destination | Immediate searchability in every query path |
| Recent query path | Query status and freshness evidence for the target window | Long-term object storage completeness |
| Stored query path | Correct tenant, time, predicates, and readable blocks | Sampling loss if the predicate simply does not match |
Engineering inference: Find the first boundary where the expected evidence changes. If the SDK never creates the payment span, start with instrumentation. If the Collector receives it but fails export, investigate that hop. If ingestion accepts it but a correct recent query errors, inspect the query path before changing the sampler.
Aggregate counts help detect a problem, but retries can make attempts exceed unique spans. Restarts and different label selections can make counters incomparable. Avoid a dramatic “15 spans lost” conclusion from subtracting totals that measure different populations.
For a synthetic reconciliation exercise, label 100 specific spans as received and 90 specific spans as successfully exported. Ten have a known export failure in that fixture. If search returns 75, the remaining 15 still require investigation: they may not satisfy the query, may be outside its window, or may be affected by query freshness. The numbers do not establish one single drop mechanism.
This ledger is an investigation aid. A local simulation can test its bookkeeping, but does not validate a real Tempo deployment or prove that a named metric represents unique accepted spans in every release.
How does the deployed architecture affect diagnosis?
Engineering inference: Begin the incident explanation with the installed Tempo version, deployment mode, and ingestion route. A component that does not exist in that deployment cannot be the cause of its failure.
For the currently documented 3.1 architecture, recent data and older stored data have distinct paths. A controlled trace that appears later can indicate freshness or lag rather than permanent loss. Capture the actual query outcome and version-appropriate evidence before promising that scaling one component will fix the symptom.
For an older deployment, use the matching documentation and inspect its own ingestion and query components. Do not rename a 2.x metric to a plausible 3.x equivalent from memory. Similarly, Collector and Alloy metric names depend on the component and release; inspect what the running endpoint exposes.
In a multi-tenant setup, verify that ingestion and query use the intended tenant through the configured gateway and datasource. A mismatch can make existing telemetry appear absent. Authentication, tenant routing, and query predicates are separate checks.
A safe diagnostic trace has a known operation name, identifiers, expected spans, timestamps, and destination. Track it through the system without logging arbitrary sensitive span contents. Remove broad diagnostic logging after collecting the needed evidence according to the deployment's normal procedure.
What would you say in a missing-span interview case?
Prompt: “Payment errors increased, but Tempo shows almost no payment spans.”
An original strong answer is: “I would verify the error increase using request metrics, then generate an authorized known payment trace. I would check instrumentation and propagated trace context, the sampling decision, exporter and Collector evidence, and Tempo acceptance. If the span is accepted, I would inspect tenant, time range, stored attribute scope, query semantics, and the version-specific recent or stored query path.”
The follow-up asks whether increasing sampling solves the incident. Explain that it helps only if sampling is the limiting boundary. It cannot repair a wrong endpoint, missing instrumentation, incompatible predicate, or query-path error. Propose a change only after evidence points to that boundary, then repeat the known trace test.