Grafana Loki Interview Questions: Labels, LogQL, and Expensive Queries

An incident dashboard for a checkout service becomes slow just as the team needs it most. A developer adds request_id as a Grafana Loki label so a single request is “easy to search,” then asks why the stream count and query latency rise. Another developer writes a broad LogQL query with a JSON parser and regex over a week of logs. Both changes may be well intentioned; neither identifies the actual cost boundary.

This is an original practice incident, not a report about a company's interview. Grafana's label guidance and LogQL documentation support the product behavior. The made-up checkout logs, stream counts, and query results below are explicit exercise assumptions and deductions. For incident reasoning beyond Loki syntax, Use Metrics and Logs to Establish a Production Root Cause is a useful related prompt.

The interview fixture: a checkout error burst

Assume an application ships JSON log lines to Loki. Each line has indexed stream labels service_name, deployment_environment, and region; its body includes event, status, and request_id. The incident is a ten-minute spike in payment failures in checkout, production, us-east-1. Six representative lines appear below. They are synthetic, not a claim about the user's cluster.

Line	Indexed labels	Log content
A	checkout, prod, us-east-1	{"event":"payment failed","status":"503","request_id":"r1"}
B	checkout, prod, us-east-1	{"event":"payment succeeded","status":"200","request_id":"r2"}
C	checkout, prod, us-east-1	{"event":"payment failed","status":"503","request_id":"r3"}
D	checkout, prod, us-east-1	payment failed: malformed-json
E	checkout, prod, eu-west-1	{"event":"payment failed","status":"503","request_id":"r4"}
F	inventory, prod, us-east-1	{"event":"payment failed","status":"503","request_id":"r5"}

The urgent questions are: Which fields belong in the index? How many streams do those choices create? Which lines survive each LogQL stage? What happens when JSON parsing fails? Does structured metadata make request lookup free? How do you lower the cost of a real incident query without hiding the incident? A strong answer traces both the ingestion model and the query pipeline.

Question 1: What exactly does a Loki label index?

Grafana's label documentation says Loki groups log entries with the same complete label set into a stream. The index selects streams by those labels; Loki does not index every word of each log line. For our fixture, {service_name="checkout", deployment_environment="prod", region="us-east-1"} identifies one label-set combination in the simplified dataset. A query still scans relevant log content inside selected streams to apply line-level filters.

Now add request_id as an indexed label. If A, B, and C each have a distinct value, those three entries no longer share the same label set, despite identical service, environment, and region. Across a million distinct request IDs, the exercise could create up to a million additional combinations for that slice. That is an illustrative upper-bound calculation—not an observed Loki metric, because real stream counts depend on all labels, collector behavior, and how many entries share an ID.

The cost is not merely a bigger count in a dashboard. Grafana warns that high-cardinality labels increase index work and generate many small chunks, hurting performance and cost. request_id, trace_id, user ID, and other unbounded identifiers usually describe an event, not a stable source of logs. A bounded service_name or environment is a stronger stream label. But “low cardinality” alone is too crude: a label also needs to be useful for narrowing real queries. A single {environment="prod"} stream selector could still make an investigation scan far too much data.

In an interview, explain the tradeoff with a label-set ledger: count distinct values and combinations over a realistic time window, estimate how often each stream receives logs, and inspect chunk size and index/query behavior. Ten low-cardinality labels can still multiply into many combinations. Removing a useful service label to minimize the number of labels may make incident queries worse. The target is a stable, selective stream model, not the fewest fields at all costs.

Question 2: Where should request_id go instead?

Keep a request identifier in the log body or, when the deployment supports it, use Loki structured metadata. Grafana describes structured metadata as per-log-line data that is not indexed as a stream label and need not be embedded in the text. It can still be available for filtering after the stream selection. This avoids a new indexed stream for every request ID, but it does not make an arbitrary ID lookup an index seek. Loki may still need to search the selected stream content and metadata.

Check the storage and ingestion contract before recommending it. The current Grafana documentation says structured metadata uses chunk format V4, schema version at least 13, and a TSDB index; it is not available with the deprecated boltdb-shipper store. Collector configuration and cloud deployment may also affect how attributes arrive. The same docs recommend revisiting default OpenTelemetry labels such as k8s.pod.name and service.instance.id for new deployments because they can have high cardinality; those defaults were retained for compatibility, not because they are ideal in every cluster.

A practical lookup begins with a short incident time range and stable labels, then filters for the request ID in structured metadata or log content. If the use case is frequent, point lookups across enormous log volumes, discuss whether traces or another index is better suited to that access pattern. Do not promise that moving the ID from a label to metadata always improves an individual lookup's latency; it primarily changes the ingest/index tradeoff. Measure both ingestion health and representative queries.

Question 3: Which lines survive a LogQL pipeline?

Grafana's log-query reference distinguishes a stream selector, a line filter, a parser, and a filter on original or extracted labels. For the six-line fixture, consider this illustrative LogQL query, applied to the incident's ten-minute time range in Grafana Explore:

{service_name="checkout", deployment_environment="prod", region="us-east-1"}
  |= "payment failed"
  | json
  | __error__=""
  | status="503"

The stream selector admits A–D and excludes E (wrong region) and F (wrong service). The literal line filter |= retains A, C, and D while dropping the successful B. | json extracts fields from A and C; D is malformed. Loki marks a parser failure with __error__ rather than automatically dropping the line. | __error__="" removes D for this analysis, and | status="503" keeps A and C. Predicted result: two lines. That count is a deduction from the synthetic table; it is not a screenshot from an executed Loki cluster.

This step-by-step result is a better answer than saying “use | json.” It shows why each stage is there and what it costs. If the JSON field is absent or named differently, the final filter may return nothing. If the log format changed during deployment, excluding parse errors can hide an important incident signal; inspect the error count separately. If request IDs contain sensitive data, avoid dumping full raw lines during a shared interview or production incident.

Grafana's docs say line filters applied early generally reduce work before parsing, and literal filters are faster than regex for the same simple task. The first index selector must also be selective enough to keep the candidate streams small. Neither a parsed field nor a structured-metadata field becomes an indexed stream label merely because a query shows it among results. That distinction is the center of this interview topic.

Question 4: Why is the dashboard query expensive?

Assume the dashboard originally uses {deployment_environment="prod"} | json |~ ".*payment.*failed.*" over seven days. It selects every production stream, parses lines before the intended filter, and uses a regex for a simple known phrase. The Grafana query best-practices guide recommends narrowing the time range, using precise stream labels, applying simple line filters early, and avoiding unnecessarily complex parsers. Each change removes work before the next stage. Our narrower query is a proposed improvement; actual speedup must be measured in the target cluster.

Separate three possible costs. At ingestion, high-cardinality labels can create many streams and small chunks. At query time, a broad selector and long range increase the streams and data Loki must inspect. In the LogQL pipeline, expensive parsing or regex on every remaining line adds CPU. Raising query limits is not a first diagnosis: Grafana's troubleshooting guide identifies series/cardinality errors as signals to narrow selectors, range, or aggregation. An error limit may protect the service from an expensive request rather than being the root cause.

For a dashboard that needs a count rather than raw lines, consider a range aggregation scoped to the needed labels, such as a count_over_time of the narrowed log pipeline, then aggregate to the dimensions the chart actually displays. Do not put request_id into the output series; that recreates high-cardinality results at query time. A metric query's range window is not a replacement for the dashboard's overall time-range selection. If this aggregate runs constantly and is stable, Grafana also discusses recording rules; evaluate their freshness and operational cost rather than assuming one is automatically necessary.

An interviewer may ask whether adding status="503" as an indexed label would make the query faster. It might narrow streams if status values are bounded and commonly queried, but it also splits streams by status and changes ingestion behavior. Status could be a parsed field or structured metadata instead. Decide from the full workload: frequency of this filter, distribution of statuses, number of label combinations, volume per stream, and the most urgent query. A single query's convenience is not enough to choose an ingestion schema.

Question 5: How do you prove the change helped?

Build a before/after ledger with the same tenant, time range, incident period, and expected lines. Verify that both queries return A and C from the synthetic fixture before talking about speed. In production, take a representative quiet interval and an incident burst; compare query duration, bytes and lines processed, number of streams/chunks touched, queue time, and result cardinality using available query statistics. These dimensions tell you whether the win came from stream narrowing, earlier filtering, or less parsing. Also watch ingestion-side stream counts, small-chunk pressure, dropped log lines, and costs after changing labels.

Observation	Likely question to ask next	Candidate action
Many tiny streams after request_id promotion	Did each request create a label-set combination?	Demote ID from index label; test representative lookups.
Same streams, far more bytes scanned over seven days	Is the selected time window unnecessarily wide?	Narrow to incident window and confirm no evidence is lost.
High CPU after JSON parsing on a broad selector	Can a literal filter remove lines before parsing?	Reorder pipeline and compare matched-line count.
maximum number of series error	Did extracted IDs become output series?	Drop/keep labels or aggregate to bounded dimensions.
Query returns zero lines	Is the selector or parsed field wrong?	Remove stages one at a time, including parse-error filter.

Treat a failed query as a diagnostic clue. Start with a simple stream selector, then add one stage at a time; Grafana recommends that workflow in its query troubleshooting guide. A zero result may reflect wrong labels, a time-zone or time-range mismatch, parsing errors, or genuinely no matching logs. It is not by itself proof that checkout recovered. Correlate with metrics, traces, collector health, and customer impact before closing the incident.

A strong interview answer in one minute

“Loki indexes label sets into streams, not every log-line field. I would keep stable source dimensions such as service and environment as stream labels, and avoid unique request IDs as index labels. I would query a short incident window with a selective stream selector, a cheap literal line filter, then parse JSON and filter extracted fields. I would inspect parser errors rather than silently assume missing matches mean no incident. Finally, I would compare correctness and query statistics before and after, and inspect stream/chunk behavior after any label change.”

The important detail is the two-sided cost model. A change that lowers query time by creating millions of streams can worsen ingestion; a change that lowers stream count may force every investigation to scan too much. Explain the access pattern and the evidence that would change your recommendation.

Sources and further reading

• Grafana Loki: Understand labels — streams, index labels, and cardinality.
• Grafana Loki: Structured metadata — high-cardinality event fields without indexed stream labels.
• Grafana Loki: Modify default OpenTelemetry labels — current guidance on pod and instance attributes.
• Grafana Loki: Log queries — selectors, line filters, parsers, errors, and label filters.
• Grafana Loki: Query best practices — time range and pipeline ordering.
• Grafana Loki: Troubleshoot log queries — zero results and cardinality-limit diagnostics.