Grafana Labs System Design Interview: Preparing for Non-Abstract Design

A system design answer can look complete on a whiteboard and still fail its first capacity check. “Add a queue, shard the database, replicate across zones” leaves important questions open: how much data enters, which resource fills first, and whether the surviving machines can carry the load after a failure.

For Grafana Labs preparation, the useful skill is turning vague requirements into a design you can challenge with numbers. This guide develops an original telemetry-ingestion exercise, including checked arithmetic and a zone-loss revision. It is not a leaked prompt, a description of Grafana’s production architecture, or a benchmark of its software.

To practice the same reasoning on a different workload, open Design a Telemetry Platform Ingesting Two Gigabytes per Second. Keep its assumptions separate from the smaller worked example here.

What is verified about the interview?

Official, dated process: Grafana Labs’ August 2023 backend hiring article explicitly describes a non-abstract system design stage and connects it to Google’s NALSD approach. It emphasizes reasoning from an underspecified problem and making the candidate’s thinking visible. The published description included a one-hour session. That is historical guidance, not a promise that every current role uses the same name, duration, or sequence.

Current official principles: The Grafana Labs hiring guide emphasizes evidence, structured criteria, collaboration, and adaptability. It does not turn the older backend article into a universal current technical-round specification. Use the invitation and recruiter guidance for your role’s actual format.

Candidate evidence boundary: A public Reddit post asks for information about an upcoming process. It is not a completed system design account. We did not establish two independent same-cycle reports, so this article makes no claims about recurring prompts or a current pass rubric.

What makes a design non-abstract?

Primary method source: Google’s NALSD chapter describes iterative design that connects requirements to concrete resource estimates and failure behavior. Its emphasis is defensible reasoning rather than a uniquely correct machine count.

Our preparation interpretation is to keep three things visible: the assumption, the calculation it drives, and the decision that follows. For example, “seven days of retained raw samples” is an assumption. Multiplying the arrival rate by record size and retention is a calculation. Deciding whether one storage tier is adequate is a design decision that still needs further evidence.

You should also identify what a calculation excludes. A raw-byte estimate does not include every index, replica, query cache, compaction buffer, or protocol overhead. Exclusions are not an excuse to avoid estimating. They make the estimate useful by showing where the next revision belongs.

Define the telemetry exercise before selecting components

Imagine a service that accepts measurements from customer applications and supports recent operational dashboards. The following numbers are editorial assumptions, chosen for practice rather than taken from Grafana infrastructure:

Assumption	Value
Average arrival rate	100,000 samples/second
Accounted bytes per sample	200 bytes
Peak multiplier	3
Raw retention	7 days
Physical copies	3
Failure objective	Continue writes after one zone is lost

Use decimal units throughout: one MB is one million bytes, and one TB is one trillion bytes. Treat the 200 bytes as the sample representation included in this model. If it describes compressed storage but not network encoding, introduce separate sizes rather than quietly reusing it for both.

Clarify whether the peak lasts seconds, minutes, or days. Ask whether “continue writes” permits delayed visibility, whether acknowledged data may be lost, and whether a failed zone must immediately regain its lost redundancy elsewhere. These are different requirements. The arithmetic below sizes a simplified physical-write budget; it does not by itself establish durability or latency guarantees.

Calculate the workload with units attached

The average logical input is:

100,000 samples/s × 200 bytes/sample
= 20,000,000 bytes/s
= 20 MB/s

Seven days contain 604,800 seconds. Therefore logical retained data is 12.096 TB. Three physical copies require 36.288 TB under the model, before extra storage costs. Peak logical input is 60 MB/s, and the modeled three-copy peak write demand is 180 MB/s.

Notice what changed between the estimates. Retention uses the average rate, assuming the stated average already accounts for normal peaks. Instantaneous write capacity uses the peak rate. Multiplying seven days by the peak again would describe a different traffic assumption. It may be a useful stress case, but it should be named as such.

A local Python calculation checked these values, the node counts below, and the backlog example. All arithmetic assertions passed. We did not deploy a telemetry service or measure storage hardware. The calculated numbers are exact consequences of hypothetical inputs, not evidence that a real node sustains the chosen throughput.

Turn the workload into a provisional resource budget

Assume a fictional storage node can sustain 25 MB/s of physical writes under a representative workload. Choose a 60% operating target, leaving an initial allowance for variation and other work. The resulting budget is 15 MB/s per node. Both the throughput figure and utilization target need validation before any production purchase or deployment.

At peak, 180 divided by 15 gives 12 nodes. That result is a healthy-cluster lower bound for this write model. It is not yet a complete design. It says nothing about read throughput, memory, partition skew, replica placement, or a failed zone.

Suppose each node also provides 4 TB of usable storage. Eighteen nodes would provide 72 TB; 36.288 TB of modeled retained copies occupies 50.4% of that. This cross-check is helpful, but it does not prove the system is comfortably sized. Temporary rebuild space, indexes, and uneven placement could consume the apparent margin. Identify those costs before interpreting “half full” as a safety guarantee.

A good explanation names the binding constraint provisionally: “With these assumptions, the zone-loss write budget determines the node count. Storage also fits in the simplified model, but I still need to examine memory and read amplification.” That is a conclusion another engineer can challenge.

Recalculate after losing one zone

Distribute 12 nodes evenly across three zones: four per zone. Losing a zone leaves eight nodes. At 15 MB/s each, they provide 120 MB/s of modeled write capacity, below the 180 MB/s requirement. Healthy capacity was not sufficient failure capacity.

Let each zone contain n nodes. After one zone is lost, 2n survive. The condition is:

2 × n × 15 MB/s >= 180 MB/s
n >= 6

The revised fleet is therefore 18 nodes, six per zone. Losing six leaves 12, which exactly meet the modeled operating budget. This is a sizing exercise, not a complete replication protocol. Replica placement, acknowledgment rules, and the ability to perform required writes on the surviving topology must be designed separately.

Do not describe the result as unlimited headroom. The 60% operating target already embeds one chosen allowance, and the failed-cluster result lands exactly on that target. If rebuild traffic must also run at full speed during peak load, add its resource demand explicitly or revise the service objective. Counting the same spare capacity twice produces reassuring but invalid arithmetic.

What if you keep twelve nodes and accept a backlog?

This is a useful alternative, provided the requirements allow delayed processing and an appropriately durable buffer. After one zone fails, the modeled deficit is 60 MB/s at peak. Ten minutes at that deficit accumulates 36 GB of pending physical-write work.

When arrivals return to the average, modeled demand falls to 60 MB/s. The surviving eight nodes provide 120 MB/s, leaving 60 MB/s for draining that work. With no other limiting resource, the backlog takes another ten minutes to clear. Our local calculation checked both results.

The 36 GB figure is expressed in the model’s physical-write units. A queue storing one logical copy would need a different byte calculation, and its own replication increases its physical footprint. State where the buffer lives and what representation it stores before using this number to size it.

Then connect the trade-off to users. Are dashboards stale during the backlog? Are alerts delayed? Can producers retry safely if admission is limited? Does one customer’s burst consume everyone’s buffer? “The queue absorbs it” is incomplete until you define limits, durability, recovery rate, and the consequence of reaching capacity.

Revise one assumption at a time

Try extending retention from seven to fourteen days while keeping the arrival distribution unchanged. The storage estimate doubles to 72.576 TB of copies, already above the fictional 72 TB fleet capacity before overhead. The write-rate calculation does not double merely because retention does. The revised requirement challenges a different resource.

Now instead keep retention at seven days and double the peak multiplier from three to six. Peak physical-write demand becomes 360 MB/s. Under the same one-zone-loss model, each surviving pair of zones must supply that capacity, requiring 12 nodes per zone, or 36 total. Again, this is a conditional result, not a recommendation for real Grafana workloads.

These revisions demonstrate why an assumption ledger is useful. A change in one input should propagate to the estimates it actually affects. Avoid restarting the entire design conversation or changing several unrelated numbers to protect your original architecture.

Next, introduce skew. If one tenant produces most of the traffic and maps to one partition, the aggregate fleet budget can look healthy while a single partition is overloaded. Discuss a partitioning strategy, a way to detect imbalance, and the cost of moving or splitting hot work. The total-byte calculation remains useful, but it is no longer the only constraint.

Keep the discussion collaborative and reviewable

Present a small viable path through the system, then challenge it. For this exercise, identify admission, buffering if required, durable storage, and a query path. Explain each responsibility before naming a technology. A product name does not establish its capacity under your workload or prove that its failure behavior matches the requirement.

When new information invalidates an estimate, say so plainly. “My first twelve-node estimate covered a healthy peak. It fails the zone-loss condition, so I revised it to eighteen under the same operating budget.” That explanation shows the reason for the change and preserves the evidence behind it.

Close a practice session by distinguishing what you established from what you would measure next. You established a conditional write and retention model. You still need representative node benchmarks, query concurrency, memory pressure, replication behavior, skew, and recovery traffic. Choose the next measurement based on the weakest assumption that could change the design, rather than listing every possible performance test.

Five questions to practice concrete design reasoning

Give each practice attempt a specific constraint to examine.

Practice question	Constraint to examine
Design a Telemetry Platform Ingesting Two Gigabytes per Second	Translate rates into physical budgets
Design a Metrics Collection System	Connect ingestion to query needs
Design a Low-Latency Metrics and Alerting Platform	Explain freshness and overload behavior
Design a Distributed File System	Separate replication from capacity
Design a Power-Plant System from an Underspecified Prompt	Turn ambiguity into explicit assumptions

Start with the telemetry-platform question, write an assumption ledger, and ask a partner to change one requirement. The preparation target is a design that can be revised with clear reasoning, not a memorized diagram that only works under its original numbers.

Sources and Further Reading

• Grafana Labs: Backend engineering hiring process, August 2023 — dated official NALSD description.
• Grafana Labs: Current hiring guide — official hiring principles.
• Google SRE Workbook: Non-Abstract Large System Design — primary methodology reference.
• Reddit: Request for Grafana interview information — evidence limitation, not a completed-round report.