Articles
LLM Gateway System Design Interview: Routing, Rate Limits, Fallbacks, and Cost Controls
Practice LLM gateway system design interview on architecture, debugging, evaluation, safety, and trade-offs with a focused 2026 guide and seven-day plan.
LLM Gateway System Design Interview: Routing, Rate Limits, Fallbacks, and Cost Controls
Quick answer: Design an LLM gateway as a policy and reliability layer, not a thin proxy. Cover authentication, tenant budgets, model routing, prompt and response handling, rate limits, retries, fallbacks, streaming, privacy, observability, and cost attribution. Be explicit about what the gateway must never log or transform.
What interviewers actually test
An LLM gateway centralizes access to one or more model endpoints. It can enforce quotas, select providers, normalize APIs, collect telemetry, and degrade gracefully, but it also becomes a high-value security and reliability boundary for prompts, outputs, credentials, and customer data.
The differentiating skill is the control plane and request path between applications and multiple model providers. An interview-ready answer turns that idea into a checkable contract: state the user or system goal, identify the component that owns each decision, name one invariant, and show the metric or test that would expose a failure.
A four-part answer framework
Use this structure for LLM gateway system design interview without forcing every question into the same architecture. Start with the smallest design that meets the requirement, then add complexity only when a stated constraint requires it.
| Part | What to explain | Evidence that makes it credible |
|---|---|---|
| Requirement | User, task, inputs, outputs, and success condition | One explicit assumption and one excluded scope |
| Mechanism | Routing, quotas, retries, streaming, privacy, and unit economics | A data flow, state transition, API contract, or invariant |
| Evidence | How you test or compare the design | Baseline, metric, failure slice, or trace |
| Recovery | What happens when the design is wrong or a dependency fails | Safe fallback, user-visible status, and next action |
LLM Gateway System Design Interview: questions and strong-answer signals
The LLM gateway system design interview questions below each target a decision that can be tested. Avoid listing components without explaining ownership, failure behavior, and evidence.
1. What belongs in an LLM gateway and what should stay outside it?
Centralize authentication, quotas, provider normalization, routing policy, telemetry, and common safety controls. Keep product-specific prompts, business workflows, and domain authorization in the owning service so the gateway does not become an untestable monolith.
What interviewers listen for: Draw a clear control-plane and data-plane boundary. Name one trade-off or condition that would change your recommendation.
2. How would you route requests across models and providers?
Define eligibility first using capability, region, privacy, context size, and tenant policy. Rank eligible routes by measured quality, latency, availability, and cost, then record the decision so it can be explained and evaluated.
What interviewers listen for: Include a stable default and a way to disable a bad route quickly. Name one trade-off or condition that would change your recommendation.
3. How do rate limits and tenant budgets interact?
Enforce per-user and per-tenant quotas before provider calls, then account for actual usage and retries. Use token buckets or similar controls for bursts, reserve capacity for critical traffic, and return actionable retry information without exposing other tenants.
What interviewers listen for: Prevent one tenant or runaway workflow from exhausting shared capacity. Name one trade-off or condition that would change your recommendation.
4. Which failures should trigger retries or fallbacks?
Retry transient, idempotent failures with bounded backoff and a deadline. Route to a fallback only when the alternative satisfies the request's capability and policy constraints; do not silently trade away privacy, quality, or output compatibility.
What interviewers listen for: Classify failures and explain how retry budgets avoid request amplification. Name one trade-off or condition that would change your recommendation.
5. How does streaming change the gateway design?
Track partial delivery, cancellation, client disconnects, provider usage, and final completion separately. Propagate backpressure and define whether a fallback may start after bytes have already reached the client.
What interviewers listen for: Make partial responses and billing status unambiguous. Name one trade-off or condition that would change your recommendation.
6. Can the gateway cache prompts or responses?
Cache only when semantics, freshness, authorization, and privacy allow it. Include model and prompt versions plus relevant context in the key, isolate tenants, and avoid storing sensitive payloads merely to save tokens.
What interviewers listen for: Explain invalidation and why semantic caching can return a plausible but wrong answer. Name one trade-off or condition that would change your recommendation.
7. What telemetry should the gateway collect?
Capture route, model and prompt versions, latency stages, status, token usage, cost, retries, and privacy-safe quality signals. Use correlation IDs across downstream tools while applying redaction, access control, sampling, and retention.
What interviewers listen for: Connect telemetry to routing changes and incident diagnosis. Name one trade-off or condition that would change your recommendation.
8. How would the gateway behave during a provider outage?
Trip route-specific circuit breakers, protect healthy providers from a retry storm, preserve tenant priority, and degrade explicitly when no compliant route remains. Recovery should be gradual and measured rather than an immediate return of all traffic.
What interviewers listen for: Include capacity assumptions, load shedding, and a truthful user-facing response. Name one trade-off or condition that would change your recommendation.
A worked interview drill
Use this question as a timed drill: How do rate limits and tenant budgets interact?
First pass, two minutes: for LLM gateway system design interview, state the requirement, your recommendation, and the main reason. Do not start by naming products. The interviewer should understand what the system or candidate must accomplish and which constraint drives the choice.
Second pass, five minutes: draw the data or control flow behind routing, quotas, retries, streaming, privacy, and unit economics. Label the component that owns validation, authorization, state, and recovery. Add one invariant that must remain true even if a model, tool, or dependency returns a plausible but incorrect result.
Interviewer challenge: in the LLM gateway system design interview scenario, assume the first dependency times out, the cost doubles, or the input violates an important assumption. Revise the design without discarding the original goal. Explain whether you retry, degrade, ask for clarification, require approval, or stop.
Evidence check: Prevent one tenant or runaway workflow from exhausting shared capacity. Add one metric or test that could prove your first recommendation wrong, then name the condition that would make you choose a simpler design. This final step turns a memorized answer into engineering judgment and gives the interviewer a concrete path for follow-up questions.
How to prepare for this topic
Choose two questions from the list and answer each at three depths: a 30-second summary, a two-minute design, and a ten-minute whiteboard discussion. For each answer, draw the boundary implied by routing, quotas, retries, streaming, privacy, and unit economics, then add one failure case and one measurement that could disprove your first idea.
Bring one project or incident that demonstrates the control plane and request path between applications and multiple model providers. Write down the original requirement, your exact contribution, the trade-off you made, the evidence you collected, and what changed afterward. For company-specific interviews, keep official information, candidate reports, and your own inference clearly separated; the current invitation remains the source of truth.
Explain the invariant before coding, test an edge case before polishing, and state the operational or complexity cost of the final choice.
A seven-day preparation plan
| Day | Focus | What to do |
|---|---|---|
| Day 1 | Map the search intent | Write a one-paragraph answer to LLM gateway system design interview, including the audience, scope, and one assumption. |
| Day 2 | Master the core system | Draw routing, quotas, retries, streaming, privacy, and unit economics and label ownership, data flow, and one invariant. |
| Day 3 | Answer questions 1-4 | Give each answer in two minutes, then add a failure case and a trade-off. |
| Day 4 | Answer questions 5-8 | Add a metric, test, or trace that could prove each recommendation wrong. |
| Day 5 | Build a project story | Rehearse one example with your contribution, evidence, limitation, and what changed after measurement. |
| Day 6 | Run a timed simulation | Complete one backend engineer practice set and review assumptions, tests, permissions, and skipped edge cases. |
| Day 7 | Final review | Confirm logistics and permitted tools, then review the five weakest answers without adding a new framework. |
Frequently asked questions
Should an LLM gateway retry every failed request?
No. Retry only transient failures and only when the operation is safe. Respect provider guidance, use backoff and budgets, and avoid multiplying a request that may already have produced a side effect through a tool.
How do you route between models?
Define a policy based on capability, latency, cost, context, privacy, and tenant requirements. Keep a fallback path and measure quality so routing does not optimize one metric blindly.
What should a gateway log?
Log structured metadata and trace identifiers needed for operations while applying retention and redaction rules. Do not assume prompts and outputs are harmless telemetry.
How do streaming responses change the design?
Track partial delivery, cancellation, client disconnects, provider timeouts, and usage accounting. Define when a stream is considered complete and how a partial response is labeled.
What is a good gateway cost metric?
Cost per successful task by tenant, model, route, and retry class is more useful than raw token volume alone. Pair it with quality and latency slices.
What a strong answer sounds like
A strong LLM gateway system design interview answer is specific enough to challenge. The interviewer should be able to point to the requirement, the owner of each decision, the evidence supporting the choice, and the condition that triggers a fallback or redesign. If any of those pieces are missing, the answer is probably still at the buzzword level.
Final takeaway
Prepare for LLM gateway system design interview by making every answer inspectable: define the requirement, show the mechanism, provide evidence, and design the recovery path.
Sources and Further Reading
- OpenAI API rate limits
- OpenAI API best practices
- OpenTelemetry GenAI semantic conventions
- NIST AI Risk Management Framework
Research note: This draft was checked on August 31, 2026. Employer processes, platform features, documentation, and model behavior can change; the reader's current instructions remain the source of truth.