Articles
Envoy Interview Questions: Listeners, Routes, Clusters, and Upstream Failures
Trace Envoy requests through listeners, routes, clusters, and endpoints, then diagnose upstream failures with logs, timeouts, and retry safety.
An Envoy interview answer becomes useful when it can locate a failed request on the actual path: downstream connection, listener, HTTP connection manager, virtual host and route, cluster, endpoint, upstream response. Saying “the proxy returned 503” is only the beginning. This guide offers five original failure prompts and a small routing map. They are practice scenarios, not reports of a particular employer's questions. Route HTTP requests to pods in a cluster is a related exercise for separating endpoint selection from the route rule that picked the upstream service.
Draw the request path before naming a fix
In Envoy's terminology, a listener is the network location downstream clients connect to. A cluster groups logically similar upstream hosts; discovery and load balancing select members of that cluster. The life-of-a-request guide follows a request through the proxy's filter and routing path. The terms matter because a listener accepting TCP does not prove an HTTP route exists, and a route naming a cluster does not prove the cluster has a healthy endpoint.
Use this original practice configuration: listener public-8080 accepts HTTP; host api.example.test has /v1/orders routed to cluster orders-v1; /v1/payments routes to payments-v1. The orders cluster has endpoints O1 and O2; payments has P1 and P2. P1 becomes unreachable. The client sends GET /v1/payments/42 with Host: api.example.test and receives 503. All names are invented. Your task is to decide which observation would identify the failing layer.
| Symptom | First boundary to inspect | Why the status alone is insufficient |
|---|---|---|
| Connection refused before HTTP | Listener bind, address, port, TLS/filter chain | No route evaluation may have occurred |
| HTTP 404 or route miss | Authority, virtual host, path/method match | Upstream endpoints may be healthy but irrelevant |
| 503 after route selection | Cluster existence, endpoint health, connection | Different upstream failures share an HTTP code |
| 504/timeout detail | Route and upstream timeout path | A slow upstream differs from no route or no endpoint |
The exact status and detail depend on configuration and the failure. Use logs and counters rather than treating this table as a deterministic code lookup.
Question 1: What is the difference between a listener, route, and cluster?
Answer in the order traffic sees them. A listener binds a downstream address and accepts a connection. For HTTP, a configured connection manager decodes requests and applies HTTP filters and a route configuration. A virtual host and route match request attributes, such as authority and path, then choose an action. A cluster represents the upstream service chosen by that route; endpoint discovery and load balancing determine where Envoy sends the request. An endpoint is one target, not the whole cluster. If the interviewer asks “where do I set the port?”, clarify whether they mean downstream listening port or upstream endpoint port.
Suppose /v1/payments accidentally names payment-v1 but the cluster is payments-v1. The listener can be healthy, the route can match, and yet forwarding fails because the route's selected cluster is absent. Adding more healthy P endpoints does nothing until the route reference is fixed. Conversely, if the host or path fails to match, changing the cluster configuration may have no effect. This is why a candidate should say which stage has been reached rather than immediately suggesting retries.
Production configurations may use static resources or dynamic discovery. The diagnostic questions remain: what is the active listener config, what route config did this worker use, what cluster was selected, and which endpoints were available? A file on disk is not proof of the active configuration after a failed rollout; inspect Envoy's active state through authorized admin/config-dump paths and deployment logs. Keep secrets out of shared artifacts when using those diagnostics.
Question 2: Which evidence distinguishes routing from upstream failure?
The Envoy response-code details documentation identifies details such as route-not-found, cluster-not-found, no-healthy-upstream, upstream reset, and response timeout. The list is not an eternal API contract; use the deployed version's detail strings. Configure access logs to include response code details, response flags, chosen upstream cluster/host, and duration where appropriate. The access-log documentation gives the available fields. A request ID then connects the client report to one proxy observation.
For the practice request, first confirm the Host and path matched the intended route. If no route matched, P1's reachability is irrelevant. If payments-v1 was selected, inspect endpoint membership and health. If P2 is healthy, ask whether the load balancer tried P1, whether connection failure was retried, and whether the request was safe to retry. If the log shows an upstream response of 503, the upstream service may have generated it; that differs from an Envoy local reply. The same surface status can have different owners.
Say what you would capture before changing the system: one redacted request line, active route, cluster and endpoint snapshot, access-log detail/flags, upstream connection errors, and a small time window. These observations make a hypothesis falsifiable. “Restart Envoy and see” is a poor first answer because it destroys state and may mask configuration drift.
Question 3: What should retries do when one endpoint fails?
Retries are a policy decision, not a default correctness fix. If P1 fails before any request body reaches it, trying P2 may recover a safe read. If a payment POST was sent and the response was lost, replaying it can duplicate the business action unless the upstream enforces an idempotency key or another deduplication contract. Ask whether the operation is idempotent, whether the client already retries, and what timeout budget remains. Otherwise proxy and client retries can multiply load during an outage.
Also separate per-try timeout from overall request timeout. A policy allowing three long tries under a short overall deadline may produce no useful additional chance; a too-generous aggregate budget harms latency. A route to an empty cluster cannot be fixed by retrying the same absence indefinitely. A route to a cluster with one bad and one good endpoint might benefit from outlier detection or health checks, but those policies also need evidence and careful thresholds. Avoid claiming that Envoy always retries a 503 or always switches to another endpoint: actual behavior is configured.
In the practice scenario, propose a bounded experiment with an idempotent GET. Record the endpoint selected on each attempt and the final response detail. Then explain how the design would differ for POST /v1/payments: upstream idempotency, limited retries, and a way to check transaction state after an ambiguous timeout. This shows that proxy routing and application semantics must be reviewed together.
Question 4: Why can an endpoint be listed but never receive traffic?
An endpoint can exist in cluster membership while health checking or priority/load-balancing policy makes it ineligible for a particular request. The cluster might also be warming, subject to circuit breakers, or selected only by another route. Conversely, traffic might reach an endpoint whose application is broken because the health check tests a weaker path. Verify cluster membership and health independently from route selection. Then inspect selected host and response logs for the failing request. “It is in the config” proves less than “it was selected as upstream host for this request.”
Walk two contrasts. P1 is registered but unhealthy, so Envoy may choose P2. That is expected if health policy excludes P1. If both P1 and P2 are absent or unhealthy, a no-healthy-upstream local response can appear. If P1 is marked healthy but its port is wrong, connection attempts may fail even though the health signal looks green; examine how the check is configured and what address Envoy actually uses. The exact stats differ by version and configuration, so describe the observation you need rather than inventing one universal counter name.
Now change the host header to an unmatched domain. Endpoint health has not changed, but the request may fail at the virtual-host/route layer. This tiny test isolates configuration selection from upstream availability. It is more useful than changing three settings at once.
Question 5: How would you roll out a route change safely?
Validate that the new route points to a real cluster, that the cluster resolves endpoints, and that its timeout/retry policy suits the method. Use a small traffic slice or staging request with the exact authority and path. Compare response-code details, chosen cluster, latency, and upstream error rate before broadening the rollout. Keep the previous config ready for rollback. If dynamic configuration is used, confirm Envoy accepted the version and is using it; “control plane published it” is not the same as “data plane applied it.”
The rollback trigger should be specific: a rise in route-not-found, cluster-not-found, no-healthy-upstream, or upstream timeouts associated with the changed route. Do not roll back a route because an unrelated upstream business error increased. A candidate can show judgment by saying which metric locates the fault and who owns the fix. For a failed payment request, the runbook must also preserve enough request identity to reconcile possible side effects; proxy telemetry alone cannot prove whether a downstream charge committed.
The two-minute interview answer
State the active request path aloud: downstream → listener → HTTP route → cluster → endpoint → upstream response. Classify the failure using response details and selected host, then test one layer at a time. Treat retries and timeouts as business-aware policies. Close with a small canary and a rollback signal tied to the route's actual failure mode. That is a more credible proxy-debugging answer than memorizing a list of 5xx codes.
Run a two-request diagnostic drill
Send two requests with the same path and different authority headers. The first uses api.example.test; the second uses an unmatched host. If only the second fails, suspect virtual-host matching before touching upstream health. Then send two requests with the same matching authority but different paths: one to /v1/orders, one to /v1/payments. If orders succeeds while payments fails, the common listener and connection manager are less likely to be the cause. Compare the selected cluster and endpoint evidence for the failed path. These comparisons isolate the stage that differs without changing configuration during the investigation.
Now imagine the payment upstream returns its own 503 with a transaction ID. A proxy-generated 503 and an upstream-generated 503 can look identical to the client, yet the operational response differs. Preserve response-code detail, chosen host, and upstream request ID before retrying. For an idempotent GET, a bounded retry may be reasonable. For a payment POST, check transaction state or use the agreed idempotency key before repeating it. The drill tests proxy knowledge and care with business effects.