Articles
HashiCorp Consul Interview Questions: Service Discovery, Health Checks, and Stale Reads
Practice Consul service discovery by separating catalog entries, passing health checks, blocking queries, stale reads, and downstream consumer caches.
A HashiCorp Consul interview answer is strong when it follows one service instance all the way from registration to a client request. “It's in the catalog” is not the same as “a healthy client should route to it,” and “the query returned quickly” is not the same as “the answer is current.” This guide uses five original prompts around that chain. They are invented practice scenarios, not candidate reports or claims about an employer's interview. Start with Route HTTP requests to pods in a cluster if you want to compare discovery decisions with request routing.
Follow one instance through the discovery chain
Original practice scenario: payments has three registered instances, A, B, and C. A and B pass their service checks. C is still registered but its check becomes critical. A consumer resolves payments.service.consul through DNS, while another calls Consul's HTTP health endpoint with a passing filter. At the same moment, one server is lagging and a client has enabled stale reads. The precise addresses and times are invented; the question is which layer explains a bad endpoint.
The Consul health API distinguishes health queries from catalog information and documents filtering options. Consul's service-discovery overview explains how registration and health contribute to discovery. A registration tells Consul an instance exists. A passing-health query asks for a narrower set. A client may further cache or load-balance that set, adding another place where an old address can persist. Draw these as separate boxes before arguing about a stale read.
| Layer inspected | Question to ask | Common misleading observation |
|---|---|---|
| Agent registration | Is C still in the catalog, and with which address? | Presence is mistaken for readiness |
| Health check | Is C passing, warning, or critical? | A local service test is mistaken for the registered check |
| Discovery query | Was /health/service filtered to passing entries? | A catalog endpoint is treated like a health endpoint |
| Consumer cache | When did it refresh, and with which consistency mode? | One fresh API response is assumed to update every client |
Question 1: Why is an unhealthy instance still visible?
If the interviewer says “C failed but I still see C,” first ask where it is visible. A catalog lookup can continue to show a registered service while its check is critical. A health lookup with passing behavior is intended to filter to passing instances under the selected query semantics. Consul does not necessarily deregister an instance the instant a health check fails; registration and health are different states. An answer that recommends deleting C immediately may destroy useful diagnostic information and sidestep the consumer's actual query bug.
Check the service ID, node, address, port, and attached checks. It is possible to register a correct service name with the wrong port, or attach a check that measures the agent rather than the application. A service may also have more than one check, so “the endpoint returned 200 once” does not prove the Consul service is passing. The health API returns check information; inspect it before guessing. If ACL filtering affects results, record the token and response headers available to the client rather than interpreting an empty array as zero registered services.
For the practice scenario, the expected healthy set is A and B if the query requests passing results and has a sufficiently current view. C remaining in the catalog is not a contradiction. If the interviewer instead asks how to remove dead registrations, discuss deregistration policy separately from health-based routing. Making the check accurate and making the consumer use it solve the immediate bad-endpoint risk.
Question 2: What do default, consistent, and stale reads mean?
The Consul consistency-mode documentation describes trade-offs among request modes. A stale request can be served by a follower and may return an older view; stronger consistency requires more coordination and can increase latency or reduce availability under leader trouble. The correct interview answer names the endpoint and mode. Do not say “Consul is eventually consistent” as though all reads have identical behavior. Also do not say a consistent flag can make an unhealthy application healthy; it only addresses the freshness of Consul's view.
DNS adds another nuance: the documentation says Consul DNS uses stale mode by default for underlying discovery APIs unless its DNS configuration changes that behavior. DNS responses may also be cached by the consumer or resolver for their TTL. Therefore, when A is removed and a user still reaches it, inspect Consul's query result and the downstream DNS/cache path separately. If the HTTP passing-only response is already correct but a sidecar still routes to A, the stale endpoint may be in the client cache or connection pool rather than the Consul server read. A hypothesis should name the observation that would confirm it.
Choose mode by consequence. A UI showing a slightly old list of noncritical services may value low latency; a leader-election decision or sensitive operational gate may need the documented stronger mode and explicit failure handling. You cannot infer the right mode from the word “service discovery” alone. Ask for the maximum tolerated staleness and what the caller does when Consul is unreachable. If the answer is “route to anything,” health filtering may not matter in the failure path.
Question 3: How do blocking queries avoid polling races?
The blocking-query API allows many endpoints to wait for a potential change, using an index supplied from a prior response. A watch loop records the returned index, sends the next query with that index and a wait parameter, then replaces its local view from the new response. This can reduce tight polling. It does not turn the API into an immutable event log or guarantee that each intermediate registration and check transition is delivered individually. A client should reconcile current results, handle timeouts, and guard against busy loops or a regressed/invalid index according to documented behavior.
In the practice scenario, C transitions from passing to critical while the consumer waits. On response, the consumer should rebuild its eligible endpoint set, drain or stop choosing C according to application policy, and issue another blocking query. If the consumer process restarts, it should fetch a fresh baseline rather than trusting a cursor saved without its associated endpoint snapshot. If Consul returns the same index after a wait, do not assume nothing in the wider system changed: the client may still have a stale connection, separate cache, or unobserved application failure.
Distinguish the notification mechanism from the routing action. A blocking query tells the consumer that the answer may need updating; the consumer must publish that update safely to request threads. A concurrent request may already have selected C. How to handle in-flight connections, retries, and idempotency belongs to the application or proxy design. Consul does not retroactively cancel a request simply because a health response changed.
Question 4: How would you debug “a stale endpoint” without guessing?
Begin with one failing request and preserve its endpoint address, timestamp, resolver output, query mode, and health state. Compare four observations taken close together: the agent's registered service, Consul's catalog entry, a /health/service/payments?passing style response under the caller's ACL and consistency mode, and the consumer's actual endpoint choice. Differences localize the fault. If catalog has C and health excludes it, the consumer may be querying the wrong endpoint or holding a cache. If health includes C as passing but the application is broken, inspect the check definition and status propagation. If one server's stale response differs from a stronger read, the consistency choice matters.
Avoid a dangerous shortcut: purging the entire catalog because one client sees a stale IP. That can hide a misconfigured check and create wider disruption. In an interview, propose a reversible test: temporarily query with explicit passing filter and stronger consistency, then compare the returned addresses while leaving production registration untouched. The result is diagnostic, not necessarily the permanent policy. Record DNS TTL and client-side caching if DNS is involved. Say what data would falsify each hypothesis.
The failure can be layered. C can be both critical in Consul and still present in a local proxy's established connection pool. In that case a fresh discovery response is necessary but not sufficient for immediate traffic removal. Ask whether the proxy drains connections or only updates endpoints for new requests. This follows from how the consumer uses the result, not from an undocumented promise by Consul.
Question 5: How would you design safe behavior during Consul trouble?
There is no universal fail-open or fail-closed answer. For internal read traffic, a bounded last-known-good endpoint set may preserve availability; for a sensitive administrative service, routing to an unverified endpoint may be unacceptable. Define the maximum age of cached data, whether active local health checks exist, and whether retries can duplicate writes. Log the last successful discovery revision/index and age so operators can tell “no service exists” from “discovery is unavailable.” Do not hide a Consul outage behind an empty service list.
Exercise three cases: C turns critical while Consul is healthy; a follower is stale while the leader has the new status; and the consumer cannot reach Consul at all. For each, write the endpoint set, consistency choice, and fallback. That turns a generic service-discovery answer into a reviewable operating contract. The product constraint—not an abstract preference for strong consistency—decides the trade-off.
A crisp interview response
Trace registration, check status, query endpoint, consistency mode, cache, and final request selection in that order. Use the exact consumer path to explain why an unhealthy service might still be routed. Make one controlled comparison between passing-filtered fresh and stale results, and state what it would prove. Close with the caller's failure policy and the maximum acceptable cache age. This gives the interviewer both a correctness model and a debugging plan.
The follow-up that catches shallow answers
Suppose your passing-only HTTP query excludes C, but the next user request still reaches C. Both observations can be true. The consumer may cache the prior answer, a DNS resolver may hold an address until TTL expiry, or an already-open connection may keep sending to C. Ask which component made the endpoint choice for that one request and when it last refreshed its view. Change one layer at a time: compare a direct Consul health query with the consumer's resolved list, then inspect the selected upstream address in request logs. If the direct query includes C as passing, investigate the check and read freshness instead.
Now suppose stronger consistency makes the bad endpoint disappear but raises discovery latency during leader trouble. “Always use consistent” is incomplete. Decide how much staleness the caller tolerates, whether local health checks can reject a stale target, and what fallback does when Consul is unavailable. State the risk in user-visible terms: wrong destination, failed request, or temporary unavailability. That is a defensible trade-off rather than a blanket preference for one flag.