Articles
Apache ZooKeeper Interview Questions: Sessions, Ephemeral Nodes, and Watch Semantics
Practice ZooKeeper session expiry, ephemeral-node ownership, classic and persistent watches, and leader recovery with a concrete incident timeline.
The most revealing Apache ZooKeeper interview question is often: “The client disconnected. Did its ephemeral node disappear?” A careful candidate does not answer yes. A disconnected client may reconnect within its session timeout; the ephemeral node is tied to the session, not to one TCP connection. This guide turns that distinction, standard watch behavior, and recovery into five original practice prompts. The scenarios are educational examples, not a report of an employer's question set. Design a configuration subscription service is a useful adjacent drill because it forces you to define what a subscriber does after missed notifications.
A compact session timeline to reason from
Imagine three ZooKeeper servers and client C. C creates ephemeral /members/c under session S and sets a standard child watch on /members. Its network connection breaks at t1. At t2, C reconnects to a different server using S before the session timeout. Later, at t3, C loses connectivity long enough for S to expire. Finally, C creates a new client with a new session S2. These times are invented for interview practice. The ZooKeeper Programmer's Guide supplies the underlying session, ephemeral-znode, and watch rules.
| Observed event | What it proves | What it does not prove |
|---|---|---|
| Client receives disconnected state | Current connection is down | Session has expired or ephemeral node was deleted |
| Reconnect under S succeeds | Session survived | Every application watch handler has caught up |
| Expired state for S | Old session is no longer valid | A new client may safely continue old ownership without re-registration |
The table is a decision aid, not an SLA. Timing depends on negotiated session timeout and ensemble behavior. If the interviewer supplies only a network exception, ask for session state before concluding what happened to ephemeral znodes.
Question 1: When does an ephemeral node go away?
ZooKeeper deletes ephemerals owned by a session when that session expires or is closed. A transient connection loss by itself is insufficient. Therefore, an observer may still see /members/c while C is disconnected; the system is allowing time for the same session to reconnect. Once the server decides S has expired, /members/c is removed and interested watchers can learn of the change. Client C must treat an expired session as terminal and create a new client/session if it wants to rejoin. It cannot “renew” the old session after expiration or assume ownership survived through S2.
The interviewer may press: “Can C safely keep processing during disconnect?” The coordination system cannot stop C from doing work against an external database. If correctness requires exclusive leadership, C should stop or fence side effects when it cannot establish current ownership. An ephemeral node is useful for membership and leader recipes, but it is not by itself a complete cross-system fencing protocol. This is a design inference from the session rule; it is not a claim that ZooKeeper automatically validates an external write. Clarify the external system's ability to reject stale leaders before promising exclusive effects.
Test with a paused process, not just a clean disconnect. C might remain alive while its session expires because it misses heartbeats. When it resumes, its in-memory “I am leader” flag is stale. A robust application transitions to a safe state on Expired, reestablishes a new session, recreates required ephemerals, and reacquires leadership under its recipe. It should not simply recreate /members/c and assume that the old work is harmless; a replacement leader may already be active.
Question 2: What exactly does a standard watch guarantee?
Standard data, existence, and child watches are notification mechanisms, not event logs. The Programmer's Guide describes standard watches as one-time triggers. After a watch fires, a client generally reads current data and sets another watch as part of its next operation. If the node changes more than once while the client is between a notification and re-registration, the client should reason about the latest state, not expect a separate callback for every intermediate mutation. The guide explicitly warns that ordinary watches are not a reliable way to see every change to a node.
For /members, the correct subscriber loop is: issue getChildren while registering a child watch, reconcile the returned set with the local view, and repeat after the event or reconnection. The data returned by the read is the state to apply; the watch means “something may have changed, read again.” Handle repeated reads and duplicate notifications idempotently. If the application needs a durable audit of every membership transition, ZooKeeper watches alone are the wrong mechanism. A common weak answer says “the watch streams all changes”; a stronger answer identifies one-time notification and the re-read/re-arm cycle.
There is a version nuance. Modern ZooKeeper also provides persistent and persistent-recursive watch modes, documented in the current AddWatchMode API. Do not say every possible watch is one-shot. If the interviewer asks about classic getData, exists, or getChildren watches, explain their one-time behavior. If they explicitly choose addWatch(PERSISTENT), explain that it stays registered until removal, while still treating a notification as a reason to reconcile state rather than a substitute for application recovery.
Question 3: What happens to watches during disconnect and expiration?
On a temporary disconnect, clients must account for missed changes and re-read after reconnection. ZooKeeper works to maintain watches across reconnects within a live session, but application code should not build correctness around “I saw a callback for every transient state.” The standard recipe is to observe connection state, then fetch the authoritative znode data or children and restore the application's view. Existence watches have edge cases when a watched znode is created and deleted while the client is disconnected. The safest interview answer focuses on state reconciliation rather than asserting perfect event replay.
After expiration, the old session and its ephemerals are gone. A new session must register its own watches and ephemerals. That is not the same operation as reconnecting S to another server. Name the state transition explicitly in your answer. A candidate who conflates “connection loss” with “session expired” will often either delete membership too early or keep stale ownership too long. Both cause production bugs, even if a local happy-path demo appears correct.
Ask the interviewer whether they want a membership cache or an exact sequence of events. For membership, re-reading getChildren after reconnect can restore current truth. For every intermediate transition, use an appropriate durable log or redesign the requirement. A znode stat version helps detect updates to one node; it does not reconstruct all events lost between reads. That distinction prevents an overclaim about watches.
Question 4: How would you design leader election with ephemerals?
A common ZooKeeper pattern uses ephemeral sequential znodes under an election path. Each contender creates one, orders candidates by sequence suffix, and the smallest becomes leader. Instead of every follower watching the current leader and stampeding when it disappears, a follower can watch its immediate predecessor. The ZooKeeper recipes explain election and lock designs. In an interview, do more than recite the recipe: explain what happens if the create operation succeeds but its response is lost, how a client finds its own znode, and how it behaves after session expiration.
Ownership is attached to the session. If a leader loses its session, its ephemeral election node disappears and another candidate can take over. The former leader may not be able to detect that instantaneously if it is paused; therefore, an external side-effect path still needs a policy for stale leaders. Say whether you use a monotonic fencing token checked downstream, make operations idempotent, or accept a weaker guarantee. Do not assume that znode deletion remotely kills the process. A good interviewer will ask exactly where the protection lives.
The original timeline gives you a quick answer: before t3, C may reconnect as S and retain its node; after t3 it must enter as S2, and its new sequential znode may be behind another contender. It must reevaluate leadership. A cached isLeader = true boolean from S cannot be carried into S2.
Question 5: How do you test the watch/session boundary?
Use a deterministic test harness around three state changes. First, create /members/c, disconnect C for less than the negotiated timeout, and verify the node remains while C reconnects under the same session. Second, keep C disconnected past expiration, observe removal, and verify the new session does not inherit the ephemeral. Third, make multiple child changes while a standard watch handler is delayed, and verify the client eventually converges on getChildren's current set without requiring one callback per mutation. Record the expected session state, znode set, and application cache after each step.
Do not turn a timing-sensitive integration test into a universal statement about the millisecond of expiry. Make timeouts explicit, log session IDs and state transitions, and retry observations within a bounded interval. Add a crash-after-create case for election to test uncertainty around a lost response. The answer is strongest when it connects ZooKeeper's documented semantics to an application invariant: one active membership record per live session, convergence after disconnect, and no stale leader effects after replacement.
How to explain this under interview pressure
Say “connection, session, ephemeral node, watch registration” as four separate objects. Then walk t1, t2, and t3 without skipping a state transition. For each watch, say whether it is standard one-time or a modern persistent mode. Use the current znode read to rebuild state, and identify where an external system must enforce ownership. This framework handles the common follow-up questions more reliably than memorizing a one-sentence definition of a znode.
A three-case answer check
Ask yourself three questions without looking back. First, C disconnects briefly, reconnects under the same session, and sees /members/c. Should another client already have declared C dead? Only if the application has an independent failure policy; ZooKeeper's session has not necessarily expired. Second, S expires while C is paused. Should C resume its old work after waking? No: it must regard old ownership as lost and reacquire under a new session, with external side effects protected by the application. Third, a standard child watch fires while three members join. Must C receive three separate callbacks? No: it should re-read children and converge on the current set.
Then change only the watch API to a persistent mode. The one-time re-registration statement changes, but the need to reconcile current application state after disconnect or restart remains. This contrast prevents a memorized “watches are one-shot” answer from becoming a false universal rule. It also shows why an interview answer should state the API version and watch mode before promising delivery behavior.