etcd Interview Questions: Leases, Watches, Revisions, and Recovery

An etcd interview often becomes a failure-recovery interview as soon as the happy path is clear. A candidate who knows put, get, and watch still needs to say what happens when a lease expires, a watcher falls behind compaction, or a process restarts with a stale revision. This guide uses five original prompts and a revision timeline to practice those boundaries. It does not claim that any employer uses these exact questions. For adjacent distributed-systems practice, Design leader election using Redis leases is useful precisely because you must explain which Redis assumptions do not automatically transfer to etcd.

First separate a key, a lease, and a revision

The etcd v3.6 API reference describes a revisioned key-value store. A transaction can change multiple keys under a single store revision, while a key also has metadata such as creation and modification revisions. A lease is a TTL-based liveness mechanism; a key can be attached to one lease, and expiry or revocation deletes attached keys. A watch streams events from a chosen revision or from its creation point. These are related mechanisms, not interchangeable guarantees. A lease does not make an external payment idempotent, and a watch event is not a substitute for reading current state after history is lost.

Use this original practice timeline throughout the article. At revision 40, /workers/a exists under lease L. At 41, another worker updates /jobs/7. A watcher has applied through 41. The client disconnects while writes advance through 60; meanwhile L expires and /workers/a is deleted at 52. The cluster compacts history through 55 before the client resumes. The exact numbers are invented. The important invariant is that the watcher cannot request an event stream starting at a revision that has already been compacted.

Time	Client belief	Cluster truth	Safe response
Applied through 41	Worker A present	Same at that moment	Persist the handled cursor with application state
Lease expires at 52	A may still appear in local cache	A's key is deleted	Do not equate cached membership with liveness
Resume after compaction through 55	Replay from 42 requested	History too old	Rebuild snapshot, then watch from the snapshot boundary

Question 1: What does a lease actually protect?

“Worker A owns a job because its lease exists” needs qualification. etcd can delete the leased key if keepalives stop long enough for expiry; the API reference says the granted TTL is server-selected and each attached key is deleted on expiry or revocation. That helps other coordinators observe lost liveness. It does not force a paused worker to stop making external writes. A process can miss keepalives during a pause, lose its lease, then resume and call an outside database unless the application prevents it. The candidate should distinguish coordination state from the effect being coordinated.

For exclusive work, propose a compare-and-swap transaction on an ownership key, attach it to a lease, and include an ownership epoch or fencing value that the downstream system checks before accepting writes. Do not assert that an etcd revision alone magically fences a payment service. A monotonic token only works as fencing when the protected resource verifies it. If the external service has no such check, use a different idempotency or reconciliation design. Ask the interviewer what “at most once” means: never start two workers, never commit two external effects, or never display two completions? Those are different requirements and failure modes.

When the owner renews successfully, it can continue under its local policy; when renewal is uncertain, it should not assume continued ownership. A delayed keepalive response, partition, or process pause needs an explicit stop-and-recheck path. The recovery step is to read current ownership and validate the token before further side effects. This is the point where an answer moves beyond reciting lease methods into designing a safe worker.

Question 2: What revision should a watcher resume from?

The Watch API lets a client specify an inclusive start_revision. If the last fully applied revision is r, a straightforward event replay begins at r + 1. That “fully applied” condition matters: if a handler acknowledges an event before committing its local projection, saving the cursor first can skip work after a crash. If the projection and cursor cannot be committed atomically, make replay idempotent and persist the cursor only after the effect. An event may be delivered again after uncertain failure, so the handler should tolerate duplicates.

On the practice timeline, asking for 42 after history has been compacted through 55 fails. The watch response can identify compaction and cancel the watch; retrying the same start_revision in a loop cannot recreate the missing events. Read a fresh consistent snapshot of the needed keys, obtain its response revision, rebuild the local projection, and start watching from the next revision so intervening changes can be seen. The exact snapshot/watch handoff must follow the client API's revision rules. A production implementation also needs to handle a compaction race during the rebuild by retrying the snapshot workflow, not by guessing at missing events.

Note the distinction between a history gap and a current-state gap. If your application only needs the current membership set, rebuilding state may fully recover. If it must execute every historical transition as a business event, compaction means those transitions are unavailable from etcd; the architecture needs a durable event log or another source. etcd's watch is a coordination stream, not an infinite audit archive.

Question 3: How do you read a consistent recovery snapshot?

The Range API distinguishes default linearizable reads from serializable member-local reads that can be stale. A fresh recovery baseline for correctness-sensitive coordination usually calls for a linearizable range read, records the response revision, and uses that revision to bridge to the watch. That is stronger than simply contacting any member and assuming its local view is current. etcd's API guarantees explain the ordering and consistency model; candidates should identify the actual read mode rather than using “consistent” as a vague adjective.

Suppose the projection has /workers/a locally but the lease expiry deleted it at 52. The snapshot at 60 removes A from the projection even though the deletion event itself is no longer replayable. Then the watcher resumes after the snapshot revision. If a write lands between the range response and watch creation, starting at the captured revision boundary preserves it, assuming history remains available. If the watch is canceled again for compaction, rebuild again. Do not replace the snapshot with a progress notification: watch progress can help clients track a recent revision, but it does not restore compacted historical events or validate every cached key.

Question 4: Which revision belongs in a transaction compare?

An interviewer may ask you to update /jobs/7 only if nobody changed it since your read. Compare its modification revision with the revision you observed, then write in the success branch of an etcd transaction. Comparing the overall header revision instead could reject a valid update because an unrelated key changed; comparing only the value may miss an ABA sequence in which the value changed and returned to the same bytes. The right compare target depends on the invariant. Creation revision can distinguish a new incarnation of a key from a later update to the same key; version may be useful when the rule concerns a key's number of modifications.

Walk a race by hand: clients A and B read /jobs/7 at modification revision 41. A's transaction compares 41 and writes, moving the key to a new modification revision. B's same compare fails. B should reread and decide whether its intended operation is still valid. This is not a license to blindly retry a non-idempotent external action. A response lost after a successful transaction also creates uncertainty; check current state or use an idempotency marker before repeating effects. The API reference's transaction and key-value fields support the compare; the external-effect policy is an application design choice.

Question 5: What is different about disaster recovery?

A normal disconnected watcher can rebuild from the live cluster. Restoring a cluster from an older snapshot is a different event: cluster membership, history, and revision expectations may change. The etcd recovery guide documents snapshot restore mechanics and revision-bump considerations for Kubernetes-style watch consumers. Do not tell a client to resume at its old cursor after the store has been restored to a point before that cursor. Treat the restored cluster as a new baseline, invalidate dependent caches as required, and test how each watch consumer reinitializes.

In an interview, clarify the recovery objective before choosing a command. Is the cluster healthy but the client missed events? Did a member fail while quorum remains? Was the whole cluster restored from backup? These are different runbooks. Back up before destructive operations, follow the documented membership/restore procedure for the deployed version, and rehearse with a nonproduction client that watches a key through restoration. The maintenance documentation also separates compaction, defragmentation, and snapshotting; none of them means “delete all old coordination bugs.”

What a strong answer sounds like

State which component owns the truth: etcd's current keys, your local projection, or an external system receiving effects. Name the revision you have durably applied, show the snapshot-to-watch boundary, and explain when replay is safe. Then test lease loss during a pause, a duplicate event after a crash, and a compaction error during reconnect. If the required guarantee spans systems, say where the fencing or idempotency check runs. This answer demonstrates recovery design without claiming that one etcd API call provides end-to-end exactly-once behavior.

Try one crash matrix before declaring recovery complete

Put the watcher at four possible crash points. If it crashes before applying revision 41, it should replay 41. If it applies 41 but crashes before persisting its cursor, it may replay 41, so the handler must tolerate that repeat. If it persists 41 before applying the local projection and crashes, it may skip the effect on restart; that ordering is unsafe unless both updates were one atomic transaction. If it requests 42 after compaction has removed it, no retry loop can provide the missing history. Reconstruct current state and bridge to a new watch. Write down the local projection and durable cursor at each point.

Now add one external charge to the event handler. Replaying an etcd event does not prove the charge failed or succeeded. A stable idempotency key, downstream fencing check, or reconciliation query must close that uncertainty. The phrase “exactly once watch processing” deserves immediate scrutiny: the watch coordinates ordered changes, but end-to-end effects depend on how the consumer commits them. Naming the uncertain window is more convincing than promising a retry-free world.

Sources and further reading

• etcd v3.6 API reference
• etcd API guarantees
• etcd maintenance and compaction
• etcd recovery guide