Configuration Service System Design Interview: Watches, Versions, and Safe Rollouts

A configuration service system design interview tests more than key-value storage. The dangerous interval is between “read the current config” and “subscribe to future changes.” If the client misses revision 42 there, two services can run incompatible settings while every component looks healthy.

etcd officially documents revisioned watches and that compacted history is no longer available. AWS AppConfig documents schema or Lambda validators and staged deployments with growth, bake time, alarms, and rollback. Those facts support the mechanisms here. The dependency model, client protocol, and SLO choices are editorial inference for the prompt.

For a closely related drill, use Design a configuration subscription service.

Configuration Service Interview: Quick Answer

Store each configuration set as an immutable version with schema version, author, reason, validation result, and content hash. Clients read a snapshot at revision R, install it atomically, then watch from R+1. If the server reports compaction or a gap, the client fetches a fresh snapshot rather than guessing.

Validate syntax, types, cross-key constraints, and reference existence before eligibility. Roll a version through test, canary, staged cohorts, and bake periods. Clients retain the last-known-good version and reject incompatible updates. Alarms stop rollout; rollback promotes a prior immutable version as a new deployment event.

The Core Contract

A configuration service is a version-distribution protocol. Correctness comes from immutable versions, a gap-free read-plus-watch handshake, client fallback rules, and a rollout that can stop or reverse.

Client state	Input	Correct action	Unsafe shortcut
no snapshot	read returns revision 80	install v80, watch 81	watch “now”
watching 81	receive 81 then 82	apply in order	keep only latest without compatibility check
watching 81	receive 83	resnapshot	assume 82 was irrelevant
offline	watch closes	serve last-known-good within policy	erase local config
reconnect after compaction	history unavailable	read new snapshot	retry old revision forever

This state machine is the first article-specific asset. It exposes the boundary where many otherwise polished answers lose updates.

Choose the Atomic Configuration Unit

One key is easy to version but related keys can become incompatible. Define a configuration set or transaction that changes endpoints, timeouts, and limits together. Give each set a monotonically ordered revision within its namespace and a globally unique content digest. The service returns value, revision, schema, and activation metadata in one read.

Separate source version from deployment version. The same approved content may be deployed to development, canary, and production at different times. This preserves auditability and lets operators answer which cohort saw which content without copying mutable blobs.

Close the Read-and-Watch Gap

The safe handshake is snapshot at R, then subscribe from R+1. The server must either replay every committed event after R or tell the client that history is unavailable. Heartbeats show liveness but do not prove continuity; the client tracks the last applied revision and rejects gaps or regression.

Coalesce updates only if the contract permits skipping intermediate states. A client may jump from v10 to v13 for an independent scalar, but not when v11 performs a required migration. Encode minimum client version, predecessor, or migration sequence in metadata. Backpressure slow watchers and force resnapshot rather than retaining unbounded history.

Validate More Than JSON Shape

Schema validation catches missing fields and wrong types. Semantic validation checks ranges, endpoint formats, mutually exclusive options, referenced resources, and cross-field constraints. A dry-run can load the configuration into the real parser and execute domain checks. High-risk changes may require a second approver.

Validation cannot predict every production interaction, which is why rollout remains necessary. Record validator versions and results with the candidate. If validation depends on mutable external state, snapshot the referenced version or the same candidate may pass today and fail tomorrow.

Roll Out and Recover Safely

Assign targets to deterministic cohorts so a restart does not reshuffle exposure. Deploy to internal/test targets, then a small canary, larger waves, and full fleet with a bake period. Observe parse/apply failures, version skew, domain error rate, latency, saturation, and fallback usage. Stop conditions must be declared before launch.

Clients apply a new set off the request path, validate locally, then swap an atomic pointer. If application fails, keep last-known-good and report the exact version and reason. During control-plane outage, clients serve cached configuration until its policy expiry. Security revocations may need a shorter fail-closed rule than ordinary tuning parameters.

Work Through a Failure Trace

A client reads revision 100. Update 101 commits before it opens the watch. Because it subscribes from 101, the server replays the event. A slow client later requests from 90 after compaction at 95; it receives a compaction response, reads snapshot 120, validates it, and watches 121. The test asserts no silent gap and preserves last-known-good on validation failure.

Configuration Service check: How does a client avoid missing an update between snapshot read and watch? Walk the trace once in event-time order, once with retries and reordering, and once after rebuilding projections from durable records. State the expected invariant after every step. If the outcome depends on message arrival order when the contract says it should not, the design is missing an identity, version, or transaction boundary. If the design intentionally permits a temporary difference, give it a watermark, maximum age, and repair owner.

Configuration Service check: What happens when watch history is compacted? This trace is an interview model, not a measured production benchmark. Add a skew test concentrated on one tenant or key, a two-times-peak capacity test, and a fault injection at the acknowledgment boundary. Preserve input versions and random seeds so a failed run can be reduced and replayed.

Turn the Requirements into Verification

Write these five questions beside the diagram: How does a client avoid missing an update between snapshot read and watch? What happens when watch history is compacted? How are syntax and semantic dependencies validated? Can related keys roll out atomically? How do clients behave during control-plane failure? Each question must point to a record, transition, test, and observable signal. If one can be answered only with 'the service handles it,' the contract is still incomplete.

Configuration Service check: How are syntax and semantic dependencies validated? Build a deterministic fixture with five to ten timestamped inputs. Include an identical retry, a conflicting reuse of an ID, a late or stale input, a worker failure after durable work but before acknowledgment, and a rebuild of the serving projection. Expected results must be written independently of the implementation. Verify the authoritative state, the serving state through a named watermark, and every emitted correction or rejection reason.

Configuration Service check: Can related keys roll out atomically? Then add property checks. Applied logical IDs remain unique; versions never move backward; reserved or consumed value obeys the topic's invariant; a rebuild through the same input boundary produces the same digest; and an unauthorized reader cannot cross its scope. Generate reordered deliveries around the boundary instead of testing only the happy sequence. A passing fixture proves these declared cases, not production throughput or every network failure.

Configuration Service check: How do clients behave during control-plane failure? For fault testing, stop a worker immediately before and after its commit, duplicate queue delivery, delay one partition, and make the read projection unavailable. The runbook should say whether intake pauses, work accumulates durably, readers serve a last-known-good generation, or the request fails closed. Record the recovery point and compare the rebuilt digest before switching traffic. This is concrete evidence that the recovery story matches the write contract.

Capacity, Operations, and Recovery

Configuration Service check: How does a client avoid missing an update between snapshot read and watch? Estimate the durable write rate, average and peak payload, partition skew, retention volume, read or query shape, and acceptable lag. Convert those assumptions into partitions, worker concurrency, storage tiers, and backpressure. Numbers are sizing inputs, not universal facts. Say which limit fails first and what the user sees when it does.

Configuration Service check: What happens when watch history is compacted? Operate from contract metrics: accepted-versus-applied counts, duplicate/conflict rate, oldest pending work, authoritative-versus-serving divergence, retry exhaustion, and p95/p99 latency. Alert on sustained error-budget consumption and correctness gaps rather than raw traffic. Keep a runbook that can pause intake or rollout, drain work, rebuild a named projection through watermark W, verify a checksum, and switch readers to the new generation.

Configuration Service check: How are syntax and semantic dependencies validated? Use least privilege between producers, workers, control plane, data plane, and investigators. Encrypt transit and storage, rotate credentials, audit high-risk actions, and avoid placing secrets or unnecessary personal data in operational logs. Security controls support the data contract; they do not replace idempotency or verification.

Misleading Answers and Better Follow-Ups

Configuration Service check: Can related keys roll out atomically? A weak answer names Kafka, Redis, and a database before defining truth. Another says “exactly once” without identifying the idempotency key, persistence boundary, or replay behavior. A third adds a cache but cannot explain invalidation or rebuild. Replace each slogan with one invariant, one linearization point, and one counterexample.

Configuration Service check: How do clients behave during control-plane failure? Ask the interviewer to change a requirement: permit temporary staleness, add multiple regions, require a correction after finalization, or remove a dependency. Explain which state transition, key, and monitor changes. That is more useful than redrawing the same boxes at a larger scale.

Configuration Service check: How does a client avoid missing an update between snapshot read and watch? Separate three evidence classes in the answer. Official documentation supports the cited product or protocol behavior. The proposed architecture, thresholds, and sequencing are inference from the stated requirements. This prevents a sensible design choice from being presented as an external fact.

A 45-Minute Interview Plan

Spend the first five minutes clarifying actors, scale, correctness, retention, and failure expectations. Use the next eight to define the authoritative records, IDs, versions, and invariants. Draw the write path and its acknowledgment boundary for ten minutes, then the read or serving path for five.

Use ten minutes for the title-specific failure trace, including one retry, one stale or late event, and one recovery. Reserve the final seven minutes for sizing, skew, access control, monitors, and trade-offs. Keep a short list of assumptions visible. When the interviewer changes one, update the affected contract before changing components.

End with the source of truth, the permitted temporary inconsistency, and the repair procedure. A compact design that can be replayed and verified is stronger than a large architecture with undefined states.

Final Takeaway

For a configuration service system design interview, lead with this thesis: A configuration service is a version-distribution protocol. Correctness comes from immutable versions, a gap-free read-plus-watch handshake, client fallback rules, and a rollout that can stop or reverse. Define the evidence boundary, run a concrete race or failure trace, and show how the system detects and repairs divergence.

Sources and Further Reading

• etcd API guarantees and revisions
• AWS AppConfig validators
• AWS AppConfig deployment strategies
• AWS AppConfig data retrieval

Official sources support only the cited mechanism. Architecture recommendations, capacity figures, and interview sequencing are editorial inference.