Kubernetes Operator Interview Questions: Reconciliation, Finalizers, and Idempotency

A Kubernetes operator interview tests whether your controller can keep moving a system toward the desired state despite retries, stale observations, and partial failures. Writing a custom resource definition is only the beginning. The difficult part is making the next reconciliation safe when the previous attempt stopped halfway through.

Imagine a custom resource that requests a managed database. The provider creates the database, but the API call times out before your controller receives the response. Should the next reconcile create another one? A strong design can rediscover the existing resource, continue setup, and report what is actually ready.

Key takeaways

• Reconcile current state against desired state; do not rely on receiving every event in order.
• Make external operations retry-safe, including recovery after a successful request with a lost response.
• Finalizers delay deletion so cleanup can run. They are not cleanup code by themselves.
• Status describes observations. A requested replica count is not proof of readiness.
• Test interruption points, conflicts, duplicate work, and deletion failures, not just creation.

What is the difference between a controller and an operator?

A controller is a control loop that observes resources and works to make actual state match intent. An operator applies that pattern to domain-specific operational knowledge, often through custom resources. For a database, that knowledge might include backup policy, upgrade sequencing, credential rotation, and deletion behavior.

A CRD defines the custom API's shape, validation, and versions. It does not by itself create a functioning database or perform recovery. The controller implements those behaviors. In an interview, name the boundary between schema validation, reconciliation, and the external provider instead of treating “install a CRD” as the whole solution.

What should a reconcile function do?

Think of reconciliation as a repeatable inspection, not an event handler that executes a command exactly once. An update notification tells the controller to look again. The object may already have changed by the time the controller reads it.

1. Fetch the custom resource. If it no longer exists, finish without attempting a new create.
2. If deletion is in progress, follow the cleanup path before the normal create/update path.
3. Validate the requested configuration and expose a useful condition if it cannot be acted on.
4. Ensure cleanup protection exists before creating an external resource that requires it.
5. Observe Kubernetes children and external state, then compute the smallest necessary change.
6. Apply a retry-safe step, update meaningful status, and arrange another observation when work remains.

Suppose the resource requests three replicas and two are ready. Reconciliation should investigate the missing or unhealthy replica, not blindly append a replica every time it is called. The desired state remains three even if the same object is queued repeatedly.

The sequence can span many reconciliations. A cloud database may take minutes to provision. Record durable identifiers, report that provisioning is in progress, and return control rather than holding a worker in an unbounded polling loop.

What does idempotent reconciliation mean in practice?

An operation is idempotent when repeating it has the same intended effect as applying it once. “Read before create” helps, but it does not solve every race or lost response. Two actors might both observe absence, or the provider may create a resource before the controller records its identifier.

Use a provider-supported idempotency key where available, derived from a stable ownership identity. Otherwise, design a reliable lookup and uniqueness scheme, and explain its limitations. A resource name alone may be insufficient when an object is deleted and recreated under the same name; the Kubernetes UID distinguishes the new object's identity.

Interruption point	Risk	Recovery design
Create succeeds; response is lost	Duplicate external resource	Idempotency key or ownership-based rediscovery
Status update fails	Controller forgets progress	Recompute from durable observed state
Configuration partly updates	Repeated destructive changes	Compare fields and converge incrementally
Delete succeeds; response is lost	Cleanup remains stuck	Treat confirmed absence as success

How do Kubernetes finalizers work?

A finalizer is a key on an object's metadata that tells Kubernetes deletion must wait for associated cleanup. When deletion is requested, the API sets a deletion timestamp. The responsible controller observes that state, performs cleanup, then removes its own finalizer. It must not remove unrelated controllers' finalizers.

For the database example, add the finalizer before provisioning the billable external database. During deletion, check whether the external database exists, request deletion if needed, confirm completion, and then remove the finalizer. If the provider cannot be reached, keep the finalizer and make the failure observable.

Distinguish “delete request accepted” from “resource is gone.” An asynchronous provider may acknowledge deletion while still retaining the resource. Your cleanup contract determines what must be confirmed before releasing the finalizer.

Are owner references a replacement for finalizers?

No. Owner references express ownership relationships used by Kubernetes garbage collection for Kubernetes objects. A finalizer supports custom cleanup that must happen before deletion completes. A cloud database outside the Kubernetes API is not automatically removed merely because a Kubernetes child object has an owner reference.

Use each mechanism for its job. An operator may set ownership on a generated Deployment and still require a finalizer for external storage. Also define a retention policy: deleting the custom resource might intentionally retain data, but that must be an explicit product decision with clear status and documentation.

What belongs in status conditions?

Status should help a user answer three questions: what has been observed, what is blocking progress, and does that observation apply to the current desired configuration? Conditions such as Ready, Progressing, or Degraded can communicate this if their meanings are consistent.

Where your API uses generation tracking, an observedGeneration field tells consumers which desired-state generation the controller evaluated. A Ready condition describing yesterday's spec must not be mistaken for confirmation of today's requested upgrade. Include concise reasons and actionable messages; do not place credentials or provider secrets in status or events.

Avoid writing the same status every pass. Unnecessary updates increase API traffic and can generate more reconciliation work. Update transition timestamps when the condition changes, not merely because the loop ran again.

How should retries, conflicts, and terminal errors differ?

Situation	Useful response
Temporary provider outage	Bounded backoff, observable error, later retry
Normal asynchronous provisioning	Progress condition and a planned recheck
Invalid configuration	Actionable condition; avoid hammering the provider
Kubernetes update conflict	Refetch, recompute, retry without erasing others' changes
Authentication failure	Clear diagnosis and recovery route without exposing secrets

The exact return values and work-queue behavior depend on the controller framework and version. Explain the intent first. A fixed rapid retry for every failure is rarely a good answer: it can overload an unavailable dependency while giving users no better information.

How would you investigate a resource stuck terminating?

Start with evidence: deletion timestamp, finalizer keys, controller health, recent events, logs for that object, and the external provider's actual state. Check whether cleanup lacks permission, the controller is absent, the resource identifier is wrong, or the provider is still working.

Then decide the recovery step. Restore the controller or its permissions, retry the provider operation, or reconcile an already-absent external resource. Manually removing a finalizer should be a documented last-resort procedure after verifying cleanup or explicitly accepting retained resources. Removing it first can orphan data or continuing charges.

A useful operational metric is the age of objects awaiting cleanup, alongside reconcile errors and queue depth. These measurements expose a slow failure that may not appear as a crashed process.

What tests demonstrate that an operator is reliable?

• Reconcile an unchanged object twice and assert that the second pass makes no unnecessary changes.
• Simulate successful external creation followed by a lost response; assert that retry discovers one resource.
• Restart between finalizer installation, external creation, and status persistence.
• Inject an API conflict and check that another actor's fields survive.
• Delete an already-absent external resource and confirm cleanup finishes.
• Change the spec during provisioning and ensure readiness reflects the current generation.

Unit tests can cover pure comparison and decision logic. API-backed integration tests are better for real object behavior and conflicts. End-to-end tests cover provider integration. A fake client alone cannot prove garbage collection, admission, or API-server behavior matches production.

A focused interview preparation exercise

Sketch a ManagedDatabase resource with a requested size, backup policy, and retention choice. Draw creation, update, and deletion separately. For every operation, name its durable input, expected observation, failure mode, and retry behavior. Then practise a two-minute answer to: “The controller restarted after the database was created but before status was written. What happens now?”

The strongest answer is not “we retry.” It explains how the retry identifies the same database, avoids duplicate side effects, and reconstructs progress.

References: Kubernetes documents the operator pattern and finalizer lifecycle. The failure scenarios and practice design here are original examples, not a universal framework API.