Kubernetes Admission Webhook Interview Questions: Failure Policies and Cluster Lockouts

A new policy webhook goes live, then its Pods disappear. Your team cannot recreate them: every replacement Pod is rejected or times out at admission. This is the admission-webhook interview case that matters most. A strong answer traces the API request, distinguishes an explicit denial from an unreachable webhook, and restores the control path without silently disabling a security requirement.

Kubernetes officially documents that failurePolicy: Fail rejects a request when a matching webhook call fails, while Ignore lets admission continue. An explicit allowed: false response rejects the request under either policy. The examples and recovery sequence below are interview exercises and operational inference, not a report of a particular employer's interview. For a broader cluster-security prompt, try Design secure Kubernetes with CI/CD.

What happens between a request and a stored object?

The Kubernetes admission-control documentation places admission after authentication and authorization, before an API object is persisted. Admission runs for writes and some other operations, rather than ordinary reads. Built-in admission controllers and configured webhook calls can affect the result. A webhook receives an AdmissionReview; a mutating webhook may return a patch, while a validating webhook allows or denies the object. Dynamic Admission Control describes the request and response contract.

In an interview, draw the path before diagnosing it: client → API server authentication and authorization → matching admission steps → persisted object or error. A successful kubectl auth can-i create pods check covers authorization, not the later admission decision. Conversely, a Forbidden authorization error is not proof a webhook ran. Ask for the API error, resource, verb, namespace, and time; those identify the branch you need to investigate.

The first urgent question is which webhook even matches. Its rules select operations, API groups, versions, resources, and scope. namespaceSelector, objectSelector, and matchConditions can narrow calls further. A Pod create may match a validator that an UPDATE to an existing Deployment does not; creating that Deployment can still lead a controller to create a matching Pod later. The difference between admitting a controller object and admitting its child Pod makes many rollout failures appear delayed.

Failure policy: what does Fail or Ignore actually change?

Kubernetes defines failurePolicy for errors calling the webhook: network failure, timeout, non-2xx response, malformed response, and other listed call errors. Fail rejects the API request on that error and is the default. Ignore skips that failed webhook call and lets the request continue through the remaining admission path. Neither setting overrides a valid webhook response with allowed: false. “Fail open” therefore does not mean “ignore a policy denial.” It means “continue when the policy service cannot provide a valid decision.”

Webhook outcome for a matching request	Fail	Ignore	Interview interpretation
Valid allowed: true response	Continue	Continue	Other admission checks can still reject.
Valid allowed: false response	Reject	Reject	The webhook made an explicit policy decision.
Call times out or cannot connect	Reject	Continue	This is the availability-versus-enforcement choice.
Invalid HTTP or AdmissionReview response	Reject	Continue	Treat this as a broken call, not a policy denial.
Rules or selectors do not match	Skip webhook	Skip webhook	failurePolicy is irrelevant to this request.

The timeout documentation says timeoutSeconds can be 1–30 seconds and defaults to 10. A long timeout may turn a modest webhook outage into API write latency and backed-up controllers. A short timeout limits that waiting but can turn occasional slow calls into admission failures. Choose a timeout based on measured latency and the consequences of failure, then monitor timeout rate; “set it to one second” is not a universal fix.

Which policy is safer depends on the control. A mutator that adds a convenience label may be able to fail open, provided the final required state is validated elsewhere. A validator that blocks dangerous privileged workloads may deliberately fail closed. That is an engineering inference from the documented semantics, not a blanket Kubernetes setting. In the latter case, plan an independent break-glass route with restricted ownership and audit, because an unplanned cluster-wide override trades a security outage for a different risk. Kubernetes' good-practices guide recommends failing open for mutating webhooks and validating final state where appropriate.

How can a webhook lock out its own recovery?

Consider a validating webhook deployed in namespace policy-system. Its configuration matches all Pod CREATE requests, including Pods in policy-system, with failurePolicy: Fail. A node failure removes its only Pod. The Deployment controller asks the API server to create a replacement, but the API server cannot reach the webhook Service. That call fails, so admission rejects the Pod. The controller retries; the service remains without an endpoint. This loop is a hypothetical worked case, grounded in the self-mutation and dependency-loop failure modes described by Kubernetes admission-webhook good practices.

Notice the exact boundary: existing workload Pods may keep serving while new matching API writes fail. It is inaccurate to say an admission outage immediately kills all running Pods. However, failed replacements, scale-outs, and rollouts can turn a control-plane admission problem into user-visible capacity loss. If a webhook also intercepts objects needed by its network or storage dependencies, recovery can become even harder. Kubernetes advises excluding the webhook's own namespace and avoiding dependency loops; it also advises avoiding mutation of critical kube-system workloads and node Lease objects.

The design correction is narrow. Scope rules to the resources and operations the policy actually needs; use a carefully controlled namespaceSelector or other match filter so the webhook can restart; run enough replicas across failure domains; and test the loss of every webhook replica. Exemptions are a risk boundary, not a magic label: if users can freely apply the exempting label to their own namespace, they may bypass the control. An interviewer should hear who can change that label and how a denied action is audited.

Diagnose a lockout without guessing at the YAML

Start with one failed request and a timestamp. kubectl create --dry-run=server -f pod.yaml -n test can reproduce the admission path without persisting the object, but only if the webhook declares compatible sideEffects for dry-run. If dry-run is rejected for that reason, use a controlled disposable namespace and an ordinary create instead. Preserve the server error: a named webhook timeout, a TLS verification failure, and an explicit denial imply different fixes.

Then inspect the configuration and the network path. The following commands are a diagnostic checklist, not a script to run blindly in production:

kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations
kubectl get validatingwebhookconfiguration <name> -o yaml
kubectl -n policy-system get service <webhook-service> -o wide
kubectl -n policy-system get endpointslice -l kubernetes.io/service-name=<webhook-service>
kubectl -n policy-system get pods -o wide
kubectl -n policy-system describe pod <webhook-pod>
kubectl -n policy-system logs deployment/<webhook-deployment> --all-containers

Read rules, selectors, clientConfig.service, caBundle, timeoutSeconds, and failurePolicy from the effective configuration. A Service with zero ready EndpointSlice addresses suggests no healthy backend; populated addresses shift attention toward API-server-to-Service routing, TLS, or application errors. A certificate name mismatch is plausible when the webhook server certificate does not cover <service>.<namespace>.svc, a requirement in the dynamic admission reference. Logs from a surviving replica and API-server admission metrics or audit events can show whether requests reached the handler.

Do not jump from one failed kubectl command to “the cluster is locked.” Test a known matching resource in an affected namespace and a deliberately nonmatching resource or namespace. Check whether the failure is one API server, all API servers, one webhook replica, or one network zone. If an explicit deny arrives, investigate the submitted object and policy logic before changing failurePolicy; that field cannot reverse the deny. If a timeout arrives, investigate endpoints, certificates, connectivity, server load, and deadline. This decision tree is more useful than memorizing kubectl get flags.

What is a safe recovery sequence during the incident?

First, state impact and freeze unrelated rollout changes. Capture the failing API response and the current webhook configuration before editing it. If a healthy webhook replica still exists, restore its Service routing or certificate and verify a matching test write. If no replica can start because the webhook intercepts its own Pods, a privileged operator may need a tightly scoped, temporary configuration change—such as excluding only the recovery namespace or changing the failing webhook's failure behavior—under the team's incident procedure. That is a proposed recovery approach, not an instruction to remove every validator. The least broad working change preserves the most policy coverage.

After a temporary change, let the webhook Pods become Ready, confirm Service EndpointSlices, and submit a known-allowed request. Also submit a known-denied request so “the API works” does not conceal a disabled policy. Restore the intended configuration only after the service is healthy; repeat both checks and watch rejection, timeout, and latency metrics. Record the window during which admission may have been bypassed, and review objects created during it if policy requires. A candidate who restores the Pod but never restores or verifies enforcement has not finished the incident.

Symptom	Evidence that separates causes	Safest next move to discuss
failed calling webhook, timeout	No ready endpoints or unreachable backend	Recover backend or narrowly scope the call; retest matching writes.
x509 or certificate error	Certificate SAN, expiry, caBundle, service name	Repair trust or serving certificate; avoid replacing policy logic.
Webhook says denied the request	Valid allowed: false with policy reason	Fix object or policy rule; changing failurePolicy will not help.
Only new webhook Pods fail	Configuration matches its own namespace or dependencies	Break the loop through controlled scope and restore replicas.
Rollout stalls after Deployment create succeeds	Pod CREATE admission error in events	Trace the child Pod request, not only the parent Deployment.

Design and rollout questions an interviewer may ask

“Would you set every webhook to Ignore?” No. Explain the exact consequence of an unavailable policy service for each resource class. For a mutator, fail-open plus final-state validation may be appropriate. For a security validator, fail-closed may be worth the operational burden. In either case, minimize matching scope, keep the handler highly available, and rehearse a controlled recovery. Describe the security invariant that must survive a webhook outage.

“How do you roll out a new validating webhook?” Deploy and test the server first. Apply a configuration scoped to a test namespace and intended operations; use server-side dry-run where supported; test allow, explicit deny, timeout, and no-endpoint cases. Widen scope in stages while watching API latency and rejection metrics. The staged plan is an operational inference from the documented matching and failure behavior. It is not a Kubernetes guarantee that a specific rollout order is safe in every cluster.

“Can two mutating webhooks assume an order?” Kubernetes cautions that mutating webhook invocation order is not guaranteed and recommends idempotent mutations. Patch only fields needed; avoid overwriting array values or labels used by controllers. Make the validator check the resulting state rather than depend on one mutator having run first. If a second invocation produces a different patch, investigate reinvocation semantics and your own idempotence before blaming the API server. The good-practices guide covers these design limits.

“What should the on-call monitor?” Measure matching-call count, reject count by reason, timeouts, API admission latency, server error rates, ready replicas, EndpointSlice readiness, and TLS expiry. Alert on both enforcement failures and write availability. A green webhook Deployment is insufficient if the API server cannot reach its Service or if a too-broad rule is blocking unrelated control-plane work. The exact alert thresholds depend on the cluster's traffic and security posture; specify how you would validate them with a controlled failure drill.

Sources and further reading

• Kubernetes: Dynamic Admission Control — webhook matching, failure policy, timeouts, certificates, dry-run, and monitoring.
• Kubernetes: Admission Webhook Good Practices — scope, availability, mutation, self-mutation, and dependency loops.
• Kubernetes: Admission Control — where admission sits in request processing.