Articles
Kubernetes Certificate Interview Questions: Expiration, Rotation, and Recovery
Practice kubernetes certificate interview questions with worked failure traces, recovery choices, and verification tests grounded in official documentation.
For Kubernetes certificate interview questions, begin with the failing TLS connection, which side presents a certificate, and who owns its issuance, trust, and reload. An expired administrator client certificate, an expired API server serving certificate, and a failed kubelet rotation can produce different symptoms. “Renew all certificates” is not a diagnosis.
This article uses official Kubernetes documentation for platform behavior. Its connection matrix and incident timeline are original interview exercises, not a production outage report. Commands are diagnostic examples for authorized environments, not instructions to change an unknown cluster. Practice Design secure Kubernetes with CI/CD with certificate ownership added to the release design.
Which certificate is involved in this connection?
Kubernetes uses multiple certificate relationships, not one cluster-wide TLS credential. The official PKI inventory distinguishes serving and client certificates for the API server, kubelets, etcd, and other components. A node may be a client on one connection and a server on another.
Use this original diagnostic matrix to keep directions explicit:
| Connection | Certificate presented for the role being checked | First evidence to collect |
|---|---|---|
| Administrator → API server | Administrator client certificate; API serving certificate on the opposite side | Active kubeconfig identity and actual endpoint TLS error |
| Kubelet → API server | Kubelet client certificate | Kubelet logs, selected credential, and client rotation CSR |
| API server → kubelet | Kubelet serving certificate; API client credential for its own authentication | Exact node endpoint, presented serving chain, and API-side error |
| API server → etcd | etcd serving certificate and API server's etcd client certificate | Both peers' logs and the configured certificate paths |
These are not claims that every distribution uses identical files or that every error has a single possible cause. Identify the deployment tool, cluster version, external issuer, and load-balancer topology before naming a repair command. Managed control planes and self-managed kubeadm clusters have different operational owners.
Notice that a 403 Forbidden response usually tells you a request reached an authorization decision. A certificate validity error occurs at a different boundary. Do not respond to every failed kubectl command by changing RBAC or replacing keys. First preserve the exact message and determine whether TLS, authentication, authorization, or connectivity failed.
Exercise: kubectl fails but nodes appear healthy
Assume this synthetic report: one engineer's kubectl fails after a weekend, another authorized administrator can list nodes, workloads still serve traffic, and node heartbeats remain current. There is not enough evidence to declare a cluster-wide serving-certificate outage.
Check which context and credential the failing client uses. A copied kubeconfig may contain an expired client certificate even though the control-plane-managed administrator credential was renewed. Alternatively, that engineer could be using a different endpoint or trust bundle. Compare identities and endpoints without sharing private keys or dumping a full kubeconfig into chat.
For a kubeadm-managed control plane, an authorized operator can inspect its local inventory:
kubeadm certs check-expiration
openssl x509 -in /etc/kubernetes/pki/apiserver.crt \
-noout -subject -issuer -dates -ext subjectAltName
kubectl get csr
These commands do not repair anything. kubectl get csr also requires working access and appropriate permissions; during an API outage, node or control-plane-local evidence may be necessary. A certificate file inspection shows what is on disk, not necessarily what a running endpoint presents.
The kubeadm certificate-management guide explains the inventory scope, renewal behavior, and copied administrator configuration. Its expiration check does not constitute a universal inventory of kubelet, ingress, service-mesh, webhook, or external-manager certificates. Say what the command covers and what you will inspect separately.
Your provisional answer should remain narrow: “The healthy alternate client makes an isolated credential or endpoint problem plausible. I would compare the failing client's certificate validity and endpoint with the known-working path before changing cluster certificates.” That is an inference from this scenario, not an official diagnostic rule that excludes every other cause.
What does kubelet client rotation actually do?
Kubernetes' kubelet rotation documentation describes requesting and retrieving a replacement client certificate as the existing one approaches expiration. The mechanism still depends on API connectivity, CSR authorization and approval, signing, and the kubelet obtaining and using the result.
Treat rotation as a pipeline with observable transitions: request created, approved, signed, retrieved, and selected for new connections. A setting that enables rotation proves only intent. It does not prove the pipeline worked or that the active file points to the renewed credential.
In an interview, name a check for each transition. If there is no CSR, inspect kubelet configuration, connectivity, and request errors. If the CSR is pending, inspect approval policy and identity. If approved but no certificate is issued, investigate signing and controller logs. If signed but the node still fails, inspect retrieval, local permissions, active credential selection, and connection errors.
Now add an expired credential. The node may no longer authenticate sufficiently to obtain its replacement through the normal path. The documented recovery route depends on how bootstrapping and issuance were configured. Do not delete its kubeconfig or approve arbitrary requests merely to restore a green node. Establish the node's identity and authorized recovery procedure first.
Why is a serving CSR not automatically trustworthy?
Client and serving certificates have different purposes. Kubelet TLS bootstrapping documentation separates client rotation from requesting a certificate for the kubelet's HTTPS endpoint. A client-authentication certificate is not a substitute for a server-authentication certificate.
The CSR reference states that kubernetes.io/kubelet-serving requests are not automatically approved by the built-in controller manager. Third-party approvers or distribution-specific mechanisms may exist, so inspect the actual cluster rather than generalizing “never approved automatically anywhere.”
Review the requester, signer name, requested usages, subject, DNS/IP SANs, and evidence that the requested endpoint belongs to that node. Approval is a security decision, not routine housekeeping to empty a queue. An attacker-controlled request for someone else's address must not become trusted merely because its YAML resembles a legitimate CSR.
The original matrix makes the consequence clear: kubelet client rotation can succeed while API-to-kubelet TLS still fails because the serving certificate is invalid for the address being used. Conversely, a valid serving certificate does not establish that the kubelet can authenticate as a client to the API. These are two directions with separate acceptance tests.
Does renewing a leaf certificate rotate the CA?
No. Renewing a leaf under a trusted issuer and replacing the trust anchor are materially different changes. A leaf renewal can preserve its identity attributes while extending validity. A CA transition changes which issuers peers must trust and can affect many clients beyond the component you are repairing.
The manual CA rotation documentation illustrates multi-component trust distribution and rollout constraints. Treat it as a version- and topology-sensitive procedure, not a universal interview command sequence. Some components and flags have restrictions on accepting bundles; do not assume every verifier supports the same overlap mechanism.
For an interview design, draw old issuer, new issuer, old leaves, new leaves, and each verifier. State which phase accepts which chain. An overlap phase is useful only if the actual components support it. Define the retirement condition: no required client remains dependent on the old trust anchor, and new connections pass verification through the intended path.
Also separate expiration from compromise. An expired leaf requires restoring valid credentials; a compromised private key may require revocation or replacement strategies, access review, and a broader incident response. Extending the validity of a compromised identity does not resolve that threat. Ask whether the problem is availability, unauthorized identity use, or both.
Plan recovery as an owned, reversible rollout
Suppose a self-managed, replicated kubeadm control plane has an expiring serving certificate. An original recovery plan begins with inventory and authorized backup of relevant configuration and keys, then validates the issuer and desired identity attributes. Do not back up secrets into a public artifact directory or copy them to an unapproved workstation.
The kubeadm guide documents manual renewal and the need to restart control-plane Pods because dynamic reload is not supported for every component/certificate combination. It also states that replicated control planes require the operation on their control-plane nodes and that externally managed certificates need their external owner's process. Confirm those conditions instead of extrapolating a local command to the whole fleet.
Before touching a node, establish the health and redundancy budget, etcd implications, load-balancer path, and rollback criteria. Renewing a certificate file with correct dates but incorrect SANs can exchange one outage for another. Preserving an old broken certificate is not a meaningful rollback if it is already expired; identify what state can actually restore service.
Do not place manifest backups inside a watched static-Pod manifest directory. A recovery plan must account for how the local kubelet observes those manifests and how components restart without depending on an unavailable API. The practical interview answer is an ordered, reviewed runbook with explicit health gates, not an improvised bulk restart.
How do you verify the certificate that is actually served?
Inspect both local state and a new TLS connection using the real hostname, SNI, port, and trust configuration. A certificate file on disk may not be loaded. A load balancer may terminate TLS before traffic reaches the API server. Different backends may present different chains, leaving failures intermittent after a partial rollout.
In a lab, compare the endpoint's issuer, validity period, SANs, and fingerprint with the intended credential. An openssl s_client connection can help collect evidence, but successful TCP connection alone is not successful certificate validation. Use verification options appropriate to your tool version and endpoint, and inspect the verification result. Never offer --insecure-skip-tls-verify as the completed repair.
Then test operations through each affected relationship: an authorized administrator request, kubelet registration/heartbeats, API-to-kubelet operations if those failed, and control-plane-to-etcd communication when implicated. Check negative behavior too: a wrong hostname or unauthorized identity must still be rejected. Restoring availability by removing trust checks fails the security acceptance test.
Set a follow-up observation window long enough to catch all intended backends and fresh connections. Existing long-lived sessions can conceal a bad replacement until they reconnect. Record which certificate changed, who owns its next renewal, which alert detects expiration, and what evidence demonstrates that replacement credentials are active.
Rehearse the symptom-to-owner explanation
| Practice question | Topic-specific follow-up |
|---|---|
| Design secure Kubernetes with CI/CD | Separate issuer, deployer, and runtime trust. |
| Explain How Microservices Run and Fail on Kubernetes | Map a failed connection to its actual TLS peer. |
| Explain CI/CD, Kubernetes delivery, and troubleshooting | Plan a controlled certificate rollout with verification. |
| Troubleshoot a production server outage | Gather evidence before a high-impact recovery. |
| Design access control and heartbeat systems | Distinguish expired identity from authorization and liveness. |
Use Design secure Kubernetes with CI/CD to present an issuance-and-reload design. Have a partner switch the failure from administrator-to-API to API-to-kubelet midway through your answer. You should change the certificate hypothesis and verification path, not merely repeat the same renewal command with a different filename.