Articles

Kubernetes Probe Interview Questions: Readiness, Liveness, and Restart Loops

Learn Kubernetes readiness, liveness, and startup probes through failure timelines, restart-loop diagnosis, and production-safe checks.

Kubernetes probes answer different questions. Readiness asks whether a container should receive Service traffic now. Liveness asks whether Kubernetes should restart it. Startup asks whether normal liveness and readiness checks should wait while initialization completes. Confusing those actions is how a temporary dependency outage becomes a restart storm.

This guide turns probe results into an event timeline you can explain in an interview. For broader context, compare it with Diagnose overloaded Kubernetes cluster.

Evidence boundary: Kubernetes documentation is the official source for probe and EndpointSlice behavior.

Give each probe one job

Probe	Question	Failure action	What it should not mean
Startup	Has initialization completed?	keep other probes disabled; restart after threshold	every dependency is healthy forever
Readiness	Can this pod serve its intended traffic now?	mark unready and remove from Service endpoints	restart the container
Liveness	Is local progress impossible without restart?	restart according to pod policy	a remote dependency is unavailable

Kubernetes says a failed readiness probe removes the pod's IP from matching Service EndpointSlices. The container keeps running. Readiness continues over the container lifecycle, so the pod can become ready again without a restart.

When a startup probe is configured, liveness and readiness do not begin until startup succeeds. Kubernetes documents the startup allowance as failureThreshold × periodSeconds. A startup probe with failureThreshold: 30 and periodSeconds: 10 allows roughly five minutes before failure action, subject to timeout and scheduling details.

Reason through a restart-loop incident

Consider this configuration:

livenessProbe:
  httpGet:
    path: /health
    port: 8080
  periodSeconds: 5
  timeoutSeconds: 1
  failureThreshold: 3
readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  periodSeconds: 5
  timeoutSeconds: 1
  failureThreshold: 2

The /health handler calls the database. At 12:00:00, the database has a 25-second outage. Three liveness checks fail by roughly 12:00:15, so kubelet restarts every affected container. Startup work adds CPU and reconnect demand just as the database recovers. More liveness checks time out, producing a synchronized restart loop.

The local process may have been perfectly capable of recovering once the database returned. The liveness dependency converted a remote failure into local churn. The readiness check was the safer place to represent inability to serve database-dependent requests: it would stop new Service traffic while keeping process state and allowing recovery.

Kubernetes documentation explicitly warns that incorrect liveness probes can cause cascading failures under high load, including restarts and additional workload. That is an official caution, not merely an interview opinion.

Choose endpoints from failure semantics

A liveness endpoint should test the smallest local property whose failure truly requires a new process: the event loop is making progress, a critical internal worker has not deadlocked, or an unrecoverable invariant is broken. It should be cheap, bounded, and independent of the same overloaded queue used by ordinary requests where possible.

A readiness endpoint may include dependencies required to serve this pod's contract, but avoid one global boolean for every route. If only payment writes require a downstream service, marking the whole pod unready can remove healthy read capacity. Alternatives include degraded responses, separate workloads, traffic policies, or readiness tied only to dependencies essential for all traffic.

Startup should model initialization such as loading a large index, replaying a log, or warming local state. Copying the liveness check with a long initialDelaySeconds is less adaptive: fast starts wait unnecessarily, and unusually slow but valid starts may still be killed.

Read the event sequence before editing YAML

When a pod restarts, collect:

kubectl describe pod <pod>
kubectl get pod <pod> -o yaml
kubectl logs <pod> --previous
kubectl get events --sort-by=.lastTimestamp
kubectl get endpointslice -l kubernetes.io/service-name=<service>

describe shows probe-failure events and restart counts. The pod status exposes container lastState, reason, and exit code. Previous logs preserve the terminated container's output when available. EndpointSlices show whether readiness affected routing. Application metrics and traces explain whether the handler timed out because the process was stuck, the node was overloaded, or a dependency failed.

Do not label every repeated restart “CrashLoopBackOff” without evidence. That status describes increasing restart delay after repeated failures; the initiating cause may be probe failure, process exit, OOM termination, or configuration error. Name the observed lastState and events.

Calculate timing accurately enough

Interviewers often provide periodSeconds, timeoutSeconds, failureThreshold, and successThreshold. Walk the sequence instead of multiplying blindly. Initial delay, startup gating, probe duration, scheduler timing, and version-specific behavior can affect exact timestamps.

For readiness with periodSeconds: 5 and failureThreshold: 2, two consecutive failures are needed before unready state. Recovery may require successThreshold consecutive successes. For liveness, failureThreshold: 3 means isolated failures do not restart immediately. timeoutSeconds: 1 means an endpoint that normally needs 800 ms has almost no overload margin.

An HTTP 500 is an application failure. A connection refusal may mean the process is not listening. A timeout may mean the handler, network namespace, CPU scheduling, or dependency path exceeded the budget. Those observations support different hypotheses.

Avoid probe-induced load

If 2,000 pods run three HTTP probes every second, the fleet generates 6,000 checks per second before business traffic. Expensive checks can overload the application or dependency they are supposed to observe. Staggering naturally occurs, but correlated deployments and shared dependency latency still create bursts.

Keep handlers constant-cost, cache expensive internal status briefly, and measure probe latency and result counts. Never let a liveness endpoint run an unbounded query. Ensure probe traffic bypasses authentication only within the intended network boundary and returns minimal information.

Probe timeouts also interact with CPU limits. A healthy process can miss a one-second deadline while throttled. Before widening the timeout, compare CPU throttling, node pressure, event-loop lag, and ordinary request latency. The correction might be resource requests/limits, endpoint isolation, or load shedding.

Connect probes to rollout behavior

Readiness controls when a new pod joins Service endpoints, but a successful readiness check does not prove every in-flight client has observed the change. During termination, use graceful shutdown, preStop only where justified, termination grace, and application drain behavior so removed endpoints stop accepting new work and finish or transfer existing work.

For deployments, minReadySeconds, readiness, surge, and unavailable settings jointly determine progress. A probe that flaps can stall rollout or shrink capacity. Inspect Deployment conditions and ReplicaSet events rather than treating pod readiness alone as proof of safe rollout.

A pod can be ready yet useless for one customer because cached configuration is stale. It can be unready yet valuable for background work. Define readiness from the workload's traffic contract.

Compare HTTP, TCP, exec, and gRPC probes

An HTTP probe can exercise an application endpoint and inspect status codes. A TCP probe proves only that a connection can be opened to the port; it does not prove the application can answer a meaningful request. An exec probe runs a command in the container, which can inspect local state but adds process-launch cost and depends on the image containing the command. Kubernetes also supports gRPC probes for services implementing the gRPC health-checking protocol.

Choose the smallest mechanism that answers the intended question. A TCP liveness probe can miss an event loop that accepts connections but never serves them. A complex exec probe can fail because a shell or helper binary is missing rather than because the application is dead. An HTTP handler routed through sidecars may test more of the networking stack than intended.

Record which port and scheme kubelet reaches. Probe traffic originates from the node, so host headers, TLS certificates, service meshes, network policy, and bind addresses can produce differences from ordinary Service traffic. Test the deployed path rather than only curling localhost inside the container.

Separate application failure from node failure

Probes are executed by kubelet. If the node is unreachable or kubelet is unhealthy, the control plane may stop receiving status, and ordinary probe tuning is not the fix. Check node conditions, heartbeats, container runtime, CNI, and resource pressure.

An OOM kill also bypasses the story that liveness restarted a healthy process. Inspect termination reason and exit code. CPU starvation can make probe handlers miss deadlines while the process remains logically correct; memory pressure can kill it outright. Read pod events in order and correlate them with node metrics.

Readiness gates Service routing, but clients with persistent connections may continue using an endpoint depending on the proxy and protocol. Endpoint removal is necessary evidence, not proof that all traffic stopped instantly. During an incident, examine connection draining and load-balancer propagation.

Design probe observability

Export application health-state transitions with reasons, but avoid making the probe response itself a verbose diagnostic endpoint. Dashboard probe successes and failures by workload, container, reason, and deployment version. Correlate them with restart count, unavailable replicas, request results, and node pressure.

Alerting directly on one failed probe duplicates controller behavior and creates noise. Alert on sustained unavailable capacity, rollout failure, restart growth, or user impact, then use probe metrics as diagnostic evidence. A deployment with one flapping pod and ample capacity differs from every replica restarting together.

During a canary, compare new and old versions. If only the new version shows probe timeouts at equal load, inspect its startup work, handler latency, resource profile, and dependency fan-out. Keep the rollback gate tied to service impact and recovery, not merely a single event count.

Interview prompts and strong answers

“The dependency is down. Which probe should fail?” Usually readiness if the pod cannot serve traffic without that dependency. Liveness should fail only if restarting the local process can restore progress. State exceptions based on the service contract.

“Why does the pod restart after 15 seconds?” Reconstruct the liveness schedule and three consecutive failures, then verify events. Do not infer solely from configuration because startup gating and previous state matter.

“Should readiness check every downstream service?” Only dependencies essential for the traffic represented by that readiness gate. Broad fan-out makes one remote failure remove otherwise useful capacity.

“How would you stop a slow-start loop?” Add a startup probe sized from measured initialization tails, keep liveness focused on local recoverability, and verify resource constraints. Do not hide a permanently broken startup with an infinite allowance.

“How do you test probes?” Inject a dependency outage, local deadlock, overload, slow startup, and handler failure separately. Assert endpoint membership, restart count, traffic result, recovery time, and rollout behavior for each.

Common weak answers

“Readiness means alive” misses routing. “Liveness should call every dependency” creates restart cascades. “Increase initial delay” ignores variable startup time. “A restart fixed it once” does not prove the probe was semantically correct. “The YAML looks right” avoids the observed event sequence.

A concise answer is: “I map each probe failure to its controller action. Readiness removes traffic, liveness restarts only for locally unrecoverable loss of progress, and startup gates both during initialization. I diagnose from events, previous state, EndpointSlices, and resource metrics, then test dependency and local failures independently.”