Articles
Cilium Interview Questions: eBPF Datapaths, Policy, and Connectivity Troubleshooting
Practice Cilium eBPF datapaths, workload identities, directional policy, DNS and Service failures, and Hubble verdicts with concrete connectivity traces.
DNS resolves the backend Service, but the frontend's HTTP request times out. Hubble shows a policy drop near the backend. Restarting DNS would change the wrong part of the system: name resolution succeeded, while a different flow was blocked.
Cilium interview questions are most useful when you connect an observed packet to its source identity, destination, policy direction, and configured service path. This guide uses a frontend-to-backend incident to explain eBPF datapaths, policy selection, and the limits of a flow verdict.
Evidence boundary: Official facts are linked to Cilium's stable documentation, which identifies v1.20.2 at the time of review. The incident, identities, matrices, and proposed answers are original exercises and engineering inference, not candidate reports. Actual packet paths depend on routing mode, kernel capabilities, deployment options, and enabled features. No example below claims to be a live-cluster test.
For a related implementation exercise, use Troubleshoot CPU, latency, and DNS issues; the practice table below explains its connection to this topic.
What does eBPF change in the network datapath?
Official fact: Cilium uses eBPF in its datapath and documents endpoint traffic paths, eBPF maps, service load balancing, and interoperability with other networking components. Use the eBPF datapath documentation to identify the relevant path for the configured deployment.
Engineering inference: An interview answer should identify what happens to a particular connection rather than saying “everything bypasses the kernel.” eBPF programs execute at kernel hooks. Their presence does not mean every application request, DNS lookup, and Layer 7 decision follows one identical path.
Original exercise: a frontend connects to a Kubernetes Service on port 80, which targets backend pods on port 8080. Depending on the installed configuration, service translation and forwarding can involve different mechanisms. Establish whether Cilium's kube-proxy replacement is enabled before attributing all Service behavior to it.
Similarly, native routing and encapsulation imply different inter-node paths. A same-node connection can work while a cross-node one fails. Ask where the source and backend are placed, what addressing is used, and which route the deployment promises.
Do not claim that installing Cilium guarantees the complete absence of iptables. Deployment features and interoperability matter. The useful explanation names the configured mechanism and the evidence that it is operating for the failing connection.
What is a security identity, and why are labels important?
Official fact: Cilium derives workload security identities from relevant labels rather than treating each pod's IP as its policy identity. The identity-based security documentation explains this approach.
An IP address still matters to forwarding and diagnosis. The distinction is that a policy can express which workloads may communicate based on their identities, while pods can be recreated with different addresses.
Original incident: a frontend endpoint has illustrative identity 1234 and labels app=frontend, team=shop. A backend endpoint has illustrative identity 5678 and labels app=backend, team=shop. The numbers are diagnostic examples, not reserved values to hard-code into production policy.
Engineering inference: When a selector seems correct, inspect the endpoint's actual relevant labels and identity. A pod's display name, Service name, and namespace label are not automatically interchangeable. A selector can fail because the expected label is absent or has a different value, even when the human-readable workload name looks right.
If a pod is recreated, recheck the observed endpoint rather than continuing to troubleshoot a stale IP from yesterday's log. Also distinguish an identity allocation problem from a rule that intentionally denies that identity.
The interview answer should connect three pieces: the selector expresses intent, the endpoint receives an identity derived from relevant labels, and the datapath enforces the applicable policy. A correct manifest alone does not establish that the endpoint is using the expected state at the time of the test.
When does policy selection enable default deny?
Official fact: In Cilium's default enforcement mode, endpoints begin unrestricted, and selecting an endpoint with policy enables enforcement for the applicable direction. Ingress and egress are considered separately. Cilium also documents always and never enforcement modes and cautions that enableDefaultDeny does not remove the effects of Layer 7 rules. See the policy introduction.
Engineering inference: Say which endpoint and direction are selected before saying “default deny.” A policy that isolates frontend egress is a different condition from one that isolates backend ingress. A connection can require permission at both sides.
This original matrix assumes default enforcement, an established identity mapping, ordinary allow-only rules, and no additional deny, Layer 7, host, or external firewall constraints:
| Frontend egress selected? | Backend ingress selected? | Permissions needed for a new frontend-to-backend connection |
|---|---|---|
| No | No | Neither direction is isolated by these policies |
| Yes | No | Frontend egress must allow the target and port |
| No | Yes | Backend ingress must allow the source and port |
| Yes | Yes | Both applicable directions must allow the connection |
The explicit assumptions are part of the answer. Do not extend an allow-only reasoning exercise to a deployment with deny rules or additional policy layers without inspecting them.
A follow-up changes a DNS-only egress rule. The rule can allow name-resolution traffic while leaving application traffic disallowed. DNS access is one permitted destination and protocol; it is not an implied grant to every address returned by the resolver.
For multiple rules, reason over the complete effective policy set. Finding one rule that mentions the backend is not enough if its selector, direction, protocol, or port does not match the observed flow.
Diagnose DNS success followed by an HTTP timeout
Use the original setup: frontend resolves backend.shop.svc, receives the intended Service address, and sends an HTTP request. The selected backend pod listens on TCP 8080. Frontend egress allows that workload and port, but backend ingress allows only a different source label.
| Step | Original observation | What it establishes | Next check |
|---|---|---|---|
| DNS query and reply | Name resolves to expected Service address | This lookup succeeded | Compare the requested hostname and selected Service |
| Service selection | Service has ready backend endpoints | There are candidates for routing | Confirm target port and the actual selected backend |
| Source policy | Frontend egress permits this destination and port | This configured direction permits the flow | Inspect backend ingress separately |
| Backend flow | Hubble reports a policy drop for the frontend identity | A policy decision blocked this observed event | Compare actual source labels with the backend's allowed selector |
| Scoped policy repair | Intended flow succeeds in a controlled test | The target behavior is restored in that test | Confirm an unauthorized source remains blocked |
Engineering inference: The leading hypothesis is a backend ingress mismatch because the observation points there. Confirm the identity and selector before changing the rule. Do not change DNS, broaden all egress, or disable policy enforcement merely because the user-visible symptom is a timeout.
This is also why port interpretation matters. A Service port and backend target port are different parts of the connection. Verify which port the relevant policy path evaluates in the actual deployment instead of assuming the application's URL port is sufficient evidence.
If the backend receives a connection but returns an application error, the next boundary changes. Authentication, request parsing, and backend dependencies are not automatically Cilium failures. Preserve the observed status and connection evidence before editing network policy.
What does a Hubble verdict prove?
Official fact: Hubble provides network observability built on Cilium. Its documentation describes visibility into workloads and network behavior. The troubleshooting guide shows flow observations and diagnostic commands for specific cases.
Engineering inference: A verdict describes the event observed at that point. A forwarded event is not a blanket statement that the full HTTP request succeeded. A dropped event with a policy reason is stronger evidence of a specific block, but still needs its source, destination, direction, port, and time matched to the incident.
Original exercise: one event says a SYN was forwarded from the frontend. Another shows a backend-side policy drop. The first event does not contradict the second: they can describe different points along the path. Reconstruct the connection rather than picking the more convenient line.
Likewise, no matching event is not proof that no packet existed. Your observation may cover the wrong node, time window, identity, or filter; visibility can be incomplete. Check the observation scope and Hubble's own health before drawing a negative conclusion.
Useful interview evidence includes the source and destination workload identities, observed endpoints, timestamp, protocol, port, verdict, and drop reason where available. Correlate that with the effective policy state. Avoid exposing unrelated sensitive traffic data in a troubleshooting artifact.
Use commands documented for the installed tools. cilium status and a controlled cilium connectivity test can help establish baseline health, while Hubble observations can narrow a failing flow. Exact flags and available checks are version-dependent; read their help and deployment documentation rather than inventing syntax under interview pressure.
A connectivity test is a diagnostic workload, not proof that every business application path works. State what the test covered and follow it with a fixture that resembles the failing route.
How do you separate Service translation from policy problems?
Original follow-up: a direct request to the backend pod succeeds from the same source, but a request through the Service fails. That shifts attention toward the Service path, though it does not prove one cause automatically.
Engineering inference: Compare the two requests precisely. Did they reach the same backend, use the same port, cross the same nodes, and carry the same application protocol? If those differ, the tests are not equivalent.
Inspect Service selectors, ready endpoints, target ports, and the configured service translation mechanism. Then correlate the selected flow with Cilium state and observations. A Service can have no suitable endpoints or point to the wrong port without an eBPF program malfunctioning.
If only cross-node traffic fails, inspect the routing or tunnel path appropriate to the deployment. If only one backend fails, compare endpoint state and application listeners. If every request fails after a policy rollout, inspect selector and enforcement changes first.
Use the smallest comparison that distinguishes hypotheses. For example, test one source against one known backend on the same node and then an equivalent backend on another node. Keep the payload and expected response fixed. This produces clearer evidence than changing the policy, DNS, and Service configuration at once.
After a repair, repeat both the intended and prohibited flows. Restoring connectivity by allowing everything fails the security requirement even if it makes the application look healthy.
How do Layer 7 rules change an otherwise allowed flow?
An allowed TCP connection and an allowed HTTP operation are different requirements. A deployment using Layer 7 policy can permit transport establishment yet reject a specific request method or path.
Engineering inference: If the observed failure is at the proxy or HTTP layer, inspect the applicable Layer 7 rules and proxy state. Do not assume that all filtering occurs in one eBPF decision or that a successful handshake establishes application authorization.
Original exercise: backend access on TCP 8080 is allowed, but only GET /health is permitted by the selected HTTP policy. A successful health request does not prove POST /orders is allowed. Test the intended operation and explain which rule evaluates it.
The official policy introduction's enableDefaultDeny caution matters here. Avoid presenting that option as a universal switch that makes inspection-only policies harmless. Read the Layer 7 behavior for the actual rule and release.
A strong answer preserves the least-access requirement: identify the necessary method, path, source identity, and destination, then verify both allowed and disallowed examples. It does not recommend turning off enforcement as a generic recovery shortcut.
Explain the incident in a reviewable interview answer
Prompt: “The frontend can resolve the Service but times out after a new Cilium policy.”
An original answer is: “I would record the failing source, Service, backend, protocol, and time. DNS success narrows name resolution for that test. I would inspect selected endpoints and ports, source egress and backend ingress, and the actual workload identities. A matching Hubble policy drop would direct me to the selector or rule at that boundary. I would apply a scoped repair and verify that the intended flow succeeds while an unauthorized source stays blocked.”
The follow-up asks whether restarting pods is enough. Explain that recreation can change endpoints and obscure the evidence without correcting the rule. Capture the relevant state first, then choose a repair tied to the demonstrated cause.