cswork / libraryOpen practice workspace

Articles

Kubernetes NetworkPolicy Interview Questions: DNS, Egress, and Unexpected Connectivity

Prepare for Kubernetes NetworkPolicy interviews with selector logic, DNS egress, connection evidence, and unexpected connectivity diagnosis.

Kubernetes NetworkPolicy interview questions often begin with an apparent contradiction: a deny policy exists, but a request still succeeds; or an application port is allowed, yet every hostname times out. A strong answer follows one concrete connection through policy selection, source egress, destination ingress, and the actual DNS path. Reading one YAML file is not enough.

Evidence boundary: Kubernetes documentation establishes standard API behavior. Implementation-specific observations must be checked against the cluster's networking plugin and configuration. The billing-worker scenario and test matrices below are original exercises, not results from a running cluster.

Start with the connection, not the policy name

In the original scenario, a Pod labeled app=billing-worker in namespace payments must resolve names and connect to app=ledger Pods in namespace finance on TCP 8443. It must not reach arbitrary workloads in finance, another team's ledger, or external HTTPS endpoints.

Write that as a source, destination, protocol, and port. Also record the source Pod's actual namespace and labels. A policy called deny-all is a human-readable name; its effect comes from its specification and the full set of policies selecting the Pod.

Official behavior: a networking solution must implement NetworkPolicy for the resource to have an effect. Ingress and egress isolation are independent. Applicable allow rules combine additively, and a Pod-to-Pod connection must satisfy both the source's egress and destination's ingress restrictions. Allowed connection replies are implicitly permitted. Kubernetes: Network Policies.

Your initial questions should therefore be concrete: Which Pod made this request? Which policies select it for egress? Which destination was actually contacted? Which policies select that destination for ingress? Does the installed data plane enforce these resources?

This separates a selection mistake from a routing failure and an enforcement gap. It also avoids treating an unrelated debug Pod as proof of what the application can do.

Read selector structure as Boolean logic

Official API semantics: a peer containing both namespaceSelector and podSelector matches Pods satisfying both selectors. Separate peer entries are alternatives. A peer with only a Pod selector is scoped to the policy's namespace. Kubernetes: NetworkPolicy API reference.

For the billing worker, this original egress fragment describes the intended ledger peer. It is a fragment, not a complete deployable manifest:

egress:
  - to:
      - namespaceSelector:
          matchLabels:
            kubernetes.io/metadata.name: finance
        podSelector:
          matchLabels:
            app: ledger
    ports:
      - protocol: TCP
        port: 8443

The two selectors sit inside one list item. Adding another dash before podSelector would create a second alternative rather than narrow the namespace selection. The resulting rule could permit every Pod in finance plus matching Pods in the policy's own namespace.

Use a small truth table before discussing tooling:

Destination in the exerciseCombined namespace and Pod selectorSeparate peer entries
finance, app=ledgerMatchesMatches
finance, app=reportingDoes not matchMatches through namespace
payments, app=ledgerDoes not matchMatches through local Pod selector
analytics, app=ledgerDoes not matchDoes not match

This table evaluates only the peer portion of that rule. It does not establish complete reachability; ports, other applicable policies, destination ingress, and network health still matter. That qualification demonstrates that you understand what the example proves.

Check all selected policies before explaining a surprising allow

Suppose the worker can reach an external endpoint even after a restrictive policy is added. List every policy selecting the worker for egress. An older policy allowing all outbound traffic can explain the result: a narrower policy does not subtract permissions granted by another applicable policy.

Do not suggest fixing this by moving the restrictive policy “later” in the file list. Standard NetworkPolicy is not an ordered firewall rule chain. Instead, find the rule that authorizes the connection and determine whether its scope is intentional.

Official walkthrough: Kubernetes demonstrates testing the same destination from Pods with and without a selecting label. That is a useful pattern for controlled allow/deny comparisons. Kubernetes: declare a NetworkPolicy.

For this exercise, test a billing worker, a non-billing Pod in payments, and a matching-label Pod in another namespace. Changing one attribute at a time makes the selector's effect visible. If you change labels, destination, and port together, a failed request tells you much less.

Include namespace ownership in the discussion. A label used as a trust boundary is meaningful only if the right people control that label. Network reachability rules complement workload identity and application authorization; they do not replace either.

DNS is a separate required flow

Allowing TCP 8443 to the ledger does not allow the worker to ask a resolver for the ledger's address. Diagnose name resolution separately from the application connection. Record the resolver configured inside the affected Pod and test the exact name the application uses.

Official diagnostic guidance: Kubernetes' DNS troubleshooting procedure checks the Pod's resolver configuration, DNS Service, endpoints, and DNS component health. That provides evidence beyond “DNS seems broken.” Kubernetes: debugging DNS resolution.

In the exercise, compare a request by the intended hostname with a direct connection to a known current destination address. Preserve protocol and port. If the direct connection succeeds while resolution fails, investigate the resolver path first. If both fail, do not assume DNS is the sole cause.

A DNS egress exception commonly needs both UDP and TCP 53 for the resolver path used by the workload. Verify the actual destination and plugin behavior rather than copying a namespace selector from another cluster. DNS transport, caching, and deployment shape can differ.

Official architecture: NodeLocal DNSCache can place a DNS caching agent on each node, changing the path a Pod uses for lookups. A policy recipe aimed only at central DNS Pods may therefore miss the resolver the workload actually contacts. Kubernetes: NodeLocal DNSCache.

Do not broaden DNS access to every destination merely because one lookup fails. Establish whether the packet targets a cluster Service address, a node-local address, or another resolver, then design the narrow exception that fits that architecture.

Separate resolution, connection, and application success

This original evidence matrix keeps several failure classes apart:

Observation from the workerWhat it suggestsNext discriminating check
Lookup fails; known destination IP connectsResolver path or DNS configuration issueResolver address, UDP/TCP behavior, DNS endpoints
Lookup succeeds; TCP connection times outA later network path or listener problemBoth policy directions, endpoint readiness, packet evidence
TCP connects; TLS handshake failsThe transport path is available for that attemptSNI, trust chain, certificate identity
TLS succeeds; application returns forbiddenApplication authorization may reject the requestWorkload identity and resource permission
Unintended destination succeedsA broader allow, selection mistake, or enforcement gapAll selected policies and actual packet identity
Old session works; new session failsConnection lifetime may affect observationsNew connections under the same policy state

A timeout alone is not proof of a NetworkPolicy denial. A missing listener, incorrect address, route failure, or overloaded component can produce a similar symptom. Conversely, an HTTP error response demonstrates that enough of the path worked to reach a responding component, although that component may be a proxy rather than the final application.

State what each observation rules in or out. “The worker resolved the name, but no new TCP connection completed” is more useful than “the network is down.” It tells the interviewer exactly where your next evidence should come from.

Account for implementation boundaries and old connections

Official caveats: address translation can affect which addresses an ipBlock rule observes, hostNetwork treatment depends on the network implementation, and policy changes need not have uniform effects on already established connections. Standard NetworkPolicy also does not provide application-layer TLS policy or built-in security-event logging. Kubernetes: Network Policies.

For the exercise, distinguish the Service address the client requested from the endpoint address the data plane ultimately used. If a CIDR rule behaves unexpectedly, collect evidence about translation and enforcement placement instead of asserting that one address must always be visible at every stage.

When validating a change, create a fresh application connection. A pooled database connection or persistent HTTP connection can continue exercising a different lifecycle than a new connection. Record whether the test opened a new socket, reused one, or ran from a different Pod entirely.

Implementation-specific tools can add evidence. Cilium-specific example: Cilium documents Layer 7 visibility for DNS and HTTP through its policy and observability features. Those capabilities belong to that implementation, not the portable Kubernetes NetworkPolicy contract. Cilium: protocol visibility.

If using a different plugin, name its corresponding policy and flow diagnostics only after checking its documentation. The transferable skill is correlating a specific flow with the policy decision and destination state.

Roll out a narrow policy and prove both outcomes

Build a dependency inventory for the billing worker: resolver, ledger endpoint, telemetry destination if required, and any explicitly approved external dependency. Separate runtime dependencies from deployment-time image pulls or control-plane operations; they may originate from different components and follow different network paths.

Then introduce the policy to a small, representative workload scope. Define success as more than the absence of alerts. The worker should resolve the intended name, establish a new ledger connection, complete the expected operation, and fail to reach the deliberately excluded destinations.

Keep a comparison case that was already working before the change. If the ledger itself is unavailable, every negative connectivity check may “pass” for the wrong reason. A useful deny test needs a known-live target and evidence that an authorized peer can reach it.

For rollback, preserve the previous policy specification and the precise reason for reverting. Deleting all policies might restore connectivity but remove unrelated protections. Prefer reverting the change responsible for the regression, then repeat the allowed and denied checks.

Your incident update can be concise: “The new policy selected the correct workers but omitted the node-local resolver path. Direct ledger connections succeeded, DNS queries failed, and a resolver-specific exception restored lookups while the excluded application destinations remained unreachable.” This is an original example of evidence-based reporting, not a claim that every DNS outage has that cause.

Sources and Further Reading