Articles
AWS VPC Troubleshooting Interview Questions: Routes, NAT, Security Groups, and NACLs
Prepare for AWS VPC troubleshooting interviews with route selection, public NAT paths, stateful security groups, NACL return traffic, and evidence.
An EC2 worker in a private subnet times out when calling a payment provider. Its security group permits outbound HTTPS. Another instance works. The interviewer asks what you would change.
The best first answer is an observation, not a rule change. Identify the worker's network interface, the resolved destination, the connection's source port, and the route table actually associated with its subnet. Then trace both directions. An outbound allowance is only one part of a working path.
Evidence boundary: AWS documentation supports the service behavior linked below. The payment-worker incident, address ledger, and diagnostic exercises are original interview preparation examples, not candidate reports or measurements from an AWS account. Documentation-only IP addresses illustrate packet reasoning and are not live test targets.
For an early rehearsal, use Explain VPCs and private/public network connectivity. Draw the return path as carefully as the request path before reading further.
What information makes a connectivity failure diagnosable?
“Cannot connect” could mean name resolution failed, the TCP connection timed out, TLS validation failed, or the remote application returned an error. Ask for the exact stage, timestamp, destination, and one comparison that changes only a small number of variables.
In the original exercise, the worker is 10.20.4.17 and its application resolves the provider to 203.0.113.20. It attempts TCP from source port 49160 to destination port 443. These invented values let you reason about a specific connection instead of an ambiguous “HTTPS rule.”
Record the source subnet, interface, security groups, route-table association, NACL association, and address family. Ask whether the client uses a proxy. A successful browser request from a laptop may use a completely different resolver, source address, and route.
A disciplined answer also preserves time. If the failing attempt preceded a deployment or route change, current configuration alone may not explain it. Align the application error, network observations, and configuration version before claiming causation.
How do you determine which route actually applies?
Official fact: A VPC route table maps destinations to targets, and subnet or gateway associations determine where that table applies. See Configure route tables. Finding a correct-looking route in an unrelated table proves little about the failing worker.
Official fact: Route selection generally prefers the most specific matching destination; IPv4 and IPv6 routes are evaluated independently. Additional priority rules apply to overlapping route types. See route priority.
Use this original route ledger for the worker's associated table:
| Destination | Target in the exercise | Consequence for this investigation |
|---|---|---|
10.20.0.0/16 | Local VPC routing | Covers addresses inside this VPC range |
203.0.113.0/24 | Old inspection path | More specific match for the example provider |
0.0.0.0/0 | Public NAT gateway | Does not win for that provider while the specific route exists |
The default route can be correct while the actual selected path is wrong. That is why “the NAT route exists” is an incomplete diagnosis.
Before deleting the specific route, explain its purpose and scope. It may intentionally send regulated traffic through inspection. The repair could be restoring the inspection target rather than bypassing it. State which destination set the proposed change affects and how you would test an unrelated destination afterward.
What must be true for the illustrated public NAT path?
Official fact: A public NAT gateway can support outbound internet access from private instances. In the public-subnet arrangement described by AWS, it has an Elastic IP and reaches the internet through the VPC's internet gateway. A private NAT gateway routed to an internet gateway does not provide equivalent internet connectivity. See NAT gateways.
Our example deliberately uses that public-subnet arrangement and IPv4. Do not generalize its drawing to every NAT deployment option or to native IPv6 routing.
Trace the intended sequence: worker subnet route, NAT translation, NAT-side route to the internet gateway, remote destination, and translated response back to the worker. Verify the gateway state and the actual subnet associations. A resource named public-nat is not evidence that its connectivity type or path is correct.
The provider may allowlist the public source address it observes. In that case, a successful route and TCP reachability can still be followed by a provider-side rejection if the egress address changed. Ask which address the remote service recorded rather than assuming it saw the worker's private address.
Official fact: AWS's NAT troubleshooting guidance includes connection and port-allocation symptoms. See Troubleshoot NAT gateways. If only bursts fail, correlate failures with connection behavior and relevant gateway metrics before treating the incident as a permanently missing route.
Why can security groups allow traffic while a NACL blocks it?
Official fact: Security groups are stateful: response traffic for an allowed connection is permitted regardless of the corresponding reverse-direction rules. They apply to associated resources. See security groups.
Official fact: NACLs act at subnet boundaries, evaluate numbered rules in ascending order until a match, and are stateless. Return traffic therefore needs an applicable allowance too. See network ACLs.
For our worker, the request heads toward destination port 443. The response comes from the provider's port 443 toward the client's ephemeral port 49160, after translation is reversed. An inbound NACL rule allowing only destination port 443 does not describe that response.
Do not memorize 49160 as the range to configure. It is one connection's example port. The operational rule must account for the actual client and NAT behavior, approved source or destination ranges, and the subnet boundary being evaluated.
Now add a rule-order twist: an early deny matches the return packet, while a later allow appears correct. Reading only the allow line misses the effective decision. Explain the first matching rule and the direction, then propose the smallest policy change consistent with the intended traffic.
A useful contrast is two same-subnet instances versus a connection crossing subnets. Avoid assigning a subnet-boundary control responsibility for every packet simply because both instances belong to a subnet with a NACL.
What can Reachability Analyzer and Flow Logs prove?
Official fact: Reachability Analyzer analyzes configuration and can identify a blocking component along a supported path. See What is Reachability Analyzer?. A favorable configuration result is not a live HTTP transaction or proof that the remote process is listening.
Official fact: VPC Flow Logs capture information about IP traffic involving network interfaces and can help investigate accepted or rejected traffic. See VPC Flow Logs. They are not full packet captures or application payload records.
Use this original evidence ladder to avoid overstating a tool result:
| Observation | Reasonable conclusion | Next unresolved question |
|---|---|---|
| Selected route points to intended target | Routing configuration matches this destination | Is traffic actually traversing it? |
| Analyzer reports a supported path reachable | Its configuration model finds a path | Is the endpoint healthy and listening? |
| Relevant flow record is ACCEPT | Recorded traffic passed the applicable recorded filtering decision | Did a complete response reach the application? |
| Relevant flow record is REJECT | A recorded network filtering decision rejected traffic | Which applicable control and rule explain it? |
| TCP handshake completes | Transport connection was established | Does TLS and application authorization succeed? |
| Provider returns an HTTP error | The request reached an HTTP-speaking endpoint | Was identity, payload, or provider policy wrong? |
Official fact: AWS provides ACCEPT and REJECT record examples and notes interpretation details in Flow log record examples. Use the record's interface, addresses, ports, protocol, and time range; do not search only for the instance name and infer the rest.
An absent record is also ambiguous. Check whether the relevant interface and traffic are covered, whether delivery has caught up, and whether the chosen filter would include the event. “No log entry” is not automatically “the application never sent a packet.”
How do you use the working instance as a controlled comparison?
Suppose an instance in a second subnet reaches the provider. Compare its actual resolved address, address family, route-table association, NACL, security groups, proxy settings, and observed public egress address. Differences are hypotheses, not conclusions.
If the second subnet lacks the more specific inspection route, that is a promising explanation for the original failure. Test the selected path directly using appropriate authorized diagnostics. Do not simultaneously change routing, NACLs, and security groups; a successful retest would no longer tell you which intervention mattered.
If both instances complete TCP but only one fails TLS, stop treating the error as a missing NAT route. Inspect the hostname, trust configuration, and certificate evidence. If both reach HTTP and receive different authorization responses, compare application credentials and provider policy rather than opening more ports.
DNS deserves special care. Official fact: NACLs cannot block DNS requests to or from the Route 53 Resolver in the ordinary way; AWS directs DNS filtering needs toward Resolver DNS Firewall. See NACL limitations. Do not propose a generic subnet deny rule as proof that AmazonProvidedDNS was blocked.
What does a convincing repair and verification plan look like?
For the original incident, assume evidence shows the stale inspection route wins and its target cannot forward the traffic. The proposed repair should identify that specific route, its intended owner, and whether to restore the target or change the approved path.
Write the acceptance conditions before the change: the worker resolves the expected destination, establishes a new connection through the intended path, receives the expected provider response, and preserves required inspection. Verify that an unrelated destination and an intentionally disallowed connection behave as designed.
Use a new connection for the retest and record the timestamp. Existing connection state can make before-and-after observations misleading. Monitor error rate and latency across a representative interval rather than celebrating one successful request during an intermittent incident.
Finally, name the rollback: restore the specific prior route or rule version if the defined regression occurs. Record the evidence that justified the change and the evidence after it. This demonstrates operational judgment without pretending every timeout has a single universal fix.
Practice VPC diagnosis with concrete interview prompts
The follow-ups are original exercises, not claims about what their original interviewers asked.
| Interview question | VPC troubleshooting follow-up |
|---|---|
| Explain VPCs and private/public network connectivity | Draw the worker's complete public NAT return path. |
| Design VPC, IP plan, management, storage, SSO | Find the associated table and winning destination route. |
| Design a secure trading app on AWS | Preserve inspection and provider allowlisting during recovery. |
| Troubleshoot CPU, latency, and DNS issues | Separate resolver failure from transport timeout. |
| Monitor Devices Over an Unreliable Network | Distinguish intermittent loss from a deterministic policy block. |
Continue with Explain VPCs and private/public network connectivity. Explain the example twice: once with a wrong specific route, then with a missing return-path NACL allowance. Your evidence and repair should change with the cause.