AWS network boundaries
Why you care
Every resource you launch in AWS lives inside a VPC, whether you chose one or not. The moment your Jenkins job's replacement becomes a Lambda or ECS task, you need to know which subnet it lands in, which route table governs its traffic, and whether a security group or a NACL is the thing blocking port 5432. Interviewers ask "what's the difference between a security group and a NACL?" because the answer predicts whether you'll spend three hours debugging a working rule that fires on outbound but forgets return traffic.
Explain like I'm 5
Think of a VPC as your own private office floor in a shared skyscraper. Subnets are rooms on that floor. The route table is the directory on the wall saying which corridor leads outside. A security group is a personal bodyguard assigned to each desk: it remembers every conversation that came in and automatically lets the reply out. A NACL is a checkpoint at the room door: it reads a numbered list of instructions top-to-bottom and has no memory — if your rule says "let port 443 in" but you forgot to add a rule for the response to go out, the conversation is silently cut.
Core concept
A VPC (Virtual Private Cloud) is a logically isolated virtual network in your AWS account. You give it a CIDR block — for example 10.0.0.0/16 — and carve it into subnets, each tied to one Availability Zone.
A subnet is public if its route table has a route sending 0.0.0.0/0 to an internet gateway. It is private if that route is absent or points to a NAT gateway instead. The label "private" on its own does not define reachability — the route table does.
Route tables contain destination/target pairs. Every subnet's traffic is evaluated top-to-bottom until a match: the most-specific CIDR wins. The local route (10.0.0.0/16 → local) is always implicit and cannot be deleted.
Security groups are stateful instance-level firewalls. Rules are allow-only; there is no explicit deny. Because they are stateful, return traffic for an allowed inbound connection is automatically permitted regardless of outbound rules. You can reference another security group as a source or destination, which is how you say "the app tier may reach the DB tier."
Network ACLs (NACLs) are stateless subnet-level checkpoints. They support both allow and deny rules, numbered 1–32766, evaluated in ascending order — first match wins. Because they are stateless, you must write explicit inbound and outbound rules for every conversation, including the ephemeral return-port range (1024–65535 for most TCP clients).
Decision rule: use security groups as your primary control; they are flexible, stateful, and can reference other groups. Add NACLs only as a second line of defence, for example to deny a specific IP range at the subnet boundary.
When not to rely on NACLs: never write a full stateless firewall policy in NACLs alone — the ephemeral port rule is easy to get wrong, and a missing outbound ALLOW breaks the connection silently.
Minimal example CIDR arithmetic in stdlib
import ipaddress
vpc = ipaddress.ip_network("10.0.0.0/16")
# Carve four /18 subnets
subnets = list(vpc.subnets(prefixlen_diff=2))
for s in subnets:
print(f"{s} hosts={s.num_addresses - 2} first={s.network_address + 1} last={s.broadcast_address - 1}")10.0.0.0/18 hosts=16382 first=10.0.0.1 last=10.0.63.254
10.0.64.0/18 hosts=16382 first=10.0.64.1 last=10.0.127.254
10.0.128.0/18 hosts=16382 first=10.0.128.1 last=10.0.191.254
10.0.192.0/18 hosts=16382 first=10.0.192.1 last=10.0.255.254In production Stateful vs stateless rule evaluator
from dataclasses import dataclass, field
from typing import Literal
@dataclass
class Rule:
number: int # for NACL ordering; ignored in SG
action: Literal["allow", "deny"]
port: int
def sg_allows(rules: list[Rule], port: int, is_return: bool) -> bool:
"""Security groups: stateful. Return traffic is always allowed."""
if is_return:
return True
return any(r.action == "allow" and r.port == port for r in rules)
def nacl_allows(rules: list[Rule], port: int) -> bool:
"""NACLs: stateless. Rules evaluated in ascending number order, first match wins."""
for rule in sorted(rules, key=lambda r: r.number):
if rule.port == port:
return rule.action == "allow"
return False # implicit deny if no rule matched
# Scenario: inbound rule allows 5432, no outbound rule for ephemeral port 49152
inbound = [Rule(100, "allow", 5432)]
outbound_nacl = [Rule(100, "allow", 443)] # forgot ephemeral ports
# SG: inbound 5432 OK; return traffic automatically allowed
print("SG inbound 5432 :", sg_allows(inbound, 5432, False))
print("SG return port :", sg_allows(inbound, 49152, True))
# NACL: same inbound, but return traffic on ephemeral port 49152 is missing
print("NACL inbound 5432:", nacl_allows(inbound, 5432))
print("NACL return 49152:", nacl_allows(outbound_nacl, 49152))SG inbound 5432 : True
SG return port : True
NACL inbound 5432: True
NACL return 49152: FalseSee it happen Packet through VPC layers
- Packet arrives at the subnet boundary. NACL evaluates inbound rules in number order.
- Rule 100 allows port 5432. NACL passes the packet inward.
- SG has an allow rule for port 5432. Traffic reaches the instance.
- Instance replies. SG is stateful: the return is automatically allowed.
- SG passes the return packet to the subnet boundary.
- NACL is stateless. Without an explicit outbound allow for the ephemeral port range 1024–65535, the reply is silently dropped. The caller sees a timeout.
In your world
The situation
Your Jenkins-run Python ETL job is being migrated to a Lambda. A teammate puts it in subnet-private-a, creates a security group allowing port 5432 from the Lambda's SG, and deploys. Lambda logs show a timeout connecting to RDS on port 5432.
What actually breaks
The SG looks correct. The actual block is a custom NACL on the subnet that has an inbound ALLOW 5432 at rule 100 but no outbound rule for the ephemeral port range. The SYN reaches RDS; the SYN-ACK is dropped at the subnet boundary. Connection times out after ~30 s. The CloudWatch log shows the attempt but no error code — just a timeout.
What you'd do on Monday
Add an outbound NACL rule: number 900, allow TCP, ports 1024–65535, destination 0.0.0.0/0. This covers the ephemeral return range for TCP clients. Alternatively, if the NACL was the wrong tool, remove the custom NACL and let the default NACL (which allows all traffic) cover the subnet while the SG does the real work.
Imagine you work at Zomato
The numbers
Illustrative: 40k req/min × ~2 TCP connections each (HTTP keep-alive not always reused across Lambda invocations) ≈ 80k active flows at peak. Each flow has a source ephemeral port in the 1024–65535 range. A NACL that only allows outbound on specific ports will drop the majority of return traffic silently.
What the Scenario-A answer does here
For a single-instance app the missing ephemeral-port rule is noticed in one five-minute test. At 80k concurrent flows the intermittent drop rate is 100% for any port outside the allowed range; CloudWatch shows thousands of Lambda timeouts per minute before anyone thinks of the NACL.
What flips at this scale
At scale, NACLs are used only for broad subnet-level deny rules (block a compromised CIDR, deny a known bad range). The fine-grained per-service rules belong entirely in security groups that reference each other. Every subnet gets a NACL with allow-all inbound and outbound except for explicit deny rules at low numbers; security groups carry the access policy.
The trade-off you'd defend
I would defend SG-only control with a minimal NACL (allow-all with one or two explicit denies) over complex NACL policies. Security groups are stateful, easier to audit, and their SG-reference model makes the data-tier boundary readable. The cost is that a misassigned SG lets traffic through; a dedicated-subnet NACL catches it, so a paranoid design uses both.
Illustrative request figures; not Zomato-published.
Gotchas
The subnet is labelled "private" but packets still reach the internet
An EC2 instance in a "private" subnet can curl ifconfig.me and get a public IP back.
The route table has 0.0.0.0/0 → igw-xxxxxx. "Private" is just a name. The route table is the truth.
Open the subnet's route table. If the 0.0.0.0/0 target is an IGW, the subnet is public regardless of its name. Point 0.0.0.0/0 to a NAT gateway for private-with-egress, or remove the route entirely for truly isolated.
Security group rules look correct but the connection still times out
Port 5432 inbound is allowed in the SG. Lambda logs show a connection timeout every time.
A custom NACL on the subnet is stateless and lacks an outbound allow for ephemeral ports (1024–65535). The SYN reaches the destination; the SYN-ACK is discarded at the subnet boundary.
Add an outbound NACL rule allowing TCP 1024–65535 to 0.0.0.0/0, or audit whether the custom NACL is needed at all.
Referencing a security group across VPCs silently fails without an active peering connection
You copy a security group ID from VPC A into a rule in VPC B; the console rejects it or the rule has no effect.
A security group reference only resolves within the same VPC, or across an active VPC peering connection in the same Region (cross-account peers need the owner's account ID prefixed to the SG ID). There is no cross-Region security-group reference at all, even with peering — you use the peer VPC's CIDR block instead.
Create and activate the VPC peering connection first, then reference the peer's security group ID directly. For cross-Region traffic, or before peering exists, reference the peer VPC's CIDR, or use PrivateLink (Endpoints and private connectivity) for a producer/consumer service model instead of peering.
Tricky questions
No — security groups take precedence over NACLs.
It depends on direction. For traffic entering the subnet, the NACL is evaluated before the security group: a NACL DENY at a low rule number drops the packet before the SG ever sees it. For traffic leaving the subnet, the order reverses — the SG's outbound rules are checked first, then the NACL checks outbound at the subnet boundary. Neither layer has overall precedence; the order depends on which direction the packet is travelling.
Yes, no outbound rules means all outbound is blocked.
A newly created security group has a default outbound rule allowing all traffic. "No outbound rules" means you deleted that default. If you delete the default allow-all outbound rule and add nothing, all outbound is blocked — but in practice most SGs retain the default outbound allow-all.
Scenario question
You're handed a VPC with three subnets: public-alb, private-app, and private-db. A teammate says "I gave the app subnet a custom NACL so only port 8080 can enter from the ALB subnet." Traffic works in staging but timeouts appear randomly in production under load. What is happening and how do you fix it?
Model answer
I'd suspect the NACL's stateless nature immediately. Port 8080 inbound is allowed but the custom NACL probably has no outbound rule for the ephemeral return ports that the ALB's clients are using. Under low load most connections succeed before the timeout; under high load the NACL drops the bursts of SYN-ACKs returning to the ALB, and the ALB retries, compounding the problem. My fix is to add an outbound NACL rule allowing TCP 1024–65535 to the ALB subnet CIDR. The trade-off is slightly broader outbound exposure from private-app, which I'd accept because the ALB subnet is already controlled by its own SG; this is not a bypass, just closing the stateless gap.
Interview corner
CommonWhat is the difference between a security group and a network ACL?
TrickYou add a NACL rule to allow inbound TCP 443, but HTTPS still doesn't work. Why?
Experience checkWhen would you actually use a NACL in production?
CommonWhat makes a subnet public or private in AWS?
0.0.0.0/0 points to an internet gateway, traffic from the subnet can reach the internet and vice versa — that's public. If it points to a NAT gateway, traffic can initiate outbound but inbound from the internet can't reach it — that's private with egress. If there's no default route at all, the subnet is truly isolated.”Memory hook
A security group is a stateful bodyguard; a NACL is a stateless door list. Forget to update the list for return traffic and the conversation dies silently.
The route table — not the word "private" — is what makes a subnet private.
Self-check
Which layer evaluates first: a security group or a NACL?
It depends on direction. For traffic entering the subnet, the NACL evaluates first at the subnet boundary — if it drops the packet, the SG never sees it. For traffic leaving the subnet, the SG evaluates first at the instance, then the NACL checks outbound rules at the boundary.
Why must a custom NACL include rules for ephemeral ports (1024–65535)?
Because NACLs are stateless. Every TCP response uses a dynamically assigned port in the ephemeral range. Without an outbound allow for those ports, the reply is dropped even when the inbound rule is correct.
Is it possible for a security group to block traffic that a NACL allows?
Yes. The NACL allows traffic into the subnet, then the security group evaluates it at the instance. If the SG has no matching allow rule, the traffic is dropped before reaching the application.
What is the implicit rule at the end of every NACL?
A wildcard DENY
*at rule number 32767 — if no numbered rule matches, the packet is denied.
In the minimal code block above, replace /16 with /24 and change prefixlen_diff=2 to prefixlen_diff=1. Run it and paste the two-subnet output. Then write a function nacl_allows_range(rules, port_range) that returns True only if every port in range(port_range[0], port_range[1]+1) is allowed by the rules list (using the first-match logic from the production block). Test it with port_range=(1024, 1025) against the outbound_nacl from the production example and verify it returns False.