Shelf · AWS · Chapter 2

Chapter 2: Networking

Your ETL job on Jenkins doesn't just write to S3 — it lives inside a network that you never drew. This chapter makes that network visible so you can diagnose "why can't my Lambda reach Postgres?" before it burns a sprint.

Difficulty
Priority
aws-02-01Card 1 of 6

AWS network boundaries

MediumP3 · Role-dependentTalk about it13 min
vpccidrsubnetsroute-tablessecurity-groupsnaclstatefulstateless

Why you care

Every resource you launch in AWS lives inside a VPC, whether you chose one or not. The moment your Jenkins job's replacement becomes a Lambda or ECS task, you need to know which subnet it lands in, which route table governs its traffic, and whether a security group or a NACL is the thing blocking port 5432. Interviewers ask "what's the difference between a security group and a NACL?" because the answer predicts whether you'll spend three hours debugging a working rule that fires on outbound but forgets return traffic.

Explain like I'm 5

Think of a VPC as your own private office floor in a shared skyscraper. Subnets are rooms on that floor. The route table is the directory on the wall saying which corridor leads outside. A security group is a personal bodyguard assigned to each desk: it remembers every conversation that came in and automatically lets the reply out. A NACL is a checkpoint at the room door: it reads a numbered list of instructions top-to-bottom and has no memory — if your rule says "let port 443 in" but you forgot to add a rule for the response to go out, the conversation is silently cut.

Core concept

A VPC (Virtual Private Cloud) is a logically isolated virtual network in your AWS account. You give it a CIDR block — for example 10.0.0.0/16 — and carve it into subnets, each tied to one Availability Zone.

A subnet is public if its route table has a route sending 0.0.0.0/0 to an internet gateway. It is private if that route is absent or points to a NAT gateway instead. The label "private" on its own does not define reachability — the route table does.

Route tables contain destination/target pairs. Every subnet's traffic is evaluated top-to-bottom until a match: the most-specific CIDR wins. The local route (10.0.0.0/16 → local) is always implicit and cannot be deleted.

Security groups are stateful instance-level firewalls. Rules are allow-only; there is no explicit deny. Because they are stateful, return traffic for an allowed inbound connection is automatically permitted regardless of outbound rules. You can reference another security group as a source or destination, which is how you say "the app tier may reach the DB tier."

Network ACLs (NACLs) are stateless subnet-level checkpoints. They support both allow and deny rules, numbered 1–32766, evaluated in ascending order — first match wins. Because they are stateless, you must write explicit inbound and outbound rules for every conversation, including the ephemeral return-port range (1024–65535 for most TCP clients).

Decision rule: use security groups as your primary control; they are flexible, stateful, and can reference other groups. Add NACLs only as a second line of defence, for example to deny a specific IP range at the subnet boundary.

When not to rely on NACLs: never write a full stateless firewall policy in NACLs alone — the ephemeral port rule is easy to get wrong, and a missing outbound ALLOW breaks the connection silently.

Minimal example CIDR arithmetic in stdlib

PythonSubnet math with no AWS callsruns
import ipaddress

vpc = ipaddress.ip_network("10.0.0.0/16")
# Carve four /18 subnets
subnets = list(vpc.subnets(prefixlen_diff=2))
for s in subnets:
    print(f"{s}  hosts={s.num_addresses - 2}  first={s.network_address + 1}  last={s.broadcast_address - 1}")
Output
10.0.0.0/18  hosts=16382  first=10.0.0.1  last=10.0.63.254
10.0.64.0/18  hosts=16382  first=10.0.64.1  last=10.0.127.254
10.0.128.0/18  hosts=16382  first=10.0.128.1  last=10.0.191.254
10.0.192.0/18  hosts=16382  first=10.0.192.1  last=10.0.255.254

In production Stateful vs stateless rule evaluator

PythonSimulates SG (stateful) vs NACL (stateless) decision logicruns
from dataclasses import dataclass, field
from typing import Literal


@dataclass
class Rule:
    number: int          # for NACL ordering; ignored in SG
    action: Literal["allow", "deny"]
    port: int


def sg_allows(rules: list[Rule], port: int, is_return: bool) -> bool:
    """Security groups: stateful. Return traffic is always allowed."""
    if is_return:
        return True
    return any(r.action == "allow" and r.port == port for r in rules)


def nacl_allows(rules: list[Rule], port: int) -> bool:
    """NACLs: stateless. Rules evaluated in ascending number order, first match wins."""
    for rule in sorted(rules, key=lambda r: r.number):
        if rule.port == port:
            return rule.action == "allow"
    return False  # implicit deny if no rule matched


# Scenario: inbound rule allows 5432, no outbound rule for ephemeral port 49152
inbound = [Rule(100, "allow", 5432)]
outbound_nacl = [Rule(100, "allow", 443)]  # forgot ephemeral ports

# SG: inbound 5432 OK; return traffic automatically allowed
print("SG inbound 5432 :", sg_allows(inbound, 5432, False))
print("SG return port   :", sg_allows(inbound, 49152, True))

# NACL: same inbound, but return traffic on ephemeral port 49152 is missing
print("NACL inbound 5432:", nacl_allows(inbound, 5432))
print("NACL return 49152:", nacl_allows(outbound_nacl, 49152))
Output
SG inbound 5432 : True
SG return port   : True
NACL inbound 5432: True
NACL return 49152: False

See it happen Packet through VPC layers

InternetNACL (subnet boundary)Security Group (instance)EC2 / LambdaTCP SYN port 5432allowed (rule 100)allowedTCP SYN-ACK (return)passedDROPPED — no outbound rule
  1. Packet arrives at the subnet boundary. NACL evaluates inbound rules in number order.
  2. Rule 100 allows port 5432. NACL passes the packet inward.
  3. SG has an allow rule for port 5432. Traffic reaches the instance.
  4. Instance replies. SG is stateful: the return is automatically allowed.
  5. SG passes the return packet to the subnet boundary.
  6. NACL is stateless. Without an explicit outbound allow for the ephemeral port range 1024–65535, the reply is silently dropped. The caller sees a timeout.
Notice: the SG never blocked anything — the silent failure lives entirely in the NACL's missing outbound rule for ephemeral ports.
Scenario A

In your world

The situation

Your Jenkins-run Python ETL job is being migrated to a Lambda. A teammate puts it in subnet-private-a, creates a security group allowing port 5432 from the Lambda's SG, and deploys. Lambda logs show a timeout connecting to RDS on port 5432.

What actually breaks

The SG looks correct. The actual block is a custom NACL on the subnet that has an inbound ALLOW 5432 at rule 100 but no outbound rule for the ephemeral port range. The SYN reaches RDS; the SYN-ACK is dropped at the subnet boundary. Connection times out after ~30 s. The CloudWatch log shows the attempt but no error code — just a timeout.

What you'd do on Monday

Add an outbound NACL rule: number 900, allow TCP, ports 1024–65535, destination 0.0.0.0/0. This covers the ephemeral return range for TCP clients. Alternatively, if the NACL was the wrong tool, remove the custom NACL and let the default NACL (which allows all traffic) cover the subnet while the SG does the real work.

Scenario B

Imagine you work at Zomato

40k restaurant-facing API calls/min peak

The numbers

Illustrative: 40k req/min × ~2 TCP connections each (HTTP keep-alive not always reused across Lambda invocations) ≈ 80k active flows at peak. Each flow has a source ephemeral port in the 1024–65535 range. A NACL that only allows outbound on specific ports will drop the majority of return traffic silently.

What the Scenario-A answer does here

For a single-instance app the missing ephemeral-port rule is noticed in one five-minute test. At 80k concurrent flows the intermittent drop rate is 100% for any port outside the allowed range; CloudWatch shows thousands of Lambda timeouts per minute before anyone thinks of the NACL.

What flips at this scale

At scale, NACLs are used only for broad subnet-level deny rules (block a compromised CIDR, deny a known bad range). The fine-grained per-service rules belong entirely in security groups that reference each other. Every subnet gets a NACL with allow-all inbound and outbound except for explicit deny rules at low numbers; security groups carry the access policy.

The trade-off you'd defend

I would defend SG-only control with a minimal NACL (allow-all with one or two explicit denies) over complex NACL policies. Security groups are stateful, easier to audit, and their SG-reference model makes the data-tier boundary readable. The cost is that a misassigned SG lets traffic through; a dedicated-subnet NACL catches it, so a paranoid design uses both.

Illustrative request figures; not Zomato-published.

Gotchas

The subnet is labelled "private" but packets still reach the internet

Symptom

An EC2 instance in a "private" subnet can curl ifconfig.me and get a public IP back.

Cause

The route table has 0.0.0.0/0 → igw-xxxxxx. "Private" is just a name. The route table is the truth.

Fix

Open the subnet's route table. If the 0.0.0.0/0 target is an IGW, the subnet is public regardless of its name. Point 0.0.0.0/0 to a NAT gateway for private-with-egress, or remove the route entirely for truly isolated.

Security group rules look correct but the connection still times out

Symptom

Port 5432 inbound is allowed in the SG. Lambda logs show a connection timeout every time.

Cause

A custom NACL on the subnet is stateless and lacks an outbound allow for ephemeral ports (1024–65535). The SYN reaches the destination; the SYN-ACK is discarded at the subnet boundary.

Fix

Add an outbound NACL rule allowing TCP 1024–65535 to 0.0.0.0/0, or audit whether the custom NACL is needed at all.

Referencing a security group across VPCs silently fails without an active peering connection

Symptom

You copy a security group ID from VPC A into a rule in VPC B; the console rejects it or the rule has no effect.

Cause

A security group reference only resolves within the same VPC, or across an active VPC peering connection in the same Region (cross-account peers need the owner's account ID prefixed to the SG ID). There is no cross-Region security-group reference at all, even with peering — you use the peer VPC's CIDR block instead.

Fix

Create and activate the VPC peering connection first, then reference the peer's security group ID directly. For cross-Region traffic, or before peering exists, reference the peer VPC's CIDR, or use PrivateLink (Endpoints and private connectivity) for a producer/consumer service model instead of peering.

Tricky questions

Can a NACL deny a specific IP that a security group allows?
Most people say

No — security groups take precedence over NACLs.

Actually

It depends on direction. For traffic entering the subnet, the NACL is evaluated before the security group: a NACL DENY at a low rule number drops the packet before the SG ever sees it. For traffic leaving the subnet, the order reverses — the SG's outbound rules are checked first, then the NACL checks outbound at the subnet boundary. Neither layer has overall precedence; the order depends on which direction the packet is travelling.

Does a security group with no outbound rules block all outbound traffic?
Most people say

Yes, no outbound rules means all outbound is blocked.

Actually

A newly created security group has a default outbound rule allowing all traffic. "No outbound rules" means you deleted that default. If you delete the default allow-all outbound rule and add nothing, all outbound is blocked — but in practice most SGs retain the default outbound allow-all.

Scenario question

You're handed a VPC with three subnets: public-alb, private-app, and private-db. A teammate says "I gave the app subnet a custom NACL so only port 8080 can enter from the ALB subnet." Traffic works in staging but timeouts appear randomly in production under load. What is happening and how do you fix it?

Model answer

I'd suspect the NACL's stateless nature immediately. Port 8080 inbound is allowed but the custom NACL probably has no outbound rule for the ephemeral return ports that the ALB's clients are using. Under low load most connections succeed before the timeout; under high load the NACL drops the bursts of SYN-ACKs returning to the ALB, and the ALB retries, compounding the problem. My fix is to add an outbound NACL rule allowing TCP 1024–65535 to the ALB subnet CIDR. The trade-off is slightly broader outbound exposure from private-app, which I'd accept because the ALB subnet is already controlled by its own SG; this is not a bypass, just closing the stateless gap.

Interview corner

CommonWhat is the difference between a security group and a network ACL?
“I'd say the key difference is stateful versus stateless. A security group tracks connections, so return traffic is automatic. A NACL evaluates every packet independently, so you must write rules for both directions. Security groups operate at the instance level and allow only; NACLs operate at the subnet level and support both allow and deny.”
TrickYou add a NACL rule to allow inbound TCP 443, but HTTPS still doesn't work. Why?
“I'd check whether there's an outbound rule for the ephemeral port range. NACL is stateless, so the SYN-ACK going back to the client uses a random port in 1024–65535. Without that outbound allow, the reply is dropped.”
Experience checkWhen would you actually use a NACL in production?
“I'd use a NACL to explicitly deny a compromised CIDR range at the subnet level — something you want to block regardless of security group rules. For all normal access control I'd stay with security groups because they're stateful, easier to reason about, and you can scope rules to other SG IDs.”
CommonWhat makes a subnet public or private in AWS?
“I'd say it's purely the route table. If 0.0.0.0/0 points to an internet gateway, traffic from the subnet can reach the internet and vice versa — that's public. If it points to a NAT gateway, traffic can initiate outbound but inbound from the internet can't reach it — that's private with egress. If there's no default route at all, the subnet is truly isolated.”

Memory hook

A security group is a stateful bodyguard; a NACL is a stateless door list. Forget to update the list for return traffic and the conversation dies silently.

If you remember only one thing:

The route table — not the word "private" — is what makes a subnet private.

Self-check

  1. Which layer evaluates first: a security group or a NACL?

    It depends on direction. For traffic entering the subnet, the NACL evaluates first at the subnet boundary — if it drops the packet, the SG never sees it. For traffic leaving the subnet, the SG evaluates first at the instance, then the NACL checks outbound rules at the boundary.

  2. Why must a custom NACL include rules for ephemeral ports (1024–65535)?

    Because NACLs are stateless. Every TCP response uses a dynamically assigned port in the ephemeral range. Without an outbound allow for those ports, the reply is dropped even when the inbound rule is correct.

  3. Is it possible for a security group to block traffic that a NACL allows?

    Yes. The NACL allows traffic into the subnet, then the security group evaluates it at the instance. If the SG has no matching allow rule, the traffic is dropped before reaching the application.

  4. What is the implicit rule at the end of every NACL?

    A wildcard DENY * at rule number 32767 — if no numbered rule matches, the packet is denied.

Prove it

In the minimal code block above, replace /16 with /24 and change prefixlen_diff=2 to prefixlen_diff=1. Run it and paste the two-subnet output. Then write a function nacl_allows_range(rules, port_range) that returns True only if every port in range(port_range[0], port_range[1]+1) is allowed by the rules list (using the first-match logic from the production block). Test it with port_range=(1024, 1025) against the outbound_nacl from the production example and verify it returns False.

aws-02-02Card 2 of 6

IGW versus NAT and egress

MediumP3 · Role-dependentTalk about it11 min
igwnat-gatewayegresspublic-subnetprivate-subnetreachability

Why you care

"Why can my Lambda read from S3 but my EC2 instance in the private subnet can't pip-install anything?" That question almost always comes back to one decision: did you attach an internet gateway and does the route table point to it, or did you use a NAT gateway for one-way egress? Get this wrong in your Jenkins-to-Lambda migration and you spend a day debugging pip timeouts in a Lambda that can't reach PyPI. Interviewers ask about IGW vs NAT because it is the first architectural decision in every VPC and the most common source of "why can't this reach that?" in production.

Explain like I'm 5

Imagine your private office floor (the VPC). An internet gateway (IGW) is a revolving door: traffic can flow both ways — people from outside can walk in, and you can walk out. A NAT gateway is a one-way turnstile from inside to outside: you can exit to do errands, but nobody from outside can push the turnstile inward to reach you. If your EC2 instance is behind a NAT, it can download packages; it cannot receive unsolicited inbound connections from the internet.

Core concept

An internet gateway (IGW) is a horizontally scaled, redundant VPC component that allows bidirectional internet traffic. An IGW by itself does nothing; traffic only reaches the internet when the subnet's route table has 0.0.0.0/0 → igw-xxxxxxxx.

For a resource to be reachable from the internet it needs three things simultaneously:

  1. A public IPv4 address (Elastic IP or auto-assigned public IP).
  2. A route table with 0.0.0.0/0 → igw.
  3. A security group inbound rule allowing the traffic.

A NAT gateway is a managed service that allows resources in private subnets to initiate outbound connections to the internet while preventing unsolicited inbound connections. The NAT gateway itself lives in a public subnet and needs an Elastic IP. The private subnet's route table then points 0.0.0.0/0 → nat-xxxxxxxx. The NAT translates the private source IP to its own Elastic IP for outbound; return traffic is routed back to the originating instance.

Decision rule:

  • Use IGW when a resource must accept inbound connections from the internet (a load balancer, a bastion, a public API). Never expose a database or application server directly via IGW without a load balancer in front.
  • Use NAT when a resource in a private subnet needs to initiate outbound connections (pull packages, call external APIs, reach AWS services that have no VPC endpoint) but must not be reachable from the internet.
  • Use neither when the resource only needs to talk to other resources within the VPC or to AWS services via VPC endpoints (Endpoints and private connectivity).

Minimal example Route lookup: specific beats default

PythonThe most-specific CIDR wins, not list order or a "default" labelruns
import ipaddress

def next_hop(route_table: dict[str, str], destination: str) -> str:
    """Longest-prefix match. Returns the target, or 'no-route'."""
    dest = ipaddress.ip_address(destination)
    best = (-1, "no-route")
    for cidr, target in route_table.items():
        net = ipaddress.ip_network(cidr, strict=False)
        if dest in net and net.prefixlen > best[0]:
            best = (net.prefixlen, target)
    return best[1]

rt = {"10.0.0.0/16": "local", "0.0.0.0/0": "nat-def456"}
print(next_hop(rt, "203.0.113.10"))  # no /16 match, falls to the /0 default
print(next_hop(rt, "10.0.8.5"))      # /16 is more specific than /0 and wins
Output
nat-def456
local

In production Diagnosing reachability step by step

This mirrors the mental checklist you run when an instance or Lambda cannot reach an external endpoint. It takes the route already resolved by next_hop above and reasons about what else has to be true.

PythonReachability diagnostic: route + public IP + SG, all three or nothingruns
def diagnose_egress(hop: str, has_public_ip: bool, sg_outbound: list[int], port: int = 443) -> list[str]:
    """hop: output of next_hop() for the internet-bound destination."""
    results = [f"1. Route to internet: {hop}"]
    if hop == "no-route":
        results.append("   FAIL: no default route. Add an IGW or NAT route.")
        return results

    if "igw" in hop and not has_public_ip:
        results.append("   FAIL: route to IGW but no public IP assigned.")
    elif "igw" in hop:
        results.append("   OK: public subnet with public IP.")
    else:
        results.append("   OK: private subnet, outbound via NAT.")

    allowed = port in sg_outbound
    results.append(f"2. SG outbound port {port}: " + ("allowed" if allowed else "BLOCKED by security group"))
    return results


# Case 1: private subnet with NAT — should work
for line in diagnose_egress("nat-def456", has_public_ip=False, sg_outbound=[443, 80]):
    print(line)

print()

# Case 2: public subnet but no public IP — the Lambda-in-VPC trap
for line in diagnose_egress("igw-abc123", has_public_ip=False, sg_outbound=[443]):
    print(line)
Output
1. Route to internet: nat-def456
   OK: private subnet, outbound via NAT.
2. SG outbound port 443: allowed

1. Route to internet: igw-abc123
   FAIL: route to IGW but no public IP assigned.
2. SG outbound port 443: allowed
Scenario A

In your world

The situation

Your team migrates a Jenkins ETL job to a Lambda. The Lambda needs to call an external payment API over HTTPS and also pull from S3. A colleague puts the Lambda in subnet-private-a and sets the Lambda's SG to allow all outbound. The Lambda can read S3 fine but the external HTTPS call times out every run.

What actually breaks

S3 access works because it uses a Gateway VPC endpoint (Endpoints and private connectivity) that bypasses the internet entirely. The external payment API uses the internet. subnet-private-a's route table has 0.0.0.0/0 → igw-abc — it looks like a NAT route but it's actually an IGW. The Lambda has no public IP (Lambdas in a VPC never get a public IP automatically), so traffic hits the IGW and is dropped because the NAT translation step never happens. Over one nightly batch of 900 orders, all 900 payment-confirmation calls time out after 30 seconds each; the Lambda's own try/except only logs and continues, so the job's CloudWatch status is still SUCCESS.

What you'd do on Monday

Move the 0.0.0.0/0 route to a NAT gateway that lives in a public subnet. The Lambda now goes private-subnet → NAT → IGW → internet. Cost: NAT gateways are billed per hour and per GB processed; check the current pricing page before finalising this architecture for high-volume ETL.

Scenario B

Imagine you work at Ola

200k ride events/min

The numbers

Illustrative: 200k events/min triggering outbound API calls ≈ ~3.3k req/s. Each request flows through the NAT gateway. NAT gateways have a per-AZ throughput limit; check the current quota page for your Region before designing around a single NAT per AZ.

What the Scenario-A answer does here

The Lambda-in-VPC-with-NAT approach works at lower volume. At 200k events/min, if all Lambdas are in one AZ (e.g., due to RDS affinity), all NAT traffic funnels through one gateway. This is a throughput and cost risk, not a hard wall, but the per-GB NAT charge adds up.

What flips at this scale

At this scale the primary pattern is: use Gateway endpoints for S3 and DynamoDB (no NAT charge, no throughput limit on the path), use Interface endpoints for other AWS services like SQS and Secrets Manager, and reserve the NAT gateway only for genuine third-party internet calls. Reduce NAT-bound traffic to the minimum required by business logic.

The trade-off you'd defend

I would defend per-service endpoint adoption over a single large NAT gateway, trading more endpoint configuration overhead for lower data-transfer cost and eliminating the NAT as a bottleneck. For truly external calls (third-party APIs), one NAT gateway per AZ with proper load distribution is unavoidable.

Illustrative numbers; not Ola-published.

Gotchas

Lambda in a VPC can't reach the internet despite a security group allowing all outbound

Symptom

Lambda times out on every external HTTP call. No error message, just a timeout after ~30 s.

Cause

The Lambda is in a private subnet whose route table has no 0.0.0.0/0 route, or the route points to an IGW. Lambdas in a VPC never receive a public IP, so IGW-routed traffic is dropped.

Fix

The private subnet needs 0.0.0.0/0 → nat-xxxxxxxx pointing to a NAT gateway in a public subnet.

NAT gateway created in a private subnet

Symptom

The NAT gateway is created, the route is updated, but outbound internet access still fails.

Cause

The NAT gateway itself requires a public subnet (one with an IGW route) and an Elastic IP.

Fix

Delete the misplaced NAT gateway and recreate it in a subnet whose route table has 0.0.0.0/0 → igw-xxxxxxxx.

Tricky questions

If both a public IP and an IGW exist, does the resource automatically get internet access?
Most people say

Yes, a public IP plus an IGW is enough.

Actually

Three things are required together: a public IP, a route 0.0.0.0/0 → igw, and a security group inbound rule allowing the traffic. Any one missing means the connection fails.

Can a NAT gateway serve multiple AZs?
Most people say

Yes, one NAT gateway covers the whole VPC.

Actually

A NAT gateway is AZ-scoped. For high-availability egress you deploy one NAT gateway per AZ and point each AZ's private route table to its own NAT gateway. Using a single NAT gateway introduces a single point of failure for all private subnet egress.

Scenario question

You're handed a Flask API on an EC2 instance that cannot respond to incoming requests, even though port 80 is open in the security group. The VPC has an IGW attached. What do you check and in what order?

Model answer

I'd check three things in order. First, the subnet's route table: does 0.0.0.0/0 point to the IGW? If not, the packet never leaves the VPC boundary. Second, the EC2 instance: does it have a public IP or Elastic IP assigned? Without one, the IGW cannot forward inbound traffic to it. Third, the security group: is there an inbound rule for port 80 from 0.0.0.0/0? All three must be true simultaneously. The trade-off between assigning a public IP directly versus putting an ALB in front is latency (a few ms) versus the operational benefit of TLS termination, path routing, and not exposing the instance IP publicly — I'd favour the ALB for any production workload.

Interview corner

CommonWhat is the difference between an internet gateway and a NAT gateway?
“I'd say an internet gateway enables bidirectional internet traffic — things can come in and go out. A NAT gateway is one-directional: private resources can initiate outbound connections, but nothing from the internet can initiate a connection inward. The NAT also needs to live in a public subnet with its own Elastic IP.”
Experience checkYour Lambda can read S3 but can't reach an external API. What do you check?
“I'd look at the route table first. Lambda in a VPC has no public IP, so I need 0.0.0.0/0 pointing to a NAT gateway, not an IGW. S3 working tells me the Lambda is in the VPC correctly, probably via a Gateway endpoint; the external call failing tells me the NAT path is broken.”
TrickA NAT gateway is AZ-redundant, right?
“I'd push back on that — a NAT gateway is scoped to one AZ. For HA egress you need one per AZ, and each AZ's private route table points to its own NAT gateway. One NAT gateway is a single point of failure for all private egress in that VPC.”

Memory hook

IGW = revolving door (both ways). NAT = one-way turnstile (inside out only, never outside in). Lambda has no public IP, so it always needs the turnstile.

If you remember only one thing:

If it's in a private subnet and needs the internet, it needs a NAT gateway, not an IGW.

Self-check

  1. Why does a Lambda in a VPC fail to reach the internet even if its SG allows all outbound?

    Lambdas in a VPC get no public IP. An IGW route requires a public IP. The Lambda needs 0.0.0.0/0 → NAT gateway in its subnet's route table.

  2. What are the three things an EC2 instance needs to receive inbound internet traffic?

    A public/Elastic IP, a route 0.0.0.0/0 → igw in its subnet, and an inbound SG rule allowing the traffic.

  3. Why deploy one NAT gateway per AZ rather than one per VPC?

    A NAT gateway is AZ-scoped. If the single NAT gateway's AZ goes down, all private-subnet egress in the VPC fails.

Prove it

Using the next_hop function from the minimal block, build a route table that represents: a VPC 10.0.0.0/16, a more-specific corporate CIDR 10.0.128.0/17 routed to vpn-gateway-1, and 0.0.0.0/0 to nat-abc. Call it with three IPs: one in 10.0.200.0, one in 10.0.50.0, and 52.94.5.1 (an Amazon IP). Verify which route wins for each, and paste the three-line output.

aws-02-03Card 3 of 6

ALB, NLB and target groups

MediumP3 · Role-dependentTalk about it10 min
albnlbtarget-groupshealth-checksl7l4listeners

Why you care

"Should I use an ALB or an NLB?" is the first question when any service needs to handle traffic from multiple clients. Get it wrong and you either lose HTTP-level routing (path-based routing to /api vs /payments), or you add unnecessary TLS termination overhead to a raw TCP service. For a Python ETL developer moving to microservices, the ALB is almost always the answer — until the day you need sub-millisecond TCP latency or a static IP for an allowlisted third-party partner, and then NLB is the answer.

Explain like I'm 5

An ALB is a smart receptionist who reads the entire letter (including the address, subject line, and body) before deciding which desk to send it to. An NLB is a high-speed post-room sorter that only reads the envelope's destination port and forwards instantly — no reading the letter, no buffering, just fast forwarding at Layer 4.

Core concept

An Application Load Balancer (ALB) operates at OSI Layer 7 (HTTP/HTTPS/gRPC). It terminates the TLS connection, reads the full HTTP request, and evaluates listener rules in priority order. Rules can match on path, hostname, HTTP header, query string, source IP, or HTTP method. Each rule action forwards to a target group, which is a pool of registered targets — EC2 instances, ECS tasks, Lambda functions, or IP addresses.

A Network Load Balancer (NLB) operates at OSI Layer 4 (TCP/UDP/TLS). It forwards packets without inspecting the application payload. It preserves the client's source IP by default and can handle millions of requests per second with microsecond latency. NLB supports static Elastic IPs per AZ, which is useful when a third-party partner requires IP allowlisting.

Target groups decouple the load balancer from the backends. Health checks are configured per target group: protocol, path (for HTTP), port, healthy/unhealthy threshold, and interval. A target that fails health checks is removed from rotation until it recovers.

Decision rule:

  • Use ALB when you need HTTP/HTTPS routing, path-based or host-based routing, WAF integration, authentication via Cognito or OIDC, or WebSocket support.
  • Use NLB when you need ultra-low latency, static IPs, TCP/UDP passthrough, or TLS offload without HTTP inspection.
  • NLB does not support Lambda targets. ALB does.
  • If you are routing to a container service in ECS, an ALB is the default choice; ECS can register dynamic port assignments directly with the target group.

Minimal example Priority number wins, not list order

PythonA lower priority number always wins, even listed after a broader ruleruns
rules = [(10, "/api", "default-tg"), (2, "/api/users", "users-tg")]

def alb_match(rules, path):
    matched = [r for r in rules if path.startswith(r[1])]
    return sorted(matched, key=lambda r: r[0])[0][2] if matched else "404"

print(alb_match(rules, "/api/users"))  # users-tg: priority 2 beats priority 10
print(alb_match(rules, "/api/other"))  # default-tg: only the broad rule matches
Output
users-tg
default-tg

In production Health check tracking per target group

PythonTracks health state transitions across pollsruns
from dataclasses import dataclass, field


@dataclass
class Target:
    id: str
    healthy_threshold: int = 2
    unhealthy_threshold: int = 3
    _consecutive_healthy: int = field(default=0, repr=False)
    _consecutive_unhealthy: int = field(default=0, repr=False)
    state: str = field(default="healthy", repr=True)

    def record_check(self, success: bool) -> str:
        if success:
            self._consecutive_unhealthy = 0
            self._consecutive_healthy += 1
            if self._consecutive_healthy >= self.healthy_threshold:
                self.state = "healthy"
        else:
            self._consecutive_healthy = 0
            self._consecutive_unhealthy += 1
            if self._consecutive_unhealthy >= self.unhealthy_threshold:
                self.state = "unhealthy"
        return self.state


# Simulate: target starts healthy, fails three checks, then recovers
t = Target("i-0abc1234")
sequence = [True, True, False, False, False, True, True]
for ok in sequence:
    state = t.record_check(ok)
    mark = "PASS" if ok else "FAIL"
    print(f"check={mark}  state={state}")
Output
check=PASS  state=healthy
check=PASS  state=healthy
check=FAIL  state=healthy
check=FAIL  state=healthy
check=FAIL  state=unhealthy
check=PASS  state=unhealthy
check=PASS  state=healthy

See it happen Request flow through ALB

HTTPS request/payments/*/api/*health check FAILClientALB(L7, HTTPS:443)payments-tg(3 targets)app-tg(5 targets)unhealthy target
  1. Client sends an HTTPS request. ALB terminates TLS and reads the HTTP headers.
  2. ALB evaluates listener rules in priority order. Path starts with /payments, so rule 1 matches.
  3. ALB forwards to payments-tg. One target is failing health checks and is out of rotation.
  4. ALB routes only to the two healthy targets in payments-tg. The client sees no error.
Notice: the ALB, not the client, handles TLS and routing logic; the client only sees one endpoint.
Scenario A

In your world

The situation

Your ETL job produces settlement files and writes to S3. A new requirement: expose a REST endpoint so the finance team's dashboard can query current-day totals. Your colleague suggests putting an NLB in front of the Flask app.

What actually breaks

An NLB passes raw TCP traffic. Your Flask app uses path-based routing (/api/totals vs /api/status). The NLB has no concept of HTTP paths — it forwards every TCP connection to a target and lets the app handle routing itself. That's fine for one service, but the moment you add a second microservice (/api/audit) you need two NLBs or the app becomes a monolithic router. TLS certificates cannot be managed at the NLB-HTTP layer without a separate terminator. WAF rules cannot be attached to an NLB.

What you'd do on Monday

Use an ALB with two listener rules: priority 1 for /api/totals → totals-tg, priority 2 for /api/audit → audit-tg. Both services scale independently. TLS is terminated at the ALB. Adding a WAF rule to block unauthenticated requests is a two-click operation.

Scenario B

Imagine you work at Atlassian

3M Jira API requests/hr

The numbers

Illustrative: 3M req/hr ≈ 833 req/s. With 20 routes (projects × environments × REST versions), ALB path routing replaces 20 separate public endpoints with one. Each unhealthy target is auto-removed within unhealthy_threshold × interval seconds — check the current ELB documentation for default values.

What the Scenario-A answer does here

An NLB in front of a monolithic app can handle 833 req/s trivially. The problem is operational: every new service or version needs a new NLB (or the monolith grows a new route), and there is no WAF, no auth offload, no per-path metrics. Debugging "why did /billing go down but not /users?" requires tailing app logs; the ALB surfaces it as a target group health metric.

What flips at this scale

ALB per service cluster, with one DNS name (api.atlassian.example.com) resolved via Route 53 (Route 53 and failover routing). Each team owns their target group. Canary deploys use weighted routing across two target groups for the same ALB path. Operations owns one load balancer, not fifty.

The trade-off you'd defend

I would defend one ALB with multiple target groups over many NLBs, trading the slightly higher per-ALB cost for centralised TLS, WAF, auth, and per-path observability. NLB is reserved for the cases that genuinely need static IPs or sub-millisecond TCP passthrough.

Illustrative request figures; not Atlassian-published.

Gotchas

ALB health check passes but traffic still goes to a stopped instance

Symptom

An instance shows "healthy" in the target group for up to 30 seconds after it stops serving. Some requests get Connection refused.

Cause

The health check interval and unhealthy threshold mean there is a window between the last passing check and the next failing one. During that window the target is still in rotation.

Fix

Lower the health check interval and unhealthy threshold (at the cost of faster deregistration noise from momentary blips). For deployments, use connection draining (deregistration delay) to gracefully remove a target before stopping it.

Path-based routing order matters

Symptom

All requests go to the catch-all target group even when a more-specific path rule exists.

Cause

Rules are evaluated in priority order. A low-priority number wins. If the catch-all is priority 1 and the specific path is priority 2, the catch-all always fires first.

Fix

Assign lower priority numbers (higher priority) to more-specific rules. The most specific rules should have the smallest numbers.

Tricky questions

Does NLB support Lambda targets?
Most people say

Yes — NLB supports EC2, ECS, IP, and Lambda.

Actually

NLB supports EC2 instances, ECS tasks, and IP addresses. Lambda is only supported as a target by ALB. This is one reason ALB is the default for serverless backends.

Do ALB health checks use the same port as the listener?
Most people say

Yes, the ALB probes the same port the traffic arrives on.

Actually

The health check port is configured separately on the target group. You can route traffic on port 443 and have the health check probe port 8080 on the internal instance.

Scenario question

You're handed a monolithic Flask API deployed behind a single NLB. The team wants to extract a /payments service into a separate deployment without changing the client-facing URL or breaking existing /api paths. What is your load balancer strategy?

Model answer

I'd replace the NLB with an ALB, which gives me path-based listener rules without changing the DNS name. I'd create two target groups: payments-tg for the new service, api-tg for the existing monolith. Rule 1 (priority 1): if path starts with /payments, forward to payments-tg. Rule 2 (priority 2): all other paths forward to api-tg. The monolith's /payments route can remain as a passthrough initially during cut-over, then be removed. The trade-off is the small ALB cost increase versus the NLB; I'd accept it for the routing flexibility and WAF integration.

Interview corner

CommonWhen do you choose ALB over NLB?
“I'd choose ALB whenever I need HTTP routing — path or host based — or TLS termination with certificate management, or WAF integration. NLB is for when I need static IPs, TLS passthrough, or genuinely sub-millisecond TCP latency. For most Python API services, ALB is the right default.”
Experience checkHow does a target group health check differ from a load balancer listener?
“I'd separate the two: the listener is what the client connects to — it defines protocol and port on the load balancer side. The health check is a separate probe the load balancer sends to each registered target. They can use different ports and paths. The target group's unhealthy threshold determines how many consecutive failures before a target is removed from rotation.”
DesignHow would you implement a canary deploy with an ALB?
“I'd create two target groups for the same path — one for the stable version, one for the canary. Then I'd use weighted forwarding on the ALB listener rule: 90% to stable-tg, 10% to canary-tg. After validating metrics I'd shift weight to 100% canary and deregister the stable targets. No DNS change, no new load balancer.”

Memory hook

ALB reads the letter (L7); NLB forwards the envelope (L4). If you need to route by /path or attach a WAF, you need ALB.

If you remember only one thing:

Use ALB for HTTP services. Use NLB for static IPs or raw TCP/UDP.

Self-check

  1. Can you attach WAF rules to an NLB?

    No. WAF integrates only with ALB (and CloudFront and API Gateway). NLB operates at Layer 4 and has no HTTP context for WAF rules.

  2. What is the default ALB routing algorithm?

    Round robin. Least outstanding requests is available as an alternative per target group.

  3. Why might you configure the health check on a different port than the traffic port?

    The application might not serve its health endpoint on the same port as production traffic (e.g., traffic on 443, health check on 8080). Separating them also prevents a broken health path from accidentally affecting traffic throughput.

Prove it

Extend alb_match to accept a headers dict and a rule tuple with a fourth field, required_header. Add two rules for path /api: priority 1 requires X-Internal: true, priority 2 has no header requirement. Call alb_match twice — once with headers={"X-Internal": "true"}, once with headers={} — and paste the two lines of output showing each call landing on a different target group.

aws-02-04Card 4 of 6

Endpoints and private connectivity

MediumP4 · DifferentiatorTalk about it11 min
vpc-endpointsgateway-endpointinterface-endpointprivatelinkvpndirect-connect

Why you care

Every time your Lambda or ECS task calls s3.put_object() inside a VPC, that traffic by default goes out through a NAT gateway and back in through the public S3 endpoint, costing you NAT data-processing charges per GB and routing traffic over the internet when neither party is on the internet. VPC endpoints short-circuit that path. Once you understand endpoints, you can design a private network with no internet gateway and still reach dozens of AWS services. Interviewers who care about cost and security ask about endpoints because most junior engineers don't know they exist.

Explain like I'm 5

Your office floor (VPC) needs to receive packages from the AWS warehouse (S3). Without an endpoint, the postman takes your package out of the building, drives on public roads, and delivers it back to your floor entrance. A Gateway endpoint is a private tunnel from your floor's mail room directly to the warehouse — no public roads, no NAT toll. An Interface endpoint puts a phone on your desk with a private number that rings directly into the AWS service's private exchange — same idea, but for services that work over TCP rather than route tables.

Core concept

There are two types of VPC endpoints:

Gateway endpoints work for S3 and DynamoDB only. You add the endpoint to a route table; the managed prefix list for the service is automatically added as a route pointing to the gateway. Traffic that matches the S3 or DynamoDB IP ranges is routed through the AWS private network instead of going to the internet. Gateway endpoints have no hourly cost (check current pricing — as of the AWS documentation there is no charge for gateway endpoints themselves, though check the current pricing page). No ENI is created; no DNS change is required.

Interface endpoints (AWS PrivateLink) work for most other AWS services: SQS, SNS, Secrets Manager, CloudWatch, EC2 API, KMS, and many more. Creating an interface endpoint deploys an Elastic Network Interface (ENI) with a private IP address in your subnet. DNS is updated so that service calls from within the VPC resolve to the private ENI IP instead of the public endpoint. Traffic never leaves the AWS network. Interface endpoints have an hourly charge per AZ and a per-GB data-processing charge — check the current pricing page.

Endpoint policies are IAM resource policies attached to the endpoint. They control which principals can use the endpoint to call which actions. The default policy allows all principals to call all actions. A restrictive endpoint policy is a useful way to ensure that, for example, only your account's S3 buckets are reachable through the endpoint, blocking data exfiltration to S3 buckets in other accounts.

Private connectivity choices at a glance:

OptionUse caseTraffic pathCost profile
Gateway endpointS3, DynamoDB from within a VPCPrivate, no ENINo hourly or per-GB charge (check current pricing)
Interface endpoint (PrivateLink)Most other AWS services, or cross-account SaaS/custom servicesPrivate, ENI per AZHourly per AZ, plus per-GB processed
Site-to-Site VPNOn-premises to VPC over the internetEncrypted, over the public internetHourly, plus data transfer; no dedicated circuit
Direct ConnectOn-premises to VPC, high and predictable bandwidthDedicated private link, no internetPort-hour plus data; highest fixed cost, most predictable throughput

The right on-premises choice depends on bandwidth requirements and tolerance for shared internet paths — check the current AWS documentation for bandwidth tiers and pricing before a purchase.

Minimal example A Resource ARN restricts even an "Allow *" action

PythonAn endpoint policy's Resource still scopes an otherwise-open Actionruns
policy = {"Statement": [
    {"Effect": "Allow", "Action": "s3:*", "Resource": "arn:aws:s3:::my-etl-bucket/*"},
]}

def allowed(policy: dict, resource: str, action: str = "s3:GetObject") -> bool:
    for stmt in policy["Statement"]:
        prefix = stmt["Resource"][:-1]  # strip trailing '*'
        if resource.startswith(prefix) and stmt["Action"] in (action, "s3:*"):
            return stmt["Effect"] == "Allow"
    return False

print(allowed(policy, "arn:aws:s3:::my-etl-bucket/data.csv"))       # True
print(allowed(policy, "arn:aws:s3:::other-account-bucket/data.csv"))  # False
Output
True
False

In production Route-table inspection for endpoint coverage

PythonChecks which services go through endpoints vs internetruns
def classify_routes(route_table: list[dict]) -> dict:
    """
    Each route: {"destination": cidr_or_prefix, "target": target_id}
    Returns classification: endpoint / nat / igw / local / other
    """
    result = {}
    for r in route_table:
        t = r["target"]
        if t.startswith("vpce-"):
            kind = "endpoint"
        elif t.startswith("nat-"):
            kind = "nat"
        elif t.startswith("igw-"):
            kind = "igw"
        elif t == "local":
            kind = "local"
        else:
            kind = "other"
        result[r["destination"]] = {"target": t, "kind": kind}
    return result


rt = [
    {"destination": "10.0.0.0/16",            "target": "local"},
    {"destination": "pl-63a5400a",             "target": "vpce-0a1b2c3d"},   # S3 managed prefix list
    {"destination": "0.0.0.0/0",              "target": "nat-0f1e2d3c"},
]

classified = classify_routes(rt)
for dest, info in classified.items():
    print(f"{dest:30s}  {info['kind']:10s}  {info['target']}")

internet_routes = [d for d, i in classified.items() if i["kind"] in ("nat", "igw")]
print(f"\nRoutes using internet/NAT: {internet_routes}")
Output
10.0.0.0/16                     local       local
pl-63a5400a                     endpoint    vpce-0a1b2c3d
0.0.0.0/0                       nat         nat-0f1e2d3c

Routes using internet/NAT: ['0.0.0.0/0']
Scenario A

In your world

The situation

Your nightly ETL Lambda runs in subnet-private-a, reads source data from an external FTP (internet), transforms it, and writes results to S3. The monthly cost report shows a significant NAT gateway data-processing charge.

What actually breaks

Every byte written to S3 is going through the NAT gateway: Lambda → NAT → internet → S3. S3 is owned by AWS and physically colocated in the same Region. The cost report shows 640 GB processed through the NAT gateway last month — all of it S3 traffic that never needed to leave the AWS network, billed at the NAT's per-GB data-processing rate on top of the hourly charge (check the current pricing page for the exact figure). There is no data-exfiltration benefit from sending your own account's data to S3 through the internet.

What you'd do on Monday

Add a Gateway VPC endpoint for S3 to the private subnet's route table. S3 SDK calls from the Lambda will now route directly through the AWS private network. No code change; the SDK resolves the same S3 DNS name. Only the FTP call still goes through the NAT gateway (because FTP is a third-party internet service with no VPC endpoint). Check the current pricing page for S3 gateway endpoint cost — gateway endpoints for S3 have no hourly or per-GB charge in the documented pricing.

Scenario B

Imagine you work at Slack

5 billion messages stored/day

The numbers

Illustrative: if even 1% of 5B daily messages trigger an S3 write in a VPC-connected Lambda, that is 50M S3 calls/day through NAT. At illustrative NAT data charges of even fractions of a cent per GB, the accumulated bytes from metadata payloads add up. Exact NAT pricing: check the current AWS pricing page for your Region.

What the Scenario-A answer does here

For a small ETL job, 1 GB/night through NAT is negligible. For a messaging platform's metadata pipeline at 50M calls/day, the NAT becomes a throughput bottleneck (NAT gateways have per-AZ throughput limits — check the current quota page) and a meaningful cost line item.

What flips at this scale

At platform scale every internal AWS service call (S3, SQS, Secrets Manager, CloudWatch, KMS) gets a VPC endpoint. The NAT gateway is sized only for genuine internet egress. The architectural rule becomes: "if it's an AWS service, it has an endpoint; if it's the internet, it uses NAT."

The trade-off you'd defend

I would defend Interface endpoints for SQS and Secrets Manager alongside the Gateway endpoint for S3, trading the per-AZ hourly endpoint cost for zero NAT charges on high-frequency service calls. For low-frequency service calls (once per Lambda cold start), the NAT path is acceptable.

Illustrative numbers; not Slack-published.

Gotchas

The S3 Gateway endpoint is configured but calls still go through NAT

Symptom

Cost Explorer still shows NAT data-processing charges for S3 calls.

Cause

The Gateway endpoint was added to a different subnet's route table, not the one the Lambda/ECS task is in. Or the endpoint was created in a different Region than the bucket.

Fix

Check every private subnet's route table and verify the S3 managed prefix list route points to the gateway endpoint. Gateway endpoints are Region-specific and apply only to buckets in the same Region.

Interface endpoint created but DNS does not resolve to private IP

Symptom

SQS SDK calls still go to the public endpoint even after the interface endpoint is created.

Cause

"Enable Private DNS Name" was not checked when creating the endpoint, or the VPC does not have DNS hostnames and DNS resolution enabled.

Fix

Ensure the VPC has both "Enable DNS hostnames" and "Enable DNS resolution" turned on, and that the interface endpoint has Private DNS enabled.

Tricky questions

Does a Gateway endpoint for S3 work across Regions?
Most people say

Yes, one endpoint routes to S3 anywhere.

Actually

A Gateway endpoint is Region-specific. It routes traffic to S3 buckets in the same Region only. For cross-Region bucket access, traffic still goes through the NAT or internet gateway.

Is PrivateLink the same as a VPC Gateway endpoint?
Most people say

Yes, PrivateLink is just the marketing name for VPC endpoints.

Actually

Gateway endpoints (for S3 and DynamoDB) do not use PrivateLink. They work by inserting routes into the route table. PrivateLink powers Interface endpoints, which deploy ENIs and support TCP-based services. The two mechanisms are architecturally different.

Scenario question

You're handed a Python Lambda that calls S3, SQS, and an external REST API. The Lambda is in a private subnet with a NAT gateway. The team wants to reduce the NAT data costs. What endpoints would you add, and which calls still need the NAT?

Model answer

I'd add a Gateway endpoint for S3 and an Interface endpoint for SQS. Both are AWS services; their traffic can flow through the private AWS network without touching the NAT. The external REST API is a genuine internet call with no VPC endpoint, so it still needs the NAT gateway. After the change, the NAT carries only the external API traffic. The trade-off is the hourly cost of the SQS Interface endpoint per AZ; if the Lambda calls SQS only a few times per day, the Interface endpoint cost may exceed the saved NAT cost — I'd check the maths with actual call volumes before committing.

Interview corner

CommonWhat are the two types of VPC endpoints and when do you use each?
“Gateway endpoints work for S3 and DynamoDB; they add a route to the route table and have no ENI. Interface endpoints use PrivateLink; they deploy a private IP in your subnet and work for most other AWS services like SQS, Secrets Manager, and CloudWatch. I use a Gateway endpoint first for S3 and DynamoDB; Interface endpoints come in when I want to eliminate all NAT traffic for services I call at high volume.”
Experience checkYou find NAT charges on S3 traffic. What do you do?
“I'd add a Gateway VPC endpoint for S3 to the private subnet's route table. It requires no code change and no ENI. S3 calls then go through the AWS private network. For any remaining NAT charges I'd check what other AWS services are being called without an Interface endpoint.”
TrickCan an endpoint policy substitute for a bucket policy?
“No — an endpoint policy and a bucket policy are evaluated independently. Both must allow the action. An endpoint policy restricts what can be called through the endpoint; the bucket policy restricts who can access the bucket. I use a restrictive endpoint policy to block access to other accounts' S3 buckets through my endpoint, as a data-exfiltration control.”

Memory hook

Gateway endpoint = private back door to S3 and DynamoDB. Interface endpoint = private phone line to everything else. Neither has anything to do with the internet.

If you remember only one thing:

Add a Gateway endpoint for S3 and DynamoDB first; your ETL job's biggest NAT bill disappears immediately.

Self-check

  1. Which two AWS services support Gateway endpoints?

    S3 and DynamoDB.

  2. What VPC setting must be enabled for Interface endpoint Private DNS to work?

    The VPC must have both "Enable DNS hostnames" and "Enable DNS resolution" enabled.

  3. Does a Gateway endpoint for S3 cover cross-Region buckets?

    No. A Gateway endpoint is Region-specific and only covers S3 in the same Region.

Prove it

Using the allowed function from the minimal block, add a second statement with "Effect": "Deny" scoped to arn:aws:s3:::my-etl-bucket/secrets/*, and rewrite allowed to check Deny statements before Allow statements so a deny always wins regardless of list order. Test with arn:aws:s3:::my-etl-bucket/secrets/db.env and arn:aws:s3:::my-etl-bucket/data.csv, and paste the two-line output showing the deny taking effect only on the first.

aws-02-05Card 5 of 6

Route 53 and failover routing

MediumP3 · Role-dependentTalk about it11 min
route-53dnsrouting-policiesfailoverhealth-checksttl

Why you care

Every public URL your service exposes is resolved by Route 53 before a single packet reaches your VPC. Understanding how TTL affects cache staleness, how failover routing reacts to health check failures, and — critically — that Route 53 failover routes traffic but does not restore data or fix your database, prevents a whole class of "the DNS switched but users still see errors" incidents. Interviewers ask about Route 53 routing policies because a wrong answer ("just use latency routing everywhere") reveals someone who doesn't understand what DNS can and can't recover from.

Explain like I'm 5

Think of Route 53 as the phone book for the internet. When a customer types api.yourcompany.com, their browser calls Route 53 and asks "what IP should I dial?" Route 53 checks its routing policy — is this simple (one answer), weighted (split the calls), failover (is the primary still alive?), or latency (who is fastest?) — and returns an answer. That answer is cached by the caller's DNS resolver for as long as the TTL says. After that, the caller asks again. The phone book can change to a new number, but anyone who already dialled the old number and is mid-conversation stays on the old connection — DNS failover switches future callers, not existing ones.

Core concept

Route 53 DNS records have a type (A, AAAA, CNAME, ALIAS, etc.) and a TTL (time to live in seconds). TTL controls how long resolvers cache the answer. Lowering TTL before a planned change speeds up propagation; raising it reduces resolver load but slows change propagation.

Routing policies:

PolicyUse caseWhat Route 53 returns
SimpleSingle resourceThe configured IP/CNAME
WeightedA/B testing, gradual rolloutOne answer, sampled by weight
LatencyRoute to lowest-latency RegionThe Region with lowest measured latency from the resolver
FailoverActive-passive HAPrimary if healthy; secondary if primary fails
GeolocationRegulatory or regional routingAnswer for the user's country/continent
Multivalue answerLightweight load distributionUp to 8 healthy IPs at random
IP-basedISP or CIDR-based routingAnswer based on source IP CIDR

Health checks can monitor an endpoint (HTTP/HTTPS/TCP), a CloudWatch alarm, or the health of other health checks. Failover routing changes the DNS response when the primary health check fails. The latency between a health check failure and a resolver cache expiry (TTL) is the gap where users may still hit the unhealthy endpoint.

Critical limit: DNS failover switches future DNS lookups to the secondary endpoint. It does not:

  • Restore a database or repair data.
  • Terminate existing TCP connections to the primary.
  • guarantee that all resolvers switch within TTL seconds (some resolvers ignore TTL).

Minimal example Worst case is old TTL plus new TTL, not just the new one

PythonTTL change propagation window (stdlib)runs
def propagation_window(old_ttl_s: int, new_ttl_s: int) -> int:
    """Worst case: a resolver just cached under the old TTL right before the change."""
    return old_ttl_s + new_ttl_s

print(propagation_window(3600, 300))  # 3900s, not the 300s a new TTL of 300 implies
Output
3900

In production Failover routing state machine

PythonModels active-passive failover record stateruns
from dataclasses import dataclass


@dataclass
class HealthCheck:
    endpoint: str
    threshold: int = 3          # consecutive failures before marked unhealthy
    _failures: int = 0

    def check(self, success: bool) -> str:
        if success:
            self._failures = 0
            return "healthy"
        self._failures += 1
        return "unhealthy" if self._failures >= self.threshold else "degraded"


def resolve_failover(primary_hc: HealthCheck, primary_ip: str,
                     secondary_ip: str, check_result: bool) -> tuple[str, str]:
    """Returns (resolved_ip, reason)."""
    state = primary_hc.check(check_result)
    if state == "unhealthy":
        return secondary_ip, f"primary unhealthy after {primary_hc._failures} failures"
    return primary_ip, f"primary {state}"


hc = HealthCheck("https://api.example.com/health")
primary = "203.0.113.10"
secondary = "203.0.113.20"

sequence = [True, True, False, False, False, True]
for ok in sequence:
    ip, reason = resolve_failover(hc, primary, secondary, ok)
    mark = "OK  " if ok else "FAIL"
    print(f"check={mark}  resolved={ip}  ({reason})")
Output
check=OK    resolved=203.0.113.10  (primary healthy)
check=OK    resolved=203.0.113.10  (primary healthy)
check=FAIL  resolved=203.0.113.10  (primary degraded)
check=FAIL  resolved=203.0.113.10  (primary degraded)
check=FAIL  resolved=203.0.113.20  (primary unhealthy after 3 failures)
check=OK    resolved=203.0.113.10  (primary healthy)
Scenario A

In your world

The situation

Your ETL results are served by a Flask API on EC2 in us-east-1. Route 53 has a simple routing record pointing api.company.com to the EC2 Elastic IP, TTL 300. The instance goes down at 14:00.

What actually breaks

The TTL is 300 seconds. Resolvers that cached the record at 13:59 will continue to send traffic to the dead IP until 14:05. There is no health check; Route 53 will never automatically switch. Even if you manually update the A record to a new IP at 14:01, resolvers that cached at 13:59 still have 4 minutes of stale resolution. Users see connection timeouts for up to 5 minutes.

What you'd do on Monday

Before the next planned deployment: (1) lower the TTL to 60 seconds. (2) Set up a failover routing policy with a health check on /health. (3) Register a secondary endpoint (another AZ, a static error page in S3, or an ALB). (4) After the health check interval × unhealthy threshold the DNS switches automatically. The mean time to DNS failover is health_check_interval × unhealthy_threshold — check the current Route 53 documentation for default and minimum health check intervals.

Scenario B

Imagine you work at Hotstar

25M concurrent viewers (IPL peak)

The numbers

Illustrative: 25M concurrent viewers, roughly 1 caching resolver per 50 viewers thanks to ISP-level resolver sharing ≈ 500k unique resolvers. Each resolver re-queries at most once per TTL window; at a 60s TTL that's 3600s ÷ 60s = 60 refreshes/hour per resolver → 500k × 60 ≈ 30M DNS lookups/hour at peak. Route 53 is designed to serve this scale globally. The risk is not query capacity; it is the lag between a health check failure and TTL expiry: 60s TTL × 3-failure threshold at a 30s check interval = at least 90 seconds where some resolvers still point to the failed origin.

What the Scenario-A answer does here

For a single-region deployment, failover has a floor of at least 90 seconds (the health-check lag computed above), plus however long stale resolver caches persist beyond that — an illustrative worst case of 3–5 minutes total. For 25M concurrent viewers, even the 90-second floor means tens of millions of reconnect attempts could hit a dead origin before the switch is visible everywhere.

What flips at this scale

Active-active with latency routing across two or more Regions. Each Region's ALB has its own health check. Route 53 multivalue or weighted policy distributes traffic. A Region's weight drops to 0 automatically when the health check fails. Combined with connection-level retries in the CDN layer, the user-visible interruption is seconds, not minutes.

The trade-off you'd defend

I would defend latency routing with health checks over a simple A record for any service with an SLA. The DNS TTL cannot be zero; you accept a bounded switchover window in exchange for the simplicity of a single global DNS name.

Illustrative viewer numbers; not Hotstar-published.

Gotchas

DNS failover switched but users still hit the failed endpoint

Symptom

Route 53 console shows the secondary record is active. Users still get errors.

Cause

Resolver caches still have the primary IP for up to TTL more seconds. Existing TCP connections are not terminated.

Fix

Pre-emptively lower the TTL (to 60s or less) before known maintenance windows. For emergency failovers, accept the TTL window as the worst case and ensure the secondary is ready before the primary fails.

Failover routing fixed DNS but the database is still in the primary Region

Symptom

Traffic switched to the secondary API endpoint. Most requests return DB connection errors.

Cause

Route 53 failover moves traffic to a new compute endpoint. It does not replicate or switch databases. The secondary API is pointing at the same primary database or a replica that is not promoted.

Fix

DNS failover is one layer of a DR plan. The database failover (RDS Multi-AZ, Aurora Global, or manual promotion) must complete before the secondary API can serve requests (Disaster recovery).

Tricky questions

Does a low TTL guarantee instant DNS propagation?
Most people say

Yes — TTL 60 means all resolvers update within 60 seconds.

Actually

TTL is a maximum cache time, not a guarantee. Some resolvers ignore TTL or have a minimum enforced floor. "Propagation" is a probabilistic window. For planned changes, treat the TTL as your best-case bound, and test with fresh resolver queries.

Does Route 53 failover routing restore a failed database?
Most people say

Yes — if you set up failover routing to a secondary Region, the whole stack fails over automatically.

Actually

Route 53 routes DNS queries. It has no knowledge of database state. A secondary API endpoint that points to an unpromoted RDS read replica will fail just as fast as the primary, just in a different AZ. Application-level DR requires coordinating database promotion with the DNS switch.

Scenario question

You're told "just set up Route 53 failover routing and we'll be fine if us-east-1 goes down." The application uses RDS PostgreSQL with a Multi-AZ standby. What questions do you ask before agreeing?

Model answer

I'd ask three things. First, is the secondary API endpoint deployed and tested in the failover Region — failover routing needs something to route to. Second, what is the RDS Multi-AZ standby in? Multi-AZ standby is in the same Region; for a Region failure you need Aurora Global or a cross-Region read replica promoted to primary. Third, what are the RTO and RPO targets? Route 53 failover cuts the DNS switch time but the database promotion adds minutes of unavailability. If the RTO is under five minutes, you need active-active, not active-passive failover routing. The trade-off is cost: active-active doubles the compute and storage bill; active-passive with tested failover procedures is cheaper but has a longer measured recovery time.

Interview corner

CommonWhat is a Route 53 routing policy and name three types.
“I'd describe it as the rule that determines how Route 53 responds to a DNS query. Simple returns one record. Failover returns the primary unless a health check says it's down, then returns the secondary. Latency returns the record for the Region with the lowest measured latency from the resolver's location.”
TrickIf you set a record TTL to 1 second, does failover happen in 1 second?
“I'd say it's not guaranteed. The health check has its own interval and failure threshold — by default Route 53 health checks have a 30-second interval and require 3 consecutive failures. So failover takes at minimum interval × threshold seconds, regardless of TTL. TTL only controls how fast resolvers refresh; it doesn't speed up health check evaluation.”
Experience checkWhat breaks in a Region failover that Route 53 alone cannot fix?
“I'd point straight to the database. Route 53 switches DNS to the secondary compute tier. If the database is only in the primary Region and isn't replicated — or if a replica isn't promoted — the secondary API immediately throws connection errors. Route 53 failover and database failover are separate operations that must be coordinated.”

Memory hook

Route 53 is a phone book that can switch numbers. It switches future callers, not existing conversations, and it never fixes a broken database.

If you remember only one thing:

Failover routing switches DNS; it does not restore data. Plan database failover separately.

Self-check

  1. What is the minimum time before all resolvers see a DNS change, given a TTL of 60?

    There is no guaranteed minimum. TTL 60 means resolvers may cache for up to 60 seconds; some may serve stale answers beyond that. Plan for a propagation window of at least old_TTL + new_TTL seconds.

  2. Name the Route 53 routing policy that returns a primary endpoint and switches to a secondary when a health check fails.

    Failover routing policy.

  3. Why does DNS failover not help if the secondary Region has no promoted database?

    Route 53 only changes which IP address DNS returns. The secondary API still needs a working database connection. Without a promoted replica the secondary tier fails for the same reason as the primary.

Prove it

Use propagation_window to compute the worst-case propagation window, in seconds, for changing a record from TTL 3600 to TTL 60 ahead of a maintenance window starting in 1 hour. Calculate: (a) whether lowering the TTL now leaves enough time for the old TTL to fully expire before the window starts, and (b) how many seconds after the window starts every resolver is guaranteed to see the new IP. Paste both numbers, converted to minutes.

aws-02-06Card 6 of 6

Draw and debug a three-tier topology

HardP3 · Role-dependentTalk about it13 min
three-tiertopologyvpc-designpacket-pathdebug

Why you care

"Draw the architecture" is the most common whiteboard question in a senior cloud interview. If you cannot sketch a three-tier topology — internet → load balancer → application → database — and walk a packet through it, you cannot pass that interview. Worse, if you cannot walk the path you cannot debug it: "My ECS task can't reach RDS" has exactly one answer once you can see the route table, SG chain, and subnet placement. This card integrates everything from the chapter into one coherent picture.

Explain like I'm 5

A three-tier architecture is like a restaurant. The dining room (public ALB subnet) is where customers arrive and waiters take orders. The kitchen (private app subnet) is where the work happens — customers never enter. The pantry (private DB subnet) holds the ingredients — only kitchen staff can open it. Each door between rooms has a lock (security group). The building's front door faces the street (internet gateway). The kitchen has a side door for deliveries that only opens outward (NAT gateway). There are no doors from the pantry to the street at all.

Core concept

A three-tier topology separates internet-facing, application, and data tiers into distinct subnets with distinct route tables and security groups.

Subnet layout (across two AZs for resilience):

SubnetAZRoute tableContents
public-alb-a, public-alb-bus-east-1a, 1b0.0.0.0/0 → igwALB ENIs
private-app-a, private-app-bus-east-1a, 1b0.0.0.0/0 → nat-a / nat-bECS tasks, Lambda, EC2
private-db-a, private-db-bus-east-1a, 1bno default routeRDS Multi-AZ

Security group chain:

  • sg-alb: inbound 443 from 0.0.0.0/0; outbound to sg-app on port 8080.
  • sg-app: inbound 8080 from sg-alb; outbound 5432 to sg-db; outbound 443 to 0.0.0.0/0 (for external API calls).
  • sg-db: inbound 5432 from sg-app only; no outbound to internet.

Packet path (inbound request): Route 53 resolves the ALB alias → TLS terminates at the ALB → a listener rule matches and forwards to the app target group → the packet crosses into private-app-a, where any custom NACL and then SG-app decide whether the ECS task's ENI accepts it → the app calls RDS on 5432, and both SG-app outbound and SG-db inbound must allow that port → the DB's response returns automatically, because SGs are stateful in both directions.

Egress path: app → NAT gateway (in a public subnet) → IGW → internet.

Debug checklist, one tier to the next: right subnet on both ends → route table has the needed route (0.0.0.0/0 for internet, local for intra-VPC) → the receiving resource's SG allows inbound from the sending resource's SG or IP → any custom NACL allows both directions, including the sender's ephemeral outbound ports → the target is healthy in its ALB target group.

Under the hood

Every VPC resource has at least one Elastic Network Interface (ENI), and that ENI — not the subnet boundary, not the instance — is where security groups are enforced. AWS evaluates the rules on the destination ENI for inbound traffic and the source ENI for outbound. That's why an SG rule can reference another SG ID instead of an IP: the data plane resolves ENI-to-SG membership dynamically on every connection attempt.

SG statefulness lives in the VPC's connection tracking table: an allowed inbound connection records a five-tuple (protocol, src/dst IP, src/dst port), and return packets matching it skip outbound evaluation. That entry has an idle timeout (check current AWS documentation for the value); once it's flushed, both directions re-evaluate from scratch. A long-idle DB connection — an ETL job pausing between batches — can be silently dropped here, with nothing logged by the SG. A related quirk: attaching a second SG to an already-tracked ENI leaves existing connections on the old entry; only new connections see the combined rule set.

Route tables are evaluated at the VPC router — the implicit + 1 address in each subnet CIDR (10.0.0.1 for 10.0.0.0/24) — before a packet reaches its destination; the most-specific CIDR wins, with local always implicit. Gateway endpoints add routes via AWS-managed prefix lists (e.g., pl-63a5400a for S3), which expand automatically as AWS adds IP ranges, with no route-table edits required.

Minimal example Both halves of the SG chain must exist

PythonAn outbound rule with no matching inbound rule still blocks the pathruns
def can_reach(src_sg: str, dst_sg: str, port: int, rules: list[tuple]) -> bool:
    """rules: list of (sg, direction, port, peer_sg)."""
    out_ok = (src_sg, "outbound", port, dst_sg) in rules
    in_ok = (dst_sg, "inbound", port, src_sg) in rules
    return out_ok and in_ok

rules = [("sg-app", "outbound", 5432, "sg-db"), ("sg-db", "inbound", 5432, "sg-app")]
print(can_reach("sg-app", "sg-db", 5432, rules))  # True: both halves present
print(can_reach("sg-alb", "sg-db", 5432, rules))  # False: sg-alb has no path to sg-db at all
Output
True
False

In production Three-tier topology renderer

PythonRenders ASCII topology diagramruns
def render_topology(tiers: list[dict]) -> str:
    lines = []
    lines.append("Internet")
    lines.append("   |")
    lines.append("   v")
    for i, tier in enumerate(tiers):
        lines.append(f"+--{tier['label']:{'-'}<40}+")
        for sub in tier["subnets"]:
            sg_str = ", ".join(tier.get("sgs", []))
            lines.append(f"   {sub}   [{sg_str}]")
        if tier.get("egress"):
            lines.append(f"   |  egress: {tier['egress']}")
        if i < len(tiers) - 1:
            lines.append("   |")
            lines.append("   v")
    return "\n".join(lines)


tiers = [
    {
        "label": "Public ALB subnets (us-east-1a, 1b)",
        "subnets": ["public-alb-a: 10.0.0.0/24", "public-alb-b: 10.0.1.0/24"],
        "sgs": ["sg-alb: in 443/0.0.0.0/0, out 8080/sg-app"],
        "egress": "IGW (internet gateway)",
    },
    {
        "label": "Private app subnets (us-east-1a, 1b)",
        "subnets": ["private-app-a: 10.0.2.0/24", "private-app-b: 10.0.3.0/24"],
        "sgs": ["sg-app: in 8080/sg-alb, out 5432/sg-db, out 443/0.0.0.0/0"],
        "egress": "NAT gateway (per AZ)",
    },
    {
        "label": "Private DB subnets (us-east-1a, 1b) -- no egress",
        "subnets": ["private-db-a: 10.0.4.0/24", "private-db-b: 10.0.5.0/24"],
        "sgs": ["sg-db: in 5432/sg-app, no outbound to internet"],
    },
]

print(render_topology(tiers))
Output
Internet
   |
   v
+--Public ALB subnets (us-east-1a, 1b)-----+
   public-alb-a: 10.0.0.0/24   [sg-alb: in 443/0.0.0.0/0, out 8080/sg-app]
   public-alb-b: 10.0.1.0/24   [sg-alb: in 443/0.0.0.0/0, out 8080/sg-app]
   |  egress: IGW (internet gateway)
   |
   v
+--Private app subnets (us-east-1a, 1b)----+
   private-app-a: 10.0.2.0/24   [sg-app: in 8080/sg-alb, out 5432/sg-db, out 443/0.0.0.0/0]
   private-app-b: 10.0.3.0/24   [sg-app: in 8080/sg-alb, out 5432/sg-db, out 443/0.0.0.0/0]
   |  egress: NAT gateway (per AZ)
   |
   v
+--Private DB subnets (us-east-1a, 1b) -- no egress+
   private-db-a: 10.0.4.0/24   [sg-db: in 5432/sg-app, no outbound to internet]
   private-db-b: 10.0.5.0/24   [sg-db: in 5432/sg-app, no outbound to internet]

See it happen Three-tier packet path

Alias recordHTTPS 443HTTP 8080TCP 54320.0.0.0/0NAT → IGWInternetRoute 53ALBpublic-alb-a/bsg-albECS / Lambdaprivate-app-a/bsg-appNAT gwpublic-alb-aRDS Multi-AZprivate-db-a/bsg-dbExternal API(internet)
  1. Client resolves api.example.com. Route 53 returns the ALB DNS alias.
  2. HTTPS request arrives at the ALB. TLS terminates here. ALB evaluates listener rules.
  3. ALB matches the path rule and forwards to the app target group in a private subnet.
  4. App calls RDS on port 5432. sg-app outbound allows 5432 to sg-db. sg-db inbound allows 5432 from sg-app.
  5. App needs to call an external payment API. Route: private-app → 0.0.0.0/0 → NAT gateway → IGW → internet.
  6. Full path visible. Database subnet has no default route — sg-db traffic cannot reach the internet at all.
Notice: the database subnet's route table has no default route; even if sg-db allowed outbound internet, there is no path — defence in depth.
Scenario A

In your world

The situation

Your ECS task (Python ETL job migrated off Jenkins) lives in private-app-a. It writes results to RDS in private-db-a and reads source files from S3. S3 writes work; RDS writes time out.

What actually breaks

You added a Gateway endpoint for S3 (Endpoints and private connectivity) so S3 works. RDS is in private-db-a. The route table is fine (10.0.0.0/16 → local). The problem: sg-db has an inbound rule allowing port 5432 from the ECS task's private IP, not from sg-app. The ETL job's IP changes every task restart (ECS tasks get new ENIs). The static IP rule misses the new IP; the connection is dropped at the SG.

What you'd do on Monday

Change the sg-db inbound rule source from the private IP 10.0.2.x/32 to the security group ID sg-app. SG-to-SG reference is dynamic: any ENI currently associated with sg-app is automatically allowed, regardless of IP.

Scenario B

Imagine you work at Stripe

100k API requests/sec

The numbers

Illustrative: 100k req/s hitting an ALB, fanning out to ~50 ECS tasks × ~5 DB connections each → 250 total RDS connections. max_connections (set by instance class and RAM) caps how many tasks you can run before saturation; at this scale a pooler like PgBouncer in front of RDS is standard — check current RDS docs for default limits per instance class.

What the Scenario-A answer does here

For a low-traffic ETL job, one connection per task is fine. At 50 ECS tasks × 5 connections each = 250 connections, a db.t3.medium hits its limit. Tasks start failing with "too many connections" before the instance is under any meaningful CPU load.

What flips at this scale

At scale the three-tier model gains a fourth implicit layer: a connection pooler (PgBouncer, RDS Proxy) between the app tier and the DB tier, in the private-app subnet or its own private-pool subnet. Its SG is the only one with inbound rights to sg-db on 5432; sg-app connects to the pooler, not directly to RDS.

The trade-off you'd defend

I would defend RDS Proxy or PgBouncer as a standard layer from the start for any ECS-to-RDS topology, trading the proxy cost and additional hop for dramatically higher connection density and built-in connection health management.

Illustrative connection numbers; not Stripe-published.

Gotchas

ECS task fails to connect to RDS after redeployment

Symptom

The app worked yesterday. After a new ECS task deployment, every DB call times out. No change to RDS or the SG.

Cause

The sg-db inbound rule allows a specific private IP, not an SG reference. New task → new ENI → new private IP → no matching SG rule.

Fix

Replace the IP-based inbound rule in sg-db with a rule sourced from sg-app. Any ENI associated with sg-app automatically satisfies the rule.

The ALB target group shows healthy but requests return 502

Symptom

ALB health check: healthy. Requests: 502 Bad Gateway.

Cause

The health check path (/health) returns 200. The real request path (/api/data) triggers a DB query that times out because sg-db blocks the task's IP. The health check is too shallow.

Fix

Deepen the health check to probe a path that actually connects to the DB (e.g., return 200 only if SELECT 1 succeeds). Fix the underlying sg-db SG rule.

The DB subnet has no internet egress but the app still can't reach it

Symptom

Route table looks correct. SG rules look correct. Connection still refused.

Cause

The DB subnet's NACL (if custom) has an inbound deny for the app subnet CIDR at a lower rule number than the allow rule.

Fix

Check the NACL associated with private-db. Ensure rule ordering: allow should have a lower number than any deny for the same CIDR, or remove the deny if it was unintentional.

Tricky questions

Can you put the ALB in a private subnet?
Most people say

No — the ALB must be in a public subnet to accept internet traffic.

Actually

An ALB must have at least two subnets. For an internet-facing ALB, those subnets need routes to an IGW. An internal ALB can live in private subnets (used for service-to-service calls within the VPC). If the requirement is internet-facing, the ALB subnet needs IGW access.

Can sg-db have an inbound rule from sg-alb to allow the ALB to query the DB directly?
Most people say

Yes — just add sg-alb as a source in sg-db inbound for port 5432.

Actually

You technically can, but you must not. Direct ALB-to-DB connections bypass the application tier entirely, expose the DB to every rule change in sg-alb, and are unauditable. The SG chain sg-alb → sg-app → sg-db exists precisely to prevent this.

Scenario question

You're handed a ticket: "Lambda in private-app-a can't connect to RDS in private-db-a on port 5432. Logs show TCP timeout. Both were working last week." Walk through your debug steps.

Model answer

I'd go layer by layer. Route tables first — do private-app-a and private-db-a both have the 10.0.0.0/16 → local route? They should, since it's intra-VPC. Second, security groups: does sg-db have an inbound rule for 5432 sourced from sg-app, not a static IP, and does sg-app have the matching outbound rule? Third, any custom NACL on private-db-a — does it allow inbound 5432 from the app subnet and outbound ephemeral ports 1024–65535 back to it? Fourth, is RDS actually Available, not mid-failover? The most common cause after "was working last week" is an SG rule pinned to a static IP that an ECS task restart invalidated. There's no real trade-off in the fix — SG references are always preferable to IPs.

Interview corner

CommonWalk me through a packet from an internet client to a private RDS database.
“I'd trace it layer by layer: Route 53 returns the ALB alias, TLS terminates at the ALB in the public subnet, and the ALB forwards to an ECS task in private-app through sg-alb outbound and sg-app inbound. The task queries RDS via sg-app outbound and sg-db inbound on 5432. RDS sits in a private-db subnet with no default route, so it can't reach the internet at all. The response is automatic — SGs are stateful, so no extra rules are needed for the return trip.”
DesignWhy separate the ALB and app tiers into different subnets?
“Separation gives me independent security control at each tier. The ALB subnet allows 443 from everywhere. The app subnet allows only 8080 from sg-alb. The DB subnet allows only 5432 from sg-app. Each tier can only communicate with the tier directly adjacent. If the ALB is compromised, the attacker still can't reach the DB directly — they'd need to compromise sg-app too.”
TrickIf you remove the default outbound allow-all rule from sg-app, what breaks?
“I'd say all outbound from ECS tasks would be blocked, including the connection to RDS on 5432 and any external API calls on 443. Security groups default to allow-all outbound; removing that rule means you must explicitly add outbound rules for every destination and port the app needs. It's more secure but requires careful enumeration of every outbound flow.”

Memory hook

The dining room, the kitchen and the pantry only ever talk to the room next door: customers never walk into the kitchen, and the kitchen never opens the pantry to the street. Name the door (the SG reference) at each hop, and you can debug anything in the VPC.

If you remember only one thing:

Always use SG-to-SG references, never static IPs, so rules survive container restarts.

Self-check

  1. Why put the ALB in a public subnet and the app tier in a private subnet?

    The ALB must be reachable from the internet and needs an IGW route. The app tier should not be directly reachable from the internet; it receives traffic only from the ALB via the SG reference. This limits the blast radius if the ALB is misconfigured.

  2. What is the recommended way to allow an ECS task to connect to RDS, given tasks restart with new IPs?

    Use a security group reference: the sg-db inbound rule sources from sg-app, not from a static IP. Any ENI associated with sg-app is automatically included.

  3. Name the five-step debug checklist for a connectivity failure between two tiers.

    (1) Confirm subnet placement. (2) Check route table for the required route. (3) Check SG outbound on the sender. (4) Check SG inbound on the receiver. (5) Check NACL inbound on the receiver and outbound ephemeral ports on the sender.

Prove it

Using can_reach, add a rule allowing sg-nlb outbound to sg-db on 5432, but do not add a matching inbound rule on sg-db from sg-nlb. Call can_reach("sg-nlb", "sg-db", 5432, rules) and confirm it returns False. Paste the output and explain in one sentence why an outbound-only rule is never enough to open a path.