Shelf · AWS · Chapter 1

Chapter 1: Foundations & the mental model

Before you write a single boto3 call you need to know where your code runs, who it runs as, and what it is allowed to do. This chapter builds those three mental models so every later chapter has a foundation to build on.

Difficulty
Priority
aws-01-01Card 1 of 6

Regions, AZs and shared responsibility

EasyP2 · Very likelyTalk about it12 min
regionsavailability-zonesshared-responsibilityfailure-domains
Needs: nothing. Start here.

Why you care

When your ETL job writes to S3 and the job fails, the first question an interviewer or an on-call engineer asks is: "was this a region event, an AZ event, or your own code?" If you cannot distinguish those three, you will either declare an incident that isn't one or miss one that is. Shared responsibility is equally concrete: S3 durability is AWS's job, but whether your IAM policy lets an attacker read your settlement files is yours. Interviewers ask about AZs and shared responsibility to find out whether you think about failure proactively or only after it bites you.

Explain like I'm 5

Think of AWS as a city. A Region is a city like Mumbai or Virginia. An Availability Zone is a separate physical building inside that city, with its own power, cooling and internet feeds, connected to other buildings in the same city by underground fibre. If one building floods, the others keep running. Your data lives in specific buildings; if you only use one building and it fails, you have an outage. If you spread across buildings, one failure is a dip, not a disaster. AWS owns the buildings and guarantees them up to a standard; what you put inside the buildings, and how you arrange it, is your responsibility.

Core concept

Regions are fully independent geographic deployments, each with its own control plane, endpoints and data storage. Data written in ap-south-1 (Mumbai) does not leave that region unless you explicitly replicate it. Region-level failures are rare; AZ-level failures are more common.

Availability Zones are one or more discrete data centres inside a Region. They are:

  • Physically separated (kilometres apart, separate power grids).
  • Interconnected by low-latency, high-bandwidth private fibre.
  • Named with a letter suffix: ap-south-1a, ap-south-1b, ap-south-1c. The letter maps to a physical AZ randomly per account, so your ap-south-1a is not the same building as your colleague's ap-south-1a.

Availability vs recoverability: availability means the system responds right now. Recoverability means you can restore data or state after a failure. S3 is 99.999999999% durable (eleven 9s) — AWS will not lose your object — but if you delete it, AWS does not undo that. Versioning is your recoverability tool. Multi-AZ RDS keeps the database available if one AZ fails; it does not protect against DROP TABLE.

Shared responsibility: AWS owns the security of the cloud (hardware, hypervisor, managed service patches, physical access). You own security in the cloud: IAM policies, bucket policies, encryption keys, OS patches on EC2, application vulnerabilities. As you move from EC2 toward managed services (RDS → Aurora Serverless → DynamoDB), AWS's share of the responsibility grows and yours shrinks on infrastructure, but your data governance and access control responsibilities never go away.

Decision rule: use at least two AZs for any stateful workload that needs to be online during an AZ failure. Use multiple Regions only when data sovereignty, latency, or business continuity across a regional disaster justifies the cost and operational complexity.

Minimal example Simulate AZ name-to-physical shuffle

PythonAZ name vs physical AZ — each account gets a different shuffleruns
import hashlib

accounts = ["111122223333", "444455556666", "777788889999"]
az_letters = ["a", "b", "c"]
physical = ["dc-alpha", "dc-beta", "dc-gamma"]

def az_to_physical(account_id: str, letter: str) -> str:
    """AWS shuffles AZ letters per account; simulate that mapping."""
    index = int(hashlib.md5(f"{account_id}{letter}".encode()).hexdigest(), 16)
    return physical[index % len(physical)]

for acct in accounts:
    mapping = {f"ap-south-1{l}": az_to_physical(acct, l) for l in az_letters}
    print(acct, mapping)
Output
111122223333 {'ap-south-1a': 'dc-gamma', 'ap-south-1b': 'dc-alpha', 'ap-south-1c': 'dc-alpha'}
444455556666 {'ap-south-1a': 'dc-alpha', 'ap-south-1b': 'dc-gamma', 'ap-south-1c': 'dc-alpha'}
777788889999 {'ap-south-1a': 'dc-gamma', 'ap-south-1b': 'dc-gamma', 'ap-south-1c': 'dc-beta'}

In production Failure domain classifier for an ETL deployment

PythonClassify AWS resources by failure domainruns
from dataclasses import dataclass
from typing import Literal

Domain = Literal["global", "regional", "az"]

@dataclass
class Resource:
    name: str
    service: str
    domain: Domain
    az: str | None = None

def classify(r: Resource) -> str:
    if r.domain == "global":
        return f"{r.name}: global — survives any single region failure"
    if r.domain == "regional":
        return f"{r.name}: regional — survives any single AZ failure in {r.service}"
    return f"{r.name}: AZ-scoped ({r.az}) — fails if {r.az} fails"

resources = [
    Resource("iam-role-etl",      "IAM",          "global"),
    Resource("s3-settlements",    "S3",            "regional"),
    Resource("rds-postgres-main", "RDS Multi-AZ", "regional"),
    Resource("ec2-jenkins",       "EC2",           "az",  "ap-south-1b"),
    Resource("ebs-jenkins-vol",   "EBS",           "az",  "ap-south-1b"),
    Resource("rds-postgres-old",  "RDS Single-AZ", "az",  "ap-south-1a"),
]

for r in resources:
    print(classify(r))

at_risk = [r for r in resources if r.domain == "az"]
print(f"\n{len(at_risk)} resource(s) are single-AZ and represent a failure risk.")
Output
iam-role-etl: global — survives any single region failure
s3-settlements: regional — survives any single AZ failure in S3
rds-postgres-main: regional — survives any single AZ failure in RDS Multi-AZ
ec2-jenkins: AZ-scoped (ap-south-1b) — fails if ap-south-1b fails
ebs-jenkins-vol: AZ-scoped (ap-south-1b) — fails if ap-south-1b fails
rds-postgres-old: AZ-scoped (ap-south-1a) — fails if ap-south-1a fails

3 resource(s) are single-AZ and represent a failure risk.

See it happen How an AZ failure affects a two-AZ ETL setup

sync replicationwriteswritesap-south-1 (Region)AZ: ap-south-1aAZ: ap-south-1bRDS PrimaryRDS StandbyS3 (regional)
  1. Normal state: primary in ap-south-1a, standby in ap-south-1b. S3 spans both AZs internally.
  2. ap-south-1a fails. RDS Multi-AZ detects the failure and begins automatic failover.
  3. Failover completes (typically 60-120 s). The standby in ap-south-1b is promoted to primary. S3 is unaffected.
  4. A single-AZ RDS in ap-south-1a would be offline for the entire duration. No automatic recovery.
Notice: S3's regional durability is invisible to AZ events; the risk is your compute and databases that are pinned to one AZ.
Scenario A

In your world

The situation

Your Jenkins job runs on a single EC2 instance in ap-south-1b. It reads CSVs from S3 and loads into an RDS PostgreSQL instance that was created without Multi-AZ (the default in many free-tier tutorials). The whole setup has been stable for a year.

What actually breaks

One Tuesday afternoon ap-south-1b has a power event. The EC2 instance and the EBS volume become unreachable. The Jenkins job fails immediately. But worse: the RDS instance is also in ap-south-1b and goes down too. S3 is fine; your files are all there. But you have no compute to run the job and no database to load into. The on-call page fires at 02:00.

What you'd do on Monday

Move RDS to Multi-AZ (a single API call that causes a brief failover window). For Jenkins, either move to a managed service (CodeBuild, GitHub Actions) or accept the AZ risk and document it. At minimum, know your failure domains before the incident, not during it.

Scenario B

Imagine you work at Netflix

200M+ streaming sessions globally

The numbers

Illustrative. 200M concurrent sessions × ~10 KB state per session ≈ 2 TB of session state. Netflix operates across multiple AWS Regions (us-east-1, eu-west-1, ap-southeast-1 and others). A single region carrying 33% of global traffic that has an outage affects ~66M sessions.

What the Scenario-A answer does here

The Scenario A fix (Multi-AZ RDS) still prevents AZ-level outages. What it does not prevent: a regional control-plane event. In 2012, an AWS us-east-1 event took down Netflix's US service; at that scale, even a 30-minute regional outage is a nine-figure revenue event.

What flips at this scale

At Netflix's scale the flip is active-active multi-region: every region can serve all traffic and has a full replica of the data it needs. Chaos Engineering (Chaos Monkey, Chaos Kong) validates that AZ and region failures are tolerable before they happen in production, not after.

The trade-off you'd defend

I would defend Multi-AZ RDS plus a documented recovery procedure for the Jenkins host over multi-region replication for a small ETL team, trading some regional risk for dramatically lower operational complexity and cost. Multi-region is only justified when the cost of a regional outage exceeds the cost of the added engineering.

Illustrative numbers; not Netflix-published. The 2012 AWS outage and Netflix's Chaos Engineering programme are publicly documented.

Gotchas

Your ap-south-1a is not your colleague's ap-south-1a

Symptom

Two engineers both say "I deployed to ap-south-1a" but their resources end up in different physical data centres, which breaks AZ-affinity assumptions.

Cause

AWS maps AZ letters to physical data centres per account to distribute load. The letters are not consistent across accounts.

Fix

Use canonical AZ IDs (e.g. aps1-az1) from the EC2 describe-availability-zones API when you need a physical guarantee, not the letter suffixes.

Deleting an S3 object is not a failure you recover from automatically

Symptom

After an accidental s3.delete_object, the eleven-9s durability guarantee does not help. The object is gone.

Cause

S3 durability protects against hardware loss, not logical deletion. Versioning is off by default.

Fix

Enable versioning on every bucket that holds data you cannot regenerate. Add a lifecycle rule to expire old versions after N days to control cost.

Shared responsibility means the managed service patch is AWS's job, but your schema migration is yours

Symptom

You upgrade RDS engine version and the application breaks because a default behaviour changed.

Cause

AWS patches the engine binary; the impact on your queries, connection strings and behaviour is your responsibility to test.

Fix

Test engine upgrades in a non-production environment first. Read the AWS engine changelog, not just the upstream PostgreSQL release notes.

Tricky questions

Does Multi-AZ RDS protect against data loss?
Most people say

Yes — it keeps a standby copy so you never lose data.

Actually

Multi-AZ protects availability, not against logical data loss. If you run DELETE FROM orders in the primary, the standby replicates that deletion synchronously. Point-in-time recovery from automated backups is what protects against logical data loss.

Is S3 availability the same as S3 durability?
Most people say

Yes, eleven 9s means it is always available.

Actually

Durability (eleven 9s) means AWS will not lose the bits. Availability is different: S3 Standard is designed for 99.99% availability, but the published SLA commitment is 99.9% monthly uptime — the number AWS actually credits you against. They are separate guarantees with separate figures.

Scenario question

You're handed a Python ETL system that has run fine for two years. The architecture review asks you to identify the failure domains. You find: one EC2 in ap-south-1a, one RDS PostgreSQL in ap-south-1a (no Multi-AZ), and data landing in S3. What do you report and what do you recommend?

Model answer

I'd report that there are two single-AZ resources that share the same AZ: the EC2 host and the RDS instance. Any ap-south-1a failure takes both down simultaneously, and because they are coupled, a failover on one without the other gives you nothing. S3 is fine; it's regional. I'd recommend enabling Multi-AZ on RDS first because that's one checkbox with no code change and a brief failover window. For EC2, I'd evaluate whether moving the Jenkins trigger to a managed service (CodeBuild, GitHub Actions) is worth the one-off migration cost. The trade-off is that Multi-AZ RDS roughly doubles the RDS cost, but for a job that feeds a production Postgres, that's justified. I'd document the EC2 AZ risk explicitly in the runbook if the team decides not to migrate it.

Interview corner

CommonWhat is an Availability Zone?
“I'd describe it as a physically separate data centre within a Region — separate power, cooling, and network feeds — connected to other AZs by low-latency private fibre. The key point is that an AZ failure does not cascade to neighbouring AZs in the same Region.”
TrickMy colleague says they're in ap-south-1a and I'm in ap-south-1a. Are we in the same building?
“I'd say not necessarily. AWS shuffles AZ letter names per account, so ap-south-1a maps to a different physical data centre depending on whose account you're in. If you need a physical guarantee, compare canonical AZ IDs, not the letter suffixes.”
Experience checkWhat's the difference between availability and durability in S3?
“Durability is the probability AWS will not lose the bits — eleven 9s, basically never. Availability is the probability you can read or write right now. S3 Standard is designed for 99.99% availability, though the published SLA — what AWS actually commits to and credits you against — is 99.9% monthly uptime. You can have high durability and a brief outage at the same time. I think about durability as 'will the data still exist' and availability as 'can I reach it today'.”
DesignWhen would you choose multi-region over multi-AZ?
“I'd choose multi-region when the cost of a regional outage — revenue loss, SLA penalties, regulatory requirements — exceeds the operational cost of active-active or active-passive replication across regions. For most ETL workloads, Multi-AZ inside one region is the right level. Multi-region adds complexity that only pays off at a certain scale or under specific compliance requirements.”

Memory hook

Regions are cities. AZs are separate buildings in the same city. S3 spans buildings; your EC2 and single-AZ RDS do not.

If you remember only one thing:

Know which of your resources are AZ-scoped before the outage, not during it.

Self-check

  1. Why do two AWS accounts see different physical data centres behind the same AZ letter?

    AWS shuffles the mapping of letter names to physical AZs per account to spread load across data centres.

  2. What does S3's eleven-9s durability guarantee protect against?

    Hardware failure and bit-level data loss. It does not protect against accidental deletion or logical corruption; versioning does.

  3. What is the difference between Multi-AZ RDS and a read replica?

    Multi-AZ keeps a synchronous standby for high availability; failover is automatic. A read replica is an asynchronous copy for read scaling; it does not provide automatic failover.

Prove it

Run the production classifier snippet above (copy-paste as-is). Then add two resources to the list: an ElastiCache cluster in ap-south-1c (AZ-scoped) and an ECS Fargate service (regional). Confirm the output classifies them correctly and that the count of single-AZ resources increases by one. Paste the new last two output lines and the updated count line.

aws-01-02Card 2 of 6

CLI profiles and identity preflight

EasyP3 · Role-dependentKnow cold11 min
aws-cliprofilescredentialsidentityiam

Why you care

Your Jenkins job runs aws s3 cp or imports boto3. It will use some AWS identity. If you do not know which identity, you will create resources in the wrong account, read stale data from the wrong region, or accidentally use a long-lived access key that never rotates. The quiet failure mode is the worst kind: the job succeeds, writes data somewhere, and the audit trail points to the wrong account. Interviewers ask about credential management because long-lived keys embedded in Jenkins environment variables or .env files are one of the most common causes of AWS security incidents.

Explain like I'm 5

Your AWS CLI is like a keyring. Each key on the ring has a label (a profile name) and opens a specific account and role. When you run a command, the CLI picks one key from the ring and uses it. If you do not tell it which key, it grabs the one labelled default. The problem is that many developers have several keys on the ring and forget which one is default. A preflight check is reading the label on the key before you turn it in the lock.

Core concept

Named profiles are sections in ~/.aws/credentials and ~/.aws/config that group an identity under a name:

Text~/.aws/config (no secrets here)
[profile etl-dev]
region = ap-south-1
role_arn = arn:aws:iam::111122223333:role/etl-dev-role
source_profile = base

[profile etl-prod]
region = ap-south-1
role_arn = arn:aws:iam::444455556666:role/etl-prod-role
source_profile = base

The CLI resolves credentials in this order (first match wins):

  1. Command-line flags (--profile, --region).
  2. AWS_PROFILE, AWS_DEFAULT_REGION, AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY environment variables.
  3. ~/.aws/credentials and ~/.aws/config.
  4. Instance profile / ECS task role / Lambda execution role (when running on AWS).
  5. No match → NoCredentialsError.

Temporary credentials come from sts:AssumeRole and expire (between 15 minutes and 12 hours, depending on the role's MaxSessionDuration). They are three values: AccessKeyId, SecretAccessKey, SessionToken. All three must be present; passing only the first two will fail with InvalidClientTokenId. The AWS CLI handles this automatically when you use role_arn in a profile.

Long-lived keys (IAM user access keys) do not expire. They belong in a secrets manager or a CI/CD platform secrets store, never in source code, never in a .env file that gets committed, never in a Jenkins job's "Description" field where they are visible in logs.

Identity preflight: before any destructive AWS operation, verify: which account (aws sts get-caller-identity), which region (check AWS_DEFAULT_REGION or the profile's region), and whether the identity has the minimum permissions needed. A five-second preflight prevents a 45-minute incident.

Decision rule: on EC2/ECS/Lambda, use the attached instance/task/execution role — never put a key on the machine. Locally and in CI, use named profiles with role_arn to assume a temporary-credential role. Never put long-lived keys in environment variables that survive beyond the current shell session.

Minimal example Simulate the CLI credential resolution chain

PythonCredential chain resolution order (no AWS calls)runs
import os

def resolve_profile(env: dict[str, str], config_profiles: list[str]) -> str:
    """Simulate CLI resolution: env var > config > default."""
    if "AWS_PROFILE" in env:
        p = env["AWS_PROFILE"]
        if p in config_profiles:
            return f"env AWS_PROFILE={p!r}"
        return f"ERROR: profile {p!r} not found in config"
    if "default" in config_profiles:
        return "config 'default' profile"
    return "ERROR: no profile found (NoCredentialsError)"

profiles = ["base", "etl-dev", "etl-prod", "default"]

print(resolve_profile({}, profiles))
print(resolve_profile({"AWS_PROFILE": "etl-prod"}, profiles))
print(resolve_profile({"AWS_PROFILE": "etl-staging"}, profiles))
Output
config 'default' profile
env AWS_PROFILE='etl-prod'
ERROR: profile 'etl-staging' not found in config

In production Identity preflight function for ETL scripts

PythonPreflight check (needs boto3 and AWS credentials)
import boto3
import sys

def preflight(expected_account: str, expected_region: str) -> None:
    """
    Call at the top of every ETL script before touching S3 or Postgres.
    Raises SystemExit if the identity or region does not match.
    Needs boto3 and valid AWS credentials; norun.
    """
    sts = boto3.client("sts")
    identity = sts.get_caller_identity()
    actual_account = identity["Account"]
    actual_arn = identity["Arn"]

    session = boto3.session.Session()
    actual_region = session.region_name

    errors = []
    if actual_account != expected_account:
        errors.append(
            f"account mismatch: expected {expected_account!r}, got {actual_account!r}"
        )
    if actual_region != expected_region:
        errors.append(
            f"region mismatch: expected {expected_region!r}, got {actual_region!r}"
        )

    if errors:
        print("PREFLIGHT FAILED:", file=sys.stderr)
        for e in errors:
            print(f"  {e}", file=sys.stderr)
        print(f"  caller ARN: {actual_arn}", file=sys.stderr)
        sys.exit(1)

    print(f"preflight OK: account={actual_account} region={actual_region} arn={actual_arn}")


if __name__ == "__main__":
    preflight(
        expected_account="111122223333",
        expected_region="ap-south-1",
    )
Scenario A

In your world

The situation

Your Jenkins job runs the ETL script with AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY set to a long-lived IAM user key in the Jenkins global credentials store. The same key has been rotating duty as default for the last 18 months. One day a team member creates a new Jenkins job for a different project and sets the same environment variables to their key, but only in their pipeline. Your pipeline still uses the global key. Nobody notices the key has s3:PutObject on all buckets.

What actually breaks

A security audit flags the key as overprivileged and rotates it. Your job fails immediately with InvalidClientTokenId. You have no idea what region the job was using. The audit log shows reads and writes from this key across three buckets in two regions; you cannot tell which ones were yours.

What you'd do on Monday

Move to an EC2 instance profile or an IAM role assumed via sts:AssumeRole. Add a preflight call at the top of every script that asserts the expected account and region and prints the caller ARN. Now every run's log tells you exactly which identity ran it.

Scenario B

Imagine you work at Capital One

10,000+ AWS accounts

The numbers

Illustrative. Capital One has publicly discussed operating at scale across thousands of AWS accounts separated by team, product and environment. At that scale, a single long-lived IAM user key that is shared across 20 CI jobs represents 20 blast-radius vectors. If the key leaks, every job using it is compromised.

What the Scenario-A answer does here

The Scenario A fix (one instance role per EC2 host) still works at team scale. What breaks: when 50 teams each manage their own IAM users manually, key rotation discipline collapses, the audit trail becomes unreadable, and least-privilege drifts to "whatever worked last time."

What flips at this scale

At Capital One's scale the flip is AWS Organizations with Service Control Policies that deny creation of long-lived IAM user access keys entirely in all accounts except a tightly controlled break-glass account. Every CI/CD pipeline uses OIDC federation (GitHub Actions → sts:AssumeRoleWithWebIdentity) or an EC2/ECS role. There are no keys to rotate because there are no keys.

The trade-off you'd defend

I would defend instance roles and OIDC federation over long-lived keys at any scale, trading a one-time setup cost for permanent elimination of the key-rotation risk class. The only argument for long-lived keys is legacy CI tooling that cannot do OIDC, and in that case I'd store them in AWS Secrets Manager with automatic rotation.

Illustrative numbers; not Capital One-published. Their cloud-first strategy and account structure are described in public conference talks (AWS re:Invent).

Gotchas

All three STS credential values must be present

Symptom

InvalidClientTokenId or AuthFailure when using temporary credentials.

Cause

You set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY from the STS response but forgot AWS_SESSION_TOKEN. The third value is required for temporary credentials.

Fix

Always extract all three: AccessKeyId, SecretAccessKey, SessionToken.

The default profile silently picks the wrong account

Symptom

Resources are created in a non-production account when you intended production, with no error.

Cause

AWS_PROFILE was not set; the CLI fell back to [default] which points to a different account.

Fix

Add a preflight assertion for account ID at the top of any script that should only run in a specific account. Never rely on the default profile in automation.

Long-lived keys in environment variables survive printenv

Symptom

A secret access key appears in Jenkins build logs or in a printenv step.

Cause

Environment variables set as plain strings (not masked secrets) are printed by many CI tools during debug steps.

Fix

Use the platform's secret store (Jenkins Credentials Store, GitHub Actions secrets) and confirm the values are masked in logs before merging.

Tricky questions

Is an IAM role an identity?
Most people say

No, a role is a permission set you attach to a user.

Actually

A role is an identity in IAM — it has an ARN, it can be the principal in a policy, and it can be assumed to get temporary credentials. The key distinction is that a role has no long-lived credentials of its own; it grants them via sts:AssumeRole.

Does setting AWS_DEFAULT_REGION override the profile's region?
Most people say

No, the profile config takes precedence.

Actually

Environment variables override the profile config. AWS_DEFAULT_REGION wins over the region set in [profile etl-prod]. The full precedence is: CLI flag > environment variable > profile config > SDK default.

Scenario question

You're handed a Python ETL job running on a bare EC2 instance. It uses AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY hardcoded in a .env file on the instance to write to S3 and query Secrets Manager. The team wants to improve security posture before an audit. What do you do?

Model answer

I'd replace the long-lived key with an EC2 instance profile. First, I'd create an IAM role with only the permissions the job needs — s3:PutObject on the specific bucket, secretsmanager:GetSecretValue on the specific secret ARN. I'd attach that role to the EC2 instance. Then I'd remove the .env file and delete the IAM user's access keys. I'd add a preflight call that asserts the expected account and region on every run. The trade-off is that the instance role cannot be used from a developer's laptop — for local testing, I'd set up a named profile that assumes the same role via sts:AssumeRole with MFA. This is slightly more setup but eliminates the entire class of credential leak via file or log.

Interview corner

CommonHow does the AWS CLI decide which credentials to use?
“I'd describe it as a chain: CLI flags first, then environment variables like AWS_PROFILE and AWS_ACCESS_KEY_ID, then the named profile in ~/.aws/config, then the instance metadata endpoint if you're on an EC2 or Lambda. First match wins.”
TrickCan you have an IAM role without any policies attached?
“I'd say yes — it assumes fine but fails every API call with AccessDeniedException because there are no permissions. It's actually a useful pattern for a break-glass role that you want to exist but not be usable without explicitly adding policies — or a bug waiting to happen if you forget to attach the policy.”
Experience checkWhy are long-lived IAM user keys a problem in CI pipelines?
“They don't expire, so if they leak — in a log, a commit, a screenshot — you have an open door until someone manually rotates them. I've moved jobs to instance roles or OIDC federation; the credential is tied to the run, not to a persistent secret.”

Memory hook

A named profile is a labelled key on your keyring. Run the preflight check before you turn it in the lock.

If you remember only one thing:

Use roles with temporary credentials; never put long-lived keys in environment variables or files.

Self-check

  1. What three values does STS return when you assume a role?

    AccessKeyId, SecretAccessKey, and SessionToken. All three are required; omitting SessionToken causes InvalidClientTokenId.

  2. What happens if AWS_PROFILE is set to a profile name that doesn't exist in ~/.aws/config?

    The CLI raises a ProfileNotFound error and does not fall back to [default].

  3. What is the difference between AWS_DEFAULT_REGION and setting region in a named profile?

    AWS_DEFAULT_REGION (environment variable) overrides the profile's region setting. Environment variables have higher precedence than profile config.

Prove it

Write a Python function simulate_chain(env, profiles) that takes a dict of environment variable overrides and a list of profile names, and returns which profile would be resolved (or an error string). Test it with: no env vars + default in profiles, AWS_PROFILE=etl-prod + etl-prod in profiles, AWS_PROFILE=staging + staging not in profiles. Print the three results. They should match the minimal block output.

aws-01-03Card 3 of 6

IAM policy evaluation

HardP2 · Very likelyTalk about it15 min
iampolicy-evaluationexplicit-denyidentity-policyresource-policysts

Why you care

Every AWS API call goes through the IAM policy evaluator first. Get the evaluation order wrong and you write policies that look correct but silently deny access, or worse, silently grant it. The most common incident: an ETL job that worked in dev fails in prod with AccessDeniedException, though the identity policy looks identical — the cause is almost always a boundary or SCP you forgot to check. Interviewers ask about this because "I'll just add AdministratorAccess and see if it works" reveals a candidate who has never debugged IAM in production.

Explain like I'm 5

Imagine getting into a concert. The identity policy is your ticket: it says what you are allowed to enter. The resource policy is the venue's guest list: it says who the venue will let in. The permission boundary is a hard ceiling set by the venue owner: even if your ticket and the guest list both say yes, you cannot get into the VIP section if the owner said no VIPs today. An SCP is the city's law: it overrides everything, including the owner. An explicit deny anywhere in that chain is a security guard at the door who says no regardless of everything else. One "no" beats every "yes."

Core concept

The IAM policy evaluator runs in a deterministic order. The final answer is either Allow or Deny (implicit or explicit).

Evaluation order (simplified for a same-account call):

  1. Explicit deny — any policy at any layer that explicitly denies the action wins immediately. No allow overrides it.
  2. SCP (AWS Organizations) — defines the ceiling for all accounts in an OU. An action not allowed by an SCP cannot be allowed by any identity or resource policy, even by the account root.
  3. Permission boundary — a managed policy attached to a user or role that acts as a maximum permissions envelope. An action outside the boundary is denied even if the identity policy allows it.
  4. Identity-based policy — the policies attached directly to the calling user or role.
  5. Resource-based policy — a policy attached to the resource (S3 bucket policy, KMS key policy, SQS queue policy). For same-account principals, a resource policy allow is sufficient even without an identity policy allow.
  6. Session policies — policies passed as part of an sts:AssumeRole call that further restrict (never expand) the role's permissions for that session.

Default deny: if no policy explicitly allows an action, it is denied. You do not need an explicit deny for an unknown action.

Cross-account calls are more restrictive: both the identity policy on the caller and the resource policy on the target must allow the action. A bucket policy alone that allows another account's role is not enough; the role's identity policy must also allow s3:GetObject.

sts:AssumeRole: to assume a role, two things must be true: the caller's identity policy must allow sts:AssumeRole on that role's ARN, and the role's trust policy must list the caller as a trusted principal.

Decision rule: when debugging an AccessDeniedException, check in this order: explicit deny, identity policy, resource policy, then any session policy.

Under the hood

The policy evaluator is not a sequential scan; it loads all applicable policies simultaneously and runs them against a request context that includes the principal, action, resource ARN, and any condition keys. Condition keys (like aws:RequestedRegion, s3:prefix, aws:PrincipalTag) are evaluated at request time, not at policy creation time. This means a tag-based condition on a role (aws:PrincipalTag/env = prod) is evaluated from the caller's session tags, not from the resource's tags — a common source of ABAC bugs covered in Cross-account IAM and ABAC.

The aws:SourceIp condition checks the caller's public IP. A Lambda or EC2 instance calling S3 within AWS has a private RFC-1918 source IP, so a bucket policy keyed on aws:SourceIp denies it. Use aws:SourceVpc or aws:SourceVpce for intra-AWS conditions instead.

Minimal example Explicit deny wins over allow — local evaluator

PythonPolicy conflict evaluator — explicit deny always winsruns
from dataclasses import dataclass

@dataclass
class Statement:
    effect: str   # "Allow" or "Deny"
    actions: list[str]
    resource: str

def res_match(policy_res: str, req_res: str) -> bool:
    if policy_res == "*":
        return True
    if policy_res.endswith("/*"):
        return req_res.startswith(policy_res[:-2] + "/") or req_res == policy_res[:-2]
    if policy_res.endswith("*"):
        return req_res.startswith(policy_res[:-1])
    return policy_res == req_res

def evaluate(statements: list[Statement], action: str, resource: str) -> str:
    """
    Simplified same-account evaluator.
    Returns "Allow", "Deny (explicit)", or "Deny (implicit)".
    """
    explicit_deny = any(
        s.effect == "Deny" and action in s.actions and res_match(s.resource, resource)
        for s in statements
    )
    if explicit_deny:
        return "Deny (explicit)"

    allowed = any(
        s.effect == "Allow" and action in s.actions and res_match(s.resource, resource)
        for s in statements
    )
    return "Allow" if allowed else "Deny (implicit)"

# IAM policy allows s3:GetObject, but SCP/boundary adds explicit deny
statements = [
    Statement("Allow", ["s3:GetObject", "s3:PutObject"], "arn:aws:s3:::settlements/*"),
    Statement("Deny",  ["s3:DeleteObject", "s3:GetObject"], "*"),  # explicit deny
]

print(evaluate(statements, "s3:GetObject",    "arn:aws:s3:::settlements/jan.csv"))
print(evaluate(statements, "s3:PutObject",    "arn:aws:s3:::settlements/jan.csv"))
print(evaluate(statements, "s3:DeleteObject", "arn:aws:s3:::settlements/jan.csv"))
print(evaluate(statements, "s3:ListBucket",   "arn:aws:s3:::settlements"))
Output
Deny (explicit)
Allow
Deny (explicit)
Deny (implicit)

In production IAM policy simulator (pure Python, no AWS calls)

PythonEvaluate a set of JSON-style policy docs against a requestruns
import json
from typing import Any

def matches_resource(policy_resource: str, request_resource: str) -> bool:
    if policy_resource == "*":
        return True
    if policy_resource.endswith("*"):
        return request_resource.startswith(policy_resource[:-1])
    return policy_resource == request_resource

def evaluate_policies(
    policies: list[dict[str, Any]],
    action: str,
    resource: str,
) -> str:
    explicit_deny = False
    any_allow = False

    for doc in policies:
        for stmt in doc.get("Statement", []):
            actions = stmt.get("Action", [])
            if isinstance(actions, str):
                actions = [actions]
            resources = stmt.get("Resource", [])
            if isinstance(resources, str):
                resources = [resources]

            action_match = action in actions or "*" in actions
            resource_match = any(matches_resource(r, resource) for r in resources)

            if action_match and resource_match:
                if stmt["Effect"] == "Deny":
                    explicit_deny = True
                elif stmt["Effect"] == "Allow":
                    any_allow = True

    if explicit_deny:
        return "Deny (explicit)"
    if any_allow:
        return "Allow"
    return "Deny (implicit)"


identity_policy = {
    "Statement": [
        {"Effect": "Allow", "Action": ["s3:GetObject", "s3:PutObject"],
         "Resource": "arn:aws:s3:::settlements-prod/*"},
        {"Effect": "Allow", "Action": "s3:ListBucket",
         "Resource": "arn:aws:s3:::settlements-prod"},
    ]
}

boundary_policy = {
    "Statement": [
        {"Effect": "Allow", "Action": "*",
         "Resource": "arn:aws:s3:::settlements-prod/*"},
        # s3:ListBucket targets the bucket itself, not the /* prefix — outside the boundary
    ]
}

# Boundary restricts even what identity policy allows
tests = [
    ("s3:GetObject",  "arn:aws:s3:::settlements-prod/jan.csv"),
    ("s3:ListBucket", "arn:aws:s3:::settlements-prod"),
    ("s3:DeleteObject", "arn:aws:s3:::settlements-prod/jan.csv"),
]

print("Identity only:")
for action, resource in tests:
    print(f"  {action}: {evaluate_policies([identity_policy], action, resource)}")

print("Identity + Boundary (intersection):")
for action, resource in tests:
    identity_result = evaluate_policies([identity_policy], action, resource)
    boundary_result = evaluate_policies([boundary_policy], action, resource)
    # Effective permission = Allow only if both allow
    if identity_result == "Allow" and boundary_result == "Allow":
        result = "Allow"
    elif "Deny (explicit)" in (identity_result, boundary_result):
        result = "Deny (explicit)"
    else:
        result = "Deny (implicit)"
    print(f"  {action}: {result}")
Output
Identity only:
  s3:GetObject: Allow
  s3:ListBucket: Allow
  s3:DeleteObject: Deny (implicit)
Identity + Boundary (intersection):
  s3:GetObject: Allow
  s3:ListBucket: Deny (implicit)
  s3:DeleteObject: Deny (implicit)

See it happen IAM evaluation pipeline for a same-account S3 call

1 SCP?2 Boundary?not allowed → denyoutside boundary → deny3 Identity policy4 Resource policyallowallow (same-acct)explicit denyAPI RequestSCP checkPermission BoundaryIdentity PolicyResource PolicyDenyAllow
  1. A request arrives. The evaluator checks all policy layers simultaneously.
  2. SCPs define the ceiling. If the action is outside any applicable SCP, the result is Deny before the identity policy is even read.
  3. Permission boundaries clip the identity policy. ListBucket allowed by identity but outside the boundary → implicit deny.
  4. The identity policy is evaluated. An explicit deny here — or from any layer — wins immediately.
  5. For same-account calls, a resource policy allow is sufficient even without an identity policy allow.
  6. Final result: Allow only if no explicit deny exists and at least one allow exists within all applicable ceilings.
Notice: the evaluator is a ceiling-and-floor system, not a union of allows. Every ceiling (SCP, boundary) must permit the action, and then at least one allow from identity or resource policy must exist.
Scenario A

In your world

The situation

Your ETL role in the dev account has an identity policy that allows s3:GetObject and s3:PutObject on arn:aws:s3:::settlements-dev/*. You promote the same role definition to prod. The prod account is in an AWS Organization OU that has an SCP restricting s3:DeleteObject and requiring aws:RequestedRegion = ap-south-1.

What actually breaks

The prod ETL job calls s3:CopyObject to move processed files to an archive prefix — which internally issues s3:DeleteObject on the source. It fails with AccessDeniedException in prod but succeeds in dev, though the identity policy looks identical. The difference is the SCP.

What you'd do on Monday

Check the SCP on the prod OU. Either add s3:CopyObject and s3:DeleteObject to its allow list, if the OU's risk profile permits it, or switch to an s3:PutObject plus a separate privileged Lambda step that removes the source.

PythonSCP evaluation check — needs boto3 and Organizations access; norun
import boto3

def list_scps_for_account(account_id: str) -> list[str]:
    """Returns names of SCPs attached to the account's OUs."""
    orgs = boto3.client("organizations")
    parents = orgs.list_parents(ChildId=account_id)["Parents"]
    scps = []
    for parent in parents:
        policies = orgs.list_policies_for_target(
            TargetId=parent["Id"], Filter="SERVICE_CONTROL_POLICY"
        )["Policies"]
        scps.extend(p["Name"] for p in policies)
    return scps
Scenario B

Imagine you work at Freshworks

60,000+ enterprise customers, multi-tenant SaaS

The numbers

Illustrative. Freshworks serves 60,000+ enterprise customers. Each enterprise customer may require data isolation — their data must not be accessible from another customer's context. At 10,000 enterprise tenants on AWS, manual IAM policy review per tenant is impossible.

What the Scenario-A answer does here

The Scenario A fix (explicit identity and resource policies) still works for 10 tenants. At 10,000, writing and maintaining a separate IAM policy document per tenant is operationally untenable. Policy drift, forgotten explicit denies, and untested condition keys pile up.

What flips at this scale

At Freshworks's scale the flip is ABAC (attribute-based access control): tag every resource with tenant-id and write a single policy that uses aws:ResourceTag/tenant-id = ${aws:PrincipalTag/tenant-id}. The condition is evaluated dynamically at request time. You maintain one policy document instead of 10,000. Cross-tenant access is structurally impossible as long as the tags are correct.

The trade-off you'd defend

I would defend ABAC with session tags over per-tenant policies for any multi-tenant system with more than a dozen tenants, trading the complexity of correct tagging enforcement for the elimination of N×M policy maintenance. The risk is tag misconfiguration, which I'd mitigate with SCPs that deny resource creation without the required tags.

Illustrative numbers; not Freshworks-published. Their customer count is from public investor relations pages; the ABAC pattern is documented in AWS IAM docs.

Gotchas

Explicit deny in an SCP blocks even the account root

Symptom

The account's root user cannot perform an action that is explicitly denied in an attached SCP.

Cause

SCPs are a ceiling on all principals in the account, including root. This is intentional for guardrail enforcement.

Fix

Only an org administrator can lift it, by editing the SCP in the management account or the OU.

aws:SourceIp blocks Lambda calling S3 from within AWS

Symptom

A Lambda function gets AccessDeniedException on S3 even though the bucket policy has a public IP range allow for your office.

Cause

Lambda's outbound IP is an AWS-private RFC-1918 address when it stays within the VPC, not your office IP.

Fix

Replace aws:SourceIp with aws:SourceVpc or aws:SourceVpce for intra-AWS access controls. Use aws:SourceIp only for direct internet-originated calls.

Permission boundary does not add permissions; it only constrains them

Symptom

You attach a boundary with s3:* to a role but the role still cannot call s3:GetObject.

Cause

A boundary alone is not an allow. The role also needs an identity policy that allows s3:GetObject. The effective permission is the intersection of identity policy and boundary.

Fix

Think of the boundary as the maximum envelope. You still need an identity policy that explicitly allows every action the role needs, within that envelope.

An explicit deny defeats an identity-policy allow

Symptom

A role's identity policy clearly allows s3:GetObject, but every call still returns AccessDeniedException with no obvious boundary or SCP involved.

Cause

A separate statement — in the same identity policy, a different attached policy, or a resource policy — has an explicit Deny on that action or resource. Explicit deny always overrides an allow, regardless of which layer it comes from.

Fix

Search every policy attached to the principal (and the resource) for "Effect": "Deny" statements before assuming the boundary or SCP is at fault. The IAM Policy Simulator surfaces which specific statement is denying the request.

Tricky questions

Does an explicit deny in an identity policy override a resource policy allow?
Most people say

No — if the resource policy allows it, the call goes through.

Actually

Yes. An explicit deny anywhere in the chain — including the identity policy — overrides every allow, including resource policy allows. "Deny wins" is unconditional.

Can an S3 bucket policy alone grant access to a principal in the same account?
Most people say

No — you always need an identity policy allow as well.

Actually

For same-account principals, a resource policy allow is sufficient even without an identity policy allow, because the account root already has implicit ownership. For cross-account principals, both sides must allow the action.

Scenario question

You're handed a Lambda function that calls s3:GetObject on a bucket in the same account. The Lambda execution role has an identity policy allowing s3:GetObject on arn:aws:s3:::data-prod/*. The S3 bucket has no bucket policy. The call fails with AccessDeniedException. What do you check?

Model answer

With no bucket policy and a matching identity policy allow, the likely cause is a permission boundary on the Lambda role or an SCP on the account. I'd check the boundary first — does it also allow s3:GetObject on that ARN? Then I'd pull the SCPs for the account's OU. If neither explains it, I'd use the IAM Policy Simulator (iam:SimulatePrincipalPolicy) to trace the exact denial against this request, which names the statement responsible. Boundary issues are safe to fix on the role; SCP issues need a conversation with the org admin.

Interview corner

CommonWhat is an explicit deny and why does it matter?
“I'd put it this way: an explicit deny in any policy layer overrides every allow everywhere else in the evaluation chain. It's the one absolute in IAM. I check for it first when debugging an AccessDeniedException, because an allow-only audit of the identity policy will miss it.”
TrickIf an SCP allows all actions, does that mean every identity in the account can do everything?
“I'd say no. An SCP allow is a ceiling, not a grant. Every identity still needs its own identity or resource policy allow. The SCP just defines what is possible in principle; the identity policy defines what a specific principal can actually do.”
Experience checkWalk me through debugging an AccessDeniedException in production.
“I'd follow the evaluation order: explicit deny in the account's SCPs, then the identity policy and any permission boundary, then the resource policy, then any narrowing session policy from sts:AssumeRole. The IAM Policy Simulator gives me the exact denial reason for that request, with the statement responsible — I use that rather than guessing.”
DesignWhen would you use ABAC instead of RBAC in IAM?
“I'd use ABAC when the number of unique permission combinations grows with the number of tenants or environments, so that managing per-entity policies becomes untenable. You write one policy with tag-based conditions instead of N policies. The trade-off is that the security guarantee depends on tag integrity — you need SCPs or service control mechanisms to prevent resource creation without the required tags.”

Memory hook

The security guard at the concert door needs only one reason to turn you away — your ticket, the guest list and the VIP pass can all say yes, but his "no" is final.

If you remember only one thing:

Check for explicit denies first — in SCPs, boundaries, and identity policies — before you add more allows.

Self-check

  1. What is the difference between an SCP and a permission boundary?

    SCPs apply to all principals in an AWS account or OU and are set by the organisation. Permission boundaries apply to a specific IAM user or role and are set by an administrator in that account. Both are ceilings; neither grants permissions.

  2. For a same-account S3 call, is a resource policy allow sufficient without an identity policy allow?

    Yes — for same-account principals, a resource policy allow is sufficient. For cross-account principals, both the identity policy on the caller and the resource policy on the target must allow the action.

  3. What happens to a session created by sts:AssumeRole if the calling role's identity policy is later changed to remove a permission?

    Active sessions aren't revoked immediately — only new sessions see the change. To revoke existing sessions now, add an inline deny keyed on aws:TokenIssueTime before a timestamp, then remove it once legitimate sessions have expired.

Prove it

Copy the minimal block's evaluate function. Extend it so it also takes an scps parameter (a list of allowed actions; if non-empty, an action not in the list is an implicit deny). Run three tests: (1) action allowed by SCP and identity, (2) action blocked by SCP but allowed by identity, (3) action allowed by SCP but no identity allow. Print the three results and confirm they are: Allow, Deny (implicit), Deny (implicit).

aws-01-04Card 4 of 6

Cross-account IAM and ABAC

HardP3 · Role-dependentTalk about it14 min
iamcross-accountabacsession-tagstrust-policysts

Why you care

When your ETL job runs in account A and writes to S3 in account B, or when a shared data platform serves multiple business units each in separate accounts, you hit cross-account IAM. The two most common failures: the job gets an AccessDeniedException even though "I added it to the bucket policy", because cross-account requires both sides to allow the action; or an ABAC condition that works in dev fails silently in prod because session tags were not propagated. Interviewers ask about cross-account IAM and ABAC when hiring for platform roles, because these are the patterns that break at scale and require deliberate design rather than trial-and-error policy edits.

Explain like I'm 5

Imagine two office buildings managed by different companies. Building A has a pass for Alice that says "Alice can visit Building B." Building B has a guest list that says "visitors from Building A can enter." Alice needs both the pass and to be on the guest list to get in. If only one is present, she is denied. ABAC adds sticky notes to Alice's pass: "department=finance". The guest list says "visitors with department=finance can access floor 3". The note is read at the door, not when the pass was made.

Core concept

Cross-account access requires permission on both sides:

  • The calling principal in account A must have an identity policy that allows sts:AssumeRole on the role ARN in account B, and s3:GetObject (or the target action) — or the resource policy in account B must explicitly allow the account A principal.
  • The role in account B must have a trust policy listing account A (or the specific principal) as a trusted entity. The trust policy is separate from the role's permission policies.
  • The resource in account B (e.g. the S3 bucket) must have a resource policy that allows the account B role to access it — or, for S3, the bucket policy must allow the account A principal directly (bypassing the role assumption step if using a resource policy approach).

The minimum cross-account flow for ETL writing to a shared S3 bucket:

  1. Account A role calls sts:AssumeRole → gets temporary credentials for account B role.
  2. Account B role's trust policy allows the account A role as a principal.
  3. Account B role's identity policy allows s3:PutObject on the bucket.
  4. The S3 bucket policy (if present) either allows the account B role or is absent (relying on the role's identity policy within account B).

ABAC (Attribute-Based Access Control) uses IAM condition keys instead of explicit ARN lists:

  • aws:PrincipalTag/<key> — a tag on the principal making the call (user or role).
  • aws:ResourceTag/<key> — a tag on the resource being accessed.
  • Session tags set during sts:AssumeRole via the Tags parameter propagate as aws:PrincipalTag in the assumed session.

A single ABAC policy: Condition: { StringEquals: { "aws:ResourceTag/env": "${aws:PrincipalTag/env}" } } means "only access resources tagged with the same env as your session tag." This replaces N per-environment policies with one.

Session identity propagation: when you chain role assumptions (account A → account B role → account C), only the immediately assumed role's ARN is the session's aws:PrincipalArn. Account C's policies cannot see that the chain started from account A unless you pass that as an explicit session tag.

Decision rule: use cross-account roles for service-to-service access across accounts; use ABAC when the number of unique access combinations grows faster than you can manage named policies.

Under the hood

aws:PrincipalTag/<key> is not read from the IAM user or role's permanent tags at request time — it is resolved from the session's tag set, which is populated at sts:AssumeRole time from two sources: the role's own IAM tags, and any Tags passed in the AssumeRole call itself (session tags), with session tags taking precedence on a key collision. This is why the same role can produce sessions with different aws:PrincipalTag/env values depending on who calls AssumeRole and what they pass.

Session tags are, by default, not transitive: if account A assumes a role in account B with Tags=[{"Key": "env", "Value": "prod"}], and that account B session then assumes a role in account C, account C's session does not see aws:PrincipalTag/env unless account A also set TransitiveTagKeys=["env"] on the first call. Each hop in a chain silently drops non-transitive tags, which is why an ABAC condition that works for a single AssumeRole hop can fail two hops downstream with no configuration change at the point of failure — the missing setting is upstream, at the first assumption in the chain.

Minimal example Cross-account trust policy and ABAC condition — pure Python

PythonValidate a cross-account trust policy and an ABAC conditionruns
import json

def check_trust_policy(trust_doc: dict, caller_arn: str) -> bool:
    """Returns True if the trust policy allows the caller to assume the role."""
    for stmt in trust_doc.get("Statement", []):
        if stmt.get("Effect") != "Allow":
            continue
        principal = stmt.get("Principal", {})
        arns = principal if isinstance(principal, list) else (
            [principal] if isinstance(principal, str) else
            principal.get("AWS", [])
        )
        if isinstance(arns, str):
            arns = [arns]
        for arn in arns:
            if arn == "*" or arn == caller_arn:
                return True
    return False

def check_abac(session_tags: dict[str, str], resource_tags: dict[str, str]) -> bool:
    """ABAC: allow if all required tag pairs match."""
    return all(resource_tags.get(k) == v for k, v in session_tags.items())

trust_policy = {
    "Statement": [{"Effect": "Allow", "Principal": {"AWS": "arn:aws:iam::111122223333:role/etl-runner"}, "Action": "sts:AssumeRole"}]
}

print("Trust allows etl-runner:", check_trust_policy(trust_policy, "arn:aws:iam::111122223333:role/etl-runner"))
print("Trust allows other role:", check_trust_policy(trust_policy, "arn:aws:iam::111122223333:role/admin"))

print("ABAC match (env=prod,team=data):", check_abac({"env": "prod", "team": "data"}, {"env": "prod", "team": "data"}))
print("ABAC mismatch (wrong env):", check_abac({"env": "prod", "team": "data"}, {"env": "dev", "team": "data"}))
Output
Trust allows etl-runner: True
Trust allows other role: False
ABAC match (env=prod,team=data): True
ABAC mismatch (wrong env): False

In production Cross-account role assumption with session tag propagation

PythonCross-account AssumeRole with ABAC session tags (needs boto3; norun)
import boto3
from typing import Any

def assume_cross_account_role(
    role_arn: str,
    session_name: str,
    session_tags: list[dict[str, str]] | None = None,
    duration_seconds: int = 3600,
) -> dict[str, Any]:
    """
    Assumes a role in another account and injects session tags for ABAC.
    Returns the credential dict; callers build a boto3 session from it.
    Needs boto3 and a valid source identity; norun.
    """
    sts = boto3.client("sts")
    kwargs: dict[str, Any] = {
        "RoleArn": role_arn,
        "RoleSessionName": session_name,
        "DurationSeconds": duration_seconds,
    }
    if session_tags:
        kwargs["Tags"] = session_tags
        # TransitiveTagKeys makes these tags propagate to any further role assumption
        kwargs["TransitiveTagKeys"] = [t["Key"] for t in session_tags]

    response = sts.assume_role(**kwargs)
    creds = response["Credentials"]
    return {
        "aws_access_key_id":     creds["AccessKeyId"],
        "aws_secret_access_key": creds["SecretAccessKey"],
        "aws_session_token":     creds["SessionToken"],
    }


def get_cross_account_s3(
    target_role_arn: str,
    bucket: str,
    key: str,
    env_tag: str,
) -> bytes:
    """
    Assumes a cross-account role with env session tag, then reads from S3.
    The S3 bucket policy uses ABAC: aws:PrincipalTag/env must match aws:ResourceTag/env.
    Needs boto3 and valid credentials; norun.
    """
    creds = assume_cross_account_role(
        role_arn=target_role_arn,
        session_name=f"etl-cross-acct-{env_tag}",
        session_tags=[{"Key": "env", "Value": env_tag}],
    )
    s3 = boto3.client("s3", **creds)
    obj = s3.get_object(Bucket=bucket, Key=key)
    return obj["Body"].read()
Scenario A

In your world

The situation

Your company has a data platform account (account B, 444455556666) that holds a shared S3 bucket company-data-lake. Your ETL job runs in the analytics account (account A, 111122223333). A data engineer adds an S3 bucket policy to company-data-lake that grants s3:GetObject to arn:aws:iam::111122223333:role/etl-runner. They tell you it's ready.

What actually breaks

Your ETL job still gets AccessDeniedException. The bucket policy is correct. The problem is that etl-runner in account A has no identity policy that allows sts:AssumeRole on any account B role, and no direct s3:GetObject allow either. For cross-account resource policy access, account A's identity policy must also explicitly allow the cross-account action unless the request is made as an already-assumed account B session. The bucket policy allows the principal, but the principal's own account hasn't granted the capability.

What you'd do on Monday

Add an identity policy to etl-runner in account A:

PythonIdentity policy JSON to add to etl-runner in account A (needs IAM console/API; norun)
import json
policy = {
    "Statement": [
        {
            "Effect": "Allow",
            "Action": ["s3:GetObject", "s3:ListBucket"],
            "Resource": [
                "arn:aws:s3:::company-data-lake",
                "arn:aws:s3:::company-data-lake/*"
            ]
        }
    ]
}
print(json.dumps(policy, indent=2))
Scenario B

Imagine you work at Airbnb

1,000+ internal services across 50+ AWS accounts

The numbers

Illustrative. Airbnb has publicly discussed microservice-to-microservice communication across hundreds of services. At 1,000 services × 50 accounts, writing and maintaining individual cross-account trust policy entries per service-to-service pair produces a combinatorial policy maintenance burden: up to 50,000 pairwise policies in the worst case.

What the Scenario-A answer does here

The Scenario A fix (explicit ARN-based trust and bucket policies) still works for five services. At 1,000, updating the bucket policy every time a new service needs access requires a change ticket, a policy document update, a deploy, and a test cycle. Services wait days for access; engineers work around it with overly broad policies.

What flips at this scale

At Airbnb's scale the flip is ABAC with mandatory session tags set by the CI/CD system, combined with a service-mesh identity system (like AWS IAM Roles Anywhere or SPIFFE/SPIRE) that injects role ARNs and tags automatically. Every resource is tagged with owner-team and env; every service's role has matching principal tags. The policy is written once: StringEquals: { aws:ResourceTag/owner-team: ${aws:PrincipalTag/allowed-team} }. No manual policy updates per service.

The trade-off you'd defend

I would defend ABAC with tag enforcement SCPs over explicit ARN-based cross-account policies for any system with more than 20 service-to-service relationships, trading tag governance complexity for elimination of the pairwise policy update cycle. The critical requirement is that tag assignment must be automated and auditable — ABAC only works if you trust the tags.

Illustrative numbers; not Airbnb-published. AWS ABAC documentation is the authoritative source for the technical pattern.

Gotchas

Cross-account requires both sides to allow, even for bucket policy

Symptom

The S3 bucket policy in account B allows the account A principal, but the call still gets AccessDeniedException.

Cause

For cross-account calls, both the bucket policy in account B and the identity policy in account A must allow the action. The account A principal cannot use a resource policy in account B as its sole grant.

Fix

Add an explicit allow to the account A identity policy for the cross-account S3 action, even if the bucket policy is already set.

Session tags are not propagated by default through chained assumptions

Symptom

An ABAC condition that works for a single-hop role assumption fails when the role is assumed a second time in a chain.

Cause

Session tags are not automatically transitive. In a chain A → B → C, B's assumed session has A's tags only if A explicitly set TransitiveTagKeys.

Fix

Set TransitiveTagKeys in the sts:AssumeRole call for any tag you need downstream.

Trust policy change takes effect immediately, but the old session continues

Symptom

You remove a principal from a role's trust policy but their existing assumed session continues to work.

Cause

An assumed session uses credentials that were issued before the trust policy change. The change prevents new assumptions; it does not revoke existing sessions.

Fix

To revoke active sessions immediately, add a condition to the role's permission policy that denies all actions for sessions issued before a specific timestamp, then remove the deny after enough time has passed.

Tricky questions

Is it enough for the S3 bucket policy in account B to allow an account A role directly?
Most people say

Yes — bucket policies grant access; you don't need anything in account A.

Actually

For cross-account access, both sides must allow. The bucket policy in account B is a necessary condition but not sufficient. The account A role's identity policy must also grant the action. (The exception: if the account A role has already assumed an account B role, it is now acting within account B's identity context, and account B's own resource policies apply without needing an account A identity policy grant.)

Do session tags survive an sts:AssumeRole call automatically?
Most people say

Yes, session tags are part of the session so they propagate.

Actually

Only tags explicitly listed in TransitiveTagKeys during the assume call are propagated to further role assumptions. Other session tags stay in the current session only.

Scenario question

You're handed a data platform design where 30 analytics services, each in separate AWS accounts, need read access to a central S3 data lake in a shared account. The existing design uses one bucket policy with 30 explicit ARN entries. A new service needs access every week. What do you propose and what are the risks?

Model answer

I'd propose ABAC: tag every service's IAM role with data-access=true and env during role creation (enforced by a CI/CD pipeline), tag every bucket prefix with the same env, and write a single bucket policy condition that matches them. The access management process becomes: provision the role with the correct tags, and access is automatic. No bucket policy updates per service. The risks I'd name: (1) tag drift — if a role's tags are wrong, access silently fails or is granted incorrectly. Mitigation: SCPs that deny IAM role creation without required tags. (2) Tag manipulation — a caller who can set their own tags could escalate to any resource. Mitigation: use IAM permission boundaries to prevent roles from modifying their own tags. I'd also add AWS IAM Access Analyzer to continuously flag any bucket policy that allows more than intended.

Interview corner

CommonWhat is required for an IAM principal in account A to access S3 in account B?
“I'd say both sides must allow the action. The identity policy in account A must allow the action on the S3 ARN in account B, and the bucket policy in account B must allow the account A principal. If either side is missing, the request is denied.”
TrickIf I assume a role in account B, am I now subject to account B's SCPs?
“I'd say yes. Once you've assumed a role in account B, you're acting as an account B principal and account B's SCPs apply to your actions. You're no longer under account A's SCPs for actions within account B.”
Experience checkDescribe an ABAC policy you would use for a multi-tenant ETL platform.
“I'd tag every ETL role with the tenant identifier as a session tag and every S3 prefix with the same tenant tag. The bucket policy condition would be StringEquals: { aws:ResourceTag/tenant-id: ${aws:PrincipalTag/tenant-id} }. A new tenant gets access by having a correctly tagged role — no policy document change needed. The risk is tag integrity; I'd enforce required tags via SCPs.”
DesignHow do you revoke a cross-account role session immediately?
“A trust policy change stops new assumptions but doesn't affect active sessions. To revoke immediately, I'd add an inline deny policy to the role that denies all actions for sessions whose aws:TokenIssueTime is before the current time, then remove that deny once enough time has passed for all legitimate sessions to expire. This is the AWS-documented revocation pattern.”

Memory hook

Alice needs both her own pass into Building B and a place on Building B's guest list; a sticky note reading "department=finance" is read at the door, not when the pass was printed.

If you remember only one thing:

For cross-account access, always check both the identity policy in account A and the resource policy in account B.

Self-check

  1. What is the minimum trust policy configuration to allow an account A role to assume an account B role?

    The account B role's trust policy must list the account A role's ARN (or the account A root) as a trusted principal with Action: sts:AssumeRole. The account A role also needs an identity policy allow for sts:AssumeRole on the account B role ARN.

  2. How do session tags relate to aws:PrincipalTag condition keys?

    Session tags set during sts:AssumeRole are accessible as aws:PrincipalTag/<key> condition values in the assumed session. They behave identically to tags on the IAM user or role.

  3. Why does ABAC require tag integrity enforcement, and how do you get it?

    Because if a caller can assign their own session tags, they can claim any attribute and access any matching resource. Use IAM permission boundaries to prevent roles from modifying their own tags, and use SCPs to require mandatory tags at resource creation.

Prove it

Write a Python function simulate_cross_account(caller_id_policy, target_trust_policy, target_perm_policy, action, resource). It returns "Allow" only if: (1) the trust policy allows the caller, (2) the target permission policy allows the action on the resource (use the evaluate_policies function from aws-01-03's production block). Test with one allow case and one case where the trust policy blocks the caller. Print both results.

aws-01-05Card 5 of 6

Accounts, tagging and environment boundaries

MediumP3 · Role-dependentTalk about it12 min
organizationstaggingaccountsenvironmentscost-allocation

Why you care

Putting dev and prod in the same AWS account is the equivalent of giving every developer the production database password. One accidental terraform apply with the wrong environment variable deletes prod. One misconfigured IAM policy in dev broadens to prod. The cost bill has no line between "experiments" and "real workloads". Companies that grew AWS accounts one at a time end up with this problem; companies that planned it upfront do not. Interviewers ask about account strategy when hiring for platform or senior roles because it signals whether you have operated production AWS or only played with it.

Explain like I'm 5

Imagine your family shares one kitchen (one AWS account). Everyone's food is in the same fridge, and when your sibling accidentally throws away leftovers, they might throw away yours too. Now imagine each family member has their own mini-fridge (separate account). What's in your fridge can't be touched by anyone else without a key. AWS Organizations is the family home: it keeps all the mini-fridges under one roof, pays one bill, and lets parents set rules (SCPs) that all fridges must follow, like "no buying ice cream after midnight."

Core concept

AWS Organizations groups accounts into a hierarchy of Organizational Units (OUs). A management account sits at the root and can:

  • Apply SCPs to an OU or account to limit what principals in those accounts can do.
  • Enable consolidated billing so all account charges appear on one invoice.
  • Share resources (RAM), delegate service access (AWS Config, Security Hub) to member accounts.

Account boundaries are the strongest isolation in AWS. An IAM policy cannot reach across accounts without explicit cross-account grants (Cross-account IAM and ABAC). This makes separate accounts the right tool for:

  • Environment isolation: one account per stage (dev, staging, prod). A terraform apply in dev cannot affect prod.
  • Blast radius reduction: a misconfigured policy or an accidental ec2:TerminateInstances in dev cannot touch prod.
  • Regulatory separation: some compliance regimes require that production data be in an account accessible only to a named set of identities.

Tagging is the lightweight, within-account complement to account separation:

  • Cost allocation tags (activated in the Billing console) appear in Cost Explorer and CUR reports per-tag value.
  • Tags do not enforce isolation by themselves — IAM ABAC with SCPs does.
  • Tag governance: an SCP can deny ec2:RunInstances if the CostCenter tag is absent, enforcing tagging at creation time.

Naming conventions: a common pattern is <org>-<env>-<team>-<purpose>. For example, acme-prod-data-etl. The account alias (set with iam:CreateAccountAlias) appears in the Console URL and in the IAM console or the iam:ListAccountAliases API. sts:GetCallerIdentity does not return the alias — it returns the numeric account ID, the ARN, and the user ID, so scripts should preflight on the numeric ID, not the alias.

Decision rule: use separate accounts for each environment that carries prod-level data or that can affect customer experience. Use tags within an account for cost allocation, ownership tracking and ABAC. Never rely on tags alone for environment isolation.

Minimal example Tag compliance checker for AWS resources

PythonValidate required tags on simulated resource metadataruns
from dataclasses import dataclass, field

REQUIRED_TAGS = {"env", "team", "cost-center"}

@dataclass
class Resource:
    resource_id: str
    resource_type: str
    tags: dict[str, str] = field(default_factory=dict)

def check_tags(resources: list[Resource]) -> list[str]:
    issues = []
    for r in resources:
        missing = REQUIRED_TAGS - r.tags.keys()
        if missing:
            issues.append(
                f"{r.resource_id} ({r.resource_type}): missing tags {sorted(missing)}"
            )
    return issues

resources = [
    Resource("i-0abc123", "EC2", {"env": "prod", "team": "data", "cost-center": "DE-01"}),
    Resource("s3-raw-data", "S3", {"env": "prod", "team": "data"}),          # missing cost-center
    Resource("rds-pg-main", "RDS", {"team": "data"}),                         # missing env + cost-center
    Resource("lambda-etl", "Lambda", {"env": "dev", "team": "data", "cost-center": "DE-02"}),
]

issues = check_tags(resources)
if issues:
    print(f"Tag compliance issues ({len(issues)}):")
    for i in issues:
        print(f"  {i}")
else:
    print("All resources compliant.")
Output
Tag compliance issues (2):
  s3-raw-data (S3): missing tags ['cost-center']
  rds-pg-main (RDS): missing tags ['cost-center', 'env']

In production Account boundary enforcer — simulate cross-env resource access attempt

PythonDetect cross-environment resource access attemptsruns
from dataclasses import dataclass
from typing import Literal

Env = Literal["dev", "staging", "prod"]

@dataclass
class Account:
    account_id: str
    env: Env
    name: str

@dataclass
class AccessAttempt:
    caller_account: Account
    target_account: Account
    action: str
    resource: str

def check_cross_env_access(attempt: AccessAttempt) -> str:
    if attempt.caller_account.account_id == attempt.target_account.account_id:
        return "same-account: allowed by default IAM policy"

    if attempt.caller_account.env == "dev" and attempt.target_account.env == "prod":
        return (
            f"BLOCKED: dev account {attempt.caller_account.name!r} "
            f"cannot access prod account {attempt.target_account.name!r} "
            f"without explicit cross-account grant"
        )
    return (
        f"cross-account: requires explicit trust policy in {attempt.target_account.name!r}"
    )

accounts = {
    "dev":     Account("111122223333", "dev",     "acme-dev-data-etl"),
    "staging": Account("444455556666", "staging", "acme-staging-data-etl"),
    "prod":    Account("777788889999", "prod",    "acme-prod-data-etl"),
}

attempts = [
    AccessAttempt(accounts["dev"],     accounts["dev"],     "s3:GetObject", "s3:::raw-data/jan.csv"),
    AccessAttempt(accounts["dev"],     accounts["prod"],    "s3:GetObject", "s3:::prod-settlements/jan.csv"),
    AccessAttempt(accounts["staging"], accounts["prod"],    "s3:PutObject", "s3:::prod-settlements/out.csv"),
    AccessAttempt(accounts["dev"],     accounts["staging"], "s3:GetObject", "s3:::staging-data/test.csv"),
]

for a in attempts:
    print(check_cross_env_access(a))
Output
same-account: allowed by default IAM policy
BLOCKED: dev account 'acme-dev-data-etl' cannot access prod account 'acme-prod-data-etl' without explicit cross-account grant
cross-account: requires explicit trust policy in 'acme-prod-data-etl'
cross-account: requires explicit trust policy in 'acme-staging-data-etl'
Scenario A

In your world

The situation

You join a company that has everything in one AWS account: dev EC2 instances, the Jenkins server, the prod RDS database, the prod S3 buckets, and the analytics sandbox. IAM policies are broadly scoped because "it's easier to manage."

What actually breaks

A junior developer runs a Terraform module in the wrong environment. The module has lifecycle { create_before_destroy = true } and accidentally matches a prod S3 bucket prefix in its resource tagging filter. The bucket is emptied. Backups exist but restoring takes six hours.

What you'd do on Monday

Separate prod into its own account with SCPs that deny destructive S3 operations (s3:DeleteObject, s3:DeleteBucket) for all principals except a designated break-glass role. Dev can destroy all it wants; it cannot reach prod. This is not an overnight migration, but starting with prod-isolation of a single critical resource (the RDS database) is the first step.

Scenario B

Imagine you work at Zoho

50M+ users, multi-product SaaS

The numbers

Illustrative. Zoho operates dozens of products (CRM, Desk, Books, Mail) each with distinct compliance and data-residency requirements. Illustrative: 50 product teams × 3 environments (dev/staging/prod) = 150 AWS accounts minimum. At 150 accounts, a central SCP set from the management account provides security guardrails that no individual product team can bypass.

What the Scenario-A answer does here

The Scenario A fix (separate prod account per product) still works as the foundation. What breaks at 50 products: account provisioning becomes a manual bottleneck. Teams wait days for a new account; they share existing accounts to avoid the wait, which defeats the isolation.

What flips at this scale

At Zoho's scale the flip is Account Vending Machine (Account Factory): a self-service pipeline (AWS Control Tower or a home-built Terraform/CodePipeline automation) that provisions a new account from a template in minutes, pre-configured with mandatory SCPs, logging destinations, a cost budget, and a baseline IAM role structure. Teams get isolation without waiting.

The trade-off you'd defend

I would defend automated account provisioning over manual account creation at any scale above 20 accounts, trading the upfront engineering cost of the vending machine for elimination of the "share this account for now" workaround that permanently degrades isolation.

Illustrative numbers; not Zoho-published. AWS Control Tower and Account Factory for Terraform are publicly documented AWS services; Zoho's multi-product scale is publicly known.

Gotchas

Tags are not enforced by default; they are advisory

Symptom

You define a tagging policy but resources are created without the required tags.

Cause

Tags are metadata; without an SCP or AWS Config rule that explicitly denies untagged resource creation, nothing enforces them.

Fix

Add an SCP to the OU that denies ec2:RunInstances (and other key services) when the CostCenter tag is absent. AWS Config can detect non-compliant resources but cannot prevent creation without an SCP.

Consolidated billing groups all charges but does not separate cost visibility automatically

Symptom

You moved to AWS Organizations for billing but the Cost Explorer still shows one undifferentiated total.

Cause

Cost allocation tags must be activated explicitly in the Billing console. Also, account-level cost separation requires viewing costs grouped by Linked Account, which is not the default view.

Fix

Activate cost allocation tags in Billing → Cost Allocation Tags. Set up Cost Explorer views grouped by Linked Account and by tag to see per-environment and per-team costs.

An SCP deny is invisible to principals in the account

Symptom

An engineer calls iam:CreateUser and gets a generic AccessDeniedException with no mention of SCPs.

Cause

SCPs produce the same error as IAM policy denies. There is no flag in the error that says "blocked by SCP."

Fix

Use the IAM Policy Simulator (iam:SimulatePrincipalPolicy) to trace the exact denial reason for the specific request, or check the Organization's attached SCPs manually when debugging unexplained denies, especially after a move to a new OU.

Tricky questions

Do cost allocation tags replace account-level cost separation?
Most people say

Yes — if you tag every resource with env=prod, you can see prod costs in Cost Explorer.

Actually

Tags give you cost visibility within an account, but not isolation. A bug that deletes prod resources still happens in the same account. Use separate accounts for isolation; use tags for cost attribution within or across accounts.

Does an SCP apply to the management account?
Most people say

Yes, SCPs attached to the root apply everywhere including the management account.

Actually

SCPs do not apply to the management account itself. This is an intentional design: the management account is the control plane, and restricting it could lock you out. This also means the management account should be used only for organisation management, not for running workloads.

Scenario question

You're handed a startup that has one AWS account with dev, staging, and prod all coexisting. The CTO wants to "tighten up AWS" before a SOC 2 audit in three months. You have two engineers available. What do you prioritise and why?

Model answer

I'd prioritise separating the production database and production S3 buckets into a dedicated prod account first, because that directly addresses the SOC 2 requirement for access controls between environments. I'd create a new account under AWS Organizations, move the prod RDS and S3 there, add SCPs that deny destructive operations except via a named deploy role, and update the ETL jobs to use cross-account role assumption. That is achievable in two to three weeks and immediately demonstrates environmental separation to an auditor. I'd defer moving the prod EC2 compute until after the audit if the timeline is tight. The trade-off is that a split migration leaves the prod account partially populated with some compute still in the original account, which means the cross-account IAM setup is mandatory immediately. I'd document the migration state clearly in the audit evidence.

Interview corner

CommonWhy should dev and prod be in separate AWS accounts?
“I'd start with this: account boundaries are the strongest isolation in AWS. An IAM policy, a Terraform run, or a misconfigured SDK call in dev cannot reach prod if they're in separate accounts. Tags can track cost, but they can't stop a delete. The account boundary stops it structurally.”
TrickIf SCPs apply to all accounts in an OU, do they apply to the management account too?
“I'd say no — SCPs explicitly do not apply to the management account. It's intentional so you can't accidentally lock yourself out of the control plane. That's also why you should run zero production workloads in the management account.”
Experience checkHow would you enforce tagging on all new EC2 instances?
“I'd add an SCP to the OU that denies ec2:RunInstances when a required tag like CostCenter is absent. That blocks creation, not just flags it. I'd pair it with an AWS Config rule for after-the-fact auditing of anything that slipped through before the SCP was applied.”
DesignWhat is an Account Vending Machine and when do you need one?
“I'd describe it as an automated pipeline that creates a new AWS account from a template — SCPs, IAM roles, logging, budgets, all pre-configured. You need it when manual account provisioning becomes a bottleneck, typically beyond 15-20 accounts. Without it, teams share accounts to avoid the wait and you lose the isolation you were trying to get.”

Memory hook

One shared kitchen means a sibling's mistake can empty everyone's fridge; separate mini-fridges keep your food safe no matter what happens in theirs.

If you remember only one thing:

Never put dev and prod in the same account — a mistaken API call in one account cannot reach the other.

Self-check

  1. Why do SCPs not apply to the management account?

    By design — the management account is the Organisation's control plane, and applying SCPs to it could cause a lockout with no recovery path.

  2. What does activating a cost allocation tag do?

    It makes that tag key and its values available as dimensions in Cost Explorer and Cost and Usage Report, enabling cost filtering and attribution by that tag.

  3. What is the difference between using tags for environment separation versus using separate accounts?

    Tags are metadata; they affect cost visibility and ABAC conditions but do not structurally prevent resources in one logical environment from being modified by code running in another. Account boundaries are a hard isolation enforced by IAM — cross-account access requires explicit grants from both sides.

Prove it

Run the minimal block's tag compliance checker. Add two more resources: a Lambda missing all three required tags, and an RDS instance with all three present. Confirm the output lists three issues total (the two from the original snippet plus the new Lambda). Paste the final output.

aws-01-06Card 6 of 6

Budgets and controlled lab spend

EasyP4 · DifferentiatorTalk about it12 min
aws-budgetscost-controlalertslab-safetyteardown

Why you care

Every AWS learning project or ETL lab ends the same way unless you plan for it: you forget a running EC2 instance, a NAT Gateway, or a data transfer job, and your card is charged $200 while you were not looking. AWS Budgets is the mechanism that alerts you when spend exceeds a threshold. But budgets are not an instant circuit breaker: there is a processing delay between spend and alert, and Budget Actions (auto-responses to budget breaches) only act within the permissions they are configured with. Interviewers rarely ask about budgets directly, but a senior engineer who cannot describe how they control AWS costs in a lab or a staging environment is a risk. The practical importance is higher than the interview probability.

Explain like I'm 5

A budget is a smoke alarm, not a fire extinguisher. The smoke alarm (budget alert) tells you there is smoke (you have spent 80% of your limit). The fire extinguisher (Budget Action) can be configured to kick in — but only if you have set it up in advance and only within the limits of what it has permission to do. And just like a smoke alarm has a small delay before it triggers, AWS Budgets can be a few hours late. Never assume a budget will stop a runaway cost immediately.

Core concept

AWS Budgets lets you set a monthly (or custom period) cost or usage threshold. When actual or forecasted spend crosses a percentage of the threshold, it can:

  • Send an SNS notification (email, Slack via SNS→Lambda, PagerDuty).
  • Trigger a Budget Action: apply an IAM policy, target an EC2 Auto Scaling group, or run an SSM document.

Budget alert latency: AWS processes cost data with a delay. Actual cost alerts typically fire within 8 hours of the spend event, not immediately. Forecasted alerts fire earlier but based on a projection. Do not rely on budgets to stop a cost spike within minutes.

Budget Actions: a budget action runs automatically when an alert threshold is crossed. Available action types:

  • Apply an IAM policy — attach a deny-all managed policy to an IAM user, group, or role.
  • Target an EC2 Auto Scaling group — modify the min/max capacity.
  • Run an SSM Automation document — execute arbitrary automation (stop instances, delete resources).

The action's IAM execution role must have the permissions to perform those actions. If the role lacks ec2:StopInstances, the action fails silently.

Lab safety checklist (do this at the end of every lab session):

  1. List all running EC2 instances: aws ec2 describe-instances --filters "Name=instance-state-name,Values=running".
  2. List NAT Gateways (hourly charge even when idle): aws ec2 describe-nat-gateways --filter "Name=state,Values=available".
  3. List RDS instances: aws rds describe-db-instances.
  4. Check S3 storage used: aws s3api list-buckets + lifecycle rules.
  5. Check Data Transfer charges in Cost Explorer for the previous 24 hours.
  6. Set a budget with an email alert at 50% and 100% of your monthly limit.

Costs that surprise people: NAT Gateway ($0.045/hr plus data transfer; check the current pricing page), idle RDS instances (charged even when not queried), data transfer out of S3 to the internet, and Elastic IPs not attached to running instances.

Decision rule: set a budget with an SNS alert before you create any AWS resource in a learning account. Use a Budget Action that applies a deny-all policy to your learning IAM role if you want automated protection — but test the action manually first to confirm the execution role has the right permissions.

Minimal example Lab safety cost simulator with daily teardown checker

PythonSimulate daily lab cost accumulation and budget breach alertruns
from dataclasses import dataclass

@dataclass
class Resource:
    name: str
    hourly_cost_usd: float
    running: bool

def daily_cost(resources: list[Resource]) -> float:
    return sum(r.hourly_cost_usd * 24 for r in resources if r.running)

resources = [
    Resource("ec2-t3-medium",   0.0416, True),   # illustrative; check current pricing
    Resource("nat-gateway",     0.045,  True),   # illustrative; check current pricing
    Resource("rds-db-t3-micro", 0.017,  True),   # illustrative; check current pricing
    Resource("s3-data-lake",    0.0,    True),   # storage billed monthly, not hourly here
]

month_spend = 0.0
budget_limit = 50.0
thresholds = [0.5, 0.8, 1.0]
fired: set[str] = set()
alert_log: list[str] = []

for day in range(1, 32):
    month_spend += daily_cost(resources)
    for t in thresholds:
        key = f"{t*100:.0f}%"
        if month_spend >= budget_limit * t and key not in fired:
            fired.add(key)
            alert_log.append(
                f"Day {day:02d}: ALERT: {month_spend:.2f} USD spent "
                f">= {t*100:.0f}% of {budget_limit:.2f} USD budget"
            )

for entry in alert_log:
    print(entry)
print(f"Month total (illustrative): {month_spend:.2f} USD")
Output
Day 11: ALERT: 27.35 USD spent >= 50% of 50.00 USD budget
Day 17: ALERT: 42.27 USD spent >= 80% of 50.00 USD budget
Day 21: ALERT: 52.21 USD spent >= 100% of 50.00 USD budget
Month total (illustrative): 77.08 USD

In production Teardown checklist runner (stdlib only, no AWS calls)

PythonLab teardown checklist — runs against a local resource manifestruns
import json
from pathlib import Path

MANIFEST = [
    {"type": "EC2",         "id": "i-0abc123",  "state": "running",   "region": "ap-south-1"},
    {"type": "NAT Gateway", "id": "nat-0xyz456","state": "available", "region": "ap-south-1"},
    {"type": "RDS",         "id": "db-main",    "state": "available", "region": "ap-south-1"},
    {"type": "EC2",         "id": "i-0stopped", "state": "stopped",   "region": "ap-south-1"},
    {"type": "Elastic IP",  "id": "eipalloc-1", "state": "unattached","region": "ap-south-1"},
]

CHARGEABLE_STATES = {
    "EC2":         ["running"],
    "NAT Gateway": ["available"],
    "RDS":         ["available", "stopping"],
    "Elastic IP":  ["unattached"],  # charged when unattached to a running instance
}

def teardown_check(manifest: list[dict]) -> None:
    flagged = []
    for r in manifest:
        chargeable = CHARGEABLE_STATES.get(r["type"], [])
        if r["state"] in chargeable:
            flagged.append(r)

    if not flagged:
        print("Teardown OK: no chargeable resources found.")
        return

    print(f"WARNING: {len(flagged)} chargeable resource(s) still running:")
    for r in flagged:
        print(f"  [{r['type']}] {r['id']} — state={r['state']} region={r['region']}")
    print("\nAction: stop or terminate these resources before ending your lab session.")

teardown_check(MANIFEST)
Output
WARNING: 4 chargeable resource(s) still running:
  [EC2] i-0abc123 — state=running region=ap-south-1
  [NAT Gateway] nat-0xyz456 — state=available region=ap-south-1
  [RDS] db-main — state=available region=ap-south-1
  [Elastic IP] eipalloc-1 — state=unattached region=ap-south-1

Action: stop or terminate these resources before ending your lab session.
Scenario A

In your world

The situation

You create a VPC with a NAT Gateway for a networking lab. The lab works. You spend Friday afternoon on it, note the results, and close your laptop. You do not delete the NAT Gateway because you plan to resume Monday.

What actually breaks

Monday comes and goes; you move on to other work. The NAT Gateway runs all weekend and into the following two weeks. The charge is approximately $0.045/hr × 24 hr × 14 days ≈ $15.12 for the NAT Gateway alone, plus data transfer. At the end of the month your AWS bill is $45 instead of the expected $5. The budget alert fired at day 9 to an email address you had not checked.

What you'd do on Monday

After every lab session, run the teardown checklist above. Set the budget alert recipient to a mobile number via SNS → SMS so it reaches you even if you miss email. Add a calendar reminder for "AWS teardown check" at end of Friday.

Scenario B

Imagine you work at PhonePe

500M+ registered users, 2,400+ crore transactions/month

The numbers

Illustrative. PhonePe processes 2,400+ crore (24 billion) transactions per month. At that scale, cost control is not a budget alert — it is an architectural constraint. A 1-rupee-per-transaction data-transfer overhead at 24B transactions/month is 24B rupees (approximately $288M USD). No budget alert prevents that; the architecture must be designed for it.

What the Scenario-A answer does here

The Scenario A fix (budget alert via SNS) still works for lab safety. At PhonePe's scale, a budget alert fires after the cost has already been incurred. An 8-hour alert latency on a workload that burns $100,000/hour means $800,000 is already spent before the alert arrives.

What flips at this scale

At PhonePe's scale the flip is proactive cost governance: unit economics tracked per transaction (cost per API call, cost per S3 put), capacity reservation (Savings Plans, Reserved Instances) for predictable baseline load, and real-time spend anomaly detection via Cost Anomaly Detection with an action that pages an on-call engineer. The budget is still there as a ceiling, but it is the last line of defence, not the first.

The trade-off you'd defend

I would defend proactive unit-cost tracking over reactive budget alerts for any workload above $10,000/month, trading the engineering effort to instrument unit costs for early warning before a budget alert would fire. Budget alerts stay as the safety net.

Illustrative numbers; not PhonePe-published. Their transaction volume is from public announcements; the hourly cost burn figures are illustrative estimates only.

Gotchas

Budget alerts fire with a delay, not immediately

Symptom

A runaway Lambda function burns $50 before the budget alert arrives.

Cause

AWS Budgets processes spend data with up to an 8-hour delay. A forecasted alert fires earlier but is based on projection, not actual spend.

Fix

Set forecasted alerts (not just actual) at 80% of budget to get earlier warning. For critical workloads, add CloudWatch billing alarms (which check the EstimatedCharges metric and have a ~6 hour update frequency) as a second layer.

Budget Actions require the execution role to have the right permissions

Symptom

You configure a Budget Action to stop EC2 instances on budget breach, but the action never fires.

Cause

The IAM role that the Budget Action executes as must have ec2:StopInstances. If it does not, the action fails silently.

Fix

Test the Budget Action manually before relying on it. Check the action execution logs in the Budgets console.

NAT Gateway charges even when no traffic is flowing

Symptom

A "zero-use" VPC still shows charges.

Cause

NAT Gateways charge an hourly fee regardless of traffic, plus per-GB data transfer. An idle NAT Gateway costs approximately $0.045/hr (check the current pricing page) just for existing.

Fix

Delete NAT Gateways at the end of any lab session. Recreate them when needed. In prod, a NAT Gateway is a fixed cost you accept; in a lab, it is optional.

Tricky questions

Does a budget alert stop a runaway cost immediately?
Most people say

Yes — once the threshold is breached, AWS stops incurring charges.

Actually

No. A budget alert notifies you; it does not stop anything. A Budget Action can take automated steps, but only within the permissions it has been configured with, and only after the alert fires (which has a delay). The cost has already been incurred when the alert triggers.

Is a forecasted budget alert more reliable than an actual cost alert?
Most people say

Yes — it fires before you actually spend the money.

Actually

A forecasted alert fires based on a spend projection, which may be wrong. It is more proactive but less certain. Use both: a forecasted alert at 80% and an actual alert at 100%.

Scenario question

You're setting up a personal AWS learning account for 60 days of labs from this book. Your budget is Rs.2,000/month (approximately $24 USD). You want to avoid surprise charges. Describe your full setup.

Model answer

I'd do three things before running the first lab. First, set a budget: $20/month in AWS Budgets with email and SMS alerts at 50%, 80%, and 100% — SMS because I check email inconsistently. Second, enable MFA on the root account and create a personal IAM user for daily work; I never use root for API calls. Third, write a teardown script that lists all running EC2 instances, available NAT Gateways, running RDS instances, and unattached Elastic IPs; I run it at the end of every session. Beyond that, I'd avoid creating NAT Gateways unless a specific lab requires them (use VPC endpoints for S3/DynamoDB instead, which are free). I'd use t2.micro or t3.micro instances that qualify for free tier where possible. The trade-off: this setup catches problems within 8 hours; it does not prevent them. For labs that could create large data-transfer charges (anything that moves data across regions or to the internet), I'd add a CloudWatch billing alarm with a $5 threshold to get a tighter warning window.

Interview corner

CommonWhat is the delay between incurring an AWS cost and an AWS Budget alert firing?
“Actual cost budget alerts can take up to 8 hours to fire after the spend is incurred — the billing data is processed in batches, not in real time. Forecasted alerts fire earlier based on projected spend. I always set both, and I don't assume a budget alert will stop a cost spike before it's already happened.”
TrickDoes a Budget Action automatically stop the cost?
“Only if you've set it up correctly and the execution role has the right permissions. A budget alert tells you; a Budget Action reacts, but it can stop instances or apply deny policies only for resources its role can touch. It's not a kill switch for all costs — I've seen teams configure a Budget Action that stops EC2 but not an active Glue job that's the actual cost driver.”
Experience checkDescribe a cost surprise you'd protect a new AWS learning account against.
“The classic one is a forgotten NAT Gateway. It charges by the hour regardless of traffic. I always add a teardown checklist for end-of-session and a budget alert at 50% so I catch creep early. I'd also set a CloudWatch billing alarm as a second layer, since it checks the EstimatedCharges metric which is updated roughly every 6 hours.”

Memory hook

A budget is a smoke alarm with an 8-hour delay. Set it before the first resource, not after the first bill.

If you remember only one thing:

Budget alerts notify; they do not stop charges.

Self-check

  1. What is the approximate delay between incurring spend and an actual cost budget alert firing?

    Up to 8 hours. Budgets process billing data in batches, not in real time.

  2. What three pieces of information does a Budget Action need to execute successfully?

    A configured threshold (the spend % that triggers it), the action type (IAM policy apply, EC2 ASG change, SSM document), and an IAM execution role that has permissions to perform the action.

  3. Why does a NAT Gateway cost money even when no traffic is flowing through it?

    NAT Gateways have an hourly provisioning charge ($0.045/hr; check the current pricing page) independent of traffic. Data transfer is an additional per-GB charge on top.

Prove it

Copy the minimal block. Change the nat-gateway resource to running = False (simulate you remembered to delete it). Observe that the 50% alert now fires on day 12 instead of day 9, and the 100% alert fires on day 23 instead of day 19. Paste the new output and confirm the month total is lower.