Shelf · AWS · Chapter 10

Chapter 10: Security in depth

IAM evaluation from chapter 1 answers "is this call allowed." This chapter answers the five questions that come after: is the data itself protected, where do secrets live, who stops a flood of bad requests, what proves an incident happened, and how do you write a policy that is actually narrow instead of merely working.

Difficulty
Priority
aws-10-01Card 1 of 6

Key management

HardP3 · Role-dependentTalk about it10 min
kmsenvelope-encryptiondata-keyskey-policygrantsrotation

Why you care

Every SSE-KMS bucket and every Secrets Manager secret sits on one mechanism: envelope encryption. Know only "KMS encrypts things" and you will eventually hit an AccessDeniedException on s3:GetObject where every S3 permission is correct, because the real blocker is a policy on the key, not the bucket. "The identity policy allows it" is the answer of someone who has never debugged a production decrypt failure — the real answer always checks two policies. IAM policy evaluation owns IAM evaluation; this card owns the one resource policy here that is default-deny for everyone, including the account root.

Explain like I'm 5

A KMS key is a vault that never leaves the room. To lock a box you ask the clerk for a one-time combination, handed to you two ways: the plain combination, and the same combination sealed in an envelope only the vault can reopen. You lock the box, throw away the plain combination, and keep only the sealed envelope. Later, to open the box, you hand the envelope back — the clerk only breaks the seal if you're on this vault's approved list, a list the vault owns, not your badge issuer.

Core concept

  • Envelope encryption: GenerateDataKey returns a data key as plaintext and ciphertext. You encrypt your payload locally with the plaintext key, store the ciphertext key alongside it, and discard the plaintext immediately. KMS never touches bulk data, only the small key — which is why one mechanism scales from a secret to a 20 GB file.
  • Decrypt: you send KMS the ciphertext key, not the payload; it returns the plaintext key only if authorized, and you decrypt locally.
  • Key policy is default-deny for everyone. No principal — not even root — has any permission unless a key policy, IAM policy, or grant explicitly allows it. The default key policy must explicitly enable IAM delegation, or an IAM allow does nothing on this key.
  • Grants are narrow, revocable permissions on a key for a specific principal, used to hand out access without editing the key policy — AWS services use these on your behalf constantly.
  • Rotation: a customer managed key can auto-rotate annually, creating new backing material under the same key ID/ARN; old material is kept so old ciphertext stays readable. AWS managed keys rotate yearly with no opt-out.

Decision rule: use an AWS managed key when you don't need to control decrypt access or cross-account sharing. Use a customer managed key the moment you need to name principals, share across accounts, or audit key usage separately.

Under the hood

A KMS key wraps an HSM backing key under a domain key AWS rotates on its own schedule. "Rotating" a customer managed key creates a new backing key version under the same key ID; the old one is kept so prior ciphertext still unwraps. The consequence: rotating a key never re-encrypts anything already written. To re-encrypt existing data under new material, you must re-write the objects yourself.

Minimal example Only the wrapped key is ever kept

PythonThe plaintext data key is used once, then discardedruns
plaintext_key = bytes([91, 14, 233, 7])               # KMS would hand you this once
wrapped_key = bytes(b ^ 0xFF for b in plaintext_key)  # ciphertext form, safe to store

stored_object = {"ciphertext_key": wrapped_key}       # plaintext_key never gets written anywhere
del plaintext_key                                     # the only copy is gone now

print(stored_object)
print("plaintext_key" in locals())
Output
{'ciphertext_key': b'\xa4\xf1\x16\xf8'}
False

In production The key policy decides decrypt, not just the identity policy

PythonIdentity allow alone is not enough without key-policy delegationruns
from dataclasses import dataclass, field


@dataclass
class KeyPolicy:
    enable_iam_delegation: bool
    allowed_principals: set = field(default_factory=set)  # explicit allow in the key policy


@dataclass
class IdentityPolicy:
    allowed_principals_actions: dict


def can_decrypt(principal: str, key_policy: KeyPolicy, identity_policy: IdentityPolicy) -> str:
    identity_allows = "kms:Decrypt" in identity_policy.allowed_principals_actions.get(principal, set())
    key_policy_allows = principal in key_policy.allowed_principals or (
        key_policy.enable_iam_delegation and identity_allows
    )
    if not identity_allows:
        return "Deny: identity policy does not allow kms:Decrypt"
    if not key_policy_allows:
        return "Deny: key policy does not grant this principal, and IAM delegation is not enabled"
    return "Allow"


identity = IdentityPolicy({"role/etl-runner": {"kms:Decrypt", "kms:GenerateDataKey"}})

key_no_delegation = KeyPolicy(enable_iam_delegation=False)
key_with_delegation = KeyPolicy(enable_iam_delegation=True)
key_explicit = KeyPolicy(enable_iam_delegation=False, allowed_principals={"role/etl-runner"})

print(can_decrypt("role/etl-runner", key_no_delegation, identity))
print(can_decrypt("role/etl-runner", key_with_delegation, identity))
print(can_decrypt("role/etl-runner", key_explicit, identity))
Output
Deny: key policy does not grant this principal, and IAM delegation is not enabled
Allow
Allow

See it happen GenerateDataKey and Decrypt round trips to KMS

ETL jobKMSS3 objectGenerateDataKey(keyId)plaintext key + ciphertext keyPUT encrypted payload + ciphertext keyGetObject later: ciphertext payload + keyDecrypt(ciphertext key)plaintext keydecrypt payload locally
  1. Before writing, the job asks KMS for a fresh data key.
  2. KMS returns both forms; only the ciphertext form is ever stored.
  3. The payload is encrypted locally; the plaintext key is discarded right after.
  4. A later read returns the wrapped key alongside the data — KMS has not been involved yet.
  5. To read it back, the job sends the small wrapped key to KMS, never the payload.
  6. KMS unwraps and returns the key only if the key policy (and identity policy) allow it.
  7. The job decrypts the payload itself; KMS never saw the bulk data at either end.
Notice: KMS only ever handles the small data key, never the object itself, which is why one mechanism covers a 200-byte secret and a 20 GB file identically.
Scenario A

In your world

The situation

Your ETL role has had s3:GetObject/s3:PutObject on settlements-prod/* for a year under an AWS managed key. The platform team migrates the bucket to a customer managed key for a compliance requirement, using a boilerplate Terraform module.

What actually breaks

The next run fails every GetObject with AccessDeniedException. The IAM role hasn't changed. The actual cause: the boilerplate key policy only grants the platform team's own admin role and never delegates to IAM or names your ETL role — your existing s3:GetObject allow is irrelevant to the failure.

What you'd do on Monday

Check the key, not the bucket. Read the current key policy:

PythonRead a KMS key policy (needs boto3 + AWS credentials)
import boto3

kms = boto3.client("kms")
key_id = "alias/settlements-prod"
policy = kms.get_key_policy(KeyId=key_id, PolicyName="default")["Policy"]
print(policy)  # inspect whether IAM delegation is enabled and whether etl-runner is named

Then add either an explicit statement naming your role, or enable IAM delegation and add kms:Decrypt/kms:GenerateDataKey to the role's identity policy, scoped to this key's ARN — not Resource: "*".

Scenario B

Imagine you work at Juspay

40M encrypted transactions/day

The numbers

Illustrative: 40M transactions/day, each with one encrypted payload, is 40M GenerateDataKey calls plus 40M Decrypt calls — about 80M KMS calls/day, under 1,000/s on average but with sharp retail-hour peaks.

What the Scenario-A answer does here

The Scenario A fix still applies, but one shared customer managed key across every transaction type means a single key-policy mistake locks out every bank integration at once, and all 80M calls compete for the same key's request allowance.

What flips at this scale

At this scale you partition keys by data classification — card tokens, PII, logs — so one bad policy or quota pressure point only affects one data class. On hot paths you also stop calling GenerateDataKey per record and reuse one data key per batch, trading a larger crypto-period for far fewer KMS calls.

The trade-off you'd defend

I would defend per-classification keys plus batched data-key reuse over one key for everything, trading more key inventory for a scoped blast radius and headroom under whatever the account's current KMS limits are.

Illustrative transaction volume; not Juspay-published. KMS request limits change over time, so none are asserted here.

Gotchas

An S3 allow is not a decrypt allow

Symptom

s3:GetObject succeeds on unencrypted objects but returns AccessDeniedException only for SSE-KMS objects, even with s3:GetObject present.

Cause

Reading an SSE-KMS object needs kms:Decrypt, authorized by both the identity policy and the key policy. An S3 permission says nothing about key permissions.

Fix

Check the key policy for that object's key, not just the bucket; name the principal explicitly or enable IAM delegation and grant kms:Decrypt scoped to the key ARN.

Rotating a key does not re-encrypt your data

Symptom

After enabling rotation, old objects still decrypt fine, and a reviewer asks why that doesn't prove "new" encryption.

Cause

Rotation creates new material for future calls; it never touches ciphertext already written, which KMS retains so old data stays readable.

Fix

If re-encryption under new material is required, read and re-write the objects yourself; rotation alone doesn't do it.

Losing the wrapped data key is losing the data

Symptom

A bug deletes the ciphertext-key column from a metadata table; the KMS key itself is healthy.

Cause

KMS can only unwrap a ciphertext key it's given — a healthy master key is irrelevant if the wrapped key is gone.

Fix

Back up the wrapped data key with the same rigor as the payload it protects.

Tricky questions

If my identity policy allows kms:Decrypt on Resource: "*", can I decrypt any key in the account?
Most people say

Yes, a wildcard resource covers every key.

Actually

Only keys whose key policy opts into IAM delegation (or names this principal) are reachable. The key policy remains the gate regardless of the identity policy's wildcard.

Does rotating a customer managed key change its ARN?
Most people say

Yes, every reference needs updating.

Actually

No. The key ID and ARN stay fixed for life; only the backing material rotates. SSE configuration and IAM references keep working unmodified.

Scenario question

You're handed a Lambda reading from an SSE-KMS bucket. Its role has s3:GetObject and kms:Decrypt on Resource: "*", yet every invocation fails AccessDeniedException on GetObject. What do you check, and what's the minimal fix?

Model answer

I would check the key's own key policy, not the IAM role again — a wildcard identity allow is only half the requirement. If the key policy doesn't name this role or enable IAM delegation, the request is denied regardless of the identity policy. I'd fix it narrowly: add a key-policy statement naming the Lambda role's ARN for kms:Decrypt, scoped to this key, rather than widening the identity side further. The trade-off: every new consumer of this bucket now needs an explicit key-policy edit, more friction than a wildcard, but it keeps the blast radius of a compromise to the principals actually listed.

Interview corner

CommonWhat is envelope encryption and why does KMS use it?
“I'd describe it as encrypting data locally with a one-time data key, then having KMS wrap only that small key. That's why the same service handles a tiny secret and a multi-gigabyte file with the same two calls: GenerateDataKey and Decrypt.”
TrickMy role has full s3:GetObject, but reading an encrypted object still fails. Why?
“I'd suspect the KMS key policy. SSE-KMS needs kms:Decrypt authorized by both the identity policy and the key's own key policy, which is default-deny for everyone, including root.”
Experience checkWhere has a KMS permission issue bitten a deployment you were close to?
“I've seen a bucket migrated to a customer managed key whose policy only names the platform team's admin role. Every consumer breaks at once, and the fix is always on the key policy, never the IAM role.”
DesignHow would you structure KMS keys for a platform with several data classes?
“I'd use one key per data classification so a policy mistake only concerns one class of data, and for high call volumes I'd batch data-key generation instead of calling GenerateDataKey per record.”

Memory hook

The vault clerk never lets your combination leave the room; you only ever carry the sealed envelope.

If you remember only one thing:

s3:GetObject allowed is not kms:Decrypt allowed — always check the key policy too.

Self-check

  1. What two things does KMS return from GenerateDataKey, and what happens to each?

    A plaintext data key, used immediately to encrypt locally then discarded, and a ciphertext data key, which is stored alongside the encrypted data permanently.

  2. Why can the account root be denied access to its own KMS key?

    Key policies are default-deny for every principal, including root; nothing is allowed unless a key policy, IAM policy, or grant explicitly says so, with no exceptions by identity.

  3. Does automatic rotation re-encrypt existing S3 objects under a customer managed key?

    No. Rotation only changes which backing key material future encryptions use; previously encrypted data stays readable using the retained old material.

Prove it

Extend the production block with a grants dict mapping principal -> {allowed_actions}, separate from KeyPolicy. Write a can_use(principal, action, grants) function and three tests: a principal with a grant for Decrypt only (succeeds for Decrypt, fails for GenerateDataKey), a principal with no grant (fails both), and a revoked grant (remove the entry, confirm it now fails). Paste all three results.

aws-10-02Card 2 of 6

Secrets Manager versus Parameter Store

MediumP3 · Role-dependentTalk about it11 min
secrets-managerparameter-storerotationconfigurationkms

Why you care

Every TCS-style Python job has a config file or environment variable holding a database password, and the first security review you survive will ask where that password actually lives. Pick the wrong AWS service for it and you either pay for rotation infrastructure you don't need, or you store a credential in a place that was never built to rotate it, and six months later a departed vendor's static password is still valid. Interviewers ask "Secrets Manager or Parameter Store?" because the honest answer requires knowing what each one is actually for, not just that both can hold a string — getting it backwards is a visible signal that you've only ever read about secrets management, never operated it.

Explain like I'm 5

Parameter Store is a labelled filing cabinet: good for anything you'd write on an index card — a URL, a feature flag, an AMI id — and some drawers lock with a key (SecureString) if the card is sensitive. Secrets Manager is a safe with a built-in clockwork mechanism: it is built specifically for things that need to change themselves on a schedule, like a password, without you remembering to swap the card. You could technically lock a password in the filing cabinet, but nothing in the cabinet will ever change it for you.

Core concept

  • Parameter Store stores named key-value configuration: String, StringList, and SecureString (KMS-encrypted at rest). Standard-tier parameters are free, support up to 10,000 per account/Region, and 4 KB values; advanced tier raises limits and adds sharing across accounts, at a cost. It has no built-in rotation — a SecureString just stays encrypted until something external changes it.
  • Secrets Manager is purpose-built for credentials: database passwords, API keys, OAuth tokens. It supports automatic rotation on a schedule, either through AWS's managed rotation for supported databases or a Lambda function you own for anything else. You pay per secret per month plus per API call; there is no free tier for the secret's existence.
  • Decision rule: if the value is a credential, or it needs rotation, cross-account sharing with fine-grained audit, or native database integration — Secrets Manager. If the value is static configuration — a URL, a flag, an approved AMI id, a tuning parameter — Parameter Store, with SecureString only if the value itself is sensitive but doesn't need to rotate.
  • Both encrypt via KMS: Parameter Store's SecureString and Secrets Manager's secret value both ultimately hit the Key management decrypt boundary — an IAM allow on ssm:GetParameter or secretsmanager:GetSecretValue is not sufficient by itself if a customer managed key is involved.
  • Retrieval cost adds up fast if you don't cache. Both services charge or throttle per API call; a Lambda or long-running job that calls GetSecretValue on every invocation, rather than once per warm container lifetime, pays for it in cost and in TPS headroom.

Minimal example One flag decides the store, not the data's sensitivity

PythonNeeding rotation, not needing encryption, is the deciding questionruns
def recommend(is_credential, needs_rotation):
    if is_credential or needs_rotation:
        return "Secrets Manager"
    return "Parameter Store"


print(recommend(is_credential=True, needs_rotation=True))    # db password
print(recommend(is_credential=False, needs_rotation=False))  # feature flag
Output
Secrets Manager
Parameter Store

In production Caching retrieval across warm invocations

PythonFewer API calls, same freshness, measured over a runruns
class SecretCache:
    def __init__(self, ttl_seconds=300):
        self.ttl = ttl_seconds
        self._value = None
        self._fetched_at = None
        self.api_calls = 0

    def get(self, now, fetch_fn):
        if self._value is None or (now - self._fetched_at) > self.ttl:
            self._value = fetch_fn()
            self._fetched_at = now
            self.api_calls += 1
        return self._value


def fetch_db_password():
    return "s3cr3t-rotated-value"  # stands in for a GetSecretValue call


cache = SecretCache(ttl_seconds=300)
invocation_times = [0, 10, 20, 310, 320, 900]  # seconds since cold start
for t in invocation_times:
    cache.get(t, fetch_db_password)

print("invocations:", len(invocation_times))
print("actual Secrets Manager API calls:", cache.api_calls)
Output
invocations: 6
actual Secrets Manager API calls: 3
Scenario A

In your world

The situation

Your Jenkins-triggered Python job calls boto3.client("secretsmanager").get_secret_value(...) at the top of every function invocation, inside a Lambda that gets invoked a few hundred times a minute during the morning batch window.

What actually breaks

Nobody notices until the monthly bill shows a line item for Secrets Manager API calls that's larger than the compute cost of the Lambda itself, and a load test later that quarter trips Secrets Manager's request throttling, failing jobs with ThrottlingException during a traffic spike — not because the secret is wrong, but because you re-fetched it on every single invocation.

What you'd do on Monday

Cache the secret at module scope so it survives across warm Lambda invocations in the same execution environment, and only re-fetch after a TTL you control (the production example above). For many services the Parameters and Secrets Lambda Extension does this caching for you via a local HTTP endpoint, which is worth checking before writing your own.

Scenario B

Imagine you work at Chargebee

3,000 customer database credentials

The numbers

Illustrative: a billing SaaS platform with, say, 3,000 customer-specific database connections across dev/staging/prod, each needing its own credential rotated on a schedule, is 3,000 secrets. At a rotation cadence of every 30 days, that's roughly 100 rotations a day system-wide, each needing a Lambda invocation and a verified cutover.

What the Scenario-A answer does here

The Scenario A fix (caching with a TTL) still matters, but at 3,000 secrets a human clicking "rotate now" per secret, or a single shared rotation Lambda hardcoded per secret name, does not scale; secrets drift out of rotation silently and nobody notices until an audit.

What flips at this scale

At this scale rotation itself becomes templated: one generic rotation Lambda parameterized by secret metadata (tags identifying the database type and connection details), triggered on a schedule across all matching secrets, with a dashboard tracking "last successfully rotated" per secret so a stalled rotation is visible before an auditor finds it, not after.

The trade-off you'd defend

I would defend one templated rotation function plus per-secret rotation-health monitoring over 3,000 bespoke rotation Lambdas, trading some flexibility per secret for the ability to actually operate rotation at this count without a dedicated team per customer.

Illustrative customer and secret counts; not Chargebee-published figures.

Parameter Store vs Secrets Manager vs AppConfig

AxisParameter StoreSecrets ManagerAppConfig
Built forStatic config, key-value lookupsCredentials needing lifecycle managementFrequently-changing app config, safe rollout
RotationNone built-inAutomatic, scheduled, Lambda or managedNot applicable
Deployment safetyNone (next read sees new value)Not applicableGradual rollout, validation, auto-rollback on alarm
Typical costFree at standard tierPer secret per month + per callPer configuration request
EncryptionOptional SecureString via KMSAlways KMS-encrypted at restAWS-managed, optional customer key

Gotchas

Calling GetSecretValue on every invocation

Symptom

Secrets Manager cost line grows roughly linearly with Lambda invocation count, and under load you see ThrottlingException.

Cause

No caching between invocations; each one re-fetches a value that almost never changes between calls.

Fix

Cache at module/container scope with a TTL shorter than your rotation window, or use the Parameters and Secrets Lambda Extension.

Expecting Parameter Store to rotate a SecureString

Symptom

A database password stored as a Parameter Store SecureString is still valid a year after a security policy mandated 90-day rotation.

Cause

Parameter Store has no rotation mechanism; SecureString only encrypts the value, it does not change it.

Fix

Move credentials that must rotate to Secrets Manager; keep Parameter Store for values that are static by nature.

ECS/Fargate environment variables from Parameter Store go stale

Symptom

You update a Parameter Store value referenced by a running ECS task's environment variables, and the running task keeps using the old value indefinitely.

Cause

Environment variables sourced from Parameter Store are resolved once, when a task starts; a running task does not re-read the parameter.

Fix

Force a new deployment to pick up the change, or move frequently-changing values to a mechanism designed for live updates (AWS AppConfig, or an application-level refresh call).

Tricky questions

Is SecureString in Parameter Store the same protection as a Secrets Manager secret?
Most people say

Yes, both are KMS-encrypted, so they're equivalent.

Actually

Encryption at rest is similar, but Secrets Manager adds rotation, fine-grained per-secret resource policies for cross-account sharing, and native database rotation integrations that Parameter Store simply does not have. Encryption alone was never the differentiator.

If a secret costs more per month, does that mean Parameter Store is always the cheaper choice for credentials?
Most people say

Yes, so use Parameter Store for passwords too and save money.

Actually

The monthly fee buys automatic rotation and audit granularity that a leaked, never-rotated password in Parameter Store would cost far more to clean up after. The comparison that matters is the cost of a credential incident, not the monthly line item.

Scenario question

You're handed a legacy script that reads a third-party vendor's API key from a .env file checked into a private repo, and your task is to "fix the secret before the audit." The key is static — the vendor doesn't support rotation — but it is sensitive. What do you move it to, and why?

Model answer

I would move it to Parameter Store as a SecureString, not Secrets Manager, because the deciding factor here is rotation capability, and this vendor doesn't support any — paying Secrets Manager's per-secret fee buys nothing extra for a key that never changes. I would scope an IAM policy so only the specific role that needs this key can call ssm:GetParameter with decryption on that one parameter path, remove it from the repo and git history, and add a comment in the code pointing to where it now lives. If the vendor ever adds key rotation support, I would reconsider Secrets Manager at that point, since the trade-off changes the moment rotation becomes possible.

Interview corner

CommonWhen would you choose Secrets Manager over Parameter Store?
“I'd choose Secrets Manager the moment the value is a credential that should rotate, or needs cross-account sharing with its own resource policy. For static configuration that doesn't need rotation, Parameter Store is simpler and free at the standard tier.”
TrickDoes putting a password in Parameter Store as a SecureString make it as safe as Secrets Manager?
“Encryption at rest, yes — both use KMS. But Parameter Store has no rotation mechanism, so a credential there only gets safer the day someone remembers to rotate it manually. Secrets Manager automates that, which is the actual security property I care about for credentials.”
Experience checkHave you seen a cost or reliability issue from secret retrieval patterns?
“I've seen a Lambda call GetSecretValue on every single invocation instead of caching it for the life of the execution environment, which showed up both as an unexpectedly large Secrets Manager bill and as throttling under a traffic spike. Caching with a TTL fixed both.”
DesignHow would you manage rotation for thousands of per-customer database credentials?
“I'd template the rotation Lambda so it's parameterized by secret metadata instead of writing one Lambda per secret, and I'd track 'last successfully rotated' per secret on a dashboard, so a stalled rotation surfaces before an audit does, not during one.”

Memory hook

Parameter Store is a filing cabinet; Secrets Manager is a safe with clockwork that changes the combination for you.

If you remember only one thing:

If it's a credential or it must rotate, it belongs in Secrets Manager — everything else is config.

Self-check

  1. What does Parameter Store lack that makes it unsuitable for most database passwords?

    Built-in rotation. A SecureString stays encrypted but never changes itself; Secrets Manager automates rotation on a schedule.

  2. Why can caching a secret with a short TTL both save money and prevent throttling?

    Both are billed and rate-limited per API call; serving the same value from memory for the TTL window turns many calls per invocation into one call per TTL window.

  3. What happens to a running ECS task's environment variable sourced from Parameter Store after you update the parameter?

    Nothing, until the task is redeployed — the value is resolved once at task start, not re-read while running.

Prove it

Write a function monthly_cost(num_secrets, calls_per_day, secret_monthly_fee, call_fee) that estimates a Secrets Manager bill, and a second function for an equivalent Parameter Store SecureString setup (no per-secret fee, only call fee, standard tier). Run both for 50 secrets at 10,000 calls/day with made-up but consistent fee constants, and print which is cheaper and by how much, labelling the fees as illustrative placeholders, not real AWS pricing.

aws-10-03Card 3 of 6

WAF, Shield and application boundaries

MediumP4 · DifferentiatorKnow it exists11 min
wafshieldddosapplication-securityrate-limiting

Why you care

A product company interviewer will not expect you to configure WAF rules from memory, but they will expect you to know instantly which layer a given attack belongs to, because picking the wrong tool wastes an incident's first ten minutes. A SYN flood pointed at WAF does nothing; a credential-stuffing bot pointed only at Shield does nothing. This card exists at awareness depth deliberately — Authentication and authorization: sessions and JWT owns authentication and Application threat boundaries: SQLi, SSRF and IDOR owns OWASP-class application threats — but you still need to be able to say, in one sentence, which AWS service sits where in the stack, because "security in depth" questions almost always start with "where would you put a WAF versus Shield."

Explain like I'm 5

Think of your application as a shop. Shield is the structural engineering of the building: it's rated to survive a mob trying to physically crush the doors (a volumetric flood) — this protection exists the moment the building is built, no extra request needed, and you can pay for a stronger-rated version. WAF is the bouncer at the door reading a list of rules: no fake ID, no more than three entries a minute from the same face, no one wearing the pattern on last week's known-shoplifter poster. The bouncer only ever deals with people who walk up looking like legitimate customers; the bouncer cannot stop a wall being driven into the building.

Core concept

  • AWS WAF is a layer 7 (application) web application firewall. It inspects HTTP/HTTPS requests to resources it protects — CloudFront, ALB, API Gateway, AppSync, Cognito, App Runner and a few others — and can allow, block, count, or challenge (CAPTCHA) requests based on rules you define: IP lists, country, header/body patterns including SQL injection and XSS signatures, request size, and rate-based rules (block an IP that exceeds N requests in a time window). AWS and marketplace vendors publish managed rule groups so you don't write every signature yourself.
  • AWS Shield Standard is automatic and free the moment you use AWS: it protects against common network and transport layer (3/4) volumetric and protocol attacks, like SYN floods, with no configuration.
  • AWS Shield Advanced is a paid, opt-in tier that extends coverage to specific resource types (EC2, ELB, CloudFront, Route 53, Global Accelerator), adds application-layer (layer 7) DDoS detection and mitigation, cost protection for scaling triggered by an attack, and access to the Shield Response Team.
  • Decision rule: WAF answers "is this specific request malicious or abusive" (SQLi, bad bot, scraping, brute force); Shield answers "is my infrastructure being flooded to the point of unavailability." Use WAF for anything you can describe as a rule about a request's content, headers, origin, or rate. Reach for Shield Advanced only when the business cost of a sustained volumetric attack justifies the subscription — most ETL-adjacent backend teams will never configure Shield directly; it's largely there whether you look at it or not.
  • Neither service replaces authentication, authorization, or input validation inside your application — a WAF rule can slow a brute-force attempt, but it does not replace rate-limiting your own login endpoint or hashing passwords correctly.

Minimal example A rate-based rule only blocks after the threshold

PythonThe 4th request from the same source trips the rule, not the 1struns
def waf_rate_rule(ip, counts, limit=3):
    counts[ip] = counts.get(ip, 0) + 1
    return "BLOCK" if counts[ip] > limit else "ALLOW"


counts = {}
for ip in ["198.51.100.1"] * 4:
    print(waf_rate_rule(ip, counts))
Output
ALLOW
ALLOW
ALLOW
BLOCK

In production Routing a traffic signature to the right mitigation layer

PythonWAF handles well-formed requests; Shield handles the flood itselfruns
from dataclasses import dataclass


@dataclass
class TrafficSignature:
    protocol_valid_http: bool
    packets_per_second: int
    distinct_source_ips: int


def classify_mitigation(sig: TrafficSignature) -> str:
    if not sig.protocol_valid_http and sig.packets_per_second > 100_000:
        return "Shield: network/transport-layer volumetric or protocol attack"
    if sig.protocol_valid_http and sig.distinct_source_ips < 50:
        return "WAF: rate-based rule, few sources sending well-formed requests"
    if sig.protocol_valid_http:
        return "WAF managed rule group: well-formed but malicious-pattern requests"
    return "Shield: unclassified non-HTTP flood, escalate to Shield Advanced + SRT"


cases = [
    TrafficSignature(protocol_valid_http=False, packets_per_second=500_000, distinct_source_ips=20_000),
    TrafficSignature(protocol_valid_http=True, packets_per_second=4_000, distinct_source_ips=12),
    TrafficSignature(protocol_valid_http=True, packets_per_second=200, distinct_source_ips=3_000),
]
for c in cases:
    print(classify_mitigation(c))
Output
Shield: network/transport-layer volumetric or protocol attack
WAF: rate-based rule, few sources sending well-formed requests
WAF managed rule group: well-formed but malicious-pattern requests
Scenario A

In your world

The situation

Your team's internal admin dashboard sits behind an ALB with no WAF attached, because "it's internal, only our office IP range hits it." A contractor's laptop gets compromised and starts sending login attempts at the dashboard from a rotating pool of residential proxy IPs, each one well within normal single-IP rate limits.

What actually breaks

Because every individual IP looks like a normal, low-rate user, nothing in your ALB access logs stands out as a single spike — it's hundreds of distinct low-and-slow sources. The attempt eventually succeeds against a reused password, and the first thing the incident review asks is "why wasn't this rate-limited at all."

What you'd do on Monday

Attach a WAF web ACL to the ALB with a rate-based rule keyed on a broader signal than source IP alone (header combinations, or AWS's bot-control managed rule group), and add a login-attempt rule independent of your application's own throttling. A WAF rule is not a replacement for strong authentication, but it is a layer that doesn't require redeploying application code to tighten.

Scenario B

Imagine you work at BrowserStack

50M requests/day

The numbers

Illustrative: a browser-testing SaaS platform fronting customer test sessions might see on the order of 50M HTTP requests a day across its public endpoints — about 580 requests/s average, with multiples of that during business-hour peaks across time zones, mixed with scraping and credential-stuffing attempts against customer-facing login pages.

What the Scenario-A answer does here

The Scenario A fix (a rate-based WAF rule) still works for one endpoint, but hand-writing and maintaining custom signatures for every known bot pattern across dozens of public endpoints does not scale with a small security team, and a single over-broad rate rule risks blocking legitimate CI traffic that behaves in bursts by nature.

What flips at this scale

At this scale you lean on managed rule groups (AWS's own, plus a bot-control group) instead of hand-written signatures, and you run new rules in Count mode against real traffic before switching them to Block, so you can measure false-positive impact on legitimate automated clients — which, for a browser-testing platform, look unusually similar to the bots you're trying to stop.

The trade-off you'd defend

I would defend managed rule groups plus a count-then-block rollout over hand-rolled rules deployed straight to blocking, trading slower rule rollout for a measured false-positive rate before anything customer-facing gets blocked.

Illustrative traffic figures for the arithmetic; not BrowserStack-published.

WAF vs Shield Standard vs Shield Advanced

AxisWAFShield StandardShield Advanced
Layer7 (application)3/4 (network/transport)3/4 and 7
CostPay per web ACL/rule/requestFree, automaticPaid subscription
What it needs from youRules, managed rule groupsNothingResource opt-in, response team access
Decides onRequest content, headers, rate, originTraffic volume/protocol anomaliesBoth, plus cost protection during an attack
Typical ownerApp/platform teamNobody — it's always onSecurity/platform team at larger scale

Gotchas

Expecting WAF to stop a volumetric flood

Symptom

An ALB becomes unreachable during a flood, and the WAF web ACL's metrics show almost nothing — few blocked requests, few counted requests.

Cause

A true layer 3/4 volumetric attack can exhaust network capacity before requests ever reach the layer where WAF inspects them; WAF only sees requests that get far enough to be parsed as HTTP.

Fix

This is Shield's job, not WAF's. Shield Standard is already active; if this becomes recurring, that's the trigger to evaluate Shield Advanced, not to write more WAF rules.

A rate-based rule blocking by single IP misses distributed low-and-slow attacks

Symptom

Credential stuffing or scraping succeeds despite a WAF rate rule, because logs show no single IP ever crossing the threshold.

Cause

The attack is distributed across many IPs, each individually under the per-IP limit.

Fix

Use a managed bot-control rule group, or key rate limits on a broader signal (a cookie, a header fingerprint) rather than IP alone.

Treating "internal" as a reason to skip WAF entirely

Symptom

An internal admin tool with no WAF is the entry point for a credential-stuffing incident.

Cause

"Internal" described network reachability, not request legitimacy; a compromised device or leaked URL erases that assumption instantly.

Fix

Apply at least a basic rate-based rule and a managed rule group to any ALB serving authentication endpoints, internal or not.

Tricky questions

Does enabling Shield Standard require any configuration?
Most people say

Yes, you need to attach it to specific resources like you do with WAF.

Actually

No. Shield Standard applies automatically to AWS resources at no extra cost the moment you use AWS; there's nothing to attach or configure. Shield Advanced is the opt-in, paid tier that requires naming specific resources.

Can WAF alone stop a brute-force login attack?
Most people say

Yes, a rate-based rule blocking frequent requesters handles it.

Actually

Only the straightforward case. A distributed, low-and-slow brute force defeats a naive per-IP rate rule; WAF reduces the easy version of the problem, but your application's own account lockout, MFA, and anomaly detection still matter.

Scenario question

You're handed metrics showing your public API's error rate spiking every day at the same hour, correlated with a surge in requests from a huge, constantly-changing set of IPs, each sending only a handful of requests. The on-call engineer wants to "just add Shield." What do you actually recommend, and why?

Model answer

I would push back on Shield first, because the signature described — many distinct low-volume sources, well-formed enough to be hitting a specific API path, at a predictable daily hour — looks like layer 7 scraping or a scheduled bot job, not a layer 3/4 volumetric flood. Shield Standard is already protecting the infrastructure layer automatically regardless; adding Shield Advanced would add cost without addressing a request-content problem. I'd start with a WAF web ACL using a managed bot-control rule group and a rate-based rule keyed on something broader than per-IP count, and I'd run it in Count mode first against real traffic for a day to measure what it would have blocked before switching to Block, so I don't take down a legitimate daily batch client by mistake.

Interview corner

CommonWhat's the difference between WAF and Shield?
“I'd say WAF inspects the content of individual HTTP requests and decides allow, block, count, or challenge — it's layer 7. Shield protects against volumetric and protocol floods at layers 3 and 4, and it's on by default at no cost in its Standard tier.”
TrickIf I'm getting flooded and my WAF metrics show almost nothing, what does that tell you?
“I'd read that as a sign the attack isn't reaching the application layer at all — it's probably a network or transport-layer flood that WAF never gets to inspect. That points me at Shield, not at writing more WAF rules.”
Experience checkHave you seen a WAF rollout cause a false-positive problem?
“The pattern I've seen described is a rate-based rule going straight to Block without first running in Count mode, and it ends up blocking a legitimate automated client whose traffic pattern happens to look like abuse. Counting first before blocking is the fix I'd insist on.”
DesignWhen would you actually recommend Shield Advanced to a team?
“I'd only recommend it once Shield Standard's automatic protection has proven insufficient — recurring, business-impacting volumetric events against named resources — because Advanced is a paid subscription with cost-protection and response-team benefits that most teams below a certain traffic and risk level don't need.”

Memory hook

Shield is the building's structural rating; WAF is the bouncer reading a list at the door.

If you remember only one thing:

If the attack is about what's inside the request, it's WAF; if it's about raw volume overwhelming the pipe, it's Shield.

Self-check

  1. What's the one-sentence test for choosing WAF vs Shield?

    If you can describe the attack as a rule about a request's content, origin, or rate, it's WAF; if it's about raw traffic volume threatening availability, it's Shield.

  2. Is Shield Standard something you have to turn on?

    No — it's automatic and free the moment you use AWS. Shield Advanced is the opt-in, paid tier.

  3. Why does a per-IP rate-based WAF rule fail against a distributed credential-stuffing attack?

    Because the attack spreads requests across many IPs, each individually staying under the per-IP threshold, so no single IP ever trips the rule.

Prove it

Extend the minimal evaluator with a third rule: a managed-rule-group stand-in that blocks any request whose simulated "body" contains the substring "' OR 1=1" (a toy SQL injection signature), checked before the IP block list. Run it against a mixed list of five requests (two clean, one blocked by IP, one blocked by the new SQLi rule, one allowed) and print each decision with the rule that fired.

aws-10-04Card 4 of 6

GuardDuty, CloudTrail and audit evidence

MediumP4 · DifferentiatorKnow it exists12 min
guarddutycloudtrailauditthreat-detectionfindings
Needs: nothing. Start here.

Why you care

Every "prove it happened" conversation after an incident, and every compliance audit before one, hinges on the same confusion: people say "check the logs" without saying which logs. CloudTrail is a record of API calls against your AWS account; GuardDuty is a detector that watches that same record (plus a few others) and flags activity it judges suspicious. Neither one is your application's request log. Mixing these up in an interview — claiming CloudTrail would show you "who looked at this customer's order" when the order was read through your own API, not an AWS API — is a fast way to signal you've never actually had to produce audit evidence under pressure.

Explain like I'm 5

Imagine a building with two systems. CloudTrail is the building's keycard log: every door that was badged, by whom, at what time — a complete, boring record of entries, kept automatically whether or not anything bad happened. GuardDuty is a security guard watching that same keycard log (plus the cameras and the Wi-Fi logs) for patterns that look wrong — someone badging into the server room at 3 a.m. who's never done that before — and radioing an alert when they see one. Neither of them knows what happens inside your own office once you're badged in; that's your own notebook, not the building's.

Core concept

  • CloudTrail records events: management events (control-plane operations like AttachRolePolicy, CreateSubnet — logged by default) and data events (high-volume resource-level operations like S3 GetObject/PutObject, Lambda Invoke — opt-in because of volume and cost). Event History in the console/API gives you the last 90 days of management events with no setup. Beyond that window, or for data events, or for durable retention, you need a trail delivering events to S3 (and optionally CloudWatch Logs/EventBridge), which you must configure before you need it, not after.
  • GuardDuty is a continuous threat-detection service, not a request log. It ingests CloudTrail management events, VPC flow logs and DNS logs by default (and more via opt-in protection plans for S3, EKS, RDS, Lambda, EC2/ECS runtime), and uses threat intelligence feeds and machine learning to produce findings — things like compromised credentials, cryptomining, or malware presence — rather than a raw searchable record of everything that happened.
  • The relationship: CloudTrail is the raw material; GuardDuty is one consumer of a subset of it, producing judgments. You use CloudTrail (via Event History or a trail) to answer "exactly what happened and when, show me the record." You use GuardDuty to answer "is anything happening right now that looks like an attack."
  • Neither is your application log. A request to your own API, your own database query, your own business logic — none of that appears in CloudTrail unless the action itself is an AWS API call. If an auditor asks "who read this customer's record through our API," that answer lives in your own application or access logs, not in AWS's audit trail.
  • Decision rule: configure a multi-Region trail to S3 before you need historical evidence — Event History's 90-day window is not retroactive once it expires. Enable GuardDuty as a baseline detector everywhere; route its findings into your alerting (Alarms, SLOs and Well-Architected reviews), because a finding nobody reads is not detection.

Minimal example Your own API request never touches CloudTrail at all

PythonOnly AWS API calls generate a CloudTrail eventruns
def shows_up_in_cloudtrail(is_aws_api_call):
    return "CloudTrail event" if is_aws_api_call else "NOT in CloudTrail: not an AWS API call"


print(shows_up_in_cloudtrail(True))   # e.g. s3:GetObject
print(shows_up_in_cloudtrail(False))  # e.g. your own /orders endpoint
Output
CloudTrail event
NOT in CloudTrail: not an AWS API call

In production Event History's 90-day window versus a durable trail

PythonWhat an audit request can actually be answered fromruns
from datetime import date

EVENT_HISTORY_WINDOW_DAYS = 90


def can_answer_from_event_history(event_date: date, today: date) -> bool:
    return (today - event_date).days <= EVENT_HISTORY_WINDOW_DAYS


def audit_query(event_date: date, today: date, has_trail_to_s3: bool) -> str:
    if can_answer_from_event_history(event_date, today):
        return "Answerable from CloudTrail Event History (console/API), no trail required"
    if has_trail_to_s3:
        return "Event History window has passed; answerable only from the trail's S3 log files"
    return "UNANSWERABLE: no trail was configured, and Event History only covers the last 90 days"


today = date(2026, 10, 1)
print(audit_query(date(2026, 9, 20), today, has_trail_to_s3=False))
print(audit_query(date(2026, 3, 1), today, has_trail_to_s3=True))
print(audit_query(date(2026, 3, 1), today, has_trail_to_s3=False))
Output
Answerable from CloudTrail Event History (console/API), no trail required
Event History window has passed; answerable only from the trail's S3 log files
UNANSWERABLE: no trail was configured, and Event History only covers the last 90 days
Scenario A

In your world

The situation

Your company has never set up a CloudTrail trail — "Event History is on by default, isn't that enough?" — and GuardDuty has never been enabled, because nobody owns "security tooling" explicitly.

What actually breaks

A customer complaint six months old escalates into a formal audit request: "show us every API call that touched this S3 prefix in the last calendar year." Event History only covers 90 days, data events were never logged anyway because no trail with data-event selectors ever existed, and the honest answer to the auditor is that the evidence doesn't exist and cannot be reconstructed.

What you'd do on Monday

This is purely preventive, not retroactive — a trail configured today protects only future evidence. Enable a multi-Region trail delivering to a versioned, access-logged S3 bucket, with data-event logging turned on for the specific S3 prefixes or Lambda functions that matter, and enable GuardDuty's S3 Protection plan so suspicious data access generates a finding, not just a line in a log nobody is reading:

PythonCreate a trail and enable GuardDuty (needs boto3 + AWS credentials)
import boto3

cloudtrail = boto3.client("cloudtrail")
cloudtrail.create_trail(
    Name="org-audit-trail",
    S3BucketName="org-cloudtrail-logs",
    IsMultiRegionTrail=True,
    IsOrganizationTrail=False,
)
cloudtrail.start_logging(Name="org-audit-trail")

guardduty = boto3.client("guardduty")
detector_id = guardduty.create_detector(Enable=True)["DetectorId"]
print("trail started; GuardDuty detector:", detector_id)
Scenario B

Imagine you work at Policybazaar

12-month audit lookback requests

The numbers

Illustrative: 40 insurer integrations, each triggering roughly 5 compliance or regulatory data requests a year — 40 × 5 = 200 audit requests/year, better than one every two business days. Each request asks for a 6-12 month lookback window; at roughly 2M CloudTrail management events logged per month account-wide, a 12-month lookback is 2M × 12 = 24M events a single request may need to search through.

What the Scenario-A answer does here

The Scenario A fix (a trail plus GuardDuty) answers "did this happen," but a single trail dumping every Region's events into one S3 prefix with no lifecycle policy becomes slow and expensive to query at a year's worth of volume, and sifting it manually per audit request does not scale with request frequency.

What flips at this scale

At this scale you route trail events into a queryable store — CloudTrail Lake, or Athena over the S3-delivered logs with sensible partitioning by date and account — so a 12-month audit query is a bounded SQL query with a cost estimate, not a manual S3 object scan. You also pre-aggregate GuardDuty findings into a ticketing workflow so each finding has an assigned owner and a closure record, which itself becomes audit evidence of a working process.

The trade-off you'd defend

I would defend queryable, partitioned audit storage with a defined retention policy over "logs exist somewhere in S3," trading some upfront setup cost for audit requests that take hours instead of becoming their own multi-week project.

Illustrative audit frequency and lookback windows; not Policybazaar-published figures.

GuardDuty vs CloudTrail

AxisCloudTrailGuardDuty
What it isA record of API activityA threat detector over that activity
OutputEvents (management/data)Findings (judgments)
Default retention90 days (Event History), no setupFindings retained per account settings
Needs configuring for durabilityYes — a trail to S3No extra config for foundational detection
Answers"What happened, exactly, and when""Does anything look like an attack"
Covers your app's own request logsNo — AWS API calls onlyNo — same underlying data sources

Gotchas

Assuming Event History is a permanent record

Symptom

An audit request for anything older than roughly three months returns nothing, even though "CloudTrail is on."

Cause

Event History is a 90-day rolling window with no setup; it is not a trail, and it is not retroactively extendable.

Fix

Configure a multi-Region trail to S3 before you need long-term evidence — there is no way to recover history that predates the trail's creation.

Confusing CloudTrail with your application's own access logs

Symptom

An auditor asks "who viewed this customer's record" and the answer is sought in CloudTrail, which comes back empty.

Cause

The record was read through your own API, not an AWS API call; CloudTrail has no visibility into your application's internal request handling.

Fix

Maintain your own application-level audit log for business-data access; treat CloudTrail as AWS control/data-plane evidence only, never a substitute for it.

Enabling GuardDuty and never routing its findings anywhere

Symptom

A post-incident review discovers GuardDuty had flagged the compromised credential days earlier, but nobody saw the finding.

Cause

GuardDuty findings sit in the console/API until something reads them; enabling the detector is not the same as having a response process.

Fix

Route findings to EventBridge and into your paging/ticketing system (Alarms, SLOs and Well-Architected reviews), with a defined severity-to-response mapping, before you need it.

Tricky questions

Does GuardDuty give you a complete audit trail of every action in your account?
Most people say

Yes, it's AWS's security log, so it has everything.

Actually

GuardDuty produces findings — its own judgment calls — from a subset of your account's activity. It does not give you a complete, queryable record of every action; that's what a CloudTrail trail is for, and the two serve different questions.

If CloudTrail is "on," are S3 object-level reads automatically logged?
Most people say

Yes, CloudTrail logs all S3 activity by default.

Actually

Only management events are logged by default. S3 object-level operations are data events, which are off by default in both trails and event data stores because of their volume, and must be explicitly enabled per bucket or prefix.

Scenario question

You're handed a request from the compliance team: "Confirm no unauthorized party accessed the pii/ prefix in our S3 bucket in the last 9 months." Your account has GuardDuty enabled but no CloudTrail trail was ever created. What do you tell them, and what do you set up going forward?

Model answer

I would tell them honestly that this specific request can't be answered: GuardDuty findings are judgments about suspicious activity, not a complete access record, and without a trail, CloudTrail's Event History only covers the last 90 days and never logged S3 data events anyway since those require explicit enabling. There is no way to reconstruct 9 months of object-level access after the fact. Going forward, I would create a multi-Region trail to a dedicated, access-controlled S3 bucket with data-event logging enabled specifically for that pii/ prefix, and enable GuardDuty's S3 Protection plan so future suspicious access on that prefix generates a finding, not just a trail entry nobody is watching. I'd flag clearly to compliance that this closes the gap from today onward, not retroactively.

Interview corner

CommonWhat's the difference between CloudTrail and GuardDuty?
“I'd say CloudTrail is a record of API activity in your account — what happened, by whom, when. GuardDuty is a detector that watches a subset of that activity, plus VPC flow and DNS logs, and raises findings when something looks like an attack. One is a record; the other is a judgment on top of a record.”
TrickIf GuardDuty is enabled, do you still need a CloudTrail trail?
“I'd say yes, still. GuardDuty doesn't give you a durable, queryable, complete record — it gives you findings on suspicious patterns. If you need to answer 'exactly what happened over the last year,' that's a trail to S3, not GuardDuty's finding history.”
Experience checkHave you seen an audit request that CloudTrail couldn't answer?
“I've seen the common one be someone assuming Event History is permanent, then getting an audit request for something eight months old. It only covers 90 days by default; without a trail configured in advance, that evidence simply doesn't exist.”
DesignHow would you make GuardDuty findings actually actionable instead of sitting in a console?
“I'd route findings through EventBridge into the team's existing alerting and ticketing system, with severity mapped to response urgency, so a finding becomes a tracked, owned task the moment it fires rather than something someone has to remember to check for.”

Memory hook

CloudTrail is the keycard log; GuardDuty is the guard watching it for patterns; neither one is your own office notebook.

If you remember only one thing:

If it wasn't an AWS API call, CloudTrail never saw it — and Event History forgets after 90 days unless a trail is already running.

Self-check

  1. What's the default retention of CloudTrail Event History, and what extends it?

    90 days by default; a configured trail delivering to S3 is what gives you durable, longer retention.

  2. Why doesn't CloudTrail show you who read a customer record through your own API?

    Because CloudTrail only logs AWS API activity; your own application's request handling is not an AWS API call, so it never appears there.

  3. Are S3 data events logged by default once a trail exists?

    No — data events (like S3 object-level GetObject) must be explicitly enabled per trail; only management events are on by default.

Prove it

Write a function reconstruct_access(requests, trail_started_at, today) that takes a list of (event_date, description) tuples and classifies each as answerable from Event History, answerable only from a trail (if trail_started_at is before the event), or unanswerable. Test it with one event inside 90 days, one event before the trail existed, and one event after the trail started but outside the 90-day window. Print all three classifications.

aws-10-05Card 5 of 6

Private-subnet security patterns

HardP3 · Role-dependentTalk about it13 min
vpcprivate-subnetsecurity-groupsendpointsegress

Why you care

"Private subnet" is a label people trust more than they should. A subnet named private with a route table pointing 0.0.0.0/0 at an Internet Gateway is not private — it's public with a confusing name — and nobody notices until a security review runs describe-route-tables instead of reading the console's subnet list. Interviewers probe private-subnet design because it's where "I know IAM" candidates and "I've actually run production networks" candidates diverge: the first group describes isolation as a property of a name or a diagram; the second checks the route table, the security group, and the egress path explicitly, every time.

Explain like I'm 5

Calling a room "soundproof" doesn't make it soundproof — what matters is whether there's actually a gap under the door. A subnet is "private" only because of what its route table actually points to, the same way a room is quiet only because of what's actually sealed, not because of a sign on the door. And even a genuinely sealed room still needs a defined way for people inside to get a message out — that's the endpoint or NAT path, which has to exist and has to be reviewed, or the room is either unreachable or has an undocumented hole in the wall.

Core concept

  • A subnet is private only if its route table has no default route to an Internet Gateway. The subnet's name, its position in a diagram, or a tag means nothing to actual traffic; only the route table decides.
  • Private-with-egress uses a NAT Gateway: instances can initiate outbound connections (package installs, API calls to the internet) but nothing from the internet can initiate inbound. Private-isolated has no default route to anywhere outside the VPC at all.
  • VPC endpoints let private resources reach specific AWS services without any internet path: gateway endpoints (S3, DynamoDB) are free and attach to route tables directly; interface endpoints (most other services, via PrivateLink) are ENIs with their own security groups and a per-hour-plus-data cost. Using an S3 gateway endpoint instead of a NAT Gateway for S3 traffic removes both the egress cost and the public-internet hop entirely for that traffic.
  • Security groups are the per-instance/per-ENI firewall, stateful (return traffic is automatically allowed) and allow-only (no explicit deny statements). For admin access, the modern pattern is no inbound SSH/RDP rule at all — use AWS Systems Manager Session Manager, which needs only outbound HTTPS (or an interface endpoint) and an IAM permission, never an open inbound port or a public IP on the instance.
  • Egress deserves the same review as ingress. A private subnet with an unreviewed, wide-open outbound rule (0.0.0.0/0 on all ports) is a quiet exfiltration path if any instance in it is ever compromised; scope outbound rules to what the workload actually needs to reach.

Decision rule: default every workload to a private-isolated or private-with-NAT subnet; promote to public only for the specific tier that must terminate internet-originated connections (typically just the load balancer). Replace every inbound admin rule with Session Manager before you replace it with anything else.

Under the hood

A security group evaluates every rule and takes the union of all matching allows — there is no deny statement and no rule ordering, unlike an IAM policy or a NACL. This is why "tightening" a security group by adding a narrower rule does nothing if a broader rule still matches; you must remove the broad rule, not merely add a narrow one alongside it. Network ACLs, by contrast, are stateless and ordered: a NACL that allows inbound 443 but has no corresponding outbound ephemeral-port rule will silently break connections, because stateless rules require you to explicitly allow the return traffic yourself — a classic cause of "security group looks fine, but it still doesn't connect" tickets that turn out to be the subnet's NACL instead.

Minimal example A route table decides privacy, never the subnet's name

PythonA subnet named "private" with an IGW route is publicruns
def is_private(default_route_target):
    if default_route_target is None:
        return True
    return not default_route_target.startswith("igw-")


print(is_private("igw-0abc"))  # the subnet is named "private-app" in every diagram
print(is_private("nat-0def"))
Output
False
True

In production Audit security groups for public admin access

PythonFlag open admin ports, recommend Session Manager insteadruns
from dataclasses import dataclass

ADMIN_PORTS = {22, 3389}


@dataclass
class SGRule:
    port: int
    cidr: str


@dataclass
class SecurityGroup:
    name: str
    inbound: list


def audit(sg: SecurityGroup) -> list[str]:
    findings = []
    for rule in sg.inbound:
        if rule.port in ADMIN_PORTS and rule.cidr == "0.0.0.0/0":
            findings.append(
                f"{sg.name}: port {rule.port} open to the internet. "
                "Use SSM Session Manager instead of inbound SSH/RDP; "
                "it needs no open port and no public IP."
            )
    return findings


groups = [
    SecurityGroup("sg-jenkins-legacy", [SGRule(22, "0.0.0.0/0"), SGRule(443, "10.0.0.0/8")]),
    SecurityGroup("sg-app-private", [SGRule(443, "10.0.1.0/24")]),
]
for sg in groups:
    issues = audit(sg)
    print(sg.name, "OK" if not issues else issues)
Output
sg-jenkins-legacy ['sg-jenkins-legacy: port 22 open to the internet. Use SSM Session Manager instead of inbound SSH/RDP; it needs no open port and no public IP.']
sg-app-private OK

See it happen A request's path in, and the app's egress path out

HTTPSSG allows ALB onlypip install (needs internet)GetObject (no internet hop)internetInternet GatewayALB (public subnet)app (private subnet)NAT GatewayS3 gateway endpointS3 bucket
  1. A request from the internet can only reach the public subnet's ALB through the IGW.
  2. The ALB forwards to the app in the private subnet; the app's security group allows only the ALB, nothing else inbound.
  3. When the app needs genuine internet egress (a package registry), it goes out through the NAT Gateway, never a direct route.
  4. When the app only needs S3, it uses the gateway endpoint instead: no NAT, no public IP, no internet hop at all for that traffic.
Notice: two outbound paths exist on purpose, and only one of them ever touches anything outside AWS's own network — picking the wrong one for a given call is both a cost and a review-surface mistake.
Scenario A

In your world

The situation

Your Jenkins box and the ETL application servers sit in a subnet labelled private-app in every diagram the team has ever drawn, and the SSH port has been open to 0.0.0.0/0 "temporarily" for two years, because that's how the original setup script configured it before anyone owned the network.

What actually breaks

A routine cost review notices unexpected data transfer charges and traces them to the private-app subnet's route table — someone added a default route to an Internet Gateway eight months ago to unblock a one-off package download, and never removed it. Combined with the open SSH rule, the subnet has been functionally public the entire time, discoverable by anyone scanning for it.

What you'd do on Monday

Fix both independently, because they're two separate failures stacked together. Remove the IGW default route and replace it with a NAT Gateway route scoped to what the app actually needs outbound; remove the open SSH rule entirely and move admin access to Session Manager, which needs only an IAM permission and outbound HTTPS (ideally through an interface endpoint, not even NAT):

PythonReplace an SSH session with SSM Session Manager (needs boto3 + SSM agent on the instance)
import boto3

ssm = boto3.client("ssm")
session = ssm.start_session(Target="i-0123456789abcdef0")
print("session started:", session["SessionId"])  # no inbound port, no public IP needed
Scenario B

Imagine you work at Cars24

150 backend services, 1,500+ SG rules

The numbers

Illustrative: a marketplace platform running on the order of 150 backend services across inventory, pricing, and logistics, each with its own security group, means roughly 150+ sets of inbound/outbound rules to keep reviewed — at even a modest 10 rules per group, that's 1,500+ individual rules to audit for drift.

What the Scenario-A answer does here

The Scenario A fix (remove the bad route, move to Session Manager) works per-instance, but doing this review manually, service by service, across 150 security groups does not scale, and nothing stops a future script from reintroducing an open admin rule the same way the original one appeared.

What flips at this scale

At this scale you flip from manual review to continuous, automated detection: AWS Config rules (or an SCP-adjacent control) that flag or auto-remediate any security group with an admin port open to 0.0.0.0/0, and a route-table check that alerts the moment any subnet tagged private gains a default route to an IGW. The review becomes "what did the automated control catch this week," not "did anyone remember to look."

The trade-off you'd defend

I would defend automated, continuous drift detection over periodic manual audits at this service count, trading some initial tooling investment for catching a misconfiguration within minutes instead of during the next scheduled review, however many months away that is.

Illustrative service and rule counts; not Cars24-published figures.

Gotchas

A route-table edit, not a subnet rename, decides privacy

Symptom

A subnet named and tagged private turns out to be reachable directly from the internet.

Cause

Someone added a default route to an Internet Gateway for a one-off need and never removed it; the name and tag never changed to reflect that.

Fix

Audit by route table content (describe-route-tables), never by subnet name or tag, and alert automatically on any private-tagged subnet acquiring an IGW default route.

Adding a narrower security group rule without removing the broad one

Symptom

A security group "tightened" by adding a specific-CIDR rule still allows the whole internet in.

Cause

Security groups are allow-only unions with no deny and no precedence; a pre-existing 0.0.0.0/0 rule keeps matching regardless of any narrower rule added alongside it.

Fix

Remove the broad rule explicitly — adding a narrower one next to it changes nothing.

A stateless NACL missing the return-traffic rule

Symptom

A security group looks correct, but connections still fail intermittently or entirely for a specific subnet.

Cause

NACLs are stateless; an inbound allow on port 443 needs a matching outbound allow for the ephemeral port range, or return traffic is dropped at the NACL before it ever reaches the security group.

Fix

Check the subnet's NACL for both directions whenever a security-group-level fix doesn't resolve a connectivity issue.

Tricky questions

Does placing a database in a private subnet make it secure?
Most people say

Yes, if it's not reachable from the internet, it's protected.

Actually

Private placement controls reachability, not authorization. A compromised instance inside the same VPC, or an overly broad security group rule allowing any instance in the VPC to connect, can still reach it. Private placement is one layer, not the whole answer — IAM policy evaluation's IAM boundary and the database's own access controls still matter.

Is a gateway endpoint for S3 the same thing as an interface endpoint?
Most people say

Yes, both just give private access to an AWS service.

Actually

A gateway endpoint (S3, DynamoDB only) attaches directly to route tables and is free; an interface endpoint is an ENI with its own security group, built on PrivateLink, and incurs an hourly-plus-data charge. Picking the wrong one for S3 means paying for an interface endpoint you didn't need.

Scenario question

You're handed a VPC diagram showing a private application subnet with no internet gateway route — looks correctly isolated. A teammate insists it still needs a security review before go-live. What else do you check, and why might the diagram still be misleading?

Model answer

I would check three things the diagram doesn't show. First, the actual route table content via the API, not the diagram, because diagrams drift from reality the moment someone makes a console change without updating it. Second, the security group's inbound rules for anything resembling an admin port open broadly, and the egress rules for an unreviewed wide-open outbound allow, since the diagram shows subnet placement, not rule content. Third, whether any VPC endpoints are actually in place for the AWS services this tier needs, because if they're missing, someone may "temporarily" add a NAT or IGW route later to unblock a dependency, exactly how these things usually drift from isolated into reachable. The diagram proves intent; only the live configuration proves the current state, and those two things diverge quietly over time.

Interview corner

CommonWhat actually makes a subnet private?
“I'd say it's the route table having no default route to an Internet Gateway — nothing about its name, its tag, or its position in a diagram. I always check the live route table, not the documentation, before trusting that label.”
TrickIf a security group has both a broad allow and a narrow allow for the same port, which one wins?
“I'd say neither one 'wins' — security groups are a union of every matching allow rule, with no deny and no precedence. The broad rule still applies in full; adding a narrower one changes nothing unless you remove the broad one.”
Experience checkHow do you give engineers SSH-equivalent access without opening port 22?
“I use SSM Session Manager — it needs an IAM permission and the agent running on the instance, with outbound HTTPS, ideally through an interface endpoint rather than NAT. No inbound port, no public IP, and every session is logged to CloudTrail as an API call, which a raw SSH session never was.”
DesignHow would you keep 100+ security groups from drifting back toward overly broad rules over time?
“I'd automate detection with AWS Config rules that flag or auto-remediate admin ports open to the whole internet, and alert on any route-table change that adds an IGW default route to a subnet tagged private, rather than relying on a periodic manual review that only catches drift whenever someone gets around to it.”

Memory hook

A subnet is private because of what its route table points to, never because of what its name claims.

If you remember only one thing:

Check the route table and the security group directly — never trust the diagram or the tag.

Self-check

  1. What single piece of configuration determines whether a subnet is private?

    Whether its route table has a default route (0.0.0.0/0) to an Internet Gateway. No IGW default route means no direct internet path in either direction.

  2. Why doesn't adding a narrow security group rule fix an overly broad one?

    Security groups are a union of all matching allow rules with no deny statements and no precedence; the broad rule keeps matching until it's removed, regardless of what else is added.

  3. What do you need to let an EC2 instance run pip install from a fully private subnet?

    A route to a NAT Gateway (or, for specific AWS services, a VPC endpoint instead) — a private-isolated subnet with no default route anywhere has no path for that outbound call at all.

Prove it

Extend the minimal block's is_private function so it also returns the actual target it classified (an IGW id, a NAT id, or None). Then write a small "drift detector" that takes a list of (subnet_name, is_tagged_private, default_route_target) tuples and prints an alert line for any subnet tagged private whose target is actually an IGW — the exact mismatch from the Scenario A story. Test it with one correctly isolated private subnet, one drifted "private" subnet with an IGW route, and one correctly public subnet (not tagged private, has an IGW route, so no alert).

aws-10-06Card 6 of 6

Least-privilege policy recipes

HardP3 · Role-dependentKnow cold12 min
least-privilegeiam-policyscoped-accesscross-accountpolicy-testing

Why you care

"Least privilege" is a phrase every candidate says and almost nobody can write under pressure. The gap between the principle and the policy is where most IAM incidents live: a role that's "basically fine" for a year, until an audit shows it could have done ten times more damage than it ever needed to. This card is marked cold on purpose — a whiteboard with three sentences of requirements expects a scoped policy and a plan to test it, not theory. IAM policy evaluation gave you the evaluator; this card is the muscle memory for using it to write policies narrow enough that getting them wrong is loud, not silent.

Explain like I'm 5

A least-privilege policy is a key cut for exactly one lock, not a master key used carefully. You describe a key by what it must open and nothing else — not "this probably opens the mail slot, let's see what else it opens along the way." The discipline isn't trusting yourself with a master key; it's never cutting one.

Core concept

Three recipes come up constantly enough to know cold:

  • Scoped S3 prefix access. Grant s3:GetObject/s3:PutObject on arn:...:bucket/prefix/*, never the bucket ARN alone and never Resource: "*". s3:ListBucket targets the bucket ARN, not the prefix, with an s3:prefix condition — a different resource shape, and a common missing statement.
  • Lambda execution role. Start from nothing: logs:CreateLogGroup/CreateLogStream/PutLogEvents scoped to this function's log group, plus exactly the data-plane actions the body calls. No iam:PassRole unless it genuinely launches other roles.
  • Cross-account roles. The trust policy names the specific account, never "*", and any third-party-assumed role requires an ExternalId condition — skipping it is the textbook confused-deputy setup.
  • Prove it, don't assert it. IAM policy evaluation's evaluator becomes a test harness: a positive and a negative test per statement, before deploying. IAM Access Analyzer generates a fine-grained policy from a role's actual CloudTrail activity, which you test and tighten rather than hand-write.
  • ABAC and the KMS boundary apply here too: a scoped S3 policy that forgets Key management's key-policy requirement is incomplete, not least-privilege — it fails loudly on first use rather than quietly over-granting.

Under the hood

Access Analyzer's policy generation reads CloudTrail activity for a role over a lookback window and emits a policy covering only what it saw — a window shorter than the role's full operational cycle (a monthly job reviewed over two weeks) produces a policy that breaks the first time the unseen path runs. It proves what was used, not what's supposed to be allowed; that gap is exactly where a generated policy needs human review before it ships.

Minimal example A scoped policy is proven by its negative test, not its positive one

PythonSame action, different key — only one of these should passruns
def s3_allows(action, key, allowed_prefix):
    return action == "s3:GetObject" and key.startswith(allowed_prefix)


print(s3_allows("s3:GetObject", "raw/orders/jan.csv", "raw/orders/"))        # positive
print(s3_allows("s3:GetObject", "raw/settlements/jan.csv", "raw/orders/"))   # negative
Output
True
False

In production Three recipes, one pass/fail test harness

PythonS3 prefix, Lambda execution role, and cross-account ExternalId — tested, not assertedruns
from dataclasses import dataclass


@dataclass
class PolicyTest:
    description: str
    actual: bool
    expected: bool


def run_tests(tests: list[PolicyTest]) -> int:
    failures = 0
    for t in tests:
        status = "PASS" if t.actual == t.expected else "FAIL"
        if status == "FAIL":
            failures += 1
        print(f"{status}: {t.description}")
    return failures


# Recipe 1: S3 prefix-scoped role
def s3_allows(action, key):
    return action in {"s3:GetObject", "s3:PutObject"} and key.startswith("curated/team-data/")


# Recipe 2: Lambda execution role, minimal actions only
LAMBDA_ALLOWED = {"logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents", "s3:GetObject"}


def lambda_allows(action):
    return action in LAMBDA_ALLOWED


# Recipe 3: cross-account assume role requires a matching ExternalId
def cross_account_allows(caller_account, trusted_account, external_id, expected_external_id):
    return caller_account == trusted_account and external_id == expected_external_id


tests = [
    PolicyTest("S3: read in-scope prefix", s3_allows("s3:GetObject", "curated/team-data/2026.parquet"), True),
    PolicyTest("S3: write out-of-scope prefix denied", s3_allows("s3:PutObject", "curated/other-team/x"), False),
    PolicyTest("Lambda: write own logs", lambda_allows("logs:PutLogEvents"), True),
    PolicyTest("Lambda: cannot call ec2:RunInstances", lambda_allows("ec2:RunInstances"), False),
    PolicyTest(
        "Cross-account: correct account + ExternalId",
        cross_account_allows("111122223333", "111122223333", "etl-prod-2026", "etl-prod-2026"),
        True,
    ),
    PolicyTest(
        "Cross-account: right account, guessed ExternalId (confused deputy) denied",
        cross_account_allows("111122223333", "111122223333", "guessed-id", "etl-prod-2026"),
        False,
    ),
]

failures = run_tests(tests)
print("failures:", failures)
Output
PASS: S3: read in-scope prefix
PASS: S3: write out-of-scope prefix denied
PASS: Lambda: write own logs
PASS: Lambda: cannot call ec2:RunInstances
PASS: Cross-account: correct account + ExternalId
PASS: Cross-account: right account, guessed ExternalId (confused deputy) denied
failures: 0
Scenario A

In your world

The situation

Your ETL role was created eighteen months ago with AmazonS3FullAccess attached, "to unblock the first sprint," and nobody has revisited it since. The job only ever touches two prefixes in one bucket.

What actually breaks

A security review during a SOC 2 audit flags the role as a finding: it can read, write, and delete every object in every bucket in the account, including buckets belonging to other teams, and the audit cannot distinguish "has never done this" from "cannot do this" — the role's blast radius is the entire account's S3 estate, and the only evidence against misuse is that it hasn't happened yet.

What you'd do on Monday

Replace the managed policy with a scoped customer policy, built from what the role actually uses, not from a guess:

PythonGenerate a scoped policy from real usage, then test it (needs boto3 + Access Analyzer)
import boto3

analyzer = boto3.client("accessanalyzer")
job = analyzer.start_policy_generation(
    policyGenerationDetails={"principalArn": "arn:aws:iam::111122223333:role/etl-runner"}
)
job_id = job["jobId"]
# After the job completes: analyzer.get_generated_policy(jobId=job_id)
# Review the generated statements against a full operational cycle (including monthly jobs)
# before replacing AmazonS3FullAccess with the generated, scoped policy.
print("policy generation job started:", job_id)
Scenario B

Imagine you work at Groww

80 microservices, 350+ scoped access relationships

The numbers

Illustrative: an investment platform with on the order of 80 microservices, each with its own execution role, and 4-5 S3 prefixes or tables per role on average, is roughly 350-400 distinct scoped access relationships to define and keep correct — far too many to hand-review individually on every change.

What the Scenario-A answer does here

The Scenario A fix (replace a wildcard managed policy with a generated, tested scoped policy) works role by role, but doing that migration manually across 80 services, with no standard test harness, means every team invents its own notion of "tested enough" and some roles never get migrated off the broad policy they started with.

What flips at this scale

At this scale the recipe itself becomes a template: a standard Terraform module that takes a bucket, a list of prefixes, and an action set, and emits both the scoped policy and a matching set of positive/negative tests automatically, so "least privilege" is the default output of provisioning a new service, not a retrofit project someone has to schedule and justify later.

The trade-off you'd defend

I would defend a standard least-privilege module with built-in tests over case-by-case manual policy authoring at this service count, trading some flexibility for every new service starting scoped by construction, which is cheaper than auditing 80 existing roles for drift one at a time.

Illustrative service and prefix counts; not Groww-published figures.

Gotchas

A broad S3 action alone can still hit a KMS wall, and that's not a bug

Symptom

A newly scoped S3 policy passes every S3-only test, then fails in staging with AccessDeniedException on an SSE-KMS bucket.

Cause

Granting an S3 action alone does not establish permission to decrypt a KMS-protected object — the Key management key-policy boundary is a separate, additional gate that a least-privilege S3 policy doesn't automatically satisfy.

Fix

Add the matching kms:Decrypt/kms:GenerateDataKey statements, scoped to the specific key ARN, and verify the key policy itself names or delegates to this role — test both layers, not just the S3 layer.

Forgetting s3:ListBucket targets the bucket, not the prefix

Symptom

Object reads and writes work under the scoped prefix, but any code path calling list_objects_v2 fails even though "the S3 permissions look complete."

Cause

s3:ListBucket is a bucket-level action; granting only prefix-scoped object actions never covers it, and a s3:prefix condition is needed to scope what the listing can return.

Fix

Add a separate statement for s3:ListBucket on the bucket ARN with a StringLike condition on s3:prefix matching the allowed prefix.

A cross-account role with no ExternalId

Symptom

A security review flags a vendor-assumed role as a confused-deputy risk, even though the trust policy correctly names only the vendor's account.

Cause

Naming the account is necessary but not sufficient; without an ExternalId condition, anyone who can get the vendor's own systems to assume a role on their behalf — intentionally or by being tricked — can reach your role too.

Fix

Require a unique sts:ExternalId in the trust policy's condition block, generated per integration, and verify the vendor's AssumeRole call actually passes it.

Tricky questions

If a generated least-privilege policy passes every test you can think of today, is it done?
Most people say

Yes, if it passes the tests, ship it.

Actually

It's done for the usage pattern the test lookback window covered. A role whose operational cycle includes a monthly or quarterly job that wasn't exercised during the Access Analyzer lookback window will pass every test today and fail the first time that job runs — review the role's full cycle, not just recent activity.

Does restricting Resource to a specific prefix also restrict s3:ListBucket on that prefix automatically?
Most people say

Yes, if the object actions are scoped, listing is scoped too.

Actually

No. s3:ListBucket is evaluated against the bucket ARN as its resource, regardless of how narrow your object-level Resource entries are; it needs its own statement with a prefix condition, or it's either missing (breaking functionality) or left ungoverned (over-granting).

Scenario question

You're handed a Lambda execution role with AWSLambdaBasicExecutionRole plus a customer policy granting s3:* and dynamodb:* on Resource: "*", because "it needs to read a config bucket and one DynamoDB table, and nobody got around to narrowing it." Walk through tightening it, and how you'd prove the tightened version still works.

Model answer

I would start from the function's actual code, not the existing policy, and list exactly which calls it makes: probably s3:GetObject on one prefix for config, and dynamodb:GetItem/PutItem on one table. I'd write the scoped policy with those four statements, each resource-scoped to the specific ARN, drop s3:* and dynamodb:* entirely, and keep the basic execution role for CloudWatch Logs since that's already minimal. Before deploying, I'd write positive tests — the config read, the two DynamoDB calls — and negative tests for the obvious lookalikes: a different S3 prefix, a different table, s3:DeleteObject, dynamodb:DeleteTable. I would also run IAM Access Analyzer's policy generation against the role's CloudTrail history across at least one full deployment cycle, not just the last few days, specifically to catch any call path my manual read of the code might have missed. The trade-off is that the scoped policy will break loudly the first time an untested code path runs — which I consider the point, not the risk, because a loud break during testing is far cheaper than a silent over-grant discovered during an audit.

Interview corner

CommonWhat does a least-privilege S3 policy actually look like, concretely?
“I'd scope the resource to the exact prefix the role touches, never the bucket ARN alone, list only the specific object actions it needs, and add a separate s3:ListBucket statement on the bucket ARN with an s3:prefix condition if it needs to list anything. Then I write a negative test for a sibling prefix to confirm it's actually denied.”
TrickYour scoped S3 policy passes all its tests, but the role still gets AccessDeniedException reading an object. What's the first thing you check?
“I'd check whether the bucket uses SSE-KMS with a customer managed key. A correctly scoped S3 policy says nothing about decrypt permission — that's a separate key-policy boundary, and it's one of the most common 'the policy is right but it still fails' causes I've seen.”
Experience checkHow would you migrate a role off an overly broad managed policy without an outage?
“I'd generate a scoped policy from the role's actual CloudTrail activity over at least one full operational cycle, write positive and negative tests for every statement, and run the new policy in parallel — attached but not yet replacing the old one, or tested in a non-prod account with the same role shape — before removing the broad policy. The risk I'm managing is an unseen, infrequent code path, not the common-case traffic.”
DesignHow do you keep least-privilege from becoming a one-time exercise that drifts back to broad access?
“I'd template the scoping pattern itself — a module that takes a bucket, prefixes, and an action set and emits both the policy and its tests — so every new service starts scoped by construction. For existing roles, I'd use IAM Access Analyzer's continuous findings on unused permissions to catch drift before the next audit does.”

Memory hook

A least-privilege policy is a key cut for one lock, tested against the lock next to it, not a master key you promise to use carefully.

If you remember only one thing:

Write the negative test before you trust the positive one — a policy that only ever gets tested on what it should allow will never catch what it still allows by accident.

Self-check

  1. Why does a prefix-scoped s3:GetObject/s3:PutObject policy still need a separate statement for s3:ListBucket?

    Because s3:ListBucket targets the bucket ARN, not the object/prefix ARN, so it falls outside any object-scoped statement and needs its own resource and condition.

  2. What does an ExternalId condition in a cross-account trust policy actually prevent?

    A confused-deputy attack, where a third party you've correctly named in the trust policy is tricked into assuming your role on behalf of someone else; the ExternalId is a shared secret only your real integration knows to pass.

  3. Why can IAM Access Analyzer's generated policy still miss a permission the role genuinely needs?

    Because it's built from the role's actual CloudTrail activity over a lookback window; any operational path that didn't run during that window — a monthly job, a rare error-handling branch — never appears in the generated policy.

Prove it

Take the production block's s3_allows recipe and extend it with a fourth test case representing the Scenario A story: a request for s3:GetObject on an in-scope prefix, but on an SSE-KMS bucket where kms_decrypt_allowed is False. Write a combined s3_and_kms_allows(action, key, kms_decrypt_allowed) function that returns True only if both the S3 scope check and the KMS flag pass, add it to the test harness, and show the full six-test run, including the new case, printing failures: 0.