Shelf · AWS · Chapter 8

Chapter 8: Infrastructure as code

Clicking through the console or re-running a bash script of AWS CLI commands is not infrastructure as code; it has no desired state, no diff and no lock. This chapter gives you the vocabulary and the failure modes of CloudFormation, Terraform and CDK so you can read a plan, defend a module boundary and explain why your team's "IaC" still lets Friday-night console changes vanish on Monday.

Difficulty
Priority
aws-08-01Card 1 of 6

Infrastructure as code

HardP3 · Role-dependentTalk about it10 min
iaccloudformationterraformcdkdesired-statedrift
Needs: nothing. Start here.

Why you care

Every AWS job above entry level expects you to provision infrastructure through code, not the console. An interviewer asking "CloudFormation or Terraform?" is testing whether you understand desired state: a file says what should exist, a tool reconciles reality to match it, and the diff is reviewable before anything changes. A Jenkins job running raw aws CLI commands has no diff, no state, and no way to answer "what will this change?" Get this wrong and you'll describe IaC as "a script that creates AWS stuff" — someone who has never had a console fix silently overwritten by a redeploy.

  • Note: CDK/CloudFormation detail below is marked spec rather than cited; verify exact flags and syntax before using them against a real account.

Explain like I'm 5

A tenant hands a landlord a signed floor plan instead of describing changes verbally. The plan is your config: wall here, door there. The landlord (the IaC tool) compares plan to building and does only the work needed: adds the missing wall, leaves the correct door, removes the shed that isn't on the plan. If a past tenant knocked a hole in a wall and told nobody, the next inspection finds the mismatch — that's drift.

Core concept

  • Desired state is the config you write: a CloudFormation template, Terraform HCL, or a CDK app that compiles to a template. All describe an end state, never steps.
  • State ownership is who remembers what exists. CloudFormation owns it per stack, inside the service; Terraform owns it in a state file you store yourself (next card); CDK has none of its own — it synthesizes to CloudFormation and lets CloudFormation own the state.
  • The reviewable diff is a CloudFormation change set or a Terraform plan: create/update/delete, shown before anything changes. cdk diff compares the same way.
  • Drift is any gap between state and reality — a console edit, a manual call, another tool. Undetected drift, not drift itself, is what causes outages.

Decision rule: CloudFormation when AWS should own state and YAML/JSON suits you; Terraform when one state model must span non-AWS resources too; CDK when your team already codes in Python/TypeScript/Java and wants real functions and tests around infrastructure, with CloudFormation still provisioning. All three fail the same way without a reviewed diff.

Under the hood

CloudFormation computes a change set against the stack's last recorded state, and you must explicitly execute it — nothing auto-applies. Terraform's plan refreshes first: it reads every resource's real value from the AWS API before diffing against your .tf config, which is why a plan can show a change you never made. CDK's synth runs entirely offline, building a construct tree from your Python code and serializing it to a CloudFormation template; AWS calls happen only once CloudFormation later diffs and applies that template.

Minimal example Desired state vs actual state, computed as a diff

PythonA resource diff: create, update, or unmanagedruns
desired = {"web_sg": 443, "web_server": "t3.micro"}
actual = {"web_sg": 22, "orphan_eip": "vpc"}

for name in desired | actual:
    if name not in actual:
        print(name, "create")
    elif name not in desired:
        print(name, "unmanaged: in account, not in config")
    elif desired[name] != actual[name]:
        print(name, "update (drift from config)")
Output
web_sg update (drift from config)
web_server create
orphan_eip unmanaged: in account, not in config

In production Rendering a change set and checking for replacements

A toy change-set renderer using CloudFormation's own symbols, plus a drift check on one SQS queue property — the same shape as describe-stack-resource-drifts.

PythonChange-set symbols and a drift check on one propertyruns
import json
from pathlib import Path

CHANGE_SET = [
    {"logical_id": "OrdersQueue", "type": "AWS::SQS::Queue", "action": "Modify",
     "replacement": False, "changed": {"VisibilityTimeout": [30, 120]}},
    {"logical_id": "SettlementsTable", "type": "AWS::DynamoDB::Table", "action": "Modify",
     "replacement": True, "changed": {"KeySchema": ["id", "settlement_date"]}},
    {"logical_id": "AuditBucket", "type": "AWS::S3::Bucket", "action": "Remove", "replacement": False},
    {"logical_id": "ReportsFunction", "type": "AWS::Lambda::Function", "action": "Add", "replacement": False},
]
SYMBOL = {"Add": "+", "Remove": "-", "Modify": "~"}


def render_change_set(changes):
    lines = []
    for c in changes:
        symbol = "-/+" if c["action"] == "Modify" and c["replacement"] else SYMBOL[c["action"]]
        lines.append(f"{symbol}  {c['logical_id']} ({c['type']})")
        if c["replacement"]:
            lines.append("     ! replacement: a new resource is created, the old one is destroyed")
    return lines


for line in render_change_set(CHANGE_SET):
    print(line)

replacements = [c["logical_id"] for c in CHANGE_SET if c.get("replacement")]
print("review required before execute:", replacements)

Path("expected.json").write_text(json.dumps({"VisibilityTimeout": 120, "MaximumMessageSize": 262144}))
Path("actual.json").write_text(json.dumps({"VisibilityTimeout": 30, "MaximumMessageSize": 262144}))
expected = json.loads(Path("expected.json").read_text())
actual = json.loads(Path("actual.json").read_text())
drift = {k: (expected[k], actual[k]) for k in expected if expected[k] != actual.get(k)}
print("drifted properties:", drift)
Output
~  OrdersQueue (AWS::SQS::Queue)
-/+  SettlementsTable (AWS::DynamoDB::Table)
     ! replacement: a new resource is created, the old one is destroyed
-  AuditBucket (AWS::S3::Bucket)
+  ReportsFunction (AWS::Lambda::Function)
review required before execute: ['SettlementsTable']
drifted properties: {'VisibilityTimeout': (120, 30)}

See it happen One reconcile cycle, three tools, one shape

refreshapply (only after review)config(.tf / template.yaml / CDK app)last known state(CFN service / tfstate)real AWS resourcesplan / change set
  1. Nothing has run yet. config, state and real resources are three separate things.
  2. A refresh reads real AWS resources into the tool's view of state before any comparison.
  3. The diff compares config against the refreshed state, not against the real resources directly.
  4. The diff is the change set or plan: create/update/delete/replace, shown before anything changes.
  5. Only after review does apply push changes to real resources, which then feeds the next refresh.
Notice: config never talks to real resources directly — every apply goes through a refresh and a reviewed diff, the step a console click skips.
Scenario A

In your world

The situation

Your team's "IaC" is a Jenkins job running a bash script of aws s3api/aws rds CLI commands recorded eighteen months ago. No file says what infrastructure should look like today, only a script of what to run.

What actually breaks

At 2 AM an on-call engineer bumps the RDS instance class in the console to relieve CPU pressure. Three days later Jenkins runs for an unrelated change, re-executes the old aws rds modify-db-instance --db-instance-class db.t3.medium, and silently reverts the fix — the script never knew the console change happened.

What you'd do on Monday

Replace the script with a declarative config so the next run diffs against reality:

Terraform
resource "aws_db_instance" "settlements" {
  identifier     = "settlements-pg-01"
  instance_class = "db.t3.large"   # the incident fix, now written down
  engine         = "postgres"
}
Shell
terraform plan -out=tfplan   # shows "update in place: instance_class" if config still says t3.medium
terraform show tfplan        # review before anyone runs apply

The 2 AM fix now has to be reflected here, or the next plan shows it as a pending revert — in review, not silently in production.

Scenario B

Imagine you work at Postman

400 stacks across 12 teams

The numbers

Illustrative: 400 stacks across 12 teams, ≈33 per team. One undocumented tweak per stack a quarter is ≈400 untracked changes company-wide; at 20 minutes to diff one by hand, reconciling all of them costs ≈133 hours a quarter (400 × 20 ÷ 60).

What the Scenario-A answer does here

The Scenario A fix still works per stack, but trusting every engineer to run plan locally doesn't scale to 400: a stale, unreviewed plan looks identical to a clean one until apply runs.

What flips at this scale

At this count, drift detection must be scheduled and centralised: a nightly job plans all 400 stacks and tickets any change nobody authored, instead of waiting for the next deploy to "fix" an incident patch by accident.

The trade-off you'd defend

I would defend centralised drift detection with a required-plan CI gate over trusting local plan runs, trading CI minutes for an audit trail across all 400 stacks, not just the ones someone remembered to check.

Illustrative numbers for the arithmetic; not Postman-published figures.

CloudFormation vs Terraform vs CDK

AxisCloudFormationTerraformCDK (Python)
State ownershipAWS service, per stackA file/backend you ownNone; delegates to CloudFormation
ScopeAWS onlyAWS plus other providersAWS only (synthesizes to CloudFormation)
Review artefactChange setPlancdk diff
AuthoringYAML/JSONHCLGeneral-purpose code
LogicMinimal, declarativeMinimal, HCL expressionsA lot; a real language

Gotchas

A console fix gets silently reverted

Symptom

A manual change that fixed an incident disappears after the next unrelated deploy, with no error logged.

Cause

The config never captured the manual change, so the diff treats the old value as desired and reapplies it.

Fix

Treat every manual production change as a same-day config update, even a one-line bump.

"No changes" is trusted without ever running a diff

Symptom

A team believes their infra is drift-free because nobody has reported a problem.

Cause

Nobody schedules drift detection; a diff only runs right before a planned apply.

Fix

Run a standalone drift-detection job on a schedule, independent of planned deploys, and alert on unexpected changes.

Tricky questions

Does CDK replace CloudFormation?
Most people say

Yes, CDK is a newer alternative that avoids CloudFormation's limitations.

Actually

No. CDK generates a CloudFormation template; cdk deploy hands it to CloudFormation, which still owns the state and computes the change set.

Is a clean Terraform plan a guarantee apply will succeed?
Most people say

Yes, if the plan is clean, apply does exactly that.

Actually

No. The plan is computed against a snapshot; a change between plan and apply, or a provider-side failure, can still change the outcome.

Scenario question

You're handed a repo with both a CloudFormation template and a Terraform config that both claim to manage the same S3 bucket, left over from a half-finished migration. What do you do before touching either?

Model answer

I would not run apply or deploy either tool blindly — two state owners for one resource means whichever runs last wins and fights the other. I'd check which tool's state tracks the bucket, pick one owner, and remove it from the other's state (terraform state rm, or dropping it from the template without DeletionPolicy: Delete so the bucket survives) so only one tool reconciles it going forward. The trade-off: the retired tool's next plan shows a false "create," which I'd flag in the PR.

Interview corner

CommonWhat's the difference between CloudFormation, Terraform and CDK?
“I'd say CloudFormation and Terraform both reconcile desired state with a reviewable diff, but CloudFormation owns its state inside the service while Terraform's state is a file I own; CDK just generates a CloudFormation template, so it inherits CloudFormation's model.”
TrickIf a CDK app has no state file, does that mean no locking issues?
“I'd push back on that — CDK deploys to a CloudFormation stack, and CloudFormation serializes updates to that stack, so two concurrent cdk deploy runs can still collide.”
Experience checkHave you seen a manual change get reverted by automation?
“I've seen a console fix get undone by an unrelated Jenkins run because a CLI script had no idea the change happened — a reviewed plan would have caught it.”
DesignHow would you catch drift before it causes an incident?
“I'd run scheduled drift detection across every stack and alert on any unexpected diff, instead of waiting for the next deploy to surface it.”

Memory hook

A floor plan and a landlord who only does the work the plan still shows is missing.

If you remember only one thing:

The diff, not the apply, is the part of IaC that actually protects you — never skip reviewing it.

Self-check

  1. What does CDK's synth step talk to over the network?

    Nothing — it writes a CloudFormation template locally; AWS calls happen only during the later diff/deploy.

  2. Why can a Terraform plan show a change you didn't make?

    plan refreshes state from real AWS resources before diffing, so it can surface someone else's console or CLI change.

  3. Is drift itself a failure?

    No — only undetected drift is; reviewed drift is the system working as intended.

Prove it

Write two dicts, desired and actual, covering three resources: one changed, one missing, one extra. Extend the minimal example's loop to also flag a replacement when a chosen property (say, a database engine) changes, and print which resources need a -/+ warning. Paste the output.

aws-08-02Card 2 of 6

Terraform state and backends

HardP3 · Role-dependentTalk about it10 min
terraformstatebackendslockingimport

Why you care

Terraform's state file is the single most dangerous file in your repo, and most teams treat it like an implementation detail. It is the only record binding your .tf resource blocks to real AWS resource IDs; lose that binding and Terraform cannot tell "this doesn't exist yet" from "this exists but I forgot where." Interviewers ask about backends and locking because the answer reveals whether you've run Terraform with more than one person touching the same environment — exactly what your current single-script setup has never tested.

  • Note: backend-specific locking and import mechanics are tool documentation, verify exact flags/behaviour before relying on them in production.

Explain like I'm 5

Think of the state file as the one ledger a shipping company keeps, mapping container numbers to physical crates on the dock. The crates don't care about the ledger; they exist either way. But the moment the ledger says "container 4471 is empty" when the crate is sitting right there full of goods, every worker who trusts the ledger walks past it as if it isn't there — or worse, orders a new one.

Core concept

  • State is a binding, not a backup. Each entry maps a resource address (aws_db_instance.settlements) to a real resource's attributes, keyed by its actual AWS ID. Terraform never infers this binding from scratch; it is written once, at apply or import time, and must be deliberately maintained.
  • Remote backends (S3, HCP Terraform, and others) store that file somewhere shared instead of on one laptop, so a team works from the same source of truth. Accessing a remote backend's state "generally requires access credentials, since state data contains extremely sensitive information" — the backend is also an access-control boundary, not just storage.
  • Locking happens automatically on any operation that could write state; you see no message, it just happens. Not every backend supports it. If locking fails, Terraform refuses to continue by default; -lock=false overrides this, and the documentation explicitly discourages it, because that's exactly how two concurrent writers corrupt one file.
  • Force-unlock is the manual escape hatch when an operation dies mid-run and never releases its lock. It requires the specific lock ID as a safety check, so you can't accidentally steal someone else's active lock; use it only on your own stuck lock.
  • Import binds an existing AWS resource into state so Terraform starts managing it, but it does not generate the matching .tf config for you — you write the resource block first, then import binds state to it. Each real object should be bound to exactly one resource address.

Decision rule: beyond a single-person sandbox, state lives in a backend that supports locking, with credentials scoped separately from the credentials Terraform uses on AWS. No locking support means no safe concurrent operators — a race condition waiting for a demo day.

Under the hood

The state file is JSON with a serial (a version counter), a lineage (a UUID so Terraform can detect two unrelated states being merged by accident), and a resources array where each entry carries a type, name and instances holding the real provider-returned attributes. The resource address — type.name, optionally inside a module path — is the lookup key everything hangs off. A lock is not an OS-level filesystem lock; it's a cooperative record every well-behaved Terraform process checks before writing. That's why -lock=false is dangerous: it doesn't disable an enforced mutex, it just tells your own process to stop checking the record, while every other honest process still respects it.

Minimal example Locking is a cooperative record, not magic

PythonA second lock attempt is refused while the first is heldruns
from pathlib import Path

LOCK_FILE = Path("state.lock")
LOCK_FILE.write_text("held by jenkins-job-412")

def acquire_lock(operator):
    if LOCK_FILE.exists():
        raise RuntimeError(LOCK_FILE.read_text())
    LOCK_FILE.write_text(f"held by {operator}")

try:
    acquire_lock("laptop-manual-apply")
except RuntimeError as e:
    print("RuntimeError:", e)
Output
RuntimeError: held by jenkins-job-412

In production Losing a resource's binding, and getting it back

PythonA manual "state rm" turns a real resource into "unmanaged"runs
import json
from pathlib import Path

STATE = {
    "version": 4, "serial": 12, "lineage": "b7f1-fixed-for-example",
    "resources": [
        {"module": "root", "type": "aws_s3_bucket", "name": "audit_logs",
         "instances": [{"attributes": {"id": "etl-audit-logs-prod", "region": "ap-south-1"}}]},
        {"module": "root", "type": "aws_db_instance", "name": "settlements",
         "instances": [{"attributes": {"id": "settlements-pg-01", "engine": "postgres"}}]},
    ],
}
Path("terraform.tfstate").write_text(json.dumps(STATE))


def addresses(state):
    return {f"{r['type']}.{r['name']}": r["instances"][0]["attributes"]["id"] for r in state["resources"]}


state = json.loads(Path("terraform.tfstate").read_text())
print("bound resources:", addresses(state))

state["resources"] = [r for r in state["resources"] if r["name"] != "settlements"]
Path("terraform.tfstate").write_text(json.dumps(state))

state_after = json.loads(Path("terraform.tfstate").read_text())
print("bound resources after state rm:", addresses(state_after))
print("real database still running in AWS:", True)
print("terraform now treats it as unmanaged:", "aws_db_instance.settlements" not in addresses(state_after))

rebound = {**state_after, "resources": state_after["resources"] + [
    {"module": "root", "type": "aws_db_instance", "name": "settlements",
     "instances": [{"attributes": {"id": "settlements-pg-01", "engine": "postgres"}}]},
]}
print("bound resources after explicit import:", addresses(rebound))
Output
bound resources: {'aws_s3_bucket.audit_logs': 'etl-audit-logs-prod', 'aws_db_instance.settlements': 'settlements-pg-01'}
bound resources after state rm: {'aws_s3_bucket.audit_logs': 'etl-audit-logs-prod'}
real database still running in AWS: True
terraform now treats it as unmanaged: True
bound resources after explicit import: {'aws_s3_bucket.audit_logs': 'etl-audit-logs-prod', 'aws_db_instance.settlements': 'settlements-pg-01'}

See it happen A lock's lifecycle around one apply

operator Abackend lock recordoperator Bapply: check for lockno lock found, acquireapply: check for locklocked by A, refuseapply completes, release lockretry apply
  1. Locking is automatic on any write operation; no message is shown when it succeeds.
  2. A's process writes its own lock record before touching state.
  3. B starts a concurrent apply against the same state.
  4. Terraform does not continue if locking fails; B's apply stops here, by default.
  5. A's process finishes and removes its own lock record.
  6. B can now acquire the lock and proceed, safely, one writer at a time.
Notice: the lock never touches the real AWS resources; it only ever blocks a second Terraform process from writing the same state file at the same time.
Scenario A

In your world

The situation

Two of you share one Terraform config for a dev environment on an S3 backend with no locking configured, "because it's just dev." During a sprint demo you both run terraform apply within the same minute.

What actually breaks

Both processes read the same state and write back independently; whichever write lands last silently overwrites the other's changes. The next plan reports a dozen resources needing re-creation, because state matches neither apply.

What you'd do on Monday

Use a backend that supports locking and never bypass it:

Shell
terraform init -reconfigure   # pick a backend that documents lock support
terraform apply               # locking is automatic; the second apply waits or refuses
terraform force-unlock <ID>   # only if a run died mid-apply and never released its own lock

If a lock is ever stuck, get the lock ID from the error and confirm it is your abandoned run before force-unlocking.

Scenario B

Imagine you work at Groww

180 Terraform-managed accounts

The numbers

Illustrative: 180 accounts running ≈40 applies a day company-wide, ≈0.22 per account per day. If 3 accounts still share one state file from an early setup, those 3 compete for a single lock: ≈0.67 applies a day funnel through it, and at 10-15 minutes held per apply, two engineers colliding in the same window becomes a near-weekly event.

What the Scenario-A answer does here

The Scenario A fix (a locking backend) still prevents corruption, but doesn't fix the design: a shared state file for 3 accounts means every lock wait blocks the other two, and a bad apply can only be rolled back by touching the file all three depend on.

What flips at this scale

At this account count, state boundaries must match blast-radius boundaries: one state file per account, never shared, so lock contention on one team's deploy can't queue another team's unrelated change.

The trade-off you'd defend

I would defend splitting shared state immediately over tolerating the contention, trading a careful state mv migration for deploying 180 accounts independently and in parallel.

Illustrative figures for the arithmetic; not Groww-published numbers.

Gotchas

Losing the state binding makes a real resource look unmanaged

Symptom

terraform plan proposes to create a resource that is already running in AWS, with the same name you expected it to already manage.

Cause

The resource's entry was removed from state (manually, by a bad merge, or by state corruption), so Terraform has no binding left even though the real resource and your config both still exist.

Fix

Re-bind with terraform import after writing the matching resource block; never let apply run against a stale plan that proposes "create" for something you know is live, until you've confirmed it's not just a lost binding.

-lock=false used to "fix" a stuck lock

Symptom

Two applies run back to back with no error, and the resulting state matches neither.

Cause

Someone bypassed the lock check instead of investigating why it was stuck, letting two writers hit the same file.

Fix

Never use -lock=false to work around contention; find the real lock holder and either wait, or force-unlock the correct, verified lock ID.

Tricky questions

If I delete a resource block from my .tf file, does the real AWS resource go away?
Most people say

No, deleting the code just stops managing it; the resource stays.

Actually

On the next apply, Terraform sees the resource in state but not in config and destroys it. Use terraform state rm if you want to stop managing something without deleting it.

Does state locking protect the real AWS resources from concurrent changes?
Most people say

Yes, locking means two people can't change the same AWS resource at once.

Actually

Locking only protects the state file. Nothing stops someone editing the same resource in the console while Terraform holds its lock; that's drift, a separate problem from locking.

Scenario question

You're handed a terraform plan that proposes to create an RDS instance with an identifier that matches one already running in production. What is your first move?

Model answer

I would not run apply. A "create" plan for something that visibly already exists usually means the state binding is missing, not that Terraform is confused about reality. I'd check terraform state list for that resource's address; if it's absent, I'd write the matching resource block and run terraform import to bind the real ID into state, then re-run plan and expect "no changes." The trade-off is time: import is manual, but far cheaper than restoring from a bad apply.

Interview corner

CommonWhat does Terraform state actually store?
“I'd describe it as the binding between my resource addresses and the real AWS IDs Terraform created, plus a serial and lineage ID so it can tell two unrelated state files apart.”
TrickDoes -lock=false disable the lock for everyone?
“No — it only tells my own process to skip checking the lock. If anyone else applies at the same time, we can still corrupt the state, because the lock was only ever respected cooperatively.”
Experience checkHave you dealt with a stuck or lost state binding?
“I've seen a resource vanish from state after a bad merge; the fix was writing the resource block fresh and running terraform import with the real AWS ID, then confirming plan came back clean.”
DesignHow would you structure state for a team of 20 engineers across 10 services?
“I'd use one state file per service per environment, in a backend that supports locking, with credentials scoped per environment, bounding both blast radius and lock contention.”

Memory hook

State is the shipping ledger, not the dock. Lose the ledger entry and the crate is still there, just invisible to everyone trusting the ledger.

If you remember only one thing:

A missing binding looks exactly like "doesn't exist yet" — never trust a "create" plan for something you can see is already running.

Self-check

  1. What two fields let Terraform detect two unrelated states being merged by mistake?

    lineage (a UUID) and serial (a version counter); a lineage mismatch is a hard stop.

  2. Does terraform import write your .tf resource block for you?

    No. You write the resource block first; import only binds an existing real resource's ID into state against that block.

  3. If locking fails, what does Terraform do by default?

    It refuses to continue, rather than risk two writers touching the same state file.

Prove it

Build a toy JSON "state" file with two resources, then write acquire_lock/release_lock functions that store a lock ID in a lock file. Simulate three operators: one that acquires, applies (sleep-free, just a print), and releases cleanly; one that tries to acquire while the first still holds it and gets refused, like the minimal example; one that force-unlocks with the wrong ID and gets refused. Paste all three outcomes.

aws-08-03Card 3 of 6

Terraform modules and reading a plan

HardP3 · Role-dependentTalk about it9 min
terraformmodulesplanvariablesoutputsdrift

Why you care

"Reading a plan" sounds passive, but it's the one skill that separates someone who runs terraform apply from someone who should be allowed to. A 200-line plan with one -/+ buried in the middle looks identical, at a glance, to one full of harmless tweaks — until "destroy, then create" drops a production table. Interviewers probe module design and plan-reading together because both test the same thing: can you say exactly what will happen to a resource before anyone clicks apply.

  • Note: module registry mechanics and exact plan-output formatting are tool documentation; verify current CLI output before quoting it to a team.

Explain like I'm 5

A module is a recipe card handed to several cooks: it lists what it needs (variables) and what comes out (outputs). A plan is the waiter reading back your order: "modify the spice level, keep the rest" is a safe update; "we have to take your dish back and remake it from scratch" is a replacement — same meal eventually, but you lost what was already cooking.

Core concept

  • A module is a reusable collection of resources with a defined interface: variable blocks declare inputs, output blocks declare what it hands back. The root module is what Terraform runs directly; every module block inside it is a child, and children can nest.
  • Reading a plan means reading the symbol and the reason: + create, - destroy, ~ update in place, -/+ destroy-then-create. A replacement changes the resource's identity — anything referencing its old ID can break even if the "new" resource looks identical.
  • Drift vs intentional change show up as the same ~ with different causes: drift is a ~ with no matching code change (a console edit, caught by refresh); an intentional change matches a diff in your own PR.
  • Saved plans (-out=tfplan) let you promote the exact reviewed plan straight to apply — but it's only valid against the exact config and state it was computed from; if either moves, Terraform refuses to apply a stale plan.

Decision rule: review every plan for -/+ lines first, since those can destroy data; review ~ lines next against an actual code diff; treat any unmatched ~ as drift to investigate.

Under the hood

A plan is computed by walking Terraform's internal dependency graph, not by diffing resources in isolation. A replacement can cascade: if resource A's ID changes because A is replaced, and B references that ID, the graph walk marks B for update too — and if B's provider treats that attribute as immutable, B's update becomes a second, unplanned -/+. This is also why a saved plan is tied to the exact snapshot it was computed from: replaying it against a different one means reasoning from a graph that no longer matches reality.

Minimal example Same key, two different reasons for a change

PythonDrift on refresh vs. an intentional config bumpruns
config = {"instance_type": "t3.micro", "desired_count": 2}
refreshed_state = {"instance_type": "t3.small", "desired_count": 2}
new_config = {"instance_type": "t3.small", "desired_count": 4}


def diff(a, b):
    return {k: (a[k], b[k]) for k in a if a[k] != b.get(k)}


print("drift found on refresh (console edit, config unchanged):", diff(config, refreshed_state))
print("plan once config catches up to the real value:", diff(refreshed_state, new_config))
print("same instance_type key, different cause:", "instance_type" in diff(config, refreshed_state),
      "instance_type" in diff(refreshed_state, new_config))
Output
drift found on refresh (console edit, config unchanged): {'instance_type': ('t3.micro', 't3.small')}
plan once config catches up to the real value: {'desired_count': (2, 4)}
same instance_type key, different cause: True False

In production A module's interface, and a promotion gate for saved plans

PythonDeclared variables/outputs, plus refusing a stale saved planruns
import hashlib
from pathlib import Path

MODULE = Path("modules/vpc_subnet")
MODULE.mkdir(parents=True, exist_ok=True)
tf = MODULE / "main.tf"
tf.write_text(
    'variable "cidr_block" {}\n'
    'variable "az_count" {\n  default = 2\n}\n'
    'output "subnet_ids" {\n  value = aws_subnet.this[*].id\n}\n'
)


def declared_names(path, keyword):
    return [line.split('"')[1] for line in path.read_text().splitlines() if line.strip().startswith(keyword)]


print("variables:", declared_names(tf, "variable"))
print("outputs:", declared_names(tf, "output"))


def config_hash(path):
    return hashlib.sha256(path.read_text().encode()).hexdigest()[:12]


Path("saved.tfplan").write_text(config_hash(tf))


def apply_saved_plan(plan_file, config_file):
    saved = plan_file.read_text()
    current = config_hash(config_file)
    if saved != current:
        raise RuntimeError("plan is stale: config changed since plan was saved, re-run plan")
    return "apply proceeds from the saved plan, no new diff is computed"


print(apply_saved_plan(Path("saved.tfplan"), tf))

tf.write_text(tf.read_text() + '\nvariable "extra" {}\n')
try:
    apply_saved_plan(Path("saved.tfplan"), tf)
except RuntimeError as e:
    print("RuntimeError:", e)
Output
variables: ['cidr_block', 'az_count']
outputs: ['subnet_ids']
apply proceeds from the saved plan, no new diff is computed
RuntimeError: plan is stale: config changed since plan was saved, re-run plan
Scenario A

In your world

The situation

Your CI pipeline runs terraform plan, posts the full output as a PR comment, and the reviewer skims it before approving. The actual config change is a one-line bump to desired_count on an ECS service.

What actually breaks

The plan is 180 lines because an unrelated upstream module version also moved, and inside that noise sits a single -/+ on a DynamoDB table whose key_schema got silently reordered. The reviewer scrolls past it, approves, and the apply drops and recreates the production table.

What you'd do on Monday

Never trust eyeballing a long plan; make the pipeline fail automatically on any unacknowledged destructive action:

Shell
terraform plan -out=tfplan
terraform show -json tfplan | jq '.resource_changes[] | select(.change.actions | index("delete") or index("create"))'
# fail CI if that query returns anything on a resource not named in the PR description

That turns "did the reviewer notice the -/+" into a hard gate that can't be skimmed past.

Scenario B

Imagine you work at Twilio

900 module instantiations across 60 services

The numbers

Illustrative: 60 services each calling a shared module library ≈15 times ≈ 900 instantiations company-wide. A breaking change to one module's default is a one-line diff in the module's repo, but once every consumer re-plans, that line can fan out into ≈900 -/+ or ~ lines across 60 downstream plans.

What the Scenario-A answer does here

The Scenario A fix still catches each replacement, but reviewing 900 flagged PRs one at a time isn't a plan-reading problem anymore, it's a rollout problem: nobody coordinates 60 teams merging in the same week.

What flips at this scale

At this fan-out, module versions must be pinned per consumer, and a bump must ship its own note on which resource types will show -/+, so each team decides when to take it instead of discovering it mid-sprint.

The trade-off you'd defend

I would defend strict per-consumer pinning plus a fan-out impact note over "always use latest," trading slower fix propagation for scheduling 900 instantiations' worth of change in batches teams can absorb.

Illustrative figures for the arithmetic; not Twilio-published numbers.

Gotchas

A -/+ buried in a long plan gets approved by accident

Symptom

An apply destroys and recreates a resource nobody intended to touch, discovered only after data loss.

Cause

Long plan output hides a single destructive line among dozens of harmless updates, and review was a visual skim.

Fix

Script the gate — parse the plan's JSON output and fail CI on any delete/replace action not called out in the PR.

A ~ with no matching code change is treated as "just Terraform being Terraform"

Symptom

Every plan on a resource shows the same ~ update, apply after apply, and it never "sticks."

Cause

Something outside Terraform keeps changing the real resource back after each apply.

Fix

Treat a repeating unexplained ~ as an ownership conflict; find and stop whatever else is touching the resource.

Tricky questions

Does a -/+ replacement always mean data loss?
Most people say

No, Terraform recreates the resource with the same data, just a new ID.

Actually

Depends on the resource. Stateful resources genuinely lose data on replacement unless a prevent_destroy rule, snapshot, or deletion policy protects them first.

If a module's output doesn't change, is it safe to assume nothing downstream changes?
Most people say

Yes, unchanged outputs mean unchanged behaviour for consumers.

Actually

No — a module can change an internal resource's config without changing its outputs. A stable interface says nothing about what the module does differently on the next apply.

Scenario question

You're handed a 300-line terraform plan output for a routine dependency bump, and asked to approve it in a stand-up in five minutes. What do you actually do?

Model answer

I would not read 300 lines top to bottom. I'd run terraform show -json and filter for anything with delete or create in the actions, since those lines can lose data or break references. If that's empty, I'd skim remaining ~ lines for stateful resources. If it's not empty, I would not approve in the stand-up — I'd ask for the specific resource and reason in writing. The trade-off is speed, but a -/+ on a database isn't something I'll approve from memory of what "usually" happens.

Interview corner

CommonHow do you review a Terraform plan before approving it?
“I'd script the triage rather than eyeball it: filter for any delete or replace action first, and treat each one as requiring an explicit explanation before I approve anything else.”
TrickDoes a long, mostly harmless plan reduce the risk of missing something dangerous?
“I'd say it increases the risk — a destructive line is easiest to miss when it's surrounded by 200 harmless ones. That's why I filter for destructive actions first instead of reading top to bottom.”
Experience checkHave you seen module changes fan out unexpectedly?
“I've seen a shared module default change ripple into several downstream plans at once, each showing a replacement nobody intended. The fix was pinning module versions per consumer.”
DesignHow would you make plan review scale across dozens of teams?
“I'd automate the gate so CI fails on unacknowledged destructive actions, and require version bumps to note which resource types will show as replaced.”

Memory hook

A waiter reading back "remake from scratch" sounds just as calm as "keep it, just add spice" — the words are what tell you which one it is.

If you remember only one thing:

Filter every plan for delete/replace first; everything else can wait for a calmer read.

Self-check

  1. What's the practical difference between ~ and -/+ in a plan?

    ~ updates in place, keeping identity; -/+ destroys and recreates, which can break references and lose data for stateful resources.

  2. Why can the same property show up as drift in one plan and as an intentional change in the next?

    Drift is a ~ with no matching code diff; an intentional change matches a diff in your own config.

  3. Why does Terraform refuse to apply an old saved plan file?

    It's only valid against the exact config and state it was computed from; applying it blind could do something other than what was reviewed.

Prove it

Write a toy "plan" as a list of dicts with logical_id, action and replacement fields, mixing a few updates with one replacement. Write a filter for destructive entries, and a function that raises if any destructive logical_id isn't in an approved set. Show it passing when approved and raising when not. Paste the output.

aws-08-04Card 4 of 6

Workspaces and multi-environment layout

MediumP3 · Role-dependentTalk about it9 min
terraformworkspacesenvironmentsisolation

Why you care

"We use Terraform workspaces for dev/staging/prod" is a common half-truth in a resume screen, because workspaces isolate state, not credentials, not blast radius, not even the account. A team that believes workspaces give them environment isolation is one export AWS_PROFILE=prod away from a "dev" apply landing in production. Interviewers who ask about multi-environment layout are checking whether you know where the isolation boundary actually sits — state file versus account versus credentials — because that gap is where real incidents happen.

Explain like I'm 5

One office building has a single front-door key that opens every floor, but each floor has its own labelled filing cabinet. CLI workspaces are the cabinets: dev's and prod's paperwork never mix in the same drawer. But the key — your AWS credentials — still opens every floor regardless of which cabinet you're filing into. That's exactly the mistake workspaces don't protect you from.

Core concept

  • A CLI workspace lets one Terraform config maintain multiple named states behind the same backend; terraform workspace select dev versus prod changes which state file commands read and write, nothing else. Only backends that support it offer this.
  • State isolation from workspaces is real: plan in one workspace never sees another's resources, and switching workspaces changes nothing about your provider block or credentials.
  • What workspaces do not isolate: credentials and access controls. The documentation is explicit that workspaces "are not appropriate for system decomposition or deployments requiring separate credentials and access controls" — one AWS profile backs every workspace unless you build separate credential-switching logic yourself.
  • Account/credential isolation comes from a directory-per-environment layout instead: separate .tf roots per environment, each with its own state key and its own provider config (different account, different role), so a mistake in one directory can't even authenticate against another's resources.
  • Tags (Accounts, tagging and environment boundaries) are a complementary control: they let you tell environments apart after the fact, but do nothing to prevent a wrong-environment apply before it happens.

Decision rule: use workspaces only for genuinely low-risk variants sharing one account (feature-branch sandboxes); the moment "prod" enters the picture, isolate it with its own directory, backend config and assumed role.

Minimal example Workspaces change the state key, not the credentials

PythonTwo "environments," one shared credentialruns
def state_key(workspace):
    return f"env:/{workspace}/terraform.tfstate"


def credentials_for(workspace):
    return "etl-shared-role"


print(state_key("dev"), state_key("prod"))
print("same credentials for both:", credentials_for("dev") == credentials_for("prod"))
Output
env:/dev/terraform.tfstate env:/prod/terraform.tfstate
same credentials for both: True

In production Directory-per-environment, with separate deploy roles

PythonSeparate backend config and role per environment directoryruns
from pathlib import Path

ENVS = {
    "dev": {"account_id": "111111111111", "role_arn": "arn:aws:iam::111111111111:role/etl-dev-deploy"},
    "prod": {"account_id": "222222222222", "role_arn": "arn:aws:iam::222222222222:role/etl-prod-deploy"},
}

for env, cfg in ENVS.items():
    path = Path(f"envs/{env}")
    path.mkdir(parents=True, exist_ok=True)
    (path / "backend.hcl").write_text(
        f'bucket   = "tf-state-{env}"\n'
        f'key      = "pipeline/terraform.tfstate"\n'
        f'role_arn = "{cfg["role_arn"]}"\n'
    )

for env in ENVS:
    print(f"--- envs/{env}/backend.hcl ---")
    print((Path(f"envs/{env}") / "backend.hcl").read_text().strip())

dev_role = [l for l in Path("envs/dev/backend.hcl").read_text().splitlines() if "role_arn" in l][0]
prod_role = [l for l in Path("envs/prod/backend.hcl").read_text().splitlines() if "role_arn" in l][0]
print("dev and prod use separate deploy roles:", dev_role != prod_role)
Output
--- envs/dev/backend.hcl ---
bucket   = "tf-state-dev"
key      = "pipeline/terraform.tfstate"
role_arn = "arn:aws:iam::111111111111:role/etl-dev-deploy"
--- envs/prod/backend.hcl ---
bucket   = "tf-state-prod"
key      = "pipeline/terraform.tfstate"
role_arn = "arn:aws:iam::222222222222:role/etl-prod-deploy"
dev and prod use separate deploy roles: True
Scenario A

In your world

The situation

Your team runs one Terraform config with dev, staging and prod workspaces, all authenticating through the same long-lived AWS profile exported in everyone's shell, "because workspaces keep them separate."

What actually breaks

An engineer debugging staging exports the prod profile to check the console, forgets to unset it, then runs terraform workspace select dev && terraform apply. Workspace selection changes the state file, but the still-exported prod credentials mean the apply authenticates against prod — creating real resources there while every log line says "workspace: dev."

What you'd do on Monday

Stop relying on workspace name as a safety signal; make environment and credentials the same decision:

Shell
cd envs/dev
terraform init -backend-config=backend.hcl   # dev's own state key AND dev's own role_arn
terraform apply                               # cannot reach prod even if a stale profile is exported

Moving to a directory-per-environment layout means the thing you cd into decides which account you can even authenticate against, independent of shell state.

Scenario B

Imagine you work at Instacart

50 environments across 3 regions

The numbers

Illustrative: ≈50 environments (dev/staging/prod for ≈17 services) across 3 regions. Under a workspace design, all 50 can share one set of long-lived credentials — one leaked CI token has a blast radius of up to 50 of 50 environments, not 1.

What the Scenario-A answer does here

The Scenario A fix still works per-service, but retrofitting it onto 50 already-live environments means a careful, one-at-a-time state mv migration, not a single PR.

What flips at this scale

At this count, isolation must cap any single credential's damage at 1 of 50: per-environment IAM roles and backend configs become a security requirement, not a tidiness preference.

The trade-off you'd defend

I would defend the migration cost of per-environment roles over keeping workspaces at this scale, trading weeks of incremental state mv work for turning a 50-environment incident into a 1-environment one.

Illustrative figures for the arithmetic; not Instacart-published numbers.

Isolating environments

AxisCLI workspacesSeparate directories, shared accountSeparate directories, separate accounts
State isolationYes, per workspaceYes, per directory/backend keyYes, per directory/backend key
Credential isolationNo — one config, one credential pathPartial — same account, can scope IAM by resourceYes — different account per environment
Blast radius of one leaked credentialUp to every workspaceBounded to one account, still broadBounded to one environment
Setup effortLowestMediumHighest (account vending, cross-account roles)

Gotchas

A stale exported credential silently targets the wrong account

Symptom

An apply in the "dev" workspace creates or modifies resources that show up in the prod AWS console.

Cause

Workspace selection changes the state key only; it does nothing to the credentials your shell or CI job happens to be using.

Fix

Tie environment selection to credential selection in one step (a directory with its own backend config and role).

Workspaces used as a substitute for account separation

Symptom

A security review finds that one IAM role can touch every environment's resources, including production.

Cause

Someone treated "we use workspaces" as "environments are isolated," which the documentation explicitly says they aren't meant for.

Fix

Reserve workspaces for genuinely low-risk, same-account variants; put production behind its own account and role.

Tricky questions

Do Terraform workspaces give you the same isolation as separate AWS accounts?
Most people say

Yes, that's what "workspace" means — an isolated environment.

Actually

Workspaces isolate state files only. Real account isolation requires separate accounts and roles, not just a workspace name.

If dev and prod are separate workspaces, can a dev apply ever touch prod resources?
Most people say

No, Terraform won't let a workspace see another workspace's resources.

Actually

It can't see prod's state, but if the credentials behind that apply are prod credentials, the apply authenticates against prod regardless of which workspace's state file records it.

Scenario question

You're handed a security review finding: "one IAM role has write access to every environment, discovered via Terraform workspace usage." How do you explain the fix to a non-technical stakeholder?

Model answer

I'd explain that workspaces are like labelled folders for paperwork, not separate locked rooms — the same key still opens every folder's door. The fix is giving each environment its own room with its own key: a separate account (or at minimum a separate IAM role) per environment, so dev credentials physically cannot authenticate against production. The trade-off is migration time: moving one role into N environment-scoped roles means touching every environment's access setup one at a time, so I'd phase it starting with production.

Interview corner

CommonWhat do Terraform workspaces actually isolate?
“I'd say they isolate state files behind one backend, nothing more — not credentials, accounts, or access controls, which is why I wouldn't use them alone for prod.”
TrickIs 'we use workspaces for dev/staging/prod' a safe answer in an interview?
“I'd flag it as a risk if that's the whole story, since it usually means one credential set reaches every environment. I'd want prod behind its own account or role.”
Experience checkHave you seen a wrong-environment apply happen?
“I've seen the near-miss version: a stale exported profile almost let a 'dev' apply touch the wrong account. That pushed me toward separate directories and roles for anything that matters.”
DesignHow would you lay out Terraform for 10 services across dev/staging/prod?
“I'd use a directory per environment per service, each with its own backend config and IAM role, and reserve workspaces only for short-lived, same-account sandboxes.”

Memory hook

One master key, many labelled filing cabinets — the cabinets don't change what the key can open.

If you remember only one thing:

Workspaces isolate state, never credentials; prod needs its own account or role, not just its own workspace name.

Self-check

  1. What exactly changes when you run terraform workspace select prod?

    Only which state file subsequent commands read and write; provider config and credentials stay the same.

  2. Why do the official docs warn against using workspaces for environment separation?

    Because they don't provide separate credentials or access controls, so one compromised credential can reach every workspace's resources.

  3. What's the minimum change needed to make dev and prod reachable only by their own credentials?

    Separate directories with their own backend config and assumed IAM role per environment, not just separate workspace names.

Prove it

Write two dicts, "workspace-based" and "directory-based," mapping environment name to credential profile. Make the workspace version share one profile and the directory version use distinct ones. Write a function reporting the blast radius (environments per shared credential) for each, and print both. Paste the output.

aws-08-05Card 5 of 6

Python CDK and CloudFormation basics

MediumP3 · Role-dependentTalk about it10 min
cdkcloudformationpythonbootstrapsynth

Why you care

CDK matters to you specifically because it lets a Python developer own infrastructure code without learning a template DSL first — the gap between "I write ETL scripts" and "I can't touch our Terraform." But teams that pick up CDK often skip the one step with no equivalent in plain CloudFormation or Terraform — bootstrapping — and burn a day debugging a failure that boils down to "this account was never prepared to run CDK." Interviewers who ask about CDK basics are checking whether you understand it's a code generator for CloudFormation, not a fourth, independent engine.

Explain like I'm 5

Writing a CDK app is like writing a function that builds a floor plan instead of drawing one by hand: an App is the whole blueprint set, a Stack is one building in it, and a Construct is a room or fixture, able to contain smaller constructs. cdk synth runs that function once, on paper — nothing gets built yet. cdk deploy hands the finished plan to the construction crew, CloudFormation, the same crew you'd use if you'd drawn the plan by hand.

Core concept

  • App → Stack → Construct is the composition hierarchy: an App is the root containing Stacks (each becomes one CloudFormation stack); each stack contains Constructs, low-level (one resource) or high-level (a bundled pattern). Constructs nest inside constructs.
  • cdk synth runs your Python app locally and produces a CloudFormation template — no AWS calls happen. This also makes CDK testable without touching an account.
  • cdk diff compares your synthesized template against what's actually deployed — the CDK equivalent of a Terraform plan or change set.
  • cdk bootstrap provisions what CDK itself needs in an account/region before any app can deploy there: an S3 asset bucket, an ECR repository, and deployment IAM roles, as its own CDKToolkit stack. Every account+region must be bootstrapped once before the first cdk deploy there.
  • cdk deploy hands the template to CloudFormation, which does the actual diffing and provisioning — CDK inherits every CloudFormation behaviour from Infrastructure as code.

Decision rule: use CDK when your team is comfortable in Python and wants real functions and tests around infrastructure; run cdk bootstrap once per account-region before the first deploy, and remember every cdk deploy is still a CloudFormation stack update underneath.

Under the hood

A construct's "shape" lives entirely in memory as a tree during synth: each construct registers itself with its parent at construction time (the scope argument every construct takes), and synth walks that tree once, asking each node to contribute its CloudFormation resource definitions into one output template per stack. Nothing about that walk talks to AWS. The bootstrap stack's template is maintained by the CDK team themselves and fetched by the CLI at bootstrap time, which is why cdk bootstrap output can change between CDK CLI versions — you're deploying whatever version of that template your current CLI knows about, not a fixed, one-time artefact. Because cdk deploy ultimately produces a CloudFormation template and asks CloudFormation to apply it, every CDK deploy pays CloudFormation's own change-set computation cost; "faster" in CDK means faster to author, not a different or faster underlying deploy engine.

Minimal example App, Stack, Construct — composed, then synthesized

PythonA construct tree is nested dicts until synth flattens itruns
tree = {"EtlStack": {
    "AuditBucket": {"Type": "AWS::S3::Bucket", "Versioned": True},
    "ScratchBucket": {"Type": "AWS::S3::Bucket", "Versioned": False},
}}


def synth(tree):
    return {"Resources": {rid: props for stack in tree.values() for rid, props in stack.items()}}


print(synth(tree))
Output
{'Resources': {'AuditBucket': {'Type': 'AWS::S3::Bucket', 'Versioned': True}, 'ScratchBucket': {'Type': 'AWS::S3::Bucket', 'Versioned': False}}}

In production Diff before deploy, and refuse an unbootstrapped environment

Pythoncdk diff logic, then a bootstrap gate before deployruns
BOOTSTRAPPED_ENVS = set()


def bootstrap(account, region):
    BOOTSTRAPPED_ENVS.add((account, region))


def deploy(template, account, region):
    if (account, region) not in BOOTSTRAPPED_ENVS:
        raise RuntimeError(f"environment {account}/{region} is not bootstrapped; run cdk bootstrap first")
    return f"deployed {len(template['Resources'])} resources"


def diff(old_template, new_template):
    changes = {}
    for logical_id, new_def in new_template["Resources"].items():
        old_def = old_template["Resources"].get(logical_id)
        if old_def is None:
            changes[logical_id] = "add"
        elif old_def != new_def:
            changes[logical_id] = "update"
    for logical_id in old_template["Resources"]:
        if logical_id not in new_template["Resources"]:
            changes[logical_id] = "remove"
    return changes


previous = {"Resources": {"AuditBucket": {"Type": "AWS::S3::Bucket", "Properties": {}}}}
current = {"Resources": {
    "AuditBucket": {"Type": "AWS::S3::Bucket",
                    "Properties": {"VersioningConfiguration": {"Status": "Enabled"}}},
    "ReportsFunction": {"Type": "AWS::Lambda::Function", "Properties": {}},
}}

print("cdk diff:", diff(previous, current))

try:
    deploy(current, "222222222222", "ap-south-1")
except RuntimeError as e:
    print("RuntimeError:", e)

bootstrap("222222222222", "ap-south-1")
print(deploy(current, "222222222222", "ap-south-1"))
Output
cdk diff: {'AuditBucket': 'update', 'ReportsFunction': 'add'}
RuntimeError: environment 222222222222/ap-south-1 is not bootstrapped; run cdk bootstrap first
deployed 2 resources
Scenario A

In your world

The situation

Excited to finally own infrastructure in your own language, you spin up a CDK Python app for a brand-new ETL project in a freshly created AWS account and run cdk deploy straight away, like running a script.

What actually breaks

The deploy fails immediately with an error about a missing S3 bucket, because the account was never bootstrapped — no CDKToolkit stack, no asset bucket, none of the roles CDK needs. It isn't a bug in your stack; the account simply isn't prepared yet.

What you'd do on Monday

Bootstrap each account/region once before the first deploy, then treat deploy as routine:

Shell
cdk bootstrap aws://222222222222/ap-south-1   # once per account+region
cdk synth                                      # local only, no AWS calls
cdk diff                                       # compare against what's actually deployed
cdk deploy                                     # hands the template to CloudFormation

Add a CI check that fails fast if the target environment's CDKToolkit stack doesn't exist, instead of letting the cryptic bootstrap error reach whoever is on call.

Scenario B

Imagine you work at Figma

300 CDK stack deploys a day across 25 teams

The numbers

Illustrative: 300 deploys a day across 25 teams ≈ 12 per team per day. If a 15 MB Lambda asset re-uploads on every deploy regardless of whether code changed, that's ≈4.5 GB a day (300 × 15 MB) of redundant S3 traffic and deploy time.

What the Scenario-A answer does here

The Scenario A fix still works at this volume, but "just deploy more often" ignores that un-cached asset uploads turn 300 small deploys into 300 full-size uploads, slowing every pipeline as traffic to the shared asset bucket grows.

What flips at this scale

At this volume, asset content-hashing must actually be relied on: unchanged code should skip the upload entirely, and cdk diff should gate whether a deploy even runs, so a no-op change doesn't trigger one.

The trade-off you'd defend

I would defend gating deploys behind cdk diff plus trusting content-addressed asset hashing over "always deploy on every merge," trading pipeline complexity for cutting redundant uploads at 300 deploys a day.

Illustrative figures for the arithmetic; not Figma-published numbers.

Gotchas

First deploy in a new account fails with a confusing bucket error

Symptom

cdk deploy fails before touching any of your resources, citing a missing S3 bucket or role you never defined.

Cause

The account/region was never bootstrapped; CDK's own supporting resources don't exist yet.

Fix

Run cdk bootstrap once per account/region, and add a CI pre-check that verifies the CDKToolkit stack exists.

"CDK has no state" is read as "CDK has no deploy coordination problem"

Symptom

Two engineers deploy the same stack minutes apart and hit a CloudFormation "stack is currently being updated" error.

Cause

CDK tracks no state, but the CloudFormation stack it deploys to serializes updates exactly like any other.

Fix

Treat cdk deploy with the same single-writer CI discipline as terraform apply.

Bootstrap behaviour differs across CDK CLI versions

Symptom

A bootstrap that worked fine for one project creates slightly different resources, or fails differently, on another project using a newer CDK CLI.

Cause

The bootstrap template is maintained and updated by the CDK team and fetched by whichever CLI version you're running; it isn't a fixed, one-time artefact pinned to your app's code.

Fix

Pin your team's CDK CLI version in CI and re-bootstrap deliberately (and test it) when you upgrade, rather than assuming bootstrap behaviour is frozen forever.

Tricky questions

Is cdk deploy a different provisioning engine than CloudFormation?
Most people say

Yes, CDK deploys directly to AWS resources using its own engine.

Actually

No. cdk deploy synthesizes a template and hands it to CloudFormation, which computes the change set and provisions resources exactly as for a hand-written template.

Does cdk synth need AWS credentials?
Most people say

Yes, it needs to check what's already deployed to build the template.

Actually

No. synth runs entirely offline; it builds the construct tree and serializes it with no AWS calls. Credentials are needed only from cdk diff onward.

Scenario question

You're handed a CDK Python app that deploys fine in the team's existing dev account, and asked to stand up the same stack in a brand-new sandbox account for a demo tomorrow. What's your first step?

Model answer

Before touching the app code, I would bootstrap the new account and region with cdk bootstrap — a fresh account has none of the supporting resources CDK needs, and working elsewhere says nothing about this account. I'd then run cdk diff to see the full stack as creates before running cdk deploy. The trade-off is a few extra minutes upfront, against discovering the same requirement from a cryptic failure during tomorrow's demo.

Interview corner

CommonWhat are the core building blocks of a CDK app?
“I'd describe App as the root, Stack as a unit mapping one-to-one to a CloudFormation stack, and Construct as anything inside a stack, from one resource to a whole reusable pattern.”
TrickIf CDK synthesizes to CloudFormation, why would I ever hit a CDK-specific error?
“I'd point to a missing bootstrap: that error has nothing to do with my stack's resources, it means the account was never prepared with what CDK needs to even start a deployment.”
Experience checkWhere have you seen CDK fail in a way that surprised someone new to it?
“I've seen a first deploy to a brand-new account fail with a bucket error that looks unrelated to the stack code. It traced back to forgetting cdk bootstrap.”
DesignWould you choose CDK over Terraform for a Python-heavy team?
“I'd lean CDK if we're staying inside AWS and already write Python, since it gets infrastructure real tests in the same language. I'd lean Terraform if we also manage non-AWS resources.”

Memory hook

A floor plan drawn by a function instead of by hand is still just a floor plan — the construction crew (CloudFormation) doesn't care who drew it.

If you remember only one thing:

cdk bootstrap once per account/region before the first deploy there, or the first cdk deploy fails on infrastructure CDK itself needs, not yours.

Self-check

  1. Does cdk synth make any AWS API calls?

    No — it produces a CloudFormation template entirely offline.

  2. What three kinds of resources does cdk bootstrap typically create?

    An S3 asset bucket, an ECR repository, and IAM roles CDK uses to perform deployments.

  3. Why can two concurrent cdk deploy runs against the same stack still conflict, even though CDK has no state file?

    Because CDK deploys to a CloudFormation stack, which serializes updates regardless of which tool triggered them.

Prove it

Extend the minimal example's tree dict with a third resource inside the same stack, and synth() again. Write a diff() call, like the production example's, comparing the two- and three-resource templates, and print which logical IDs were added. Paste the output.

aws-08-06Card 6 of 6

Secrets and recoverable IaC delivery

HardP3 · Role-dependentTalk about it11 min
secretsstatedriftrecoveryleast-privilege

Why you care

The gotcha every IaC team learns the hard way is that marking a value "sensitive" in your tool only hides it from the terminal — it does not stop that value from being written, in cleartext, into the state file and the saved plan that your backup, backend bucket and CI logs all have a copy of. Interviewers who ask "where could a secret leak in your pipeline" are testing whether you know the leak surface is bigger than the .tf file you remembered not to hardcode a password into.

Explain like I'm 5

Marking a field "sensitive" on a form is like asking the printer to blur one line on the printed copy you hand someone. The original digital file behind that printout still has the real text, unblurred, on the shared drive where other systems can open it directly. Blurring the printout protects whoever glances at it over your shoulder; it does nothing for the file itself.

Core concept

  • Never embed literal secrets in .tf/CloudFormation config. A hardcoded password or key in a resource argument lives in git history forever, independent of what happens to the file later.
  • State and plan files store sensitive values in cleartext, by design. Terraform's own documentation says plainly that it "stores values with the sensitive argument in both state and plan files, and anyone who can access those files can access your sensitive values" — marking a variable sensitive only suppresses it from CLI/log output, not from storage. Saved plan files (-out=tfplan) are explicitly called out as containing the full configuration and all planned values in cleartext.
  • Reference secrets externally instead of embedding them. Pull credentials from Secrets Manager or Parameter Store at apply/runtime via a data source or ARN reference, so only the reference, not the value, lands in state.
  • Mitigate the storage you can't avoid: store state remotely, encrypt the backend at rest, restrict who can read it, and enable audit logging. Terraform 1.10+ ephemeral values and 1.11+ write-only arguments exist specifically to keep certain values out of state and plan files entirely, where the provider supports them.
  • Deployment roles need their own least-privilege boundary. The identity your CI pipeline assumes should only have the permissions the config actually needs; a leaked state file plus an overly broad deploy role is a much bigger incident than a leaked state file alone.

Decision rule: treat anything that ever touched a .tf, state, or plan file as compromised the moment you discover it was a real secret, and rotate it — don't rely on deleting the file or rewriting history, since backups and CI logs likely still hold a copy.

Under the hood

A sensitive = true attribute in Terraform is purely a CLI/UI-layer redaction: the plan renderer and the state-display commands check that flag and print (sensitive value) instead of the real value, but the underlying JSON written to disk for both state and the saved plan file contains the actual value unredacted — the redaction is applied at render time, not at write time. This is also why terraform output -json or -raw on a sensitive output bypasses the redaction entirely: those flags are asking for the raw stored value, and the stored value was never redacted in the first place, only the default human-facing render was. CloudFormation's drift and change-set tooling has the analogous gap: a change set computed against a template with a plaintext default parameter value shows that value in the change set details unless you've used NoEcho on the parameter, and even NoEcho is a console/CLI display hint, not an at-rest encryption guarantee for the underlying stored parameter history.

Minimal example A password literal is detectable before it ever applies

PythonA literal secret is distinguishable from a safe ARN referenceruns
import re

lines = [
    'password = "Sup3rSecret!2026"',
    'DB_SECRET_ARN = "arn:aws:secretsmanager:ap-south-1:111111111111:secret:db-creds"',
]
pattern = re.compile(r'(password|secret)\s*=\s*"([^"]+)"', re.IGNORECASE)

for line in lines:
    match = pattern.search(line)
    if match and not match.group(2).startswith("arn:aws:secretsmanager"):
        print("embedded secret:", match.group(1))
Output
embedded secret: password

In production Scanning a saved plan, then redacting it for audit

PythonCleartext-in-plan detection, then redaction for a safe audit copyruns
import json
from pathlib import Path

PLAN_JSON = {
    "resource_changes": [
        {"address": "aws_db_instance.settlements", "change": {
            "after": {"username": "etl_admin", "password": "Sup3rSecret!2026"}}},
    ]
}
Path("tfplan.json").write_text(json.dumps(PLAN_JSON))


def scan_plan_for_secrets(path):
    data = json.loads(path.read_text())
    found = []
    for change in data["resource_changes"]:
        after = change["change"].get("after", {})
        for key in after:
            if "password" in key or "secret" in key:
                found.append((change["address"], key))
    return found


print("cleartext secrets inside the saved plan file:", scan_plan_for_secrets(Path("tfplan.json")))


def redact(path):
    data = json.loads(path.read_text())
    for change in data["resource_changes"]:
        after = change["change"].get("after", {})
        for key in list(after):
            if "password" in key or "secret" in key:
                after[key] = "***redacted***"
    return data


audit_copy = redact(Path("tfplan.json"))
print("audit-safe copy:", audit_copy["resource_changes"][0]["change"]["after"])
Output
cleartext secrets inside the saved plan file: [('aws_db_instance.settlements', 'password')]
audit-safe copy: {'username': 'etl_admin', 'password': '***redacted***'}

See it happen Where one secret can leak, long after the PR looked clean

planapplyconsole outputmain.tf(literal password)git historysaved plan filestate file(backend bucket)CI apply log
  1. A literal password is typed into main.tf. It is already compromised the moment it's committed.
  2. git history keeps the literal value forever, even if a later commit removes the line.
  3. terraform plan writes the full value into the saved plan file, in cleartext, by design.
  4. terraform apply carries that same value into the state file in the backend bucket.
  5. The CI job's console log can also show the plan's rendered output, a fifth copy of the same secret.
Notice: marking the variable sensitive would only have changed what the CLI prints at the bottom-right node — every other copy in this diagram would still hold the real value.
Scenario A

In your world

The situation

Under deadline pressure, you hardcode password = "TempPass123!" directly into an RDS resource block "just to get the demo working today," planning to move it to Secrets Manager "right after."

What actually breaks

The PR merges, Jenkins runs terraform apply, and that literal password now exists in three places: git history, the state file in the S3 backend bucket, and the Jenkins console log — none cleaned up by a refactor that only edits the current file.

What you'd do on Monday

Reference the secret, never embed it, and treat anything already committed as compromised:

Terraform
data "aws_secretsmanager_secret_version" "db" {
  secret_id = "prod/settlements/db-creds"
}

resource "aws_db_instance" "settlements" {
  username = "etl_admin"
  password = data.aws_secretsmanager_secret_version.db.secret_string
}

Then rotate the password that was ever hardcoded, independent of fixing the config — rewriting history doesn't undo that a real secret already existed in those files.

Scenario B

Imagine you work at Notion

1,200 secrets across environments

The numbers

Illustrative: ≈1,200 secrets referenced by ARN from ≈3 configs each ≈ 3,600 reference points company-wide. If even 5% (60) were ever hardcoded before being moved to Secrets Manager, that's 60 plaintext values sitting in state, plans and git history.

What the Scenario-A answer does here

The Scenario A fix still works per secret, but "find and rotate the 60" assumes you know which 60 — at this count, nobody remembers which secrets were ever literal, so a manual audit misses some.

What flips at this scale

At this scale, secret hygiene must be automatic: a mandatory pre-apply scan blocks any literal pattern from reaching a plan or state file again, and "not sure if hardcoded" becomes "assume it was and rotate," since checking costs more than rotating.

The trade-off you'd defend

I would defend a hard CI gate plus blanket rotation of unclear history over a one-time audit, trading some unnecessary rotations for not missing any of the 60 that genuinely weren't clean.

Illustrative figures for the arithmetic; not Notion-published numbers.

Gotchas

"Sensitive" marking is mistaken for actual protection

Symptom

A variable marked sensitive still shows up in full, in cleartext, in the state file and in terraform output -json.

Cause

The sensitive flag only redacts the default human-facing CLI/plan render; it never changes what gets written to storage.

Fix

Treat state and plan files as sensitive regardless of any sensitive flags inside them — encrypt the backend, restrict access, never attach a plan file to a public ticket.

Leaked state or plan artifacts reveal secrets even when console output looks redacted

Symptom

A support ticket includes a terraform plan screenshot that looks clean, but the attached raw plan JSON file (shared "for completeness") contains the real values.

Cause

Redaction happens in the human-facing renderer, not in the underlying file; sharing the file bypasses whatever the terminal showed.

Fix

Never attach raw state or plan files to tickets or chats; share only the redacted renderer output, or a deliberately sanitized copy.

Tricky questions

Does marking a Terraform variable sensitive keep it out of the state file?
Most people say

Yes, that's what the sensitive flag is for.

Actually

No. It only suppresses the value from the default CLI/plan rendering; the real value is still written to state and any saved plan in full, and -json/-raw output shows it unredacted.

If a saved plan file was deleted right after apply, is the secret inside it gone?
Most people say

Yes, deleting the file removes that copy of the secret.

Actually

That copy, yes — but the apply already carried the same value into the state file, and the plan may still exist in CI artefact storage or backups taken before deletion.

Scenario question

You're handed a ticket: "we think a database password was briefly hardcoded in Terraform a few months ago, not sure if it's still an issue." How do you respond?

Model answer

I would not try to confirm or rule it out from the current .tf files, since the question is about history, not the repo's present state. I'd assume the worst — that the value reached a state file, plan, or CI log at some point — and rotate that credential immediately, since rotation is cheap and removes the risk regardless. In parallel, I'd add a pre-apply secret scan so this stops being a "not sure" question going forward. The trade-off is some unnecessary rotation if it turns out the secret was never exposed, which I'd accept over leaving a possibly-leaked credential active.

Interview corner

CommonWhere can a secret leak in a Terraform pipeline besides the config file?
“I'd point to the state file and any saved plan file first, since both store sensitive values in cleartext by design. CI console logs that print plan output are a third place people forget about.”
TrickDoes marking a variable sensitive solve the leak risk?
“I'd push back — sensitive only redacts what's printed to the terminal or in output without -json/-raw. The underlying file on disk still has the real value.”
Experience checkHave you had to respond to a possibly-leaked credential?
“I've treated an uncertain case as a real one: when it wasn't clear if a secret had ever been hardcoded, I rotated it anyway, since rotation is cheap and proving a negative isn't.”
DesignHow would you prevent secrets from ever reaching a plan or state file?
“I'd put a scanning gate before any plan or apply runs, reference secrets by ARN instead of embedding them, and restrict and encrypt the state backend, since some sensitive data in state is often unavoidable.”

Memory hook

Blurring the printout doesn't touch the original file sitting unblurred on the shared drive.

If you remember only one thing:

State and plan files store secrets in cleartext regardless of any sensitive flag — protect the files, don't trust the flag.

Self-check

  1. Does Terraform's sensitive = true prevent a value from being written to the state file?

    No. It only redacts the default CLI/plan display; the state and saved plan still contain the real value.

  2. Why doesn't deleting a plan file after apply fully contain a leaked secret?

    Because apply already carried the same value into the state file, and the plan may persist in CI artefact storage or backups made before deletion.

  3. What's the right response to "we're not sure if this secret was ever hardcoded"?

    Assume it was and rotate it; confirming history usually costs more than rotating.

Prove it

Run the production example's scan_plan_for_secrets against three toy plan structures: one with a literal password, one with only an ARN reference, one with both a literal API key and an ARN reference. Print the findings for each, then redact() the one with both. Paste all outputs.