Infrastructure as code
Why you care
Every AWS job above entry level expects you to provision infrastructure through code, not the console. An interviewer asking "CloudFormation or Terraform?" is testing whether you understand desired state: a file says what should exist, a tool reconciles reality to match it, and the diff is reviewable before anything changes. A Jenkins job running raw aws CLI commands has no diff, no state, and no way to answer "what will this change?" Get this wrong and you'll describe IaC as "a script that creates AWS stuff" — someone who has never had a console fix silently overwritten by a redeploy.
- Note: CDK/CloudFormation detail below is marked spec rather than cited; verify exact flags and syntax before using them against a real account.
Explain like I'm 5
A tenant hands a landlord a signed floor plan instead of describing changes verbally. The plan is your config: wall here, door there. The landlord (the IaC tool) compares plan to building and does only the work needed: adds the missing wall, leaves the correct door, removes the shed that isn't on the plan. If a past tenant knocked a hole in a wall and told nobody, the next inspection finds the mismatch — that's drift.
Core concept
- Desired state is the config you write: a CloudFormation template, Terraform HCL, or a CDK app that compiles to a template. All describe an end state, never steps.
- State ownership is who remembers what exists. CloudFormation owns it per stack, inside the service; Terraform owns it in a state file you store yourself (next card); CDK has none of its own — it synthesizes to CloudFormation and lets CloudFormation own the state.
- The reviewable diff is a CloudFormation change set or a Terraform plan: create/update/delete, shown before anything changes.
cdk diffcompares the same way. - Drift is any gap between state and reality — a console edit, a manual call, another tool. Undetected drift, not drift itself, is what causes outages.
Decision rule: CloudFormation when AWS should own state and YAML/JSON suits you; Terraform when one state model must span non-AWS resources too; CDK when your team already codes in Python/TypeScript/Java and wants real functions and tests around infrastructure, with CloudFormation still provisioning. All three fail the same way without a reviewed diff.
Under the hood
CloudFormation computes a change set against the stack's last recorded state, and you must explicitly execute it — nothing auto-applies. Terraform's plan refreshes first: it reads every resource's real value from the AWS API before diffing against your .tf config, which is why a plan can show a change you never made. CDK's synth runs entirely offline, building a construct tree from your Python code and serializing it to a CloudFormation template; AWS calls happen only once CloudFormation later diffs and applies that template.
Minimal example Desired state vs actual state, computed as a diff
desired = {"web_sg": 443, "web_server": "t3.micro"}
actual = {"web_sg": 22, "orphan_eip": "vpc"}
for name in desired | actual:
if name not in actual:
print(name, "create")
elif name not in desired:
print(name, "unmanaged: in account, not in config")
elif desired[name] != actual[name]:
print(name, "update (drift from config)")web_sg update (drift from config)
web_server create
orphan_eip unmanaged: in account, not in configIn production Rendering a change set and checking for replacements
A toy change-set renderer using CloudFormation's own symbols, plus a drift check on one SQS queue property — the same shape as describe-stack-resource-drifts.
import json
from pathlib import Path
CHANGE_SET = [
{"logical_id": "OrdersQueue", "type": "AWS::SQS::Queue", "action": "Modify",
"replacement": False, "changed": {"VisibilityTimeout": [30, 120]}},
{"logical_id": "SettlementsTable", "type": "AWS::DynamoDB::Table", "action": "Modify",
"replacement": True, "changed": {"KeySchema": ["id", "settlement_date"]}},
{"logical_id": "AuditBucket", "type": "AWS::S3::Bucket", "action": "Remove", "replacement": False},
{"logical_id": "ReportsFunction", "type": "AWS::Lambda::Function", "action": "Add", "replacement": False},
]
SYMBOL = {"Add": "+", "Remove": "-", "Modify": "~"}
def render_change_set(changes):
lines = []
for c in changes:
symbol = "-/+" if c["action"] == "Modify" and c["replacement"] else SYMBOL[c["action"]]
lines.append(f"{symbol} {c['logical_id']} ({c['type']})")
if c["replacement"]:
lines.append(" ! replacement: a new resource is created, the old one is destroyed")
return lines
for line in render_change_set(CHANGE_SET):
print(line)
replacements = [c["logical_id"] for c in CHANGE_SET if c.get("replacement")]
print("review required before execute:", replacements)
Path("expected.json").write_text(json.dumps({"VisibilityTimeout": 120, "MaximumMessageSize": 262144}))
Path("actual.json").write_text(json.dumps({"VisibilityTimeout": 30, "MaximumMessageSize": 262144}))
expected = json.loads(Path("expected.json").read_text())
actual = json.loads(Path("actual.json").read_text())
drift = {k: (expected[k], actual[k]) for k in expected if expected[k] != actual.get(k)}
print("drifted properties:", drift)~ OrdersQueue (AWS::SQS::Queue)
-/+ SettlementsTable (AWS::DynamoDB::Table)
! replacement: a new resource is created, the old one is destroyed
- AuditBucket (AWS::S3::Bucket)
+ ReportsFunction (AWS::Lambda::Function)
review required before execute: ['SettlementsTable']
drifted properties: {'VisibilityTimeout': (120, 30)}See it happen One reconcile cycle, three tools, one shape
- Nothing has run yet. config, state and real resources are three separate things.
- A refresh reads real AWS resources into the tool's view of state before any comparison.
- The diff compares config against the refreshed state, not against the real resources directly.
- The diff is the change set or plan: create/update/delete/replace, shown before anything changes.
- Only after review does apply push changes to real resources, which then feeds the next refresh.
In your world
The situation
Your team's "IaC" is a Jenkins job running a bash script of aws s3api/aws rds CLI commands recorded eighteen months ago. No file says what infrastructure should look like today, only a script of what to run.
What actually breaks
At 2 AM an on-call engineer bumps the RDS instance class in the console to relieve CPU pressure. Three days later Jenkins runs for an unrelated change, re-executes the old aws rds modify-db-instance --db-instance-class db.t3.medium, and silently reverts the fix — the script never knew the console change happened.
What you'd do on Monday
Replace the script with a declarative config so the next run diffs against reality:
resource "aws_db_instance" "settlements" {
identifier = "settlements-pg-01"
instance_class = "db.t3.large" # the incident fix, now written down
engine = "postgres"
}terraform plan -out=tfplan # shows "update in place: instance_class" if config still says t3.medium
terraform show tfplan # review before anyone runs applyThe 2 AM fix now has to be reflected here, or the next plan shows it as a pending revert — in review, not silently in production.
Imagine you work at Postman
The numbers
Illustrative: 400 stacks across 12 teams, ≈33 per team. One undocumented tweak per stack a quarter is ≈400 untracked changes company-wide; at 20 minutes to diff one by hand, reconciling all of them costs ≈133 hours a quarter (400 × 20 ÷ 60).
What the Scenario-A answer does here
The Scenario A fix still works per stack, but trusting every engineer to run plan locally doesn't scale to 400: a stale, unreviewed plan looks identical to a clean one until apply runs.
What flips at this scale
At this count, drift detection must be scheduled and centralised: a nightly job plans all 400 stacks and tickets any change nobody authored, instead of waiting for the next deploy to "fix" an incident patch by accident.
The trade-off you'd defend
I would defend centralised drift detection with a required-plan CI gate over trusting local plan runs, trading CI minutes for an audit trail across all 400 stacks, not just the ones someone remembered to check.
Illustrative numbers for the arithmetic; not Postman-published figures.
CloudFormation vs Terraform vs CDK
| Axis | CloudFormation | Terraform | CDK (Python) |
|---|---|---|---|
| State ownership | AWS service, per stack | A file/backend you own | None; delegates to CloudFormation |
| Scope | AWS only | AWS plus other providers | AWS only (synthesizes to CloudFormation) |
| Review artefact | Change set | Plan | cdk diff |
| Authoring | YAML/JSON | HCL | General-purpose code |
| Logic | Minimal, declarative | Minimal, HCL expressions | A lot; a real language |
Gotchas
A console fix gets silently reverted
A manual change that fixed an incident disappears after the next unrelated deploy, with no error logged.
The config never captured the manual change, so the diff treats the old value as desired and reapplies it.
Treat every manual production change as a same-day config update, even a one-line bump.
"No changes" is trusted without ever running a diff
A team believes their infra is drift-free because nobody has reported a problem.
Nobody schedules drift detection; a diff only runs right before a planned apply.
Run a standalone drift-detection job on a schedule, independent of planned deploys, and alert on unexpected changes.
Tricky questions
Yes, CDK is a newer alternative that avoids CloudFormation's limitations.
No. CDK generates a CloudFormation template; cdk deploy hands it to CloudFormation, which still owns the state and computes the change set.
Yes, if the plan is clean, apply does exactly that.
No. The plan is computed against a snapshot; a change between plan and apply, or a provider-side failure, can still change the outcome.
Scenario question
You're handed a repo with both a CloudFormation template and a Terraform config that both claim to manage the same S3 bucket, left over from a half-finished migration. What do you do before touching either?
Model answer
I would not run apply or deploy either tool blindly — two state owners for one resource means whichever runs last wins and fights the other. I'd check which tool's state tracks the bucket, pick one owner, and remove it from the other's state (terraform state rm, or dropping it from the template without DeletionPolicy: Delete so the bucket survives) so only one tool reconciles it going forward. The trade-off: the retired tool's next plan shows a false "create," which I'd flag in the PR.
Interview corner
CommonWhat's the difference between CloudFormation, Terraform and CDK?
TrickIf a CDK app has no state file, does that mean no locking issues?
cdk deploy runs can still collide.”Experience checkHave you seen a manual change get reverted by automation?
DesignHow would you catch drift before it causes an incident?
Memory hook
A floor plan and a landlord who only does the work the plan still shows is missing.
The diff, not the apply, is the part of IaC that actually protects you — never skip reviewing it.
Self-check
What does CDK's
synthstep talk to over the network?Nothing — it writes a CloudFormation template locally; AWS calls happen only during the later diff/deploy.
Why can a Terraform plan show a change you didn't make?
planrefreshes state from real AWS resources before diffing, so it can surface someone else's console or CLI change.Is drift itself a failure?
No — only undetected drift is; reviewed drift is the system working as intended.
Write two dicts, desired and actual, covering three resources: one changed, one missing, one extra. Extend the minimal example's loop to also flag a replacement when a chosen property (say, a database engine) changes, and print which resources need a -/+ warning. Paste the output.