Cost engineering
Why you care
Every AWS resource you provision has at least one of four pricing shapes attached to it, and picking the wrong one is invisible until the invoice arrives. A nightly job that runs for two hours on an On-Demand instance left running 24/7 costs twelve times what it needs to, with a green Jenkins build every single day. Interviewers ask "walk me through how you'd estimate the monthly cost of this design" not to test memorised prices, but to see whether you reason in unit cost — cost per unit of work done — instead of quoting a total that means nothing without a denominator. This is also the card that connects your lab budget discipline from Budgets and controlled lab spend to running cost like production.
Explain like I'm 5
Think of four ways to pay for a cab. On-Demand is the metered taxi: you pay for exactly the ride you take, at the highest per-kilometre rate, and you can stop anytime. Spot is a shared ride-pool car: much cheaper, but the driver can bump you out mid-ride if a better-paying passenger shows up, and they only give you two minutes' notice. A Savings Plan or Reserved Instance is a monthly cab pass: you pre-pay for a committed amount of travel at a discount, and if you don't use it that month, the money is still spent. None of the four is "the cheap one" in general — the cheap one depends on how predictable and how interruptible your trip is.
Core concept
Four pricing shapes cover almost everything you'll provision:
- On-Demand: pay per second or per hour (service-dependent), no commitment, highest unit rate, instantly available, instantly cancellable. The right default for anything you haven't measured yet.
- Spot: a steep discount off On-Demand for spare EC2 capacity, in exchange for the possibility of interruption. AWS gives a two-minute Spot Instance interruption notice before stop, terminate, or (for most behaviours) action, delivered as an EventBridge event and in instance metadata, on a best-effort basis — not a guarantee. Right for retryable, checkpointable, stateless batch work.
- Savings Plans: you commit to spend a fixed dollar amount per hour, for a 1- or 3-year term, in exchange for a discount on matching usage. Unused commitment is not refunded — you pay for it whether you use it or not. Savings Plans, RIs and right-sizing owns the Savings-Plans-versus-RI depth.
- Reserved Instances (RIs): similar commitment shape, but reserved against a specific instance family/size (Standard) or exchangeable within limits (Convertible), scoped to a region or a specific AZ. Also non-refundable if unused.
Workload sizing comes before any of this: measure actual CPU/memory utilization over a real period (not a guess, not the peak), and size the instance or the commitment to that measured baseline, never to "whatever felt safe." A workload sized to peak and then run On-Demand around the clock is the single most common invisible overspend in ETL shops.
Unit cost tells you whether a design got cheaper or more expensive: cost ÷ successful output, for example dollars per 1,000 rows loaded, or dollars per successful pipeline run (Cost per successful pipeline run goes deep on this). A bill that rises while unit cost falls is a good month; a flat bill with rising unit cost means you're doing less for the same money.
Decision rule: use On-Demand for anything new, bursty, or unmeasured; move to Spot once you've proven the workload is interruption-tolerant and checkpointed; move to a Savings Plan or RI only after 30-60 days of measured, steady utilization you're confident will continue — never commit to guess at a baseline. Budget alerts from Budgets and controlled lab spend are the safety net under all four, not a substitute for choosing correctly.
Under the hood
AWS bills most compute per second (with a historical one-minute minimum on some instance types — confirm current billing granularity before relying on it), Lambda per millisecond rounded up, and S3 storage per GB prorated hourly. None of that detail reaches you directly: the Cost and Usage Report (CUR) aggregates it into hourly, daily or monthly line items in an S3 bucket you own, broken down by product, usage type, operation and your own tags. Every CUR update before the billing period closes is an estimate — AWS states the report is cumulative and subject to change until finalized at month end, and a savingsPlan/ column only appears if you used a Savings Plan that month. FinOps talk track and feedback loop builds on this: never report a mid-month number as if it were the invoice.
Minimal example Unit cost makes a bigger bill mean something
def unit_cost(total_cost: float, successful_units: int) -> float:
return total_cost / successful_units if successful_units else float("inf")
month_a = unit_cost(482.00, 1_200_000) # rows loaded successfully
month_b = unit_cost(610.00, 2_050_000)
print(f"month A: ${month_a * 1000:.3f} per 1,000 rows")
print(f"month B: ${month_b * 1000:.3f} per 1,000 rows")
bill_change = (610.00 - 482.00) / 482.00 * 100
print(f"bill grew {bill_change:.0f}%, unit cost {'fell' if month_b < month_a else 'rose'}")month A: $0.402 per 1,000 rows
month B: $0.298 per 1,000 rows
bill grew 27%, unit cost fellIn production Pick a pricing model from measured workload traits
A classifier that mirrors the decision rule: no guessing about interruption tolerance or steady demand, only measured inputs.
from dataclasses import dataclass
@dataclass
class Workload:
name: str
hours_per_day: float
days_observed: int # how long you've actually measured this
interruption_tolerant: bool # checkpointed, retryable, no live user waiting
utilization_pct: float # measured average, not peak
def recommend(w: Workload) -> str:
if w.days_observed < 30:
return f"{w.name}: On-Demand (only {w.days_observed}d observed, too early to commit)"
if w.interruption_tolerant and w.utilization_pct < 70:
return f"{w.name}: Spot (interruption-tolerant, {w.utilization_pct:.0f}% utilized, big discount available)"
if w.utilization_pct >= 70:
return f"{w.name}: Savings Plan / RI ({w.days_observed}d steady at {w.utilization_pct:.0f}%)"
return f"{w.name}: On-Demand (steady but under-utilized, right-size before committing)"
fleet = [
Workload("nightly-settlement-etl", hours_per_day=2.0, days_observed=45, interruption_tolerant=True, utilization_pct=35.0),
Workload("api-ingest-worker", hours_per_day=24.0, days_observed=120, interruption_tolerant=False, utilization_pct=78.0),
Workload("new-reconciliation-job", hours_per_day=4.0, days_observed=9, interruption_tolerant=True, utilization_pct=50.0),
Workload("report-batch-runner", hours_per_day=24.0, days_observed=90, interruption_tolerant=False, utilization_pct=42.0),
]
for w in fleet:
print(recommend(w))nightly-settlement-etl: Spot (interruption-tolerant, 35% utilized, big discount available)
api-ingest-worker: Savings Plan / RI (120d steady at 78%)
new-reconciliation-job: On-Demand (only 9d observed, too early to commit)
report-batch-runner: On-Demand (steady but under-utilized, right-size before committing)See it happen The decision rule as a flow
- Every workload starts On-Demand. There is no shortcut past this.
- You run it for real for 30-60 days and measure utilization, not peak.
- If it's interruption-tolerant and under-utilized, Spot gives the discount without a commitment.
- If it's steady and genuinely busy, a Savings Plan or RI pays off because you will use the commitment.
- Skipping straight from new workload to a 1- or 3-year commitment is the mistake this whole chapter is about.
In your world
The situation
You inherit a nightly settlements job that runs for about two hours on a c5.xlarge EC2 instance. The instance was provisioned On-Demand "to get it working" eighteen months ago and nobody ever revisited it. It has been running 24 hours a day since.
What actually breaks
The job needs the instance for roughly 60 hours a month. On-Demand, running continuously, bills for all 720 hours in the month. You are paying for twelve times the compute the job actually uses, every month, with no error anywhere — the job succeeds every night, so nothing ever flags it.
What you'd do on Monday
Measure the real run window from CloudWatch, then stop paying for idle hours instead of picking a cheaper instance type for the same 24/7 pattern:
import boto3
ec2 = boto3.client("ec2", region_name="ap-south-1")
def stop_if_idle(instance_id: str) -> None:
"""Call from an EventBridge Scheduler rule right after the nightly job's
own completion marker, not on a fixed clock guess."""
ec2.stop_instances(InstanceIds=[instance_id])
def start_before_window(instance_id: str) -> None:
"""Call a few minutes before the job's known start time."""
ec2.start_instances(InstanceIds=[instance_id])Two hours billed for two hours is a ~90% cost reduction with zero change to the job.
Imagine you work at Postman
The numbers
Illustrative: 2,000,000 collection runs/day, each executing on a 1-vCPU container task for an average of 20 seconds. That's 2,000,000 × 20s = 40,000,000 vCPU-seconds/day ÷ 3,600 ≈ 11,100 vCPU-hours/day. At an illustrative On-Demand container rate (check the current pricing page) of roughly $0.04/vCPU-hour, that's about $444/day, or roughly $13,300/month, for compute alone before memory charges.
What the Scenario-A answer does here
The Scenario A fix — stop paying for idle time — doesn't apply here, because these tasks are already ephemeral; they start and stop per run. What dominates at this volume is the billing floor: if the platform's minimum billable task duration is, say, 60 seconds (confirm the current minimum before relying on it) and the average run is 20 seconds, every run is billed for roughly 3× the compute it actually used.
What flips at this scale
At 2M runs/day you stop picking a pricing model per task and instead batch many short runs onto a smaller pool of longer-lived workers covered by a Compute Savings Plan, so the per-invocation floor stops being multiplied by two million. The metric you track changes from "$/instance-hour" to "$/1,000 runs," because that's the number that actually reflects whether batching helped.
The trade-off you'd defend
I would defend batching bursty work onto a shared, committed worker pool over paying the per-invocation billing floor at scale, trading a few seconds of queueing latency for a materially lower unit cost, with a small On-Demand burst pool for spikes the committed pool can't absorb.
Illustrative figures for the arithmetic; not Postman-published. Confirm current container billing granularity and minimums on the AWS pricing pages before using them in a real estimate.
Gotchas
A budget alert firing does not mean spending stopped
You get the budget notification email, acknowledge it, and the bill keeps growing for the rest of the month exactly as before.
AWS Budgets alerts are notifications, not circuit breakers, unless you've explicitly configured a Budget Action (like applying an IAM policy or stopping instances); by default nothing is throttled.
Treat the alert as a prompt to act, and if you need an automatic stop, wire a real Budget Action or your own Lambda response to the alert, as covered in Budgets and controlled lab spend.
24/7 On-Demand for a part-time job
A nightly or weekly job's compute bill is flat every single day of the month, including weekends when nothing runs.
The instance was never stopped between runs; On-Demand bills for every hour it exists, used or not.
Measure the actual run window and stop/start on a schedule, or move the workload to something that only bills while executing (Lambda, Fargate scheduled tasks, Batch).
Comparing total bills instead of unit cost
"Our AWS bill went up 30%" gets treated as bad news with no context.
Total cost ignores growth in work done. A 30% cost increase alongside a 60% rise in rows processed is an improvement, not a regression.
Pair every cost delta with a work-done delta, and report unit cost alongside the total.
Tricky questions
Yes, Spot discounts are usually the deepest, so it's always cheapest.
Spot is usually cheaper per hour, but comes with interruption risk and no SLA. For a workload that can't tolerate interruption, Spot's rate is irrelevant — fit the workload's tolerance first, then compare prices only within models that qualify.
No — unused commitment just means a smaller discount, not extra cost.
You're charged the committed hourly amount regardless of usage. Under-utilizing a Savings Plan or RI means paying for capacity you aren't consuming, strictly worse than On-Demand for that unused slice.
Scenario question
You're handed a monthly AWS invoice that grew from $8,200 to $11,400 and asked "why did cost go up 39%, fix it." Before touching anything, what do you ask for and what do you report back?
Model answer
I'd ask for the CUR broken down by service and cost-allocation tag for both months, plus the work-done metric for the main workloads — rows processed, pipeline runs, requests served. A total-dollar comparison alone tells me nothing. If rows processed also grew 39%, unit cost is flat and this is growth, not an incident, and I'd report that explicitly. If output only grew 10%, I'd break the delta down by service to find which line item moved — looking for an idle resource, a leak from Five recurring cost leaks, or a pricing-model mismatch like 24/7 On-Demand for part-time work. The trade-off: proper investigation takes a day or two against tag-complete billing data; guessing from the total risks "fixing" something that wasn't the cause.
Interview corner
CommonWalk me through the four main AWS pricing models.
TrickOur bill doubled this month — is that bad?
Experience checkTell me about a time you reduced AWS cost without changing functionality.
DesignHow would you decide whether to commit to a Savings Plan for a new workload?
Memory hook
Four ways to pay for a cab: metered, shared-and-interruptible, prepaid pass. Measure before you buy the pass.
Never commit to a Savings Plan or RI before you've measured 30+ days of real, steady utilization.
Self-check
What's the difference between Spot's discount and a Savings Plan's discount, in terms of what you give up for each?
Spot trades a discount for interruption risk (best-effort, two-minute notice); a Savings Plan trades a discount for a non-refundable spend commitment over 1 or 3 years, regardless of interruption.
Why is "the bill went up 30%" not enough information to act on?
Cost without a work-done denominator (unit cost) can't tell you whether the business got more expensive to run or just did more work for proportionally more money.
Does an AWS Budgets alert stop further spending by default?
No. It's a notification unless you've explicitly attached a Budget Action; without one, spending continues exactly as before.
Take the recommend() function from the production example. Add a fifth workload with days_observed=15, interruption_tolerant=False, utilization_pct=85. Run it through recommend() and explain in a comment why it still returns On-Demand despite the high utilization. Paste the function's output for all five workloads.