Shelf · AWS · Chapter 11

Chapter 11: Cost engineering

Writing a job that works and writing a job that is cheap to run are different skills, and the second one is what turns an ETL developer into someone a product company trusts with infrastructure spend. This chapter builds the vocabulary, the arithmetic and the talk track for owning AWS cost like an engineer, not like someone who only reads the invoice.

Difficulty
Priority
aws-11-01Card 1 of 6

Cost engineering

HardP4 · DifferentiatorTalk about it13 min
cost-engineeringpricing-modelsunit-coston-demandsavings-plansspot

Why you care

Every AWS resource you provision has at least one of four pricing shapes attached to it, and picking the wrong one is invisible until the invoice arrives. A nightly job that runs for two hours on an On-Demand instance left running 24/7 costs twelve times what it needs to, with a green Jenkins build every single day. Interviewers ask "walk me through how you'd estimate the monthly cost of this design" not to test memorised prices, but to see whether you reason in unit cost — cost per unit of work done — instead of quoting a total that means nothing without a denominator. This is also the card that connects your lab budget discipline from Budgets and controlled lab spend to running cost like production.

Explain like I'm 5

Think of four ways to pay for a cab. On-Demand is the metered taxi: you pay for exactly the ride you take, at the highest per-kilometre rate, and you can stop anytime. Spot is a shared ride-pool car: much cheaper, but the driver can bump you out mid-ride if a better-paying passenger shows up, and they only give you two minutes' notice. A Savings Plan or Reserved Instance is a monthly cab pass: you pre-pay for a committed amount of travel at a discount, and if you don't use it that month, the money is still spent. None of the four is "the cheap one" in general — the cheap one depends on how predictable and how interruptible your trip is.

Core concept

Four pricing shapes cover almost everything you'll provision:

  • On-Demand: pay per second or per hour (service-dependent), no commitment, highest unit rate, instantly available, instantly cancellable. The right default for anything you haven't measured yet.
  • Spot: a steep discount off On-Demand for spare EC2 capacity, in exchange for the possibility of interruption. AWS gives a two-minute Spot Instance interruption notice before stop, terminate, or (for most behaviours) action, delivered as an EventBridge event and in instance metadata, on a best-effort basis — not a guarantee. Right for retryable, checkpointable, stateless batch work.
  • Savings Plans: you commit to spend a fixed dollar amount per hour, for a 1- or 3-year term, in exchange for a discount on matching usage. Unused commitment is not refunded — you pay for it whether you use it or not. Savings Plans, RIs and right-sizing owns the Savings-Plans-versus-RI depth.
  • Reserved Instances (RIs): similar commitment shape, but reserved against a specific instance family/size (Standard) or exchangeable within limits (Convertible), scoped to a region or a specific AZ. Also non-refundable if unused.

Workload sizing comes before any of this: measure actual CPU/memory utilization over a real period (not a guess, not the peak), and size the instance or the commitment to that measured baseline, never to "whatever felt safe." A workload sized to peak and then run On-Demand around the clock is the single most common invisible overspend in ETL shops.

Unit cost tells you whether a design got cheaper or more expensive: cost ÷ successful output, for example dollars per 1,000 rows loaded, or dollars per successful pipeline run (Cost per successful pipeline run goes deep on this). A bill that rises while unit cost falls is a good month; a flat bill with rising unit cost means you're doing less for the same money.

Decision rule: use On-Demand for anything new, bursty, or unmeasured; move to Spot once you've proven the workload is interruption-tolerant and checkpointed; move to a Savings Plan or RI only after 30-60 days of measured, steady utilization you're confident will continue — never commit to guess at a baseline. Budget alerts from Budgets and controlled lab spend are the safety net under all four, not a substitute for choosing correctly.

Under the hood

AWS bills most compute per second (with a historical one-minute minimum on some instance types — confirm current billing granularity before relying on it), Lambda per millisecond rounded up, and S3 storage per GB prorated hourly. None of that detail reaches you directly: the Cost and Usage Report (CUR) aggregates it into hourly, daily or monthly line items in an S3 bucket you own, broken down by product, usage type, operation and your own tags. Every CUR update before the billing period closes is an estimate — AWS states the report is cumulative and subject to change until finalized at month end, and a savingsPlan/ column only appears if you used a Savings Plan that month. FinOps talk track and feedback loop builds on this: never report a mid-month number as if it were the invoice.

Minimal example Unit cost makes a bigger bill mean something

PythonUnit cost: dollars are meaningless without a denominatorruns
def unit_cost(total_cost: float, successful_units: int) -> float:
    return total_cost / successful_units if successful_units else float("inf")


month_a = unit_cost(482.00, 1_200_000)   # rows loaded successfully
month_b = unit_cost(610.00, 2_050_000)

print(f"month A: ${month_a * 1000:.3f} per 1,000 rows")
print(f"month B: ${month_b * 1000:.3f} per 1,000 rows")

bill_change = (610.00 - 482.00) / 482.00 * 100
print(f"bill grew {bill_change:.0f}%, unit cost {'fell' if month_b < month_a else 'rose'}")
Output
month A: $0.402 per 1,000 rows
month B: $0.298 per 1,000 rows
bill grew 27%, unit cost fell

In production Pick a pricing model from measured workload traits

A classifier that mirrors the decision rule: no guessing about interruption tolerance or steady demand, only measured inputs.

PythonPricing-model recommender from measured, not guessed, traitsruns
from dataclasses import dataclass


@dataclass
class Workload:
    name: str
    hours_per_day: float
    days_observed: int          # how long you've actually measured this
    interruption_tolerant: bool # checkpointed, retryable, no live user waiting
    utilization_pct: float      # measured average, not peak


def recommend(w: Workload) -> str:
    if w.days_observed < 30:
        return f"{w.name}: On-Demand (only {w.days_observed}d observed, too early to commit)"
    if w.interruption_tolerant and w.utilization_pct < 70:
        return f"{w.name}: Spot (interruption-tolerant, {w.utilization_pct:.0f}% utilized, big discount available)"
    if w.utilization_pct >= 70:
        return f"{w.name}: Savings Plan / RI ({w.days_observed}d steady at {w.utilization_pct:.0f}%)"
    return f"{w.name}: On-Demand (steady but under-utilized, right-size before committing)"


fleet = [
    Workload("nightly-settlement-etl", hours_per_day=2.0, days_observed=45, interruption_tolerant=True, utilization_pct=35.0),
    Workload("api-ingest-worker",      hours_per_day=24.0, days_observed=120, interruption_tolerant=False, utilization_pct=78.0),
    Workload("new-reconciliation-job", hours_per_day=4.0, days_observed=9, interruption_tolerant=True, utilization_pct=50.0),
    Workload("report-batch-runner",    hours_per_day=24.0, days_observed=90, interruption_tolerant=False, utilization_pct=42.0),
]

for w in fleet:
    print(recommend(w))
Output
nightly-settlement-etl: Spot (interruption-tolerant, 35% utilized, big discount available)
api-ingest-worker: Savings Plan / RI (120d steady at 78%)
new-reconciliation-job: On-Demand (only 9d observed, too early to commit)
report-batch-runner: On-Demand (steady but under-utilized, right-size before committing)

See it happen The decision rule as a flow

interruption-tolerant,low steady utilizationsteady,high utilizationNew or unmeasuredworkloadOn-DemandMeasure 30-60 days:utilization, patternSpotSavings Plan / RI
  1. Every workload starts On-Demand. There is no shortcut past this.
  2. You run it for real for 30-60 days and measure utilization, not peak.
  3. If it's interruption-tolerant and under-utilized, Spot gives the discount without a commitment.
  4. If it's steady and genuinely busy, a Savings Plan or RI pays off because you will use the commitment.
  5. Skipping straight from new workload to a 1- or 3-year commitment is the mistake this whole chapter is about.
Notice: the measurement step is not optional scaffolding — it is the only thing standing between you and a non-refundable commitment to the wrong shape.
Scenario A

In your world

The situation

You inherit a nightly settlements job that runs for about two hours on a c5.xlarge EC2 instance. The instance was provisioned On-Demand "to get it working" eighteen months ago and nobody ever revisited it. It has been running 24 hours a day since.

What actually breaks

The job needs the instance for roughly 60 hours a month. On-Demand, running continuously, bills for all 720 hours in the month. You are paying for twelve times the compute the job actually uses, every month, with no error anywhere — the job succeeds every night, so nothing ever flags it.

What you'd do on Monday

Measure the real run window from CloudWatch, then stop paying for idle hours instead of picking a cheaper instance type for the same 24/7 pattern:

PythonScheduled start/stop instead of 24/7 On-Demand (needs boto3 + an EventBridge schedule)
import boto3

ec2 = boto3.client("ec2", region_name="ap-south-1")

def stop_if_idle(instance_id: str) -> None:
    """Call from an EventBridge Scheduler rule right after the nightly job's
    own completion marker, not on a fixed clock guess."""
    ec2.stop_instances(InstanceIds=[instance_id])

def start_before_window(instance_id: str) -> None:
    """Call a few minutes before the job's known start time."""
    ec2.start_instances(InstanceIds=[instance_id])

Two hours billed for two hours is a ~90% cost reduction with zero change to the job.

Scenario B

Imagine you work at Postman

2M automated API collection runs/day

The numbers

Illustrative: 2,000,000 collection runs/day, each executing on a 1-vCPU container task for an average of 20 seconds. That's 2,000,000 × 20s = 40,000,000 vCPU-seconds/day ÷ 3,600 ≈ 11,100 vCPU-hours/day. At an illustrative On-Demand container rate (check the current pricing page) of roughly $0.04/vCPU-hour, that's about $444/day, or roughly $13,300/month, for compute alone before memory charges.

What the Scenario-A answer does here

The Scenario A fix — stop paying for idle time — doesn't apply here, because these tasks are already ephemeral; they start and stop per run. What dominates at this volume is the billing floor: if the platform's minimum billable task duration is, say, 60 seconds (confirm the current minimum before relying on it) and the average run is 20 seconds, every run is billed for roughly 3× the compute it actually used.

What flips at this scale

At 2M runs/day you stop picking a pricing model per task and instead batch many short runs onto a smaller pool of longer-lived workers covered by a Compute Savings Plan, so the per-invocation floor stops being multiplied by two million. The metric you track changes from "$/instance-hour" to "$/1,000 runs," because that's the number that actually reflects whether batching helped.

The trade-off you'd defend

I would defend batching bursty work onto a shared, committed worker pool over paying the per-invocation billing floor at scale, trading a few seconds of queueing latency for a materially lower unit cost, with a small On-Demand burst pool for spikes the committed pool can't absorb.

Illustrative figures for the arithmetic; not Postman-published. Confirm current container billing granularity and minimums on the AWS pricing pages before using them in a real estimate.

Gotchas

A budget alert firing does not mean spending stopped

Symptom

You get the budget notification email, acknowledge it, and the bill keeps growing for the rest of the month exactly as before.

Cause

AWS Budgets alerts are notifications, not circuit breakers, unless you've explicitly configured a Budget Action (like applying an IAM policy or stopping instances); by default nothing is throttled.

Fix

Treat the alert as a prompt to act, and if you need an automatic stop, wire a real Budget Action or your own Lambda response to the alert, as covered in Budgets and controlled lab spend.

24/7 On-Demand for a part-time job

Symptom

A nightly or weekly job's compute bill is flat every single day of the month, including weekends when nothing runs.

Cause

The instance was never stopped between runs; On-Demand bills for every hour it exists, used or not.

Fix

Measure the actual run window and stop/start on a schedule, or move the workload to something that only bills while executing (Lambda, Fargate scheduled tasks, Batch).

Comparing total bills instead of unit cost

Symptom

"Our AWS bill went up 30%" gets treated as bad news with no context.

Cause

Total cost ignores growth in work done. A 30% cost increase alongside a 60% rise in rows processed is an improvement, not a regression.

Fix

Pair every cost delta with a work-done delta, and report unit cost alongside the total.

Tricky questions

Is Spot always cheaper than Savings Plans for the same workload?
Most people say

Yes, Spot discounts are usually the deepest, so it's always cheapest.

Actually

Spot is usually cheaper per hour, but comes with interruption risk and no SLA. For a workload that can't tolerate interruption, Spot's rate is irrelevant — fit the workload's tolerance first, then compare prices only within models that qualify.

If I'm not using my Savings Plan commitment, does AWS pause the charge?
Most people say

No — unused commitment just means a smaller discount, not extra cost.

Actually

You're charged the committed hourly amount regardless of usage. Under-utilizing a Savings Plan or RI means paying for capacity you aren't consuming, strictly worse than On-Demand for that unused slice.

Scenario question

You're handed a monthly AWS invoice that grew from $8,200 to $11,400 and asked "why did cost go up 39%, fix it." Before touching anything, what do you ask for and what do you report back?

Model answer

I'd ask for the CUR broken down by service and cost-allocation tag for both months, plus the work-done metric for the main workloads — rows processed, pipeline runs, requests served. A total-dollar comparison alone tells me nothing. If rows processed also grew 39%, unit cost is flat and this is growth, not an incident, and I'd report that explicitly. If output only grew 10%, I'd break the delta down by service to find which line item moved — looking for an idle resource, a leak from Five recurring cost leaks, or a pricing-model mismatch like 24/7 On-Demand for part-time work. The trade-off: proper investigation takes a day or two against tag-complete billing data; guessing from the total risks "fixing" something that wasn't the cause.

Interview corner

CommonWalk me through the four main AWS pricing models.
“On-Demand is pay-as-you-go with no commitment, Spot is a steep discount on spare capacity with a two-minute interruption notice, and Savings Plans and RIs are pre-committed spend over one or three years, non-refundable if unused. I pick based on measured interruption tolerance and steady demand, not the lowest headline rate.”
TrickOur bill doubled this month — is that bad?
“Not necessarily. I'd ask for the work-done metric before answering. If output doubled too, the unit cost is flat and this is healthy growth. I'd never report a cost delta without a corresponding output delta.”
Experience checkTell me about a time you reduced AWS cost without changing functionality.
“I found a nightly job's EC2 instance running 24/7 for a two-hour workload. I measured the real run window from the logs, added a start/stop schedule around it, and cost dropped roughly 90% with the job's behaviour unchanged. Nobody had revisited the provisioning decision since launch.”
DesignHow would you decide whether to commit to a Savings Plan for a new workload?
“Not on day one. I'd run it On-Demand and measure utilization for at least 30 days. If it's steady and busy, I'd commit sized to that baseline, not the peak. If it's bursty or interruption-tolerant, I'd look at Spot instead. Committing before real data exists is how you end up paying for capacity you don't use.”

Memory hook

Four ways to pay for a cab: metered, shared-and-interruptible, prepaid pass. Measure before you buy the pass.

If you remember only one thing:

Never commit to a Savings Plan or RI before you've measured 30+ days of real, steady utilization.

Self-check

  1. What's the difference between Spot's discount and a Savings Plan's discount, in terms of what you give up for each?

    Spot trades a discount for interruption risk (best-effort, two-minute notice); a Savings Plan trades a discount for a non-refundable spend commitment over 1 or 3 years, regardless of interruption.

  2. Why is "the bill went up 30%" not enough information to act on?

    Cost without a work-done denominator (unit cost) can't tell you whether the business got more expensive to run or just did more work for proportionally more money.

  3. Does an AWS Budgets alert stop further spending by default?

    No. It's a notification unless you've explicitly attached a Budget Action; without one, spending continues exactly as before.

Prove it

Take the recommend() function from the production example. Add a fifth workload with days_observed=15, interruption_tolerant=False, utilization_pct=85. Run it through recommend() and explain in a comment why it still returns On-Demand despite the high utilization. Paste the function's output for all five workloads.

aws-11-02Card 2 of 6

Savings Plans, RIs and right-sizing

HardP4 · DifferentiatorTalk about it13 min
savings-plansreserved-instancesright-sizingspotgraviton

Why you care

"Should we buy Reserved Instances or Savings Plans?" is a question finance will eventually ask you, and getting it wrong is expensive in a specific way: both are non-refundable if you commit to the wrong size or the wrong flexibility. A developer who answers "Reserved Instances, they're cheaper" without asking about workload stability has memorised a price list, not understood a trade-off. Interviewers in platform and senior ETL roles ask this because it reveals whether you right-size before committing, which is the one step almost everyone skips under deadline pressure.

Explain like I'm 5

Imagine two gym memberships. A Standard Reserved Instance is a membership locked to one specific gym branch, with your exact equipment reserved — cheapest, but if you move cities, the membership is wasted. A Convertible RI is a membership you can swap to a different branch of the same chain, for a slightly higher price. A Compute Savings Plan is a membership that works at any gym chain, any city, any equipment — most flexible, broadest discount band, and it even covers a home gym (Lambda, Fargate) if you switch to working out there instead. None of them refund you for months you don't go.

Core concept

Reserved Instances (RIs):

  • Standard RI: deepest discount among RIs. For Linux/UNIX, default-tenancy, size-flexible instance families, you can modify AZ, scope and instance size within the same instance family and generation; changing instance family or OS is not a modification — it requires exchanging a Convertible RI instead.
  • Convertible RI: can be exchanged for a different instance family, size, OS or tenancy, for equal or greater value, at a smaller discount than Standard.
  • Scope: Regional (applies to any AZ in the region, flexible placement) or Zonal (locked to one specific AZ). Both carry the same discount rate — Zonal's trade is a guaranteed capacity reservation in that AZ, not a deeper discount, in exchange for giving up AZ flexibility.

Savings Plans:

  • Compute Savings Plans: the most flexible commitment. You commit to a dollar-per-hour spend, and it applies across EC2 instance family, size, AZ, region, OS and tenancy, and even across EC2, Fargate and Lambda — you can genuinely move a workload from EC2 to Fargate mid-term and keep the discount.
  • EC2 Instance Savings Plans: deeper discount than Compute SPs, but locked to a specific instance family within a specific region; you retain flexibility to change size, AZ, OS and tenancy within that family.
  • Both are use-it-or-lose-it: your committed hourly spend is charged whether or not matching usage exists that hour.

Right-sizing before committing: measure actual CPU, memory and (for databases) connection/IOPS utilization over real traffic, not the instance size someone guessed at launch. A workload consistently running at 25% CPU on an m5.2xlarge should be downsized before you reserve anything — reserving the oversized instance locks in the waste for 1-3 years. EC2 baseline and Auto Scaling owns the EC2 operational depth this feeds into.

Graviton and Spot compatibility: Graviton (ARM) instances are generally cheaper per unit of compute than equivalent x86 instances for workloads that are already compiled/tested for ARM — right-sizing and architecture choice happen together, not as separate steps. Spot works independently of commitment type: you can run Spot and hold a Savings Plan for different parts of the same fleet, using Spot for the interruption-tolerant burst layer and the commitment for the steady baseline.

Decision rule: right-size first, always. Then choose Compute Savings Plans by default for flexibility unless you have a genuinely stable, long-running fleet in one instance family, in which case an EC2 Instance Savings Plan or Convertible RI buys a deeper discount for a small loss of flexibility you've already proven you don't need.

Under the hood

A Spot Instance interruption notice is delivered two minutes before AWS stops or terminates the instance (hibernation is the exception — you get the notice but not the two-minute window, because hibernation begins immediately), as an EventBridge event with an instance-action field (stop, terminate or hibernate) and as an item in the instance's own metadata, polled from http://169.254.169.254/latest/meta-data/spot/instance-action. AWS explicitly documents this as best-effort, recommending you poll metadata every five seconds rather than assume the notice always arrives in time to act. A job that treats the two-minute window as a hard SLA, rather than as "usually enough time if you're already checkpointing," will eventually lose work on the run where the notice is delayed or coincides with other load on the metadata service.

Minimal example Right-size before you reserve, not after

PythonA reservation locks in whatever size you reserve, waste includedruns
def monthly_cost(hourly_rate: float, hours: int = 720) -> float:
    return hourly_rate * hours

oversized_ondemand = monthly_cost(0.384)     # m5.2xlarge, illustrative On-Demand rate
rightsized_ondemand = monthly_cost(0.096)    # m5.large, same workload at measured utilization

# A 3-year Standard RI locks in whichever size you reserve
reserved_oversized = oversized_ondemand * 0.60     # illustrative ~40% discount for commitment
reserved_rightsized = rightsized_ondemand * 0.60

print(f"On-Demand, oversized:  ${oversized_ondemand:.2f}/mo")
print(f"On-Demand, right-sized: ${rightsized_ondemand:.2f}/mo")
print(f"3yr RI,     oversized:  ${reserved_oversized:.2f}/mo  (discount locked onto waste)")
print(f"3yr RI,     right-sized: ${reserved_rightsized:.2f}/mo (discount on real need)")
print(f"3-year waste from reserving before right-sizing: ${(reserved_oversized - reserved_rightsized) * 36:.2f}")
Output
On-Demand, oversized:  $276.48/mo
On-Demand, right-sized: $69.12/mo
3yr RI,     oversized:  $165.89/mo  (discount locked onto waste)
3yr RI,     right-sized: $41.47/mo (discount on real need)
3-year waste from reserving before right-sizing: $4478.98

In production Spot-safe checkpoint loop with a best-effort interruption check

sqlite3 stands in for a real checkpoint store; the structure — poll for the notice, finish the current unit, persist a checkpoint, exit cleanly — is what makes a Spot-based batch worker safe.

PythonCheckpointed batch loop that respects a Spot interruption noticeruns
import sqlite3
from itertools import batched

def fake_interruption_check(tick: int) -> bool:
    """Stand-in for polling instance metadata every 5s; interrupts on tick 3."""
    return tick == 3

def run_batches(rows, batch_size, db):
    for tick, batch in enumerate(batched(rows, batch_size), start=1):
        if fake_interruption_check(tick):
            print(f"tick {tick}: interruption notice received, finishing current batch then exiting")
            with db:
                db.executemany("INSERT INTO processed VALUES (?)", [(r,) for r in batch])
            last = batch[-1]
            with db:
                db.execute("UPDATE checkpoint SET last_id = ?", (last,))
            print(f"checkpoint saved at {last}, safe to resume later")
            return "interrupted"
        with db:
            db.executemany("INSERT INTO processed VALUES (?)", [(r,) for r in batch])
            db.execute("UPDATE checkpoint SET last_id = ?", (batch[-1],))
        print(f"tick {tick}: committed batch ending at {batch[-1]}")
    return "complete"

db = sqlite3.connect(":memory:")
db.execute("CREATE TABLE processed (id INTEGER)")
db.execute("CREATE TABLE checkpoint (last_id INTEGER)")
db.execute("INSERT INTO checkpoint VALUES (0)")

status = run_batches(range(1, 51), batch_size=10, db=db)
count = db.execute("SELECT COUNT(*) FROM processed").fetchone()[0]
last = db.execute("SELECT last_id FROM checkpoint").fetchone()[0]
print(f"status={status} rows_committed={count} resume_from={last}")
Output
tick 1: committed batch ending at 10
tick 2: committed batch ending at 20
tick 3: interruption notice received, finishing current batch then exiting
checkpoint saved at 30, safe to resume later
status=interrupted rows_committed=30 resume_from=30
Scenario A

In your world

The situation

Your team's nightly ETL fleet has run on three m5.2xlarge On-Demand instances for a year, sized for "worst case" volume that happens twice a year during peak season. A new finance directive says "cut compute spend 20%, don't break anything."

What actually breaks

You check CloudWatch and find average CPU utilization across the fleet is 22% outside peak season — ten months of the year you're paying for roughly 4.5× the capacity you use, and nobody right-sized after the initial "worst case" guess became the permanent size.

What you'd do on Monday

Right-size to the measured non-peak baseline first (m5.large covers it with headroom), add Auto Scaling for the two peak months, and only then buy a 1-year Compute Savings Plan sized to the new, correct baseline — buying the commitment before right-sizing would have locked in the same 4.5× waste for a year instead of removing it.

Scenario B

Imagine you work at Groww

40,000 nightly batch jobs across a trading-data pipeline fleet

The numbers

Illustrative: 40,000 nightly batch jobs, each averaging 6 minutes on a 1-vCPU task, interruption-tolerant and checkpointed every 60 seconds. Total compute: 40,000 × 6 min = 240,000 vCPU-minutes/night ÷ 60 = 4,000 vCPU-hours/night. At an illustrative On-Demand rate (check current pricing) of $0.04/vCPU-hour that's $160/night, or roughly $4,800/month, if run On-Demand end to end.

What the Scenario-A answer does here

The Scenario A fix (right-size, then commit) still matters here, but a fleet this size has a second lever: Spot. If even 70% of the 4,000 nightly vCPU-hours move to Spot at a steep discount, the naive assumption "just flip everything to Spot" ignores that a two-minute, best-effort interruption notice is not guaranteed to arrive before every interruption — jobs without a checkpoint younger than roughly two minutes can lose work on exactly the runs where the notice is delayed.

What flips at this scale

At this volume you split the fleet: a small Compute-Savings-Plan-covered baseline for anything that cannot tolerate a lost run (final reconciliation, anything feeding a downstream SLA), and the bulk of the interruption-tolerant batch work on Spot with checkpoints no more than 60 seconds apart. The commitment and the discount are no longer a single decision; they're a split decision per job class.

The trade-off you'd defend

I would defend a split fleet — committed baseline plus Spot majority — over an all-Spot or all-committed fleet at this scale, trading the operational complexity of two capacity pools for a materially lower blended unit cost without betting SLA-critical jobs on best-effort interruption notices.

Illustrative figures for the arithmetic; not Groww-published. Spot interruption notice timing and best-effort delivery are documented by AWS; confirm current pricing and discount bands before relying on exact numbers.

Gotchas

Committing before you understand steady demand locks in a poor fit

Symptom

A 1- or 3-year Savings Plan or RI is purchased based on a single busy week, and six months later the fleet is consistently under-utilizing the commitment.

Cause

The commitment was sized to a snapshot of demand, not a measured steady baseline, and demand regressed toward the mean after the snapshot period.

Fix

Measure utilization over at least 30 days spanning normal variation (not a launch week or a peak event) before sizing any commitment, and size to the baseline, with Spot or On-Demand absorbing bursts above it.

Right-sizing after reserving wastes the discount on the wrong size

Symptom

A right-sizing review recommends downsizing an instance, but the team hesitates because "we already bought the 3-year RI for the bigger size."

Cause

The reservation locked in a specific instance size; downsizing now means the old RI sits unused (still charged) while a new, correctly sized instance runs separately.

Fix

Right-size before purchasing any commitment. If you're already locked in, a Convertible RI or Savings Plan can be modified or exchanged with less loss than a Standard RI, which is exactly the flexibility trade-off this card is about.

Treating the Spot interruption notice as a guaranteed SLA

Symptom

A Spot-based job occasionally loses in-flight work with no interruption notice ever observed in its logs.

Cause

AWS documents the two-minute notice as best-effort, delivered on a best-effort basis; it is not a contractual guarantee of advance warning in every case.

Fix

Checkpoint far more often than once per two minutes for anything expensive to redo, and design the job to resume cleanly from the last checkpoint regardless of whether a notice was ever seen.

Tricky questions

Is a Convertible RI just a worse version of a Compute Savings Plan?
Most people say

Yes, Compute Savings Plans are strictly better since they're more flexible.

Actually

A Convertible RI gives a deeper discount than a Compute Savings Plan in exchange for narrower flexibility (exchange within the RI marketplace rules, not free movement across EC2/Fargate/Lambda). If you've already proven your fleet is stable and EC2-only, that narrower flexibility costs you nothing and the deeper discount is pure upside.

Does Spot pricing stay fixed once you launch an instance?
Most people say

Yes, you lock in the Spot price for the life of the instance like a Savings Plan.

Actually

Spot has no price lock-in and no term; your instance can be interrupted whenever AWS needs the capacity back, due to capacity, your own price limit being exceeded, or constraint violations, regardless of how long it's been running.

Scenario question

You're handed a fleet of twelve EC2 instances running a batch pipeline, currently all On-Demand, and told "buy whatever reservation saves the most money, we have budget approved this week." What do you do before buying anything?

Model answer

I would not buy anything this week. First I'd pull at least 30 days of CloudWatch utilization data per instance to find the real steady baseline, separating out any instances that are only busy during known peak windows. For the steady, always-busy instances, I'd check whether they're correctly sized — right-sizing often changes which instance family or size you'd even reserve. Only with that baseline would I compare a Compute Savings Plan against an EC2 Instance Savings Plan or Convertible RI for the steady portion, and leave the peak-only capacity on-demand or Spot rather than reserving for it. The trade-off I'd explain to whoever approved the budget: buying this week on a guess risks locking in the wrong size for 1-3 years, while a two-week delay to measure properly costs a small amount of foregone discount but avoids a non-refundable mistake.

Interview corner

CommonSavings Plan or Reserved Instance — how do you choose?
“I'd start with a Compute Savings Plan by default, because it covers EC2, Fargate and Lambda and is flexible on family, size, region and OS. I'd only move to an EC2 Instance Savings Plan or a Convertible RI if I had a proven, stable, single-family fleet and wanted the deeper discount that narrower flexibility buys.”
TrickWe have unused Reserved Instance capacity — are we losing money compared to not reserving at all?
“I'd say yes, for the unused portion. You're paying the committed rate regardless of usage, so an under-utilized RI can cost more than On-Demand would have for that slice of capacity. I'd fix it by right-sizing the fleet to match the reservation, or, for a Convertible RI, exchanging it for something that matches current demand.”
Experience checkDescribe a right-sizing exercise you'd run before committing to a Savings Plan.
“I'd pull 30-plus days of CPU, memory, and for databases, connection and IOPS utilization, averaged, not peak. Anything consistently under 40% utilization gets downsized first. Only after that would I size the commitment to the new baseline, because committing to the old, oversized number just locks in the waste for the term.”
DesignHow would you build a fleet that uses both Spot and a Savings Plan safely?
“I'd split the fleet by SLA tolerance: a small committed baseline for anything that can't afford a lost run, and the bulk of interruption-tolerant batch work on Spot with frequent checkpoints, since the interruption notice is best-effort, not guaranteed. The commitment covers the floor; Spot covers everything above it.”

Memory hook

A reservation locks in whatever size you reserve — waste included. Right-size first, commit second.

If you remember only one thing:

Measure 30+ days of steady utilization before buying anything non-refundable.

Self-check

  1. What's the key flexibility difference between a Compute Savings Plan and an EC2 Instance Savings Plan?

    Compute Savings Plans apply across instance family, size, region, OS, tenancy and even EC2/Fargate/Lambda; EC2 Instance Savings Plans are locked to one instance family in one region but give a deeper discount.

  2. Why does right-sizing have to happen before buying a reservation, not after?

    Because the reservation locks in whatever size you bought; downsizing afterward leaves the old reservation under-utilized (still charged) instead of removing the waste.

  3. Is a Spot Instance's two-minute interruption notice guaranteed to arrive in time to act?

    No — AWS documents it as best-effort; design checkpoints far more frequent than two minutes for anything expensive to lose.

Prove it

Using the monthly_cost function from the minimal example, model a fleet of 5 instances currently On-Demand at an oversized rate. Compute the 1-year cost of (a) staying On-Demand oversized, (b) right-sizing then going On-Demand, (c) right-sizing then buying a 1-year commitment at an illustrative 40% discount. Print all three totals and the dollar gap between the worst and best option.

aws-11-03Card 3 of 6

Five recurring cost leaks

MediumP4 · DifferentiatorTalk about it11 min
cost-leaksidle-resourcesfinopsorphaned-storagedata-transfer

Why you care

Nobody budgets for a cost leak; it just accumulates quietly until someone asks "why is this line item three times what it was six months ago." Five patterns account for most of the AWS waste a backend or data engineer will personally cause or inherit, and knowing them cold means you can run a 30-minute audit instead of staring at a Cost Explorer dashboard hoping something jumps out. Interviewers ask "what are common ways AWS bills get out of control" because the answer separates people who've only ever used AWS from people who've had to explain an invoice to finance.

Explain like I'm 5

Imagine a house where every room has a light switch, but half the switches don't turn anything off, they just dim the light a little. You keep paying full power while thinking you saved something. The five leaks in this card are the five most common rooms where this happens: lights left on in rooms nobody uses, boxes in the attic you're still paying rent on, taking the long way round to get somewhere nearby, asking the shop a thousand small questions instead of one big one, and leaving the oven on all day for one meal.

Core concept

Five recurring leaks, in rough order of how often an ETL/backend developer personally introduces them (editorial grouping, not a measured prevalence ranking):

  • Idle or oversized compute and databases. An EC2 instance or RDS instance provisioned for peak and left running at a fraction of that size 90% of the time. RDS in particular keeps billing even when nobody's connected — "idle" is not "free" the way a stopped Lambda is.
  • Orphaned storage and excess retention. EBS volumes left behind after an instance is terminated, old snapshots nobody deletes, S3 objects (including non-current versions once versioning is on) kept forever because no lifecycle rule was ever set. Storage classes and lifecycle design owns the lifecycle design that prevents this.
  • Avoidable NAT, cross-AZ and egress paths. A NAT Gateway bills an hourly charge plus a per-GB data processing charge for every byte that passes through it; traffic that could have stayed inside the AWS network (via a VPC endpoint) or inside one AZ instead travels the expensive route by default, because nobody drew the path. Transfer and S3 request traps goes deep on this specific trap.
  • Chatty S3 requests and small files. Thousands of tiny objects instead of fewer, larger ones multiply PUT/GET/LIST request costs and multiply Athena/Glue scan overhead, even when total bytes stored are small.
  • Wasteful scans or always-on data services. An Athena query without a partition filter scanning an entire table, or a Redshift/EMR cluster running 24/7 for a workload that's actually used three hours a day.

Decision rule for an audit: start with anything billed by time regardless of use (idle compute, idle databases, always-on clusters) — those are the leaks you can usually fix with a schedule or a stop/start, no redesign needed. Then look at anything billed by volume (storage retention, request counts, data scanned) — those need a lifecycle policy, a batching change, or a partitioning fix, which takes longer but compounds for as long as the data keeps growing.

Minimal example Idle doesn't mean free

PythonLow utilization still bills in full — "idle" and "cheap" are not the same claimruns
def is_leak(avg_utilization_pct: float, threshold: float = 15.0) -> bool:
    return avg_utilization_pct < threshold


instances = {"rds-reporting-replica": 6.0, "ec2-active-etl": 68.0}
for name, util in instances.items():
    flag = "LEAK: idle, still billed in full" if is_leak(util) else "ok"
    print(f"{name}: {util:.0f}% avg utilization -> {flag}")
Output
rds-reporting-replica: 6% avg utilization -> LEAK: idle, still billed in full
ec2-active-etl: 68% avg utilization -> ok

In production Estimate the dollar impact of each leak class, not just flag it

Flagging a leak without an estimated dollar impact gets ignored; a number gets prioritised.

PythonRank leaks by illustrative monthly dollar impactruns
from dataclasses import dataclass


@dataclass
class LeakEstimate:
    name: str
    category: str
    illustrative_monthly_cost: float
    one_line_fix: str


estimates = [
    LeakEstimate("rds-reporting-replica", "idle compute/db", 310.00, "downsize or stop outside business hours"),
    LeakEstimate("ebs-vol-old-worker", "orphaned storage", 24.00, "snapshot then delete after a grace window"),
    LeakEstimate("s3-raw-landing", "excess retention", 540.00, "add lifecycle: IA at 90d, Glacier at 365d"),
    LeakEstimate("s3-event-log", "chatty requests", 95.00, "batch small objects before writing to S3"),
    LeakEstimate("nightly-athena-scan", "wasteful scans", 410.00, "add partition filter, convert CSV to Parquet"),
]

ranked = sorted(estimates, key=lambda e: e.illustrative_monthly_cost, reverse=True)
total = sum(e.illustrative_monthly_cost for e in ranked)

print(f"{'resource':<24} {'category':<18} {'$/mo':>8}  fix")
for e in ranked:
    print(f"{e.name:<24} {e.category:<18} {e.illustrative_monthly_cost:>8.2f}  {e.one_line_fix}")
print(f"\ntotal illustrative monthly leak: ${total:.2f}")
Output
resource                 category               $/mo  fix
s3-raw-landing           excess retention     540.00  add lifecycle: IA at 90d, Glacier at 365d
nightly-athena-scan      wasteful scans       410.00  add partition filter, convert CSV to Parquet
rds-reporting-replica    idle compute/db      310.00  downsize or stop outside business hours
s3-event-log             chatty requests       95.00  batch small objects before writing to S3
ebs-vol-old-worker       orphaned storage      24.00  snapshot then delete after a grace window

total illustrative monthly leak: $1379.00
Scenario A

In your world

The situation

You're asked to do a quick cost audit before a quarterly review. You have read access to Cost Explorer, CloudWatch and the S3 console, and two hours.

What actually breaks

You open Cost Explorer and see S3 and RDS are the two biggest line items, but no obvious single resource jumps out as "wrong" — the leak is spread across many small things, not one dramatic one.

What you'd do on Monday

Run the five-leak checklist in order of time-billed-regardless-of-use first: list RDS instances by CloudWatch average CPU/connections over 14 days (catches idle-and-oversized), list unattached EBS volumes (catches orphaned storage), then move to volume-billed items: S3 buckets without a lifecycle rule on objects older than a year, and any Athena queries scanning without a partition filter. Two hours is enough to find the first three categories; the Athena/Glue scan review usually needs a follow-up ticket.

Scenario B

Imagine you work at Instacart

180M+ grocery delivery events logged monthly

The numbers

Illustrative: 180M events/month, each currently written as its own small S3 object averaging 4KB, for roughly 180,000,000 PUT requests/month. At an illustrative request rate (check current pricing) of $0.005 per 1,000 PUT requests, that's about $900/month in PUT requests alone, before any GET/LIST cost for downstream processing.

What the Scenario-A answer does here

The Scenario A audit (list buckets without a lifecycle rule) still finds genuine excess-retention waste here, but at this event volume the dominant leak is request count, not storage bytes — 180M tiny objects cost far more in PUT/GET/LIST requests than the handful of gigabytes they occupy in storage, and no lifecycle rule touches request cost.

What flips at this scale

At this volume the fix is architectural, not a lifecycle policy: batch events client-side or in a buffering layer (Kinesis Data Firehose, or a simple time/size-based batcher) before they ever reach S3, so 180M events become a few thousand larger objects instead of 180M small ones. The unit to optimise shifts from "$/GB stored" to "$/million events written," because that's where the actual cost now lives.

The trade-off you'd defend

I would defend batching events before the S3 write over "just delete old data faster," trading a small amount of write latency (events wait in a buffer before landing) for an order-of-magnitude reduction in request count, which is the leak that actually dominates at this scale.

Illustrative figures for the arithmetic; not Instacart-published. Confirm current S3 request pricing before relying on exact numbers.

Gotchas

Idle is not free, even when nobody's connected

Symptom

An RDS instance with zero active connections for weeks still appears at full price on the monthly bill.

Cause

RDS (and most managed databases) bill for provisioned capacity existing, not for queries run against it; "nobody's using it" and "it costs nothing" are unrelated facts.

Fix

Stop instances that are genuinely idle outside a known window, or downsize to match actual connection/query volume, verified from CloudWatch, not from asking around.

Deleting a current S3 object doesn't delete its history

Symptom

A bucket with versioning enabled keeps growing in billed storage even as the team "cleans up" by deleting old files.

Cause

Deleting the current version of an object leaves every non-current version in place and billable, unless a lifecycle rule expires them.

Fix

Add a lifecycle rule that expires non-current versions after a defined retention window, covered in depth in Storage classes and lifecycle design.

Small files multiply request cost invisibly

Symptom

A bucket holding only a few GB of data still shows a surprisingly large request-cost line item.

Cause

Thousands or millions of small objects each cost a PUT (and later a GET) independent of their size; request pricing is per-object-operation, not per-byte.

Fix

Batch small records into fewer, larger objects before writing, or use a format/service designed for high-frequency small writes (Kinesis Data Firehose buffering) instead of direct per-event S3 puts.

Tricky questions

If an S3 bucket's total storage is small, is its cost automatically small?
Most people say

Yes, S3 cost scales with GB stored, so a small bucket is a cheap bucket.

Actually

Storage is only one billable dimension. A bucket with 50 million 2KB objects has trivial storage cost but can have a request-cost line item that dwarfs it, because PUT/GET/LIST are billed per request, not per byte.

Does stopping an EC2 instance stop all its charges?
Most people say

Yes, a stopped instance costs nothing until you start it again.

Actually

Stopping an EC2 instance stops the compute charge, but its attached EBS volumes keep billing for provisioned storage the entire time it's stopped. "Stopped" removes one leak category, not every leak attached to that resource.

Scenario question

You're handed a bucket that costs more in requests than in storage, and the team's instinct is "let's move everything to a cheaper storage class." Do you agree, and what do you check first?

Model answer

I'd push back before moving anything. If the request cost dominates, changing storage class won't help — Glacier and Deep Archive still charge per request, and for cold classes, retrieval requests cost more, not less. I'd first check object count and average object size; if it's millions of small objects, the fix is batching the writes upstream, not a storage-class migration. I'd only look at storage class once the object-size problem is fixed and storage, not requests, is the dominant cost. The trade-off I'd flag: fixing the write pattern touches producer code and takes longer than flipping a lifecycle rule, but a storage-class change here would cost engineering time for close to zero benefit.

Interview corner

CommonWhat are the most common ways AWS bills get out of control?
“I'd name five: idle or oversized compute and databases, orphaned storage and excess retention, avoidable NAT or cross-AZ data transfer, chatty small-file S3 request patterns, and wasteful or always-on scans. I'd check the time-billed ones first because they're usually a quick schedule fix, then the volume-billed ones, which need a design change.”
TrickA bucket's storage is tiny but its bill is large — what's wrong?
“I'd check request count before anything else. Small total storage with a large bill almost always means millions of small objects driving PUT/GET/LIST cost, which a storage-class change won't fix because it's a request problem, not a storage problem.”
Experience checkDescribe a cost leak you found and how you confirmed it before acting.
“I found an RDS replica with near-zero connections over two weeks of CloudWatch data. Before touching it, I confirmed with the team that nothing depended on it being warm, then stopped it rather than guessing from the console alone that it was safe to remove.”
DesignHow would you prioritise fixing five different cost leaks with limited engineering time?
“I'd estimate the dollar impact of each, not just flag that they exist, then start with whichever combination of highest dollar impact and lowest fix effort wins — usually the time-billed idle resources, since those are a schedule change, before the request-pattern or scan-tuning fixes that need code changes.”

Memory hook

Five rooms where the light's left on: idle machines, forgotten storage, the long way round the network, a thousand small knocks instead of one, and an oven left on for one meal.

If you remember only one thing:

Check what's billed by time (idle resources) before what's billed by volume (storage, requests, scans) — the first is usually a schedule fix, the second needs a design change.

Self-check

  1. Why doesn't stopping an EC2 instance eliminate all its cost?

    Its attached EBS volumes keep billing for provisioned storage while the instance is stopped; only the compute charge stops.

  2. Why can a bucket with little total storage still have a large bill?

    Request cost (PUT/GET/LIST) is billed per operation, independent of object size, so millions of small objects can dominate the bill even with few total gigabytes.

  3. What's the difference between "idle" and "free" for a managed database?

    Idle means no active queries or connections; it is not free, because provisioned capacity bills regardless of use unless the instance is explicitly stopped or downsized.

Prove it

Extend the find_leaks function with a sixth resource that is an Athena-style query logged with scanned_gb and filtered_gb attributes, where a leak is flagged if filtered_gb / scanned_gb < 0.1 (less than 10% of scanned data was actually useful). Add one compliant and one non-compliant example query to the fleet and show both the flagged and unflagged output lines.

aws-11-04Card 4 of 6

Transfer and S3 request traps

MediumP4 · DifferentiatorTalk about it13 min
data-transfers3-requestsnat-gatewayvpc-endpointsegress

Why you care

"It worked in the architecture diagram" and "it's cheap to run" are different claims, and the gap between them is almost always a transfer path or a request pattern nobody traced. A NAT Gateway bills for every gigabyte that passes through it on top of its hourly charge; a cross-region copy that looked like one line of code in a migration script can be the single largest line item on the bill. Interviewers probe this because tracing a byte's actual path — same-AZ, cross-AZ, cross-region, or out to the internet — is the one skill that separates "I used S3" from "I understand what AWS actually charges for."

Explain like I'm 5

Imagine your office has a free internal mail cart that moves paper between desks on the same floor (same-AZ), a slightly slower cart between floors (cross-AZ), a courier between buildings in the same city (cross-region), and a international shipping company for anything leaving the company entirely (internet egress). Every one of those except the same-floor cart costs money, and the shipping company is the most expensive by far. A NAT Gateway is a specific toll booth: everything headed outside your building that isn't carried directly by the free internal mail system has to pass through it, and the toll booth charges by the kilogram, every time, in both directions.

Core concept

Trace every path before trusting a design, in increasing order of what it typically costs:

  • Same-AZ: traffic between resources in the same Availability Zone. Free in most cases.
  • Cross-AZ, same region: traffic between AZs incurs a per-GB charge in each direction, even though it never leaves the region. This is easy to introduce by accident when instances and their database or cache land in different AZs with no attempt to colocate frequent-talkers.
  • Cross-region: a per-GB charge based on the source region, applies to replication, cross-region copies, and any client calling a bucket in a different region than it runs in.
  • Internet egress: traffic leaving AWS entirely to the public internet, generally the most expensive per-GB tier, and the one most teams actually watch.

NAT Gateway bills an hourly charge for existing, plus a per-GB data processing charge for every byte it processes, plus any internet transfer charge for traffic it forwards onward. Any private-subnet resource talking to the public internet, or to an AWS service without a dedicated endpoint, goes through the NAT Gateway by default and pays both charges.

VPC endpoints are the fix for AWS-service traffic specifically: a Gateway endpoint (S3, DynamoDB) is free to create and use, routes via a route-table entry, and bypasses the NAT Gateway entirely for that service's traffic. An Interface endpoint (most other AWS services, via PrivateLink) has its own hourly and per-GB charge, but it's still generally cheaper than paying the NAT Gateway's processing charge for the same bytes, and it keeps the traffic off the public internet path.

S3 request dimensions, separate from transfer: PUT/COPY/POST/LIST requests bill per 1,000 requests; GET/SELECT requests bill per 1,000 at a different rate; retrieving from an infrequent-access or archive class adds a data-retrieval charge on top; DELETE and CANCEL are free. Lifecycle transitions (moving an object between storage classes) bill per 1,000 transition requests — a rule applied to millions of existing objects triggers millions of billable transitions on day one, not a free reclassification.

Replication stacks several of the above: it bills PUT requests for the replica writes, inter-region data transfer if the destination is in another region, and retrieval charges if the source objects are in a storage class that charges for reads.

Decision rule: before accepting a design, draw the path every meaningful byte takes, end to end, and name which of same-AZ/cross-AZ/cross-region/internet it crosses, whether it passes a NAT Gateway, and which S3 request category it falls into. If you can't draw that path, you can't estimate the cost of the design, no matter how confident the total-dollar guess feels.

Minimal example Classify a transfer path and estimate its per-GB tier

PythonName the path before estimating its costruns
def classify_path(source_az: str, dest_az: str, source_region: str, dest_region: str, via_internet: bool) -> str:
    if via_internet:
        return "internet egress (most expensive tier)"
    if source_region != dest_region:
        return "cross-region (per-GB, source-region rate)"
    if source_az != dest_az:
        return "cross-AZ, same region (per-GB, both directions)"
    return "same-AZ (free in most cases)"

paths = [
    ("ap-south-1a", "ap-south-1a", "ap-south-1", "ap-south-1", False),
    ("ap-south-1a", "ap-south-1b", "ap-south-1", "ap-south-1", False),
    ("ap-south-1a", "ap-south-1a", "ap-south-1", "eu-west-1", False),
    ("ap-south-1a", "ap-south-1a", "ap-south-1", "ap-south-1", True),
]

for p in paths:
    print(classify_path(*p))
Output
same-AZ (free in most cases)
cross-AZ, same region (per-GB, both directions)
cross-region (per-GB, source-region rate)
internet egress (most expensive tier)

In production Estimate NAT Gateway cost versus a Gateway endpoint for the same S3 traffic

PythonNAT Gateway charges versus a free Gateway endpoint, same trafficruns
def nat_monthly_cost(gb_per_month: float, hourly_rate: float = 0.045, per_gb_rate: float = 0.045) -> float:
    """Illustrative rates; confirm current NAT Gateway pricing before relying on this."""
    hours_in_month = 730
    return (hourly_rate * hours_in_month) + (per_gb_rate * gb_per_month)


def gateway_endpoint_cost(gb_per_month: float) -> float:
    """S3 Gateway endpoints are free to create and use; this models that explicitly."""
    return 0.0


workloads = [
    ("etl-worker-small", 50),
    ("etl-worker-medium", 800),
    ("etl-worker-heavy", 4_000),
]

print(f"{'workload':<20} {'GB/mo':>8} {'via NAT':>10} {'via endpoint':>14}  savings")
for name, gb in workloads:
    nat_cost = nat_monthly_cost(gb)
    ep_cost = gateway_endpoint_cost(gb)
    print(f"{name:<20} {gb:>8} {nat_cost:>10.2f} {ep_cost:>14.2f}  ${nat_cost - ep_cost:.2f}")
Output
workload                GB/mo    via NAT   via endpoint  savings
etl-worker-small           50      35.10           0.00  $35.10
etl-worker-medium         800      68.85           0.00  $68.85
etl-worker-heavy         4000     212.85           0.00  $212.85

See it happen Which paths a byte can take out of a private subnet

S3 callrouted via gateway endpointno endpoint configuredforwarded, billed per GBApp inprivate subnetS3, same regionGateway Endpoint(free)NAT Gateway(hourly + per-GB)Public internet
  1. The app only ever intends to talk to S3 in the same region.
  2. With a Gateway endpoint and route-table entry, the call never leaves the AWS network and costs nothing extra.
  3. Without the endpoint, the exact same S3 call defaults to the route out of the subnet: the NAT Gateway.
  4. The NAT Gateway forwards it to the internet gateway, billing its hourly rate plus a per-GB processing charge for traffic that never needed to leave AWS at all.
Notice: the only difference between the cheap path and the expensive path is a route-table entry. The traffic, the bytes, and the destination are identical.
Scenario A

In your world

The situation

Your ETL workers run in a private subnet and read/write S3 several times per job, routed through a NAT Gateway because that's how the VPC was set up by whoever built it first, with no S3 Gateway endpoint ever added.

What actually breaks

The NAT Gateway's data-processing charge grows every month as the pipeline's data volume grows, even though every byte involved is just the job talking to S3 — traffic that never needed to leave the AWS network, let alone pass through a per-GB toll booth.

What you'd do on Monday

Add an S3 Gateway endpoint to the VPC and a route-table entry for it; it's free to use and removes this traffic from the NAT Gateway entirely:

PythonAdd an S3 Gateway endpoint to the private subnet's route table (needs boto3 + VPC/subnet IDs)
import boto3

ec2 = boto3.client("ec2", region_name="ap-south-1")

def add_s3_gateway_endpoint(vpc_id: str, route_table_ids: list[str]) -> str:
    resp = ec2.create_vpc_endpoint(
        VpcEndpointType="Gateway",
        VpcId=vpc_id,
        ServiceName="com.amazonaws.ap-south-1.s3",
        RouteTableIds=route_table_ids,
    )
    return resp["VpcEndpoint"]["VpcEndpointId"]
Scenario B

Imagine you work at Snowflake

50 TB/day of cross-region data egress for a reporting pipeline

The numbers

Illustrative: a reporting pipeline reads 50 TB/day from a data lake in ap-south-1 and ships results to a reporting cluster in eu-west-1. At an illustrative cross-region transfer rate (check current pricing) of $0.02/GB, that's 50,000 GB × $0.02 = $1,000/day, or roughly $30,000/month, just for crossing the region boundary.

What the Scenario-A answer does here

The Scenario A fix (Gateway endpoint for same-region S3) doesn't touch this at all, because the cost here isn't NAT or request count — it's a genuine cross-region egress charge that no endpoint type eliminates, since the bytes have to physically move between regions by design.

What flips at this scale

At this volume the flip is reducing what crosses the boundary, not how it crosses: pre-aggregate or filter at the source region before shipping (send a 2 GB summary instead of a 50 TB extract), or ask whether the reporting workload can run in the source region at all and ship only the final rendered output. Moving compute to the data, instead of moving data to the compute, is the only lever that changes the arithmetic at this scale.

The trade-off you'd defend

I would defend pushing aggregation to the source region over continuing to ship raw data cross-region, trading the engineering cost of a source-region aggregation job for a transfer bill that drops by one or two orders of magnitude. If the destination genuinely needs row-level data, I'd argue for replicating the warehouse into the destination region instead of a daily full-volume transfer.

Illustrative figures for the arithmetic; not Snowflake-published. Confirm current cross-region transfer pricing before relying on exact numbers.

Gotchas

Traffic to S3 defaults to the expensive route without an endpoint

Symptom

A NAT Gateway's per-GB data-processing charge grows in step with S3 traffic volume, even though the destination is an AWS service in the same account and region.

Cause

Without a Gateway (or Interface) endpoint configured, a private subnet's only route to S3 is out through the NAT Gateway and back, incurring both the NAT charge and (if it crosses the public internet gateway) potential internet transfer charges.

Fix

Add an S3 (or DynamoDB) Gateway endpoint — free to create and use — and a matching route-table entry; this is almost always a pure win with no downside.

Replication quietly bills three things, not one

Symptom

Enabling cross-region replication on a bucket produces a bill increase larger than "just the extra storage" the team expected.

Cause

Replication bills the replica PUT requests, the inter-region data transfer for every replicated byte, and, if the source objects are in a storage class with retrieval charges, the retrieval cost on top.

Fix

Model all three dimensions before enabling replication on a large or frequently-written bucket, not just the destination storage cost.

A lifecycle rule applied to existing data isn't free on day one

Symptom

Adding a lifecycle transition rule to a bucket with millions of existing objects produces an immediate spike in request-type charges.

Cause

Each object's transition to a new storage class is a billable transition request; applying the rule to the whole existing population triggers that many billable transitions at once, not gradually.

Fix

Model the one-time transition-request cost against millions of existing objects before applying a new lifecycle rule retroactively, and consider phasing the rollout by date range if the one-time cost is material.

Tricky questions

Does an Interface (PrivateLink) endpoint cost more than just going through NAT?
Most people say

Yes, since it has its own hourly and per-GB charge, NAT must be cheaper since it's "already there."

Actually

NAT's hourly charge exists regardless, and it also charges per-GB processed; an Interface endpoint typically costs less overall for AWS-service traffic because it avoids the NAT data-processing charge for that traffic, even though it adds its own smaller per-GB and hourly cost. Compare the two per-GB rates directly, don't assume "already paying for NAT" means routing more through it is free.

Is cross-AZ traffic free because it never leaves the region?
Most people say

Yes, "same region" sounds like "no transfer charge."

Actually

Cross-AZ transfer within the same region is billed per-GB in each direction. Only genuinely same-AZ traffic (and traffic via a VPC endpoint to certain services) is typically free. "Same region" and "free" are not the same claim.

Scenario question

You're handed a monthly bill where the NAT Gateway's data-processing line item is the single largest item, larger than EC2 compute. What do you check, in order, before proposing a fix?

Model answer

First I'd check what traffic is actually flowing through the NAT Gateway — CloudWatch's BytesOutToDestination/BytesInFromSource metrics on the NAT Gateway, cross-referenced with VPC Flow Logs if I need destination detail. If most of it is AWS-service traffic (S3, DynamoDB, or other services with endpoint support), that's a near-free fix: add Gateway or Interface endpoints and route around the NAT Gateway entirely. If it's genuinely internet-bound traffic to a third party, the fix is different — look at whether the destination has a CDN or caching option, or whether the call volume itself can be reduced. I would not propose "a bigger NAT Gateway" or "more NAT Gateways" as a cost fix; that only ever increases the hourly charge without addressing the per-GB driver. The trade-off I'd flag: endpoint changes are low-risk and fast; reducing genuine internet traffic usually means a design or vendor conversation that takes longer.

Interview corner

CommonWhat's the difference between a Gateway endpoint and an Interface endpoint, cost-wise?
“I'd say Gateway endpoints, for S3 and DynamoDB, are free to create and use and route via a route-table entry. Interface endpoints, for most other AWS services via PrivateLink, have their own hourly and per-GB charge, but both bypass the NAT Gateway, so they're usually still cheaper than paying NAT's data-processing charge for the same traffic.”
TrickOur bill shows a big NAT Gateway charge — is that always a mistake?
“Not always — if the traffic genuinely needs to leave the VPC to the internet, NAT or an equivalent is doing its job. The mistake is specifically AWS-service traffic routed through NAT when a free or cheaper endpoint exists. I'd check what's actually flowing through it before assuming it's wrong.”
Experience checkWalk me through how you'd trace a byte's cost from source to destination.
“I name the AZ and region on both ends, check whether it crosses the public internet, and separately check which S3 request category it falls into if S3's involved — PUT-family versus GET-family versus a retrieval from a cold storage class. Each of those is a separate billable dimension, and I don't trust an estimate until I've named all of them for the actual path.”
DesignHow would you reduce a 50 TB/day cross-region transfer cost?
“I'd look at moving the aggregation to the source region instead of shipping raw data across the boundary, since no endpoint type removes a genuine cross-region egress charge. Shipping a small aggregated result instead of the full extract usually cuts the transferred volume by an order of magnitude or more.”

Memory hook

The same bytes, the same destination, one route-table entry apart: free via an endpoint, metered via NAT.

If you remember only one thing:

Trace every path (same-AZ, cross-AZ, cross-region, internet) and every S3 request category before trusting a cost estimate.

Self-check

  1. Why is adding an S3 Gateway endpoint usually a pure win with no downside?

    It's free to create and use, and it removes AWS-service traffic from the NAT Gateway's billed data-processing path entirely.

  2. What three things does S3 cross-region replication bill, beyond destination storage?

    Replica PUT requests, inter-region data transfer, and (if the source storage class charges for reads) retrieval cost.

  3. Why can applying a new lifecycle rule to an existing bucket cause an immediate cost spike?

    Each object's transition to a new storage class is a billable transition request; applying the rule retroactively to millions of existing objects triggers that many transitions at once.

Prove it

Using the nat_monthly_cost and gateway_endpoint_cost functions, model a workload that grows from 100 GB/month to 10,000 GB/month in 100 GB increments. Print the GB level at which the NAT Gateway's cost would exceed an illustrative Interface endpoint cost of $0.01/hour + $0.01/GB, assuming the Gateway endpoint option doesn't apply (pretend it's a non-S3/DynamoDB service). Show the crossover point.

aws-11-05Card 5 of 6

Cost per successful pipeline run

HardP4 · DifferentiatorTalk about it13 min
unit-costpipeline-costcost-allocationfinopstagging

Why you care

"How much does one successful run of this pipeline cost?" is a question almost nobody in an ETL team can answer honestly, because the honest answer requires allocating compute, storage, requests, orchestration and transfer costs across all runs — including the retries, backfills and failures that produced zero usable output — and dividing by the runs that actually succeeded. Skipping the failures makes every pipeline look artificially cheap. Interviewers ask this in senior and platform rounds because it's the question that separates "I can read a Cost Explorer graph" from "I can tell finance what a feature actually costs to operate."

Explain like I'm 5

Imagine a bakery that makes 100 loaves a day but burns 15 of them. If the owner says "each loaf costs me the flour and oven time for 85 loaves," that's a lie, even if it feels like the honest number for the loaves on the shelf. The oven ran, the flour was used, and the gas was burned for all 100 attempts, burnt or not. The true cost per sellable loaf has to include the ovens that produced nothing sellable, divided only across the loaves you can actually sell. A pipeline's retries and failed runs are the burnt loaves: they cost real money and produced zero usable rows, and pretending otherwise understates the real unit cost.

Core concept

Allocating cost per successful pipeline run means summing every billable dimension the run touches, across all attempts in the period, then dividing by successful, validated runs only:

  • Compute: the hours or GB-seconds billed for every attempt — the job that failed at minute 40 still billed 40 minutes of compute.
  • Scan: for Athena/Glue-style engines, the data scanned by every query attempt, including ones that errored out partway or were re-run after a bad partition filter.
  • Storage: the marginal storage this pipeline's outputs and intermediate artifacts add, including anything left behind by failed runs that never got cleaned up (a classic link to Five recurring cost leaks's orphaned-storage leak).
  • Requests: S3/API request counts across every attempt, not just the successful one.
  • Orchestration: Step Functions transitions, MWAA/Airflow task-slot time, or EventBridge invocations for every attempt and every retry.
  • Transfer: data movement for every attempt, including a retry that re-reads the same source data a second or third time.

Divide that total by the count of runs that were validated as successful — passed data-quality checks, reconciled against the source, and actually delivered usable output — not merely "the job exited zero." A run that exits zero but loads zero rows is not a successful run for cost-allocation purposes any more than it is for data-quality purposes.

The only trustworthy source for the numerator is actual billing data, from the Cost and Usage Report filtered by the pipeline's cost-allocation tags, cross-referenced with the pipeline's own run log for which runs succeeded. Estimating cost per run from a instance's hourly rate times its average duration, without checking the CUR, misses Spot interruption retries, orchestration overhead, and anything billed outside the obvious compute line item. "We estimate it saves 20%" is not evidence; a CUR query showing the before/after unit cost is.

Decision rule: tag every resource a pipeline touches with a consistent pipeline identifier, pull actual CUR data filtered by that tag for a full billing period, divide by validated-successful runs in that same period, and never report cost-per-run from an estimate when the billing data to compute it actually exists.

Under the hood

A savingsPlan/ set of columns only appears in the CUR for a billing period if a Savings Plan was actually consumed that period, which means a pipeline's allocated unit cost can shift between months purely because the Savings Plan utilization pattern changed, with zero change to the pipeline's own code or run count — a trap when comparing month-over-month unit cost without checking whether the underlying discount mix moved. The CUR is also delivered as an estimate throughout the month, updated at least daily and cumulative, and only finalized after the invoice closes; comparing a mid-month CUR snapshot for one period against a finalized CUR for an earlier period will show a spurious gap that has nothing to do with the pipeline's actual behaviour.

Minimal example Allocate cost over successful runs only, not every attempt

PythonIncluding failed attempts in the denominator understates true costruns
def cost_per_success(total_billed_cost: float, attempts: int, successful: int) -> dict:
    return {
        "cost_per_attempt": total_billed_cost / attempts,
        "cost_per_success_honest": total_billed_cost / successful,
        "wasted_on_failures": total_billed_cost * (attempts - successful) / attempts,
    }


# 1,000 nightly attempts in a month; 60 failed partway and were retried, 940 succeeded
result = cost_per_success(total_billed_cost=4_700.00, attempts=1_000, successful=940)
for k, v in result.items():
    print(f"{k}: ${v:.2f}")
Output
cost_per_attempt: $4.70
cost_per_success_honest: $5.00
wasted_on_failures: $282.00

In production Allocate billed cost across pipeline stages from tagged CUR-style line items

sqlite3 stands in for querying the actual CUR via Athena; the shape — group by tag, sum by dimension, divide by validated successes pulled from the run log — is identical with real billing data.

PythonCost per successful run, broken down by billable dimensionruns
import sqlite3

db = sqlite3.connect(":memory:")
db.execute("""
    CREATE TABLE cur_lines (
        pipeline_tag TEXT, dimension TEXT, billed_cost REAL, usage_date TEXT
    )
""")
db.execute("CREATE TABLE run_log (pipeline_tag TEXT, run_date TEXT, validated_success INTEGER)")

cur_rows = [
    ("settlements-nightly", "compute", 820.00, "2026-09"),
    ("settlements-nightly", "scan", 140.00, "2026-09"),
    ("settlements-nightly", "storage", 60.00, "2026-09"),
    ("settlements-nightly", "requests", 35.00, "2026-09"),
    ("settlements-nightly", "orchestration", 25.00, "2026-09"),
    ("settlements-nightly", "transfer", 90.00, "2026-09"),
]
db.executemany("INSERT INTO cur_lines VALUES (?, ?, ?, ?)", cur_rows)

# 30 nightly attempts in September; 4 failed validation (retried the next night)
run_rows = [("settlements-nightly", f"2026-09-{d:02d}", 0 if d in (6, 14, 15, 27) else 1) for d in range(1, 31)]
db.executemany("INSERT INTO run_log VALUES (?, ?, ?)", run_rows)

total_cost = db.execute(
    "SELECT SUM(billed_cost) FROM cur_lines WHERE pipeline_tag = ?", ("settlements-nightly",)
).fetchone()[0]
successful_runs = db.execute(
    "SELECT COUNT(*) FROM run_log WHERE pipeline_tag = ? AND validated_success = 1", ("settlements-nightly",)
).fetchone()[0]

print(f"total CUR-tagged cost for September: ${total_cost:.2f}")
print(f"validated successful runs: {successful_runs}")
print(f"cost per successful run: ${total_cost / successful_runs:.2f}")

print("\nbreakdown per successful run, by dimension:")
for dim, cost in db.execute(
    "SELECT dimension, billed_cost FROM cur_lines WHERE pipeline_tag = ? ORDER BY billed_cost DESC",
    ("settlements-nightly",),
):
    print(f"  {dim:<14} ${cost / successful_runs:.2f}")
Output
total CUR-tagged cost for September: $1170.00
validated successful runs: 26
cost per successful run: $45.00

breakdown per successful run, by dimension:
  compute        $31.54
  scan           $5.38
  transfer       $3.46
  storage        $2.31
  requests       $1.35
  orchestration  $0.96

See it happen One month's CUR cost divided down to one successful run

30 nightly attempts26 success4 failed/retriedCUR total, Sept$1,170.00cost / successful run$45.00 (honest)
  1. The run log shows 30 attempts for the month; 4 failed validation and were retried the next night.
  2. The CUR, filtered by this pipeline's tag, shows the total billed across every attempt: $1,170.00, including the 4 failures.
  3. Dividing by all 30 attempts would understate the real cost at $39.00/run.
  4. Dividing by the 26 validated successes gives the honest number: $45.00 per successful run.
Notice: the CUR total doesn't change between the two divisions — only the denominator does, and that's the difference between an honest unit cost and a flattering one.
Scenario A

In your world

The situation

Finance asks "what does the settlements pipeline cost per run?" You compute it the fast way: this month's EC2 bill for the job's instance type, divided by 30 (one run per night).

What actually breaks

Your number is $31/run. The real number, once you pull the CUR for everything tagged to this pipeline and divide by the runs that actually passed validation (not just the ones that exited zero), is $45/run — the fast estimate silently dropped the scan, storage, requests, orchestration and transfer costs, and silently counted four failed-then-retried nights as if they were free.

What you'd do on Monday

Tag every resource the pipeline touches with the same pipeline tag, pull the actual Cost and Usage Report filtered by that tag for the full month, and divide by validated-successful runs from the pipeline's own run log, exactly as the production example does. Report the breakdown by dimension, not just the total, so finance can see which part of the $45 is the lever if they ask "how do we bring this down."

Scenario B

Imagine you work at Blinkit

1.2M order-fulfilment pipeline runs/month across dark-store ETL jobs

The numbers

Illustrative: 1.2M pipeline runs/month across all dark-store sync jobs, with a 2% validation-failure rate requiring same-day retry. That's 24,000 failed-then-retried attempts/month. If each attempt (successful or not) costs an illustrative $0.08 in blended compute, scan and transfer, the 24,000 failed attempts alone cost about $1,920/month in pure waste before counting the retry's own cost on top.

What the Scenario-A answer does here

The Scenario A fix (CUR-filtered, validated-success denominator) still gives the honest unit cost, but at 1.2M runs/month, computing it by hand per pipeline doesn't scale — someone must automate the CUR pull and run-log join, or the honest number only gets computed quarterly.

What flips at this scale

At this volume, cost-per-successful-run becomes a tracked metric with its own dashboard and alert threshold, built from an automated daily CUR-to-run-log join, the same way error rate or latency already is. A unit-cost regression (cost per success creeping up while run count stays flat) becomes something that pages someone, the same way a latency regression would, instead of surfacing three months later in a quarterly review.

The trade-off you'd defend

I would defend building the automated CUR/run-log join once, even though it takes real engineering time upfront, over recomputing unit cost by hand each quarter, trading initial setup cost for continuous visibility that catches a regression in weeks instead of months.

Illustrative figures for the arithmetic; not Blinkit-published.

Gotchas

Dividing by every attempt instead of successful runs hides the real cost

Symptom

A reported "cost per run" number looks healthy, but finance later finds the actual monthly bill is much higher than that number times the run count.

Cause

The denominator counted every attempt, including failures, which lowers the apparent per-run cost even though every attempt billed real money.

Fix

Divide total billed cost by validated-successful runs only, and separately report what the failed attempts cost, as waste, so it isn't hidden inside an averaged-down number.

A mid-month cost comparison fights a moving target

Symptom

Comparing this month's cost-per-run (checked on the 15th) against last month's final number shows a confusing, unexplained swing.

Cause

The CUR is an estimate until the billing period closes; a mid-month pull is necessarily incomplete and not comparable to a finalized prior-month number.

Fix

Compare finalized period to finalized period, or clearly label a mid-month figure as provisional and re-check it after month-end close.

Reporting an estimated saving as if it were a measured one

Symptom

A migration or optimisation is reported as "saves 20%" before and after data actually exists to measure it.

Cause

A pre-migration estimate gets repeated as fact after the fact, without anyone pulling the actual before/after CUR-tagged cost per successful run.

Fix

Hold the "estimated savings" claim separate from a "measured savings" claim, and only report the second once real CUR data from both periods confirms it — this is the distinction FinOps talk track and feedback loop builds the FinOps talk track around.

Tricky questions

If a pipeline's run count stays flat and total cost stays flat, is unit cost definitely stable?
Most people say

Yes, flat total cost over flat run count means nothing changed.

Actually

Not necessarily — if the validation-failure rate crept up (more attempts producing the same number of successes), total cost and run count can both look flat while cost per successful run quietly rises, because more of that flat total is now being spent on failed attempts.

Does tagging a resource after the fact recover its historical cost allocation?
Most people say

Yes, once I tag it, I can see what it cost all along.

Actually

Cost allocation tags only apply going forward from when they're activated and attached; CUR data from before the tag existed on a resource generally can't be retroactively attributed by that tag. Tag pipelines from day one, not when someone first asks for the number.

Scenario question

You're handed two numbers for the same pipeline: "$0.03 per record processed" from the engineering team's spreadsheet, and a Cost Explorer total for the service that implies something closer to $0.11 per record. Which do you trust and how do you reconcile them?

Model answer

I would trust neither number until I've traced what each includes. The spreadsheet figure likely comes from one line item — probably just compute — divided by every attempt rather than successful records, making it look artificially low. The Cost Explorer total is closer to the truth but might include costs not attributable to this pipeline if tagging is incomplete, making it look artificially high. I'd reconcile by pulling the CUR filtered strictly by this pipeline's tag, summing every dimension — compute, scan, storage, requests, orchestration, transfer — and dividing by validated-successful records from the run log for the same period, then report that reconciled number going forward instead of picking a side between two guesses.

Interview corner

CommonHow would you calculate the true cost of one successful pipeline run?
“I'd sum every billable dimension the run touches — compute, scan, storage, requests, orchestration, transfer — across all attempts in the period, including retries and failures, pulled from the actual Cost and Usage Report filtered by the pipeline's cost-allocation tag. Then I'd divide by validated-successful runs only, not every attempt, because dividing by attempts hides the real cost of failures.”
TrickOur cost-per-run number looks great this month — any reason not to trust it immediately?
“I'd check whether it's a finalized CUR period or a mid-month estimate, and whether the denominator is successful runs or every attempt. Both of those can make a number look better than reality without anything actually improving.”
Experience checkTell me about a time an estimated cost turned out to be wrong once you measured it.
“A team estimated a pipeline's per-run cost from just its EC2 line item and reported $31. Once I pulled the full CUR tagged to that pipeline and divided by validated-successful runs, the real number was $45, because scan, storage, requests, orchestration and failed-retry costs were all invisible in the quick estimate.”
DesignHow would you build ongoing visibility into cost-per-successful-run instead of calculating it ad hoc?
“I'd automate a daily join between the CUR, filtered by cost-allocation tag, and the pipeline's own run log for validated successes, publish it as a tracked metric, and alert on a regression the same way I'd alert on error rate or latency, so a creeping unit cost gets caught in weeks, not surfaced three months later in a review.”

Memory hook

A burnt loaf still used the flour and the oven. Divide the real cost by the loaves you can actually sell, not by every attempt.

If you remember only one thing:

Cost per successful run = all billed dimensions across every attempt, divided by validated-successful runs only.

Self-check

  1. Why does dividing total cost by every attempt (not just successes) understate the real unit cost?

    It spreads the real, already-billed cost of failed attempts across a larger denominator, making the reported per-run figure lower than what a successful run actually costs to produce.

  2. Why can't you retroactively recover historical cost allocation by tagging a resource today?

    Cost allocation tags apply from the point they're activated and attached; CUR data from before that point generally can't be attributed by a tag that didn't exist yet.

  3. What's the risk of comparing a mid-month CUR figure to a prior finalized month?

    The mid-month figure is a provisional estimate, not the finalized invoice; comparing it to a finalized prior period can show a swing that reflects incompleteness, not an actual change.

Prove it

Using the sqlite3 production example's schema, add a second pipeline (reconciliation-weekly) with its own CUR line items and run log, including at least one failed run. Write a query that returns cost-per-successful-run for both pipelines side by side, sorted by cost descending. Paste the two-row result.

aws-11-06Card 6 of 6

FinOps talk track and feedback loop

MediumP4 · DifferentiatorTalk about it13 min
finopscost-reviewtalk-trackforecast-vs-invoice

Why you care

Knowing the mechanics in this chapter is only half the job; the other half is being able to walk into a cost review and say something more useful than "AWS is expensive" or "I think we saved some money." A credible FinOps talk track — baseline, ownership, budget, a specific experiment, a reliability guardrail, a measured result, and a review cadence — is what makes a cost conversation sound like engineering instead of guesswork. Interviewers in platform and senior roles ask you to walk through a cost review specifically to check whether you confuse an estimate with a measured result, which is the single most common credibility failure in this space.

Explain like I'm 5

Imagine reporting your household budget to a partner. "I think we'll save money on groceries" is a forecast — a guess about the future. "The grocery bill was ₹200 lower than last month" is an invoice — a measured fact about what already happened. A good FinOps talk track never presents the first as if it were the second. It says: here's what we spent before (baseline), here's whose job it is to watch it (ownership), here's the tripwire if it goes wrong (budget), here's the specific thing we tried (experiment), here's how we made sure nothing broke (guardrail), and here's what actually happened, with real numbers, not a hope.

Core concept

Seven pieces make a cost conversation credible, in the order you'd actually present them:

  1. Baseline: the measured cost (and unit cost) before any change, from actual CUR data for a defined period — never a guess, never "around $X."
  2. Ownership and tags: who is accountable for this cost, enforced by consistent cost-allocation tags so the number can actually be isolated and tracked over time, as covered in Accounts, tagging and environment boundaries.
  3. Budget: a configured AWS Budget with an alert threshold tied to the baseline, as covered in Budgets and controlled lab spend — remembering that the alert is a notification, not an automatic stop, unless a Budget Action is explicitly attached.
  4. A specific utilization or design experiment: one named change (right-size this instance class, add this lifecycle rule, move this traffic off NAT) with a clear hypothesis, not "we're generally optimizing."
  5. A reliability guardrail: what you checked to confirm the change didn't degrade correctness or availability — a right-sizing change that saves money but causes timeouts under peak load is not a win, it's a trade you haven't finished evaluating.
  6. A measured result: the after-state from the same CUR-based measurement as the baseline, over a comparable, finalized period — not a mid-month estimate.
  7. A review cadence: when this gets checked again, because cost drifts, and a one-time win with no follow-up silently erodes.

The one discipline that holds this whole structure together: never let a forecast stand in for an invoice, and never let an estimate stand in for a proven saving. "We expect this to save 20%" is a hypothesis you're about to test, stated plainly as a hypothesis. "This saved 20%, measured against the same CUR-tagged baseline over the following finalized month" is a result. Mixing the two — presenting an estimate with the confidence of a measurement — is the fastest way to lose credibility the first time someone checks the actual invoice.

Decision rule: structure every cost conversation as baseline → ownership → budget → experiment → guardrail → result → cadence, and label every number explicitly as either "estimated" or "measured" — never let the room assume one is the other.

Minimal example Keep forecast and invoice labelled, never merged

PythonA forecast and an invoice are not interchangeable numbersruns
def report_number(label: str, value: float, kind: str) -> str:
    if kind not in ("estimate", "measured"):
        raise ValueError("kind must be 'estimate' or 'measured'")
    tag = "ESTIMATE (forecast, not yet billed)" if kind == "estimate" else "MEASURED (from finalized CUR)"
    return f"{label}: ${value:.2f}  [{tag}]"

print(report_number("expected savings from right-sizing", 1_200.00, "estimate"))
print(report_number("actual savings, August vs July CUR", 940.00, "measured"))

expected, actual = 1_200.00, 940.00
gap_pct = (expected - actual) / expected * 100
print(f"\nestimate overstated the measured result by {gap_pct:.0f}% -- report the measured number, not the estimate")
Output
expected savings from right-sizing: $1200.00  [ESTIMATE (forecast, not yet billed)]
actual savings, August vs July CUR: $940.00  [MEASURED (from finalized CUR)]

estimate overstated the measured result by 22% -- report the measured number, not the estimate

In production Build the seven-part talk track as structured data, not a slide

PythonA FinOps review assembled from its seven required partsruns
from dataclasses import dataclass


@dataclass
class CostReview:
    pipeline: str
    baseline_monthly: float
    owner_team: str
    budget_threshold: float
    experiment: str
    guardrail_checked: str
    measured_monthly: float
    next_review: str

    def summary(self) -> str:
        delta = self.baseline_monthly - self.measured_monthly
        pct = delta / self.baseline_monthly * 100
        lines = [
            f"[{self.pipeline}]",
            f"  baseline (measured, finalized CUR): ${self.baseline_monthly:,.2f}/mo",
            f"  owner: {self.owner_team} (tagged, tracked)",
            f"  budget alert threshold: ${self.budget_threshold:,.2f}/mo",
            f"  experiment: {self.experiment}",
            f"  reliability guardrail checked: {self.guardrail_checked}",
            f"  result (measured, finalized CUR): ${self.measured_monthly:,.2f}/mo ({pct:.0f}% reduction)",
            f"  next review: {self.next_review}",
        ]
        return "\n".join(lines)


review = CostReview(
    pipeline="settlements-nightly",
    baseline_monthly=1_350.00,
    owner_team="data-platform",
    budget_threshold=1_200.00,
    experiment="right-sized EC2 fleet from m5.2xlarge to m5.large after 45d utilization measurement",
    guardrail_checked="p99 job duration and row-count reconciliation unchanged across 14 days post-change",
    measured_monthly=980.00,
    next_review="first week of next quarter",
)

print(review.summary())
Output
[settlements-nightly]
  baseline (measured, finalized CUR): $1,350.00/mo
  owner: data-platform (tagged, tracked)
  budget alert threshold: $1,200.00/mo
  experiment: right-sized EC2 fleet from m5.2xlarge to m5.large after 45d utilization measurement
  reliability guardrail checked: p99 job duration and row-count reconciliation unchanged across 14 days post-change
  result (measured, finalized CUR): $980.00/mo (27% reduction)
  next review: first week of next quarter

See it happen The talk track as a cycle, not a one-time slide

try a named changemeasure, don't estimatebecomes next baselinere-measureBaseline(measured CUR)Experiment +guardrail checkResult(measured CUR)Review cadence
  1. Every cycle starts from a measured baseline, never a guess.
  2. One named experiment runs, with its reliability guardrail checked before anyone calls it a win.
  3. The result is pulled from the same finalized CUR method as the baseline, not from the pre-change estimate.
  4. The review cadence turns this quarter's result into next quarter's starting baseline, so cost drift gets caught again.
  5. Skipping the review step turns a one-time win into a slide nobody re-checks while the number quietly drifts back up.
Notice: the talk track is a loop, not a single slide — the review cadence is what stops a one-time saving from silently eroding.
Scenario A

In your world

The situation

You're asked to present last quarter's cost-saving work to your manager and, two levels up, to finance. You right-sized an EC2 fleet and your pre-change spreadsheet estimated "about 30% savings."

What actually breaks

In the meeting, someone asks "so we saved 30%?" and you only have the pre-change estimate to point to — you never went back to pull the actual finalized CUR for the months before and after the change. The honest answer is "I don't actually know the measured number," which is a much worse answer to give live than if you'd checked beforehand.

What you'd do on Monday

Before the meeting, pull the finalized CUR for the baseline month and the first full finalized month after the change, filtered by the fleet's cost-allocation tag, and compute the actual delta — using the same method as Cost per successful pipeline run's cost-per-run allocation. Walk in with "we estimated 30%, we measured 27%, here's the CUR query," which is a stronger answer than either the estimate alone or an unverified "yes, 30%."

Scenario B

Imagine you work at Udaan

300+ internal cost-tagged workloads under one FinOps review cadence

The numbers

Illustrative: 300 tagged workloads reviewed on a staggered monthly cadence, 25 workloads reviewed each week. If each review takes an engineer 90 minutes when done manually (pulling CUR, checking the run log, writing the summary), that's 25 × 1.5 = 37.5 engineer-hours/week just on review mechanics, before any actual optimisation work happens.

What the Scenario-A answer does here

The Scenario A fix (measure before presenting) still produces the right single-pipeline answer, but at 300 workloads, manually pulling CUR data and writing a seven-part summary for each one doesn't scale — the review cadence either collapses to "whenever someone remembers" or consumes a disproportionate amount of engineering time relative to the savings any single review finds.

What flips at this scale

At this scale the talk track itself gets automated: a scheduled job queries the CUR for every tagged workload, joins it against each pipeline's own success/run log, and generates the baseline/result pair automatically, leaving engineers to supply only the experiment description and the guardrail check — the two parts that genuinely require human judgment. The review cadence becomes a dashboard with an exception list, not 25 manually-written documents a week.

The trade-off you'd defend

I would defend automating the baseline/result pair over scaling the manual review process with more people, trading the upfront cost of building the automated CUR join for a review cadence that's actually sustainable at hundreds of workloads instead of being the first thing that gets skipped under deadline pressure.

Illustrative figures; not Udaan-published.

Gotchas

Presenting an estimate with the confidence of a measurement

Symptom

A stated "we saved X%" claim turns out, under questioning, to be the pre-change forecast, never checked against the actual post-change invoice.

Cause

The forecast was never explicitly re-verified, and nobody distinguished "estimated" from "measured" when the number was first written down, so it drifted into being treated as fact.

Fix

Label every cost figure as estimate or measured at the moment you write it, and don't present a saving claim in a review until the measured figure exists from a finalized CUR period.

A budget alert mistaken for an enforcement mechanism

Symptom

A team reports "we have a budget cap, so overspend isn't possible," and then overspends.

Cause

As covered in Cost engineering, AWS Budgets alerts are notifications by default; without an explicitly configured Budget Action, nothing stops further spending.

Fix

State explicitly in the review whether the budget is alert-only or has an attached enforcement action, so nobody assumes a guarantee that doesn't exist.

A saving that quietly broke something isn't a saving yet

Symptom

A right-sizing or commitment change reduces the bill, but a few weeks later an incident is traced back to that same change.

Cause

The reliability guardrail step was skipped or checked too briefly, so a correctness or availability regression wasn't caught before the "saved money" result was reported.

Fix

Treat the guardrail check as mandatory and time-boxed (for example, two full weeks of unchanged p99 latency and reconciliation accuracy) before reporting a cost change as a confirmed win.

Tricky questions

If this month's bill is lower than last month's, did the cost optimisation work?
Most people say

Yes, lower bill after the change means the change worked.

Actually

Not necessarily — demand could have dropped for unrelated reasons (fewer runs, less data, a seasonal dip), which would lower the bill with or without the optimisation. Compare unit cost (cost per successful unit of work, as in Cost per successful pipeline run), not just the total, before crediting the change.

Is a Cost Explorer graph showing "this month" good enough evidence for a review?
Most people say

Yes, Cost Explorer is the official AWS cost tool, so its numbers are final.

Actually

Cost Explorer and the CUR both show provisional, estimated figures for the current, unfinished billing period; they update daily and are explicitly subject to change until the period closes. Use a finalized prior period for any claim you're presenting as a confirmed result.

Scenario question

You're handed a slide from last quarter's review that says "FinOps initiative saved the company $40,000 this year" with no further detail, and asked to present an update using the same framing. What do you do before building the new slide?

Model answer

I would not reuse that number without finding its basis first. I'd ask whether $40,000 was ever reconciled against actual finalized CUR data, baseline versus result, for the specific changes it claims credit for, or whether it was the sum of pre-change estimates that were never re-verified. If it's the latter, I'd say so plainly rather than compounding an unverified number into this quarter's slide. For the update itself, I'd structure it as the seven-part talk track: for each change made this quarter, show the measured baseline, the owner, the budget, the specific experiment, the guardrail result, the measured outcome, and the next review date — and I'd explicitly mark anything still at the estimate stage as not yet confirmed. The trade-off is that this is a less flattering slide than "$40,000 saved," but it's one I can defend line by line if finance asks for the query behind any single number.

Interview corner

CommonWalk me through how you'd present a cost-saving initiative to leadership.
“I'd structure it as baseline, ownership, budget, the specific experiment, a reliability guardrail I checked, the measured result, and a review cadence — in that order. I'd label every number as estimated or measured, and I would never present an estimate as if it were a confirmed, invoiced saving.”
TrickA dashboard shows this month's AWS spend is down 15% versus last month — can we announce that as a saving?
“Not yet. I'd check whether this month's figure is a finalized CUR period or a provisional, still-updating one, and I'd check whether the drop is tied to a specific change or just lower demand. I'd only announce it as a measured saving once I can tie the delta to a specific, guardrail-checked change over a finalized period.”
Experience checkTell me about a cost result you reported that turned out to be wrong, and what you changed.
“I once presented a pre-change estimate as if it were the actual saving, because I hadn't gone back to check the finalized invoice. The estimate was close but not exact, and someone asked for the query behind the number, which I didn't have. I now never report a saving in a review without the finalized CUR comparison in hand.”
DesignHow would you scale a FinOps review process across hundreds of workloads without burning engineering time?
“I'd automate the mechanical part — the CUR pull, the tag join, the baseline-versus-result comparison — so it runs on a schedule and surfaces as a dashboard with an exception list. I'd keep the judgment parts, naming the experiment and checking the reliability guardrail, as the only manual steps, because those genuinely need a person.”

Memory hook

A forecast is "I think we'll save money." An invoice is "we did." Never let the first one speak for the second.

If you remember only one thing:

Label every cost figure estimate or measured, and never report an estimate as a confirmed saving.

Self-check

  1. What are the seven parts of a credible FinOps talk track, in presentation order?

    Baseline, ownership and tags, budget, a specific experiment, a reliability guardrail, a measured result, and a review cadence.

  2. Why can a lower total bill not be proof that a cost optimisation worked?

    Demand could have dropped for unrelated reasons; comparing unit cost, not just total cost, is needed to credit a specific change.

  3. Why shouldn't a mid-month Cost Explorer figure be presented as a confirmed result?

    It's a provisional estimate that updates daily and is subject to change until the billing period closes; only a finalized period supports a confirmed claim.

Prove it

Using the CostReview dataclass, build two reviews for the same pipeline: one where measured_monthly is only pulled from a mid-month, unfinalized CUR snapshot (label it as such in the experiment or a new field), and one from the finalized month-end CUR. Print both summaries and write one sentence on which one you would actually present to finance and why.