Shelf · AWS · Chapter 4

Chapter 4: S3 & storage mastery

S3 is already in your toolkit, but most ETL engineers who use it are one unexpected version or lifecycle rule away from a cost spike or a silent data loss. This chapter builds the mental model that turns S3 from a remote file system into a service you can design with.

Difficulty
Priority
aws-04-01Card 1 of 6

Object storage semantics

MediumP2 · Very likelyKnow cold12 min
s3object-storageversioningconsistencybuckets
Needs: nothing. Start here.

Why you care

Every ETL developer knows how to put_object and get_object. The gap that costs you in interviews and incidents is the three things S3 is not: it is not a file system, it has no true folders, and a deleted current version is not the same as deleted data when versioning is on. Interviewers use S3 semantics questions to separate engineers who upload to S3 from engineers who can design storage contracts around it. If you cannot explain what "strongly consistent" means in S3 or why listing an empty prefix still costs you API requests, you will lose the round.

Explain like I'm 5

Imagine a giant post office with a single room of numbered mailboxes. The room has no aisles or sections. Every mailbox has a full address printed on it, like raw/2024/01/orders.csv. The slashes are part of the name, not actual dividers. Asking for "everything in raw/2024/" means the clerk reads every label and hands you the matching ones. There are no folders to peek inside. And if you send a second parcel with the same address while versioning is on, both parcels are still in the box; you just get a sticker saying which one arrived last.

Core concept

Buckets are globally-named containers. A bucket lives in exactly one region; S3 does not replicate across regions unless you configure Cross-Region Replication. Bucket names must be globally unique across all AWS accounts.

Keys are the full name of an object. raw/2024/01/orders.csv is one atomic key, not a path through three directories. The slash is a convention that the S3 console uses to fake a folder view, and that Athena and Glue use for partition discovery, but S3 itself knows nothing about it.

Strong consistency (in effect since December 2020): every successful PUT or DELETE is immediately visible to all subsequent GETs and LISTs for that key. There are no eventual-consistency windows to work around for single-object operations. However, listing is not transactional across all keys: two concurrent PUTs may appear in any order in a subsequent LIST.

Versioning adds a VersionId to every PUT. When enabled:

  • A DELETE without a VersionId adds a delete marker; it does not remove the data or stop billing.
  • All noncurrent versions continue to accrue storage charges until a lifecycle rule or an explicit delete_object(VersionId=...) removes them.
  • Versioning cannot be turned off once enabled; it can only be suspended (new PUTs get a null version).

Object operations are atomic: a PUT either succeeds completely or fails. There is no partial write, no append and no in-place edit. To update a 10 MB CSV, you write the whole 10 MB under the same key (or a new key).

Decision rule: use S3 for objects you write once and read many times, or for objects you replace as a whole. Use EFS or a local filesystem for random reads, appends or POSIX semantics (EBS versus EFS versus S3). Link large object transfers to multipart (Multipart and presigned transfers).

Minimal example S3 is a flat key-value store; slashes are part of the key

PythonObject namespace: flat keyspace under a bucketruns
import hashlib

store = {}   # simulates one bucket's namespace: key -> (body, etag)

def put(key, body):
    etag = hashlib.md5(body.encode()).hexdigest()[:8]
    store[key] = (body, etag)

def list_prefix(prefix):
    return sorted(k for k in store if k.startswith(prefix))

put("raw/2024/01/orders.csv", "id,amt\n1,100")
put("raw/2024/01/settlements.csv", "id,settled\n1,true")
put("raw/2024/02/orders.csv", "id,amt\n2,200")
put("curated/orders/year=2024/data.parquet", "<parquet>")

print(list_prefix("raw/"))
print(list_prefix("raw/2024/01/"))

# Deleting one key does NOT remove a placeholder for the prefix
del store["raw/2024/01/orders.csv"]
print(list_prefix("raw/2024/01/"))   # settlements.csv is still there
print("raw/2024/01/ is not a key:", "raw/2024/01/" not in store)
Output
['raw/2024/01/orders.csv', 'raw/2024/01/settlements.csv', 'raw/2024/02/orders.csv']
['raw/2024/01/orders.csv', 'raw/2024/01/settlements.csv']
['raw/2024/01/settlements.csv']
raw/2024/01/ is not a key: True

In production Versioning semantics and delete markers

This code runs on boto3 and a real bucket; it illustrates the four-state version model that trips up ETL jobs that "delete" objects and expect them to be gone.

PythonVersioning state machine (needs boto3 + S3 bucket with versioning enabled)
import boto3

s3 = boto3.client("s3")
BUCKET = "my-etl-bucket"
KEY = "raw/2024/01/orders.csv"

# 1. Upload: creates version V1
v1 = s3.put_object(Bucket=BUCKET, Key=KEY, Body=b"id,amt\n1,100")["VersionId"]

# 2. Overwrite: V1 becomes noncurrent, V2 is current
v2 = s3.put_object(Bucket=BUCKET, Key=KEY, Body=b"id,amt\n2,200")["VersionId"]

# 3. Soft-delete: adds a delete marker; V1 and V2 still exist and still bill
s3.delete_object(Bucket=BUCKET, Key=KEY)

# GET now raises NoSuchKey even though both versions are on disk
try:
    s3.get_object(Bucket=BUCKET, Key=KEY)
except s3.exceptions.NoSuchKey:
    print("Soft-deleted: versions still exist, GET raises NoSuchKey")

# 4. Recover by deleting the delete marker (get its version id first)
versions = s3.list_object_versions(Bucket=BUCKET, Prefix=KEY)
marker_id = versions["DeleteMarkers"][0]["VersionId"]
s3.delete_object(Bucket=BUCKET, Key=KEY, VersionId=marker_id)
print("Recovered:", s3.get_object(Bucket=BUCKET, Key=KEY)["Body"].read())

See it happen Version states for one key

PUTPUT same keyDELETE (no VersionId)No objectV1 currentV2 currentV1 noncurrentDelete markerV1,V2 noncurrent
  1. Before any write: the key does not exist and costs nothing.
  2. First PUT creates V1 as the current version. GET returns V1.
  3. Second PUT makes V2 current and demotes V1 to noncurrent. Both versions are stored and billed.
  4. DELETE without VersionId adds a delete marker. GET now raises NoSuchKey. V1 and V2 are untouched on disk and still billed.
  5. To truly delete: remove the delete marker, then delete each version by VersionId. Only then does billing stop.
Notice: a DELETE on a versioned bucket is almost never what you think. The data, and the bill, survives until you delete by explicit VersionId.
Scenario A

In your world

The situation

Your nightly job writes raw/settlements/{date}/input.csv to S3, then reads it back to process. You turned on versioning last quarter for audit compliance. The job worked for months.

What actually breaks

A runbook typo last week ran the job twice on the same date. The second run overwrote the file. The version from the first run is now noncurrent, still exists, and still costs storage. Worse, the previous quarter had a lifecycle policy that expired objects after 30 days, but that policy targets current versions. The 60 noncurrent versions from six months of double-runs have never been cleared, and your bill has quietly risen.

What you'd do on Monday

Add a lifecycle rule that expires noncurrent versions after N days (N = your audit retention window). Confirm it with a daily CloudWatch metric on NumberOfObjects with StorageType: AllStorageTypes versus StorageType: StandardStorage, and alert if they diverge by more than a threshold. The Monday code in a lifecycle rule JSON:

PythonLifecycle rule body calculator (pure Python, no boto3)runs
import json

def noncurrent_expiry_rule(rule_id, prefix, days):
    return {
        "ID": rule_id,
        "Status": "Enabled",
        "Filter": {"Prefix": prefix},
        "NoncurrentVersionExpiration": {"NoncurrentDays": days},
    }

policy = {"Rules": [noncurrent_expiry_rule("audit-noncurrent", "raw/settlements/", 90)]}
print(json.dumps(policy, indent=2))
Output
{
  "Rules": [
    {
      "ID": "audit-noncurrent",
      "Status": "Enabled",
      "Filter": {
        "Prefix": "raw/settlements/"
      },
      "NoncurrentVersionExpiration": {
        "NoncurrentDays": 90
      }
    }
  ]
}
Scenario B

Imagine you work at Myntra

50M product images, 3 writes per image (original, thumb, watermark)

The numbers

Illustrative: 50M products × 3 versions × ~400 KB average = 60 TB stored. If each product update creates a new version on a versioned bucket without a noncurrent expiry rule, and products are updated monthly, then after the initial upload plus 12 monthly updates (13 versions total: 1 current, 12 noncurrent), you hold 12 × 60 TB = 720 TB of noncurrent objects alongside the 60 TB current set. Storage charges scale linearly with total bytes; noncurrent versions bill at the same rate as current ones.

What the Scenario-A answer does here

The Scenario A fix (lifecycle rule for noncurrent expiry) still applies, but at 50M objects, listing all versions to audit the policy takes hundreds of thousands of LIST calls at $0.005 per 1,000 LIST requests (check current S3 pricing), making ad-hoc audits non-trivial.

What flips at this scale

At this scale the object naming convention becomes a contract enforced at write time. Each image variant gets a deterministic key (products/{id}/{variant}/{timestamp}.jpg), and a lifecycle rule per prefix with a tight noncurrent expiry keeps the version backlog bounded. The LIST cost pushes you to use S3 Inventory (a daily or weekly object manifest delivered as CSV or Parquet) instead of scanning with list_object_versions.

The trade-off you'd defend

I would defend prefix-per-variant with tight noncurrent expiry and S3 Inventory for audits, trading an extra key-design constraint for predictable storage costs. The alternative, a single key per image overwritten in place, is simpler but loses the version history the audit requirement demands.

Illustrative figures for the arithmetic; 3 variants and 400 KB per image are approximations, not Myntra-published values.

Gotchas

Deleting an object on a versioned bucket does not free storage

Symptom

Your delete_object call returns 200 OK and the key disappears from a plain GET, but the S3 storage metric and billing do not drop.

Cause

Without a VersionId, DELETE creates a delete marker. All previous versions remain and continue to accrue charges.

Fix

Add lifecycle rules with NoncurrentVersionExpiration and ExpiredObjectDeleteMarker. For immediate cleanup, list versions and delete each by VersionId.

Prefix listing charges API requests even if the result is empty

Symptom

A cron job that checks whether yesterday's file arrived costs money even on days the file is missing.

Cause

Every list_objects_v2 call is billed as a LIST request regardless of how many objects it returns.

Fix

Use S3 event notifications to receive a PUT event when the file arrives, rather than polling with LIST. If polling is unavoidable, increase the interval.

Concurrent PUTs to the same key with versioning suspended

Symptom

Two parallel job shards write to the same key; one write appears to be silently lost.

Cause

Without versioning, two PUTs to the same key are last-writer-wins. With versioning suspended (not enabled), there is no VersionId and the second write overwrites the first.

Fix

Use unique, deterministic keys per shard (include the shard id or timestamp in the key), or enable versioning and accept both copies, then reconcile.

Tricky questions

Is an S3 bucket with versioning enabled still eventually consistent?
Most people say

Yes, S3 was eventually consistent and versioning does not change that.

Actually

Since December 2020, S3 provides strong read-after-write consistency for all GETs and LISTs, including versioned buckets. The old eventual-consistency window no longer exists for single-object operations.

Does deleting all objects in a bucket reduce the storage bill to zero?
Most people say

Yes, no objects means no storage charge.

Actually

Only if versioning is disabled or you explicitly delete every version and delete marker. On a versioned bucket, delete_object without a VersionId adds a delete marker and leaves every prior version intact and billable.

Scenario question

You're handed an ETL job that writes output/report.csv to an S3 bucket every hour, overwriting the same key each time. A teammate enables versioning on the bucket for a compliance audit. The job still passes, but the storage bill triples over the next month. Why, and what do you change?

Model answer

Before versioning, each overwrite replaced the previous object; only the current version was billed. After versioning, each overwrite creates a new current version and demotes the previous one to noncurrent; 24 versions per day × 30 days = 720 versions accumulate, and all of them bill. I would add a lifecycle rule with NoncurrentVersionExpiration set to however many days the audit requires (say, 30 days), which caps the backlog at 720 × (retention / 30) versions. I would also add ExpiredObjectDeleteMarker: true to clean up orphaned markers. The trade-off is that versions older than the retention window are permanently deleted; if the compliance requirement is to keep them forever, I would move them to Glacier instead of expiring them.

Interview corner

CommonWhat is the difference between a bucket and an object in S3?
“I'd say a bucket is a globally-named container tied to one region, and an object is any byte sequence stored under a key inside it. The key is just a string; slashes are a naming convention, not real directory structure.”
TrickYou delete an object in a versioned bucket and the GET starts returning NoSuchKey. Is the data gone?
“I'd say no. Without a VersionId, DELETE adds a delete marker. The previous versions are still on disk and still billing. To actually remove the data I'd have to list versions and delete each by VersionId, or have a lifecycle rule do it.”
CommonWhat does strong consistency mean for S3 since December 2020?
“I'd say a successful PUT or DELETE is immediately visible to all subsequent GETs and LISTs for that key. I no longer have to add retry loops or sleeps to work around an eventual-consistency window.”
Experience checkWhere has S3 versioning surprised you in real work?
“On a project where we enabled versioning for compliance, the noncurrent versions from daily overwrites accumulated unnoticed for two months. The fix was a lifecycle rule to expire noncurrents after the audit window and S3 Inventory for ongoing audits instead of scanning with list_object_versions.”

Memory hook

S3 buckets are post offices with one flat room; slashes are labels, not aisles.

If you remember only one thing:

A DELETE on a versioned bucket adds a marker; it does not delete data or stop billing.

Self-check

  1. Why does a GET return NoSuchKey after a DELETE on a versioned bucket, even though the object is still stored?

    The DELETE added a delete marker, which becomes the current version and makes GET return NoSuchKey. The previous versions are still stored.

  2. What is the correct test to tell if versioning is in the way of your delete?

    Check if NumberOfObjects drops after delete_object. If not, list versions for that key; if you see versions or delete markers, the data is still there.

  3. Can you turn off versioning on a bucket that has it enabled?

    No. You can suspend it, which stops creating new versions, but the existing versions remain and new PUTs get a null VersionId.

Prove it

Create a Python dict-based VersionedBucket class that tracks the full version history (each PUT assigns a new VersionId, a DELETE without VersionId adds a delete marker, a GET returns NoSuchKey when the current version is a marker). Test: put V1, overwrite with V2, delete (marker), confirm GET raises, restore by removing the marker, confirm GET returns V2, then delete V2 by VersionId and confirm GET returns V1. Paste the test output.

aws-04-02Card 2 of 6

Storage classes and lifecycle design

MediumP3 · Role-dependentTalk about it13 min
s3storage-classeslifecycleglaciercost

Why you care

A senior backend engineer is expected to justify storage class choices in a design review. The wrong answer, "just use Standard", passes the interview of a junior but fails the systems-design round at product companies, where an interviewer will push on what happens to that 2 TB audit archive in three years. More immediately, an ETL job that writes curated Parquet to Standard when the data will never be read after 90 days is burning money every month until someone notices. Interviewers ask about lifecycle design because it predicts whether you think about the cost of your systems, not just the correctness.

Explain like I'm 5

Think of storage class as the difference between keeping a document on your desk (Standard), in a filing cabinet you open twice a month (Infrequent Access), in an archive box in the basement (Glacier Instant), in a deep-freeze warehouse with same-day shipping (Glacier Flexible Retrieval), and in a vault you access once a year (Glacier Deep Archive). Moving a document to a colder tier costs a small fee, but you pay rent forever, so choosing the right tier early saves money every day.

Core concept

AWS S3 has seven storage classes. Know the access pattern each is designed for:

  • S3 Standard: frequent access, no retrieval fee, no minimum storage duration. Use for active ETL inputs and outputs.
  • S3 Standard-IA (Infrequent Access): less than once a month. Minimum storage duration: 30 days. Retrieval fee per GB. Do not put here unless you are confident access is rare; one unexpected re-read can cost more than Standard would have.
  • S3 One Zone-IA: same as Standard-IA but stored in a single AZ. Lower price, but data is lost if that AZ is destroyed. Suitable for reproducible data (re-derivable from a source) only.
  • S3 Glacier Instant Retrieval: quarterly access, millisecond restore. Minimum storage duration: 90 days. The right class for compliance archives you may need to access quickly but rarely do.
  • S3 Glacier Flexible Retrieval: annual access, restore in minutes to hours (Expedited: 1–5 minutes; Standard: 3–5 hours; Bulk: 5–12 hours). Minimum storage duration: 90 days. Check current pricing page for retrieval fees per GB and per-request charges.
  • S3 Glacier Deep Archive: seven-year compliance archives, restore in 12–48 hours. Minimum storage duration: 180 days. Lowest storage price; highest restore latency and cost.
  • S3 Intelligent-Tiering: automatic tier movement based on access frequency; a per-object monitoring charge applies (check current pricing). Use when access patterns are unknown or variable and the monitoring charge is smaller than the expected savings.

Lifecycle rules automate tier transitions and expiration. Key mechanics:

  • Transitions have a minimum days-after-creation value (e.g., Standard → Standard-IA requires at least 30 days).
  • Expiration deletes current versions; NoncurrentVersionExpiration deletes noncurrent ones (Object storage semantics).
  • A rule with only an expiration and no filter applies to the entire bucket.
  • The minimum storage duration billing still applies even if you delete an object early: a file moved to Glacier Instant Retrieval on day 5 and deleted on day 10 still bills for 90 days.

Decision rule: Standard for active data. Standard-IA for data accessed less than once per month with no re-derivability guarantee. Glacier Instant for compliance archives with unpredictable but rare access. Glacier Flexible or Deep Archive for long-term retention where you can wait hours. Intelligent-Tiering for mixed patterns where monitoring cost is worth it.

Minimal example Lifecycle cost simulator with illustrative unit prices

PythonDays-before-transition vs effective cost per GBruns
# Illustrative prices to show the arithmetic; check current AWS pricing page
# before relying on these for budget decisions.

CLASSES = {
    "standard":        {"storage_per_gb_month": 0.023, "retrieval_per_gb": 0.0,   "min_days": 0},
    "standard_ia":     {"storage_per_gb_month": 0.0125,"retrieval_per_gb": 0.01,  "min_days": 30},
    "glacier_instant": {"storage_per_gb_month": 0.004, "retrieval_per_gb": 0.03,  "min_days": 90},
    "glacier_flex":    {"storage_per_gb_month": 0.0036,"retrieval_per_gb": 0.01,  "min_days": 90},
    "deep_archive":    {"storage_per_gb_month": 0.00099,"retrieval_per_gb":0.02,  "min_days": 180},
}

def cost_over_months(cls, size_gb, months, retrievals_per_month=0):
    c = CLASSES[cls]
    min_months = c["min_days"] / 30
    billed_months = max(months, min_months)
    storage = c["storage_per_gb_month"] * size_gb * billed_months
    retrieval = c["retrieval_per_gb"] * size_gb * retrievals_per_month * months
    return round(storage + retrieval, 4)

# 1 TB audit archive, kept 12 months, retrieved once
gb = 1024
months = 12
for cls in CLASSES:
    total = cost_over_months(cls, gb, months, retrievals_per_month=1/12)
    print(f"{cls:<20} ${total:,.2f}")
Output
standard             $282.62
standard_ia          $163.84
glacier_instant      $79.87
glacier_flex         $54.48
deep_archive         $32.65

In production A lifecycle policy for a three-tier ETL bucket

This is the JSON you would PUT to a bucket's lifecycle configuration; presented here as a Python dict so it is runnable and shows the structure.

PythonLifecycle rule set as Python dict (serialises to valid lifecycle JSON)runs
import json

rules = [
    # Active ETL inputs: transition to IA after 30 days, expire after 365
    {
        "ID": "raw-zone-lifecycle",
        "Status": "Enabled",
        "Filter": {"Prefix": "raw/"},
        "Transitions": [
            {"Days": 30, "StorageClass": "STANDARD_IA"},
            {"Days": 90, "StorageClass": "GLACIER_INSTANT_RETRIEVAL"},
        ],
        "Expiration": {"Days": 365},
        "NoncurrentVersionExpiration": {"NoncurrentDays": 30},
    },
    # Curated Parquet: stays Standard for Athena; expire after 3 years
    {
        "ID": "curated-zone-lifecycle",
        "Status": "Enabled",
        "Filter": {"Prefix": "curated/"},
        "Expiration": {"Days": 1095},
        "NoncurrentVersionExpiration": {"NoncurrentDays": 7},
    },
    # Compliance archive: move to Deep Archive day 1, keep 7 years
    {
        "ID": "archive-lifecycle",
        "Status": "Enabled",
        "Filter": {"Prefix": "archive/"},
        "Transitions": [
            {"Days": 1, "StorageClass": "DEEP_ARCHIVE"},
        ],
        "Expiration": {"Days": 2555},  # ~7 years
    },
]

for r in rules:
    trans = [f"{t['Days']}d->{t['StorageClass']}" for t in r.get("Transitions", [])]
    exp = r.get("Expiration", {}).get("Days", "none")
    nc = r.get("NoncurrentVersionExpiration", {}).get("NoncurrentDays", "none")
    print(f"{r['ID']}: transitions={trans} expire={exp}d nc_expire={nc}d")
print(f"total rules: {len(rules)}")
Output
raw-zone-lifecycle: transitions=['30d->STANDARD_IA', '90d->GLACIER_INSTANT_RETRIEVAL'] expire=365d nc_expire=30d
curated-zone-lifecycle: transitions=[] expire=1095d nc_expire=7d
archive-lifecycle: transitions=['1d->DEEP_ARCHIVE'] expire=2555d nc_expire=noned
total rules: 3

See it happen Lifecycle transition flow

≥30d≥90d≥180dday 1 directStandard(active)Standard-IA(infrequent)Glacier Instant(rare)Deep Archive(years)
  1. New object lands in Standard. No retrieval fee, no minimum duration. Any GET costs nothing extra.
  2. After 30 days, a lifecycle rule can move it to Standard-IA. The 30-day minimum billing starts from the move date, not the original upload.
  3. After 90 days total, Glacier Instant gives millisecond restore but adds a per-GB retrieval fee. Good for compliance archives you might need urgently.
  4. Deep Archive at 180+ days or day-1 direct (e4). Lowest storage price; 12-48h restore. Any object deleted before its minimum duration still bills for the full minimum.
Notice: transitions are one-way. You cannot move an object back to a warmer tier via lifecycle; restore Glacier objects to Standard for active re-use, but they return to Glacier unless you explicitly PUT to Standard.
Scenario A

In your world

The situation

Your team writes 50 GB of Parquet files to curated/ each month. The files are queried heavily for the first two weeks, then almost never. No lifecycle rule is set. After 18 months, S3 Standard storage is a significant line item in your AWS bill.

What actually breaks

You add a blanket lifecycle rule transitioning everything in curated/ to Standard-IA after 30 days, expecting immediate savings, without accounting for retrieval fees. A quarterly compliance report re-reads the full 18-month accumulation (900 GB) once a quarter. At Standard-IA's $0.01/GB retrieval fee, that single report run costs $9 on top of storage, a charge that never existed when the data sat in Standard, where GETs are free. Nobody traces it for two quarters, since $9 looks like a rounding error against the total bill, until someone reruns the report weekly for a board deck and the retrieval charges alone reach $36/month, quietly erasing the storage savings the transition was supposed to deliver.

What you'd do on Monday

Scope the lifecycle rule with the ObjectSizeGreaterThan filter so objects under 128 KB never transition, since IA's minimum billable size makes them cost more, not less, in IA. More importantly, split any data you know will be re-read on a schedule onto its own prefix and keep it in Standard, so a report that runs weekly is never sitting in IA to begin with. Measure with S3 Storage Lens or a Cost Explorer filter on the bucket to confirm both the storage-class distribution and the retrieval-fee line item move as expected.

Scenario B

Imagine you work at Pinterest

5 billion pins, write-once read-rarely after 1 year

The numbers

Illustrative: 5B pins × 100 KB average thumbnail = 500 TB. If kept in Standard forever: 500 TB × $0.023/GB/month × 12 months ≈ $141,000/year (illustrative, at the unit price above). Moving thumbnails older than 365 days to Deep Archive ($0.00099/GB/month): 500 TB × $0.00099 × 12 ≈ $6,100/year for the cold tier. That is a ~23× storage-cost reduction for data that is never read in steady state.

What the Scenario-A answer does here

The Scenario A fix (lifecycle rule) still applies, but at 5B objects the rule must be bucketed by creation date, not just prefix, because new pins and old pins share the same prefix pattern. A single Transitions rule on the whole bucket moves everything, including fresh pins.

What flips at this scale

At this scale you design prefix structure as a lifecycle primitive. Pins are written to pins/{year}/{month}/, giving the lifecycle rule date-based precision without scanning object metadata. A transition rule per month-prefix moves objects in discrete cohorts, and S3 Inventory verifies the distribution monthly.

The trade-off you'd defend

I would defend year/month prefix partitioning as both a lifecycle primitive and an Athena partition key, accepting slightly longer keys and a fixed prefix pattern, for clean lifecycle cohorts and no cross-shard scan cost. Intelligent-Tiering is an alternative but the per-object monitoring charge on 5B objects exceeds the savings at low storage prices.

Illustrative unit prices and sizes; not Pinterest-published figures. Check current S3 pricing page before any budget decision.

Gotchas

Early deletion still bills the minimum storage duration

Symptom

You move a file to Glacier Deep Archive on day 5 and delete it on day 10. You still see a charge for 180 days of storage.

Cause

Each Glacier storage class has a minimum billable duration regardless of when you delete.

Fix

Only use Glacier for data you will keep longer than its minimum duration. Calculate break-even: if you are uncertain about retention, Standard-IA's 30-day minimum is cheaper for short-lived data.

Transition to IA on objects smaller than 128 KB is not cost-effective

Symptom

After adding a Standard-IA lifecycle rule, your per-request charges increase and overall bill barely changes.

Cause

Standard-IA has a minimum billable object size of 128 KB. Small objects (metadata, config files) are billed as 128 KB regardless of actual size, and the per-request overhead can exceed the storage savings.

Fix

Add an ObjectSizeGreaterThan filter to your lifecycle rule so only large objects transition. Leave small objects in Standard.

A lifecycle rule without a filter applies to the whole bucket, including versioned objects

Symptom

Compliance data under archive/ starts disappearing on schedule, but so do active ETL files that share the bucket.

Cause

A lifecycle Expiration rule without a filter matches every object.

Fix

Always specify a Prefix or tag filter on every lifecycle rule. Bucket-wide rules are rarely intentional.

Tricky questions

Is Standard-IA always cheaper than Standard for data kept 6 months?
Most people say

Yes, Standard-IA costs less per GB-month, so it must be cheaper over any retention period.

Actually

Only if retrievals are infrequent. Standard-IA charges a per-GB retrieval fee. If your Athena queries scan the data monthly, the retrieval fees can exceed the storage savings. The break-even depends on your read pattern, not just retention length.

Can you transition objects directly from Standard to Glacier Deep Archive on day 1?
Most people say

No, you have to go through Standard-IA first.

Actually

You can transition directly to any colder class including Glacier Deep Archive on day 1 (or any day). The multi-hop requirement only existed for older legacy rules. Check the current S3 documentation for the exact transition waterfall constraints, as they have changed over time.

Scenario question

You're handed a data lake bucket with 200 TB in Standard storage, three years of raw CSV files, and no lifecycle rules. The team wants to cut storage costs without losing the ability to run ad-hoc Athena queries on recent data (< 90 days) and infrequent re-runs on older data.

Model answer

I would define three lifecycle tiers by prefix and age. Raw files less than 90 days stay in Standard for full-speed Athena access. Files 90–365 days old transition to Standard-IA: Athena can still read them with a retrieval fee, but that fee is worth it for occasional ad-hoc queries. Files older than 365 days transition to Glacier Instant Retrieval, which Athena cannot query directly but can be restored to Standard in milliseconds if you need a re-run. The trade-off is that Glacier Instant has a 90-day minimum and a retrieval fee, so if older files end up being queried more often than expected, the retrieval charges will erode the savings. I would add S3 Storage Lens to monitor the class distribution and actual retrieval frequency for a quarter before committing to the Glacier tier.

Interview corner

CommonWhich S3 storage class would you use for audit logs kept 7 years but never read?
“I'd use Glacier Deep Archive: it's the cheapest per GB-month and the 12–48 hour restore window is fine for a once-per-decade audit. I'd make sure the data stays longer than the 180-day minimum to avoid early deletion charges.”
TrickWhat happens to cost if you add a Standard-IA lifecycle rule for all objects, including 1 KB metadata files?
“I'd expect cost to go up, not down. Standard-IA has a 128 KB minimum billable size, so every 1 KB file is billed as 128 KB, and there's a per-request retrieval fee on every access. Small files should stay in Standard.”
DesignHow would you design a three-year data lake lifecycle for a read-heavy first 90 days, rare after?
“I'd partition by year/month in the prefix so lifecycle rules land on date cohorts. Standard for the first 90 days, Standard-IA from 90 to 365, Glacier Instant from 365 to three years so re-runs are possible without a multi-hour wait. I'd verify with S3 Storage Lens monthly.”

Memory hook

Storage classes are a rent-versus-access-cost trade-off: colder is cheaper per month but charges a toll when you retrieve.

If you remember only one thing:

Never put data in IA or Glacier without checking both the minimum storage duration and the retrieval fee first.

Self-check

  1. What is the minimum storage duration for S3 Glacier Instant Retrieval?

    90 days. Deleting an object earlier still bills for the full 90 days.

  2. Why do small objects cost more per GB in Standard-IA than in Standard?

    Standard-IA has a minimum billable object size of 128 KB. An object smaller than 128 KB is billed as 128 KB, inflating the effective per-GB rate.

  3. A lifecycle rule expires objects after 30 days. The bucket has versioning enabled. What happens to the noncurrent versions?

    They are not affected unless the rule also has a NoncurrentVersionExpiration element. A plain Expiration rule only removes the current version and adds a delete marker.

Prove it

Using the cost model in the minimal example (with the disclaimer that prices are illustrative), calculate the crossover point in days where Glacier Instant Retrieval becomes cheaper than Standard-IA for a 1 GB object with zero retrievals. Vary minimum storage billing and storage price; print a table with days-kept from 30 to 180 in steps of 30.

aws-04-03Card 3 of 6

Multipart and presigned transfers

MediumP3 · Role-dependentKnow cold13 min
s3multipart-uploadpresigned-urllarge-objectstransfers

Why you care

Your Jenkins ETL job calls put_object and it works fine for 100 MB files. The day the vendor sends a 6 GB file, it fails halfway through with a network hiccup and you restart from byte zero, again. Multipart upload is the answer to that failure mode, and presigned URLs are how you let an untrusted client upload directly to S3 without ever touching your server. Both come up in system-design interviews when you whiteboard a file-upload or ETL ingest service. The cold-level tag means you need to be able to code the basic multipart split and explain presigned URL mechanics without looking them up.

Explain like I'm 5

Multipart upload is like sending a large parcel in separate boxes. You open a shipment (initiate), pack each box (upload part), and tell the warehouse to assemble them (complete). If one box is lost in transit, you re-send that box only. A presigned URL is a time-limited access code: you write the code on the parcel label before handing it to the customer, and the warehouse accepts the delivery without asking who the customer is, as long as the code has not expired.

Core concept

Multipart upload has three phases:

  1. Initiate (create_multipart_upload): returns an UploadId. No data is stored yet.
  2. Upload parts (upload_part): each part except the last must be at least 5 MB. Parts are numbered 1–10,000. Parts can be uploaded in parallel; the service stores them under the UploadId.
  3. Complete (complete_multipart_upload): you send a manifest of part numbers and ETags; S3 assembles the object. Only then does the object appear and billing for the assembled object begins.

If you never complete (network failure, crashed process), the parts continue to accrue storage charges indefinitely under the UploadId. This is the abandoned upload cost trap. Fix it with a lifecycle rule: AbortIncompleteMultipartUploads with DaysAfterInitiation: N.

Maximum object size: 5 TB. Maximum part count: 10,000. Minimum part size (except last): 5 MB.

Presigned URLs let any HTTP client perform a specific S3 operation (GET or PUT) using credentials embedded in the URL signature. Key rules:

  • A presigned URL for get_object lets anyone who has the URL download the object. It expires at the ExpiresIn time.
  • A presigned URL for put_object lets an untrusted client upload directly to S3. Your server never receives the bytes; S3 does.
  • Maximum expiry when signing with long-term IAM credentials: 7 days (604,800 seconds).
  • When signing with temporary STS credentials (role assume, instance profile), the presigned URL expires at whichever comes first: the ExpiresIn you specified or the credential expiry. A URL signed by a role with a 1-hour session and an ExpiresIn of 24 hours expires in 1 hour.
  • The URL embeds the signing credentials; rotate the IAM key and all outstanding presigned URLs signed by that key become invalid immediately.

Decision rule: use put_object for objects under 100 MB. Use multipart above 100 MB. Use managed transfer utilities (TransferManager in boto3) for 5 MB–5 TB objects when you want automatic parallelism and retry logic. Use presigned URLs to let browsers or mobile clients upload/download without proxying through your servers.

Minimal example Chunking a file for multipart upload

PythonCompute multipart chunk plan for any file sizeruns
import math

def multipart_plan(file_size_bytes, part_size_bytes=8 * 1024 * 1024):
    """
    Returns a list of (part_number, offset, length) tuples.
    Minimum part size (except last): 5 MB.
    Maximum parts: 10000.
    """
    MIN_PART = 5 * 1024 * 1024   # 5 MB documented AWS minimum
    MAX_PARTS = 10_000
    if part_size_bytes < MIN_PART:
        raise ValueError(f"part_size must be >= {MIN_PART} bytes (5 MB)")
    n_parts = math.ceil(file_size_bytes / part_size_bytes)
    if n_parts > MAX_PARTS:
        raise ValueError(f"{n_parts} parts exceeds maximum {MAX_PARTS}")
    parts = []
    for i in range(n_parts):
        offset = i * part_size_bytes
        length = min(part_size_bytes, file_size_bytes - offset)
        parts.append((i + 1, offset, length))
    return parts

# 50 MB file with default 8 MB parts
plan = multipart_plan(50 * 1024 * 1024)
print(f"parts: {len(plan)}")
for pnum, off, length in plan:
    print(f"  part {pnum}: offset {off:>10,}  length {length:>9,}")

# Edge: 4.9 MB file → 1 part (under minimum threshold → use put_object instead)
small = multipart_plan(4_900_000)
print(f"small file parts: {len(small)}, single part length: {small[0][2]:,}")
Output
parts: 7
  part 1: offset          0  length 8,388,608
  part 2: offset  8,388,608  length 8,388,608
  part 3: offset 16,777,216  length 8,388,608
  part 4: offset 25,165,824  length 8,388,608
  part 5: offset 33,554,432  length 8,388,608
  part 6: offset 41,943,040  length 8,388,608
  part 7: offset 50,331,648  length 2,097,152
small file parts: 1, single part length: 4,900,000

In production Resumable multipart upload pattern

Parts that successfully uploaded are recorded in a checkpoint; on restart only the missing parts are re-sent.

PythonResumable multipart upload with checkpoint (needs boto3 + S3 bucket)
import json
import math
from pathlib import Path

import boto3

BUCKET = "my-etl-bucket"
PART_SIZE = 8 * 1024 * 1024   # 8 MB
CHECKPOINT = Path("upload_checkpoint.json")


def load_checkpoint():
    if CHECKPOINT.exists():
        return json.loads(CHECKPOINT.read_text())
    return {}


def save_checkpoint(state):
    CHECKPOINT.write_text(json.dumps(state))


def upload_large_file(local_path, s3_key):
    s3 = boto3.client("s3")
    size = Path(local_path).stat().st_size
    ckpt = load_checkpoint()

    upload_id = ckpt.get("upload_id")
    if not upload_id:
        upload_id = s3.create_multipart_upload(Bucket=BUCKET, Key=s3_key)["UploadId"]
        ckpt = {"upload_id": upload_id, "completed_parts": {}}
        save_checkpoint(ckpt)

    completed = ckpt["completed_parts"]
    parts_manifest = []
    n_parts = math.ceil(size / PART_SIZE)

    with open(local_path, "rb") as f:
        for pnum in range(1, n_parts + 1):
            if str(pnum) in completed:
                parts_manifest.append({"PartNumber": pnum, "ETag": completed[str(pnum)]})
                continue
            offset = (pnum - 1) * PART_SIZE
            f.seek(offset)
            data = f.read(PART_SIZE)
            etag = s3.upload_part(
                Bucket=BUCKET, Key=s3_key, UploadId=upload_id,
                PartNumber=pnum, Body=data
            )["ETag"]
            completed[str(pnum)] = etag
            parts_manifest.append({"PartNumber": pnum, "ETag": etag})
            save_checkpoint({"upload_id": upload_id, "completed_parts": completed})
            print(f"part {pnum}/{n_parts} done")

    s3.complete_multipart_upload(
        Bucket=BUCKET, Key=s3_key, UploadId=upload_id,
        MultipartUpload={"Parts": parts_manifest}
    )
    CHECKPOINT.unlink(missing_ok=True)
    print(f"Upload complete: s3://{BUCKET}/{s3_key}")

See it happen Multipart upload lifecycle

ETL jobS3Incomplete parts storagecreate_multipart_upload(key)UploadIdupload_part(UploadId, part=1, data)store part 1upload_part(UploadId, part=2…7)complete_multipart_upload(manifest)200 OK, final ETag(crash before complete)
  1. Initiate: S3 returns an UploadId. No data stored yet, no charges yet.
  2. The job stores the UploadId in a checkpoint file so a restart can resume.
  3. Part 1 uploaded. Minimum 5 MB except the last part; maximum 10,000 parts.
  4. Parts are stored separately under the UploadId. Not yet visible as an object.
  5. Remaining parts sent, possibly in parallel. Each returns an ETag.
  6. Job sends part numbers + ETags. S3 assembles and creates the final object.
  7. Object is now visible to GETs. Parts storage is freed. Billing starts for the assembled object.
  8. If the job crashes here, parts linger indefinitely. A lifecycle AbortIncompleteMultipartUploads rule is the only automatic cleanup.
Notice: storage charges for uploaded parts begin immediately after each upload_part. If you never call complete_multipart_upload, you are billed indefinitely for invisible data.
Scenario A

In your world

The situation

Your Jenkins job uploads a 2 GB monthly dump using put_object with a 10-minute Jenkins timeout. The network link to S3 is stable, but last month a brief VPC flap at minute 8 killed the upload. The job shows failed, restarts from zero, and misses the SLA window.

What actually breaks

put_object is a single HTTP PUT. Any failure at any point requires a full restart. A 2 GB upload on a 40 Mbps upload link takes about 400 seconds; the 10-minute timeout is marginal. More importantly, there is no resumability: you always restart from byte zero.

What you'd do on Monday

Switch to multipart upload with a checkpoint file. Divide the 2 GB into 8 MB parts (250 parts). Each part is a separate HTTP call. On failure, only the incomplete part needs to be retried. Also add the AbortIncompleteMultipartUploads lifecycle rule to the bucket immediately:

PythonCompute retry savings for multipart vs single-PUTruns
def restart_cost(file_gb, part_mb, failure_at_fraction):
    """
    How much data must be re-uploaded on a restart?
    Single PUT: always the whole file.
    Multipart: the current part (at most one part_mb).
    """
    file_mb = file_gb * 1024
    single_put_retry_mb = file_mb
    # worst case: failure at start of last-but-one part
    parts_done = int(failure_at_fraction * file_mb / part_mb)
    multipart_retry_mb = file_mb - parts_done * part_mb
    return single_put_retry_mb, multipart_retry_mb

single, multi = restart_cost(2, 8, failure_at_fraction=0.95)
print(f"Single-PUT restart: re-upload {single:,} MB")
print(f"Multipart restart:  re-upload at most {multi:,} MB")
print(f"Savings: {single - multi:,} MB ({(single-multi)/single*100:.0f}%)")
Output
Single-PUT restart: re-upload 2,048 MB
Multipart restart:  re-upload at most 104 MB
Savings: 1,944 MB (95%)
Scenario B

Imagine you work at Canva

10 billion asset uploads per year

The numbers

Illustrative: 10B uploads/year ÷ 365 days ÷ 86,400 s ≈ 317 uploads/second. Average asset is ~2 MB (mixed images, templates, videos). At 317 uploads/s × ~2 MB = ~634 MB/s aggregate ingest. Assets under ~8 MB use a single PUT; assets above use multipart. About 20% of assets are videos over 100 MB; that is 63 large-file uploads/second concurrently.

What the Scenario-A answer does here

The Scenario A fix (multipart with checkpoint) works per upload, but at 63 concurrent large-file uploads, abandoned upload cleanup becomes critical. If 1% of large uploads are abandoned (network drops, user cancels), that is ~0.63 abandoned initiations per second, or ~54,000 per day, each holding parts indefinitely without a lifecycle rule.

What flips at this scale

At this scale, the abandoned-upload lifecycle rule is not optional hygiene; it is a line item in your cost model. You set AbortIncompleteMultipartUploads: DaysAfterInitiation: 1 at minimum, and you add a CloudWatch metric alarm on S3 - NumberOfMultipartUploadParts to detect anomalies. Upload sessions get server-side tracking so user cancellations trigger an explicit abort_multipart_upload call rather than waiting for the lifecycle rule.

The trade-off you'd defend

I would defend explicit server-side abort on cancellation plus a 1-day lifecycle fallback, accepting an extra API call per cancelled upload, for storage cost that stays predictable at high volume.

Illustrative figures for the arithmetic; not Canva-published upload metrics.

Gotchas

Abandoned multipart uploads silently accrue charges

Symptom

S3 storage costs rise gradually without a matching increase in visible object count.

Cause

Every started but never-completed multipart upload stores its parts indefinitely. They do not appear in a normal list_objects_v2; you need list_multipart_uploads.

Fix

Add AbortIncompleteMultipartUploads lifecycle rule with DaysAfterInitiation: 1 (or whatever threshold fits your max upload duration). This is defensive from day one, not a fix after the fact.

Presigned URL expiry shorter than the credential lifetime

Symptom

A presigned URL for a large upload starts failing before the user finishes, even though the ExpiresIn was set to 24 hours.

Cause

The signing credentials (IAM role session) expire in 1 hour. Presigned URLs signed by STS credentials expire at min(ExpiresIn, credential_lifetime).

Fix

Sign presigned URLs with a long-lived IAM user key if you need expiries up to 7 days. If using roles (recommended for security), design workflows so the upload completes within the credential lifetime, or refresh presigned URLs.

Part size below 5 MB (except the last part) causes CompleteMultipartUpload to fail

Symptom

complete_multipart_upload raises EntityTooSmall.

Cause

AWS enforces a 5 MB minimum on all parts except the last.

Fix

Choose a part size of at least 5 MB. The production example uses 8 MB, which balances parallel efficiency and retry granularity.

Tricky questions

Can you upload parts in any order and any number of threads?
Most people say

No, parts must be uploaded sequentially; S3 needs them in order.

Actually

Parts can be uploaded in any order and fully in parallel. The assembly order is specified in the complete_multipart_upload manifest by part number, not upload order. Parallel part uploads are exactly the performance benefit of multipart.

Does a presigned PUT URL let the client upload arbitrary content to arbitrary keys?
Most people say

Yes, the URL is a general upload grant.

Actually

No. A presigned PUT URL is scoped to one specific key. The client can upload only that key, only with the method the URL was signed for (GET or PUT), and only before expiry. You can add conditions at signing time (e.g., maximum file size via a presigned POST with Content-Length-Range).

Scenario question

You're handed a mobile app that lets users upload videos up to 1 GB directly to your backend server, which then copies them to S3. At 10,000 active users, the server becomes a bottleneck and your egress doubles. Redesign the upload path.

Model answer

I would remove the server from the upload path. The backend generates a presigned PUT URL scoped to a specific key (like uploads/{user_id}/{uuid}.mp4) with a 15-minute expiry (well within even the shortest STS session). The mobile client uploads directly to S3. The backend never receives the bytes. For videos this large I would use a presigned multipart upload: the backend initiates the upload, pre-signs each part URL, and sends the list to the client; the client uploads parts in parallel and signals the server to call complete_multipart_upload. The trade-off is added round trips for the presigning ceremony, but upload throughput is now S3's problem, not mine. I would add an S3 event notification to trigger processing once the upload is complete, and set AbortIncompleteMultipartUploads: 1 day on the bucket from day one.

Interview corner

CommonWhy use multipart upload instead of a single PUT for large files?
“I'd say three reasons: on a network failure you only re-upload one part instead of the whole file; you can upload parts in parallel and saturate the network; and the 5 GB single-PUT limit means you must use multipart for objects over 5 GB anyway.”
TrickWhat happens to parts from a multipart upload that was never completed?
“I'd say they stay in S3 indefinitely and accrue storage charges. They don't appear in a normal list_objects call; you need list_multipart_uploads. The fix is an AbortIncompleteMultipartUploads lifecycle rule.”
Experience checkHow would you let an untrusted client upload directly to S3 without exposing your credentials?
“I'd generate a presigned PUT URL on the server, scoped to a specific key with a short expiry, and send that URL to the client. The client POSTs directly to S3. The server's credentials are never transmitted; only the signature is.”
DesignA presigned URL expires in 1 hour but the upload takes 3 hours. What breaks?
“The PUT will fail after the first hour. Presigned URLs expire at the signing credential's lifetime if that's shorter than ExpiresIn. I'd switch to multipart with per-part presigned URLs, refreshing parts as needed, or use long-lived IAM user credentials for signing to get up to 7 days.”

Memory hook

Multipart is sending a parcel in numbered boxes: fail one box, resend only that box.

If you remember only one thing:

Abandoned multipart uploads bill forever. Add the AbortIncompleteMultipartUploads lifecycle rule from day one.

Self-check

  1. What is the minimum size for each uploaded part except the last?

    5 MB. Parts smaller than 5 MB (except the last) cause complete_multipart_upload to fail with EntityTooSmall.

  2. A presigned URL is signed with an STS role whose session expires in 2 hours. You set ExpiresIn to 24 hours. When does the URL expire?

    In 2 hours. The URL expires at min(ExpiresIn, credential_lifetime).

  3. How do you find and abort abandoned multipart uploads?

    Use list_multipart_uploads to find in-progress uploads, then call abort_multipart_upload for each, or rely on an AbortIncompleteMultipartUploads lifecycle rule to clean them up automatically.

Prove it

Write a function split_file(path, part_size_mb) that reads a file and yields (part_number, bytes) chunks. Use it on a 50 MB temp file (created in the snippet with random-ish content). Assert that: all parts except the last are exactly part_size_mb MB, the last part may be smaller, no parts are empty, and sum(len(part) for _, part in parts) == file_size. Print a summary. Target part size: 8 MB.

aws-04-04Card 4 of 6

S3 events and secure access

HardP3 · Role-dependentTalk about it13 min
s3-eventsnotificationsevent-bridgeiambucket-policyencryption

Why you care

S3 event notifications are how a file arriving in a bucket triggers your ETL pipeline without polling. Most backend engineers who have used them underestimate two things: S3 guarantees at-least-once delivery, not exactly-once, and an S3 read after an event is strongly consistent but that does not make your event handler idempotent. An interviewer asking about S3-triggered pipelines is really asking whether you understand duplicate-safe side effects. The hard difficulty means you need to explain the event contract, sketch a deduplication design, and describe IAM plus bucket policy interaction without prompting.

Explain like I'm 5

S3 event notifications are like parcel delivery notes. When a parcel (object) is dropped in your mailbox (bucket), S3 scans a list of addresses (notification configurations) and mails a note to everyone registered: "parcel arrived at raw/2024/01/orders.csv". The postal service (SNS/SQS/Lambda/EventBridge) might deliver the same note twice if the first delivery was not acknowledged. Your job is not to unpack the same parcel twice.

Core concept

Event notification targets: Lambda functions, SQS queues, SNS topics, and EventBridge (S3 → default event bus). EventBridge is the most flexible: it supports content-based routing, archiving and replay.

Event types (use exact names in notification configurations):

  • s3:ObjectCreated:* covers PUT, POST, COPY and CompleteMultipartUpload
  • s3:ObjectCreated:Put, s3:ObjectCreated:CompleteMultipartUpload for specific operations
  • s3:ObjectRemoved:*, s3:ObjectRemoved:Delete, s3:ObjectRemoved:DeleteMarkerCreated
  • s3:ObjectRestore:* for Glacier restores
  • s3:Replication:* for replication status events

Delivery contract: S3 publishes each event at least once. In rare cases the same event may be delivered more than once (network retry on S3's side). Your consumer must be idempotent: the same s3:ObjectCreated for the same key and ETag must not produce a second business effect (End-to-end idempotency and effect boundaries).

Event ordering: S3 does not guarantee that events arrive in the order the PUTs occurred. A fast second PUT followed by a slow first PUT can deliver events in reverse order. Design consumers that are insensitive to order, or filter on x-amz-request-id to deduplicate at the event level.

IAM and bucket policy:

  • By default a bucket is private. A GetObject action on the bucket requires an IAM identity with s3:GetObject on the bucket/key resource.
  • Block Public Access is a bucket-level and account-level setting that overrides bucket and ACL policies. Even with a bucket policy granting s3:GetObject to *, Block Public Access prevents it. Check this setting before debugging "403 despite a correct policy".
  • For cross-account access, the bucket policy must grant the other account explicitly, and that account's IAM policy must also permit the action. Both sides must allow.
  • For encryption with KMS, the IAM role also needs kms:Decrypt on the key (Key management). S3 GetObject alone is not enough.

Decision rule: use Lambda for low-volume, low-latency triggers. Use SQS for buffering and retry. Use EventBridge for multi-target fan-out, schema registry or event replay. Never use S3 notifications for business logic that cannot be safely retried.

Under the hood

S3 event notifications are published asynchronously after the write is durably committed. The strong consistency guarantee (immediate GET visibility) is separate from and faster than event delivery. A consumer that receives an s3:ObjectCreated:Put event for key K and immediately calls get_object(Key=K) will always succeed: the strong-consistency guarantee means the object is visible. However, the event may arrive seconds after the write, and in high-throughput scenarios (thousands of PUTs per second to the same bucket) the notification queue can introduce additional delay. Do not design latency-critical pipelines around S3 events; use Kinesis Data Streams for millisecond-latency triggers.

Events fired to EventBridge are in a standard envelope; the S3-specific fields are nested inside "detail". Lambda events receive an array of S3 record objects with a slightly different schema. Test your consumer against both schemas if you plan to switch targets.

Minimal example IAM policy evaluator for S3 access

PythonSimulate cross-account S3 access decision treeruns
def can_access(principal_account, bucket_owner_account,
               iam_allows, bucket_policy_allows, block_public_access_on):
    """
    Returns (allowed: bool, reason: str).
    Rules:
    1. Block Public Access overrides any Allow for public principals.
    2. Cross-account: BOTH bucket policy AND IAM policy must allow.
    3. Same-account: IAM policy OR bucket policy Allow is sufficient.
    """
    same_account = principal_account == bucket_owner_account
    if block_public_access_on and principal_account == "public":
        return False, "Block Public Access prevents public access"
    if same_account:
        if iam_allows or bucket_policy_allows:
            return True, "same-account: IAM or bucket policy allows"
        return False, "same-account: neither IAM nor bucket policy allows"
    else:
        if iam_allows and bucket_policy_allows:
            return True, "cross-account: both IAM and bucket policy allow"
        if not iam_allows:
            return False, "cross-account: IAM policy does not allow"
        return False, "cross-account: bucket policy does not allow"

cases = [
    ("111", "111", True,  False, False),  # same account, IAM only
    ("111", "111", False, True,  False),  # same account, bucket policy only
    ("222", "111", True,  True,  False),  # cross-account, both allow
    ("222", "111", True,  False, False),  # cross-account, missing bucket policy
    ("public","111",True, True,  True ),  # public but block public access on
]
for args in cases:
    allowed, reason = can_access(*args)
    print(f"{'ALLOW' if allowed else 'DENY ':5} | {reason}")
Output
ALLOW | same-account: IAM or bucket policy allows
ALLOW | same-account: IAM or bucket policy allows
ALLOW | cross-account: both IAM and bucket policy allow
DENY  | cross-account: bucket policy does not allow
DENY  | Block Public Access prevents public access

In production Idempotent S3 event handler

The handler records processed (bucket, key, etag) tuples to deduplicate at-least-once delivery.

PythonDeduplicating S3 event handler (simulates SQS batch with duplicates)runs
import hashlib
import json

# Simulate a processed-events log (in production: DynamoDB or Postgres table)
processed = set()

def event_id(record):
    """Stable id: bucket + key + etag. Re-processing the same object version is a no-op."""
    key_str = f"{record['bucket']}/{record['key']}/{record['etag']}"
    return hashlib.sha256(key_str.encode()).hexdigest()[:16]

def handle_event(record):
    eid = event_id(record)
    if eid in processed:
        print(f"  SKIP (duplicate) {record['key']}")
        return
    # --- business effect goes here ---
    processed.add(eid)
    print(f"  PROCESS {record['key']} etag={record['etag'][:8]}")

# Simulate SQS batch with one genuine duplicate (same ETag) and one re-upload (new ETag)
batch = [
    {"bucket": "my-etl", "key": "raw/orders.csv", "etag": "abc12345"},
    {"bucket": "my-etl", "key": "raw/orders.csv", "etag": "abc12345"},  # duplicate
    {"bucket": "my-etl", "key": "raw/orders.csv", "etag": "def99999"},  # new version
]
print("Processing batch:")
for record in batch:
    handle_event(record)
print(f"total processed (unique): {len(processed)}")
Output
Processing batch:
  PROCESS raw/orders.csv etag=abc12345
  SKIP (duplicate) raw/orders.csv
  PROCESS raw/orders.csv etag=def99999
total processed (unique): 2

See it happen S3 event delivery and consumer deduplication

PUT callerS3SQSETL workerPutObject(key, body)publish eventreceive_message: eventget_object(key)delete_message (ack)re-deliver same eventidempotency check
  1. Write is durable before S3 sends any notification.
  2. S3 publishes the event asynchronously. Delivery can take seconds.
  3. Worker receives the event. SQS visibility timeout starts.
  4. Strong consistency: GET always finds the object that caused the event.
  5. Worker deletes the message only after completing the effect. If it crashes before here, SQS redelivers.
  6. At-least-once: the same event arrives again (crash or timeout). Worker must deduplicate on (bucket, key, etag).
  7. If (bucket, key, etag) is already in the processed log, skip. Otherwise process and record.
Notice: the gap between event delivery and the business effect is where duplicates are created. The key/etag combination is the stable deduplication token, not the SQS message id.
Scenario A

In your world

The situation

Your pipeline writes a daily Parquet file to S3 and an S3 event notification triggers a Lambda that upserts it into Redshift. One morning the Lambda runs twice: a transient SQS redelivery caused by a cold-start Lambda timeout.

What actually breaks

The second invocation finds the same file (same key, same ETag) and upserts the same rows again. Because the ETL uses INSERT INTO without conflict handling, you get duplicate rows. The downstream report shows double the revenue for that day.

What you'd do on Monday

Change the Lambda to check whether it has already processed this (key, etag) pair. Use a DynamoDB table with a conditional write: PutItem with ConditionExpression: attribute_not_exists(event_id). If the condition fails, the event is a duplicate and the Lambda exits 200 without touching Redshift. Also switch the Redshift load to an upsert (merge/INSERT … ON CONFLICT DO NOTHING) as a second line of defence:

PythonConditional-write deduplication logic (stdlib simulation)runs
def conditional_put(table, event_id, payload):
    """Raises if event_id already exists (simulates DynamoDB ConditionExpression)."""
    if event_id in table:
        raise ValueError(f"ConditionalCheckFailed: {event_id} already processed")
    table[event_id] = payload

dedup_table = {}
try:
    conditional_put(dedup_table, "evt-abc123", {"key": "daily.parquet"})
    print("first: processed")
    conditional_put(dedup_table, "evt-abc123", {"key": "daily.parquet"})
except ValueError as e:
    print("second:", e)
Output
first: processed
second: ConditionalCheckFailed: evt-abc123 already processed
Scenario B

Imagine you work at Dropbox

500M files processed per day via S3 events

The numbers

Illustrative: 500M files/day ÷ 86,400 s ≈ 5,787 events/second at sustained peak, with spikes to 2× at business hours. Each event triggers a Lambda consumer; at 5,787/s that is ~5,787 concurrent Lambdas (each invocation < 1 s). S3 → Lambda fan-out at this scale saturates the default Lambda concurrency limit (check current service quotas page); you would request a quota increase.

What the Scenario-A answer does here

The Scenario A fix (deduplication table) works at low volume, but at 5,787 events/second each DynamoDB conditional write is a separate API call. At sustained rate, DynamoDB write capacity becomes a bottleneck and adds latency to every event handler.

What flips at this scale

At this scale, the deduplication contract shifts. Instead of per-event deduplication, you design for exactly-once delivery at the output level: use a Redshift MERGE or a Postgres INSERT … ON CONFLICT DO UPDATE keyed on the natural key of the data, so any duplicate load is idempotent at the data layer, not the event layer. The event handler becomes a simple forwarder; correctness lives in the write path.

The trade-off you'd defend

I would defend natural-key upsert at the storage layer over per-event DynamoDB deduplication, accepting that the upsert must be correct for all possible orderings and duplicates, for simpler event handlers and no DynamoDB bottleneck. Event-level deduplication remains valuable for expensive side effects (sending emails, calling payment APIs) that the storage layer cannot absorb.

Illustrative figures for the arithmetic; not Dropbox-published event volume. Check current Lambda and DynamoDB service quotas before relying on any limit stated above.

Gotchas

An S3 event notification does not guarantee delivery order

Symptom

A fast overwrite followed by a slow original write delivers events in reverse order; the consumer processes an older version after a newer one.

Cause

S3 does not guarantee event ordering across concurrent PUTs to the same key.

Fix

Use the ETag in the event to distinguish versions. If order matters, use the LastModified timestamp in the S3 object metadata to discard events for objects older than what you have already processed.

Block Public Access prevents bucket policy Allows for public principals

Symptom

You add a bucket policy granting s3:GetObject to "Principal": "*" and GETs still return 403.

Cause

Block Public Access (BPA) is a guard rail that overrides bucket and ACL policies for public principals. BPA is on by default for new buckets since April 2023.

Fix

Check both bucket-level and account-level BPA settings in the S3 console or with get_public_access_block. Only disable BPA if you truly intend public access; for cross-account access, use explicit account ARNs in the bucket policy instead.

Lambda cold-start timeout causes SQS redelivery and duplicate processing

Symptom

Every few hundred events, the Lambda processes the same file twice with no error in CloudWatch.

Cause

If Lambda does not delete the SQS message within the visibility timeout (default 30 s), SQS redelivers it to another consumer.

Fix

Set the Lambda timeout well below the SQS visibility timeout, add idempotency logic, and monitor the ApproximateNumberOfMessagesNotVisible SQS metric.

Tricky questions

Does a GET immediately after an S3 event notification always find the object?
Most people say

No, S3 was eventually consistent, so you might still need a retry loop.

Actually

Yes. Since December 2020, S3 provides strong read-after-write consistency. An s3:ObjectCreated event is published after the write is durable, so a GET on that key will always succeed. You no longer need a retry loop for the read after an event.

Is cross-account S3 access granted by an IAM policy in the accessing account alone?
Most people say

Yes, if the IAM role in Account B has s3:GetObject on the bucket ARN, access is allowed.

Actually

No. For cross-account access, both the bucket policy in Account A must explicitly grant Account B, and the IAM policy in Account B must also allow the action. One side allowing is not enough; both must allow.

Scenario question

You're handed a pipeline where an S3 PUT triggers an SQS event that triggers a Lambda that writes to Postgres. The team reports intermittent duplicate rows in Postgres. Diagnose and fix.

Model answer

The root cause is at-least-once SQS delivery combined with a non-idempotent Lambda. SQS redelivers any message that is not deleted within the visibility timeout; if Lambda times out or crashes before deleting, the message reappears. I would fix it in two places. First, make the Lambda idempotent: use the (S3 bucket, key, ETag) as a deduplication key, checked against a DynamoDB table with a conditional write before doing any work. Second, make the Postgres write idempotent: switch from INSERT to INSERT … ON CONFLICT (natural_key) DO UPDATE so a duplicate load is a no-op. Finally, set the Lambda timeout to the SQS visibility timeout minus a safety margin so messages are always processed or fail before redelivery. I would monitor ApproximateNumberOfMessagesNotVisible to detect slow consumers.

Interview corner

CommonWhat delivery guarantee does S3 event notification provide?
“I'd say at-least-once. In rare cases the same event can be delivered more than once, so every consumer must be idempotent. The simplest approach is deduplicating on bucket, key and ETag before applying any side effect.”
TrickYou add a bucket policy allowing GetObject for all principals. Users still get 403. What do you check first?
“I'd check Block Public Access at both bucket and account level. BPA overrides bucket policies for public principals and it is on by default for new buckets. If BPA is on, the bucket policy Allow for star is silently overridden.”
Experience checkWhere has at-least-once S3 event delivery caused problems for you?
“I was on a project with S3-triggered Lambda loading into Redshift where transient SQS redeliveries caused duplicate rows. We fixed it with a conditional DynamoDB write as a dedup check before the load, and changed the INSERT to an upsert on the natural key.”
DesignHow would you design an S3-triggered pipeline that processes a file exactly once even at 5,000 events per second?
“I'd rely on natural-key upserts at the storage layer rather than per-event deduplication, because per-event DynamoDB writes become the bottleneck at that rate. The event handler is a forwarder; correctness is enforced at the write. For expensive side effects like payments, I'd use a separate idempotency table keyed on the event ID.”

Memory hook

S3 events are at-least-once postal deliveries; build every handler as if the parcel might arrive twice.

If you remember only one thing:

Deduplicate on (bucket, key, etag), not on event id. The same object version delivered twice has the same etag.

Self-check

  1. What is the earliest you can GET an object after receiving its S3 ObjectCreated event?

    Immediately. S3 publishes the event after the write is durable, and strong consistency guarantees the GET will succeed.

  2. What two policies must both allow for cross-account S3 access?

    The bucket policy in the owning account and the IAM policy in the accessing account. Both must grant the action; one alone is not sufficient.

  3. What lifecycle rule prevents abandoned multipart uploads from accumulating?

    AbortIncompleteMultipartUploads with a DaysAfterInitiation value. This applies to the current card because large-file PUTs use multipart, and incomplete uploads fire no ObjectCreated event.

Prove it

Write a Python simulation of a SQS-like at-least-once queue with a 30% random redelivery rate. Process 20 events through it with an idempotent handler that records (bucket, key, etag) in a set. Show that despite redeliveries, the set contains at most 20 unique entries and no side effect runs more than once. Print the total deliveries, unique events processed, and duplicate deliveries skipped.

aws-04-05Card 5 of 6

S3 data-lake access patterns

MediumP2 · Very likelyTalk about it14 min
s3data-lakepartitioningzonesglueathenaetl

Why you care

If you write to S3 from a Python ETL job, you are already building a data lake, whether you know it or not. Engineers who advance to senior roles are the ones who can explain why key structure affects Athena query cost, why the raw zone must be immutable, and why a curated zone exists at all. This card bridges "upload a CSV to S3" to "design a storage layer a data team can query, replay and trust." Interviewers weigh it at an 80% probability, because it applies directly to your current role at TCS and the next one.

Explain like I'm 5

Think of a data lake as a library with three sections. The raw section is the original documents as they arrived, locked in glass cabinets: no one may alter them. The curated section is the reading room with indexed, cross-referenced copies in a standard format. The archive section is the basement with older editions. A catalogue at the entrance tells you which shelf each document is on. Without the catalogue, finding anything means walking every shelf.

Core concept

Zone design is the most common pattern for S3 data lakes:

  • Raw zone (s3://my-bucket/raw/): immutable source data, written once. Never overwrite or delete raw inputs; they are your audit trail and your re-processing safety net. Enable versioning and, optionally, Object Lock in compliance mode for regulatory requirements.
  • Curated zone (s3://my-bucket/curated/): cleaned, validated, typically Parquet-formatted outputs. Partitioned by a dimension that matches query patterns (usually year=YYYY/month=MM/day=DD for time series).
  • Archive zone (s3://my-bucket/archive/): older curated data transitioned to a colder storage class via lifecycle rules.

Partition key design is the decision that determines Athena query cost. Athena charges per TB scanned (check current pricing). A WHERE dt = '2024-01-15' on an unpartitioned table scans the entire dataset. The same query on a table partitioned year=2024/month=01/day=15 scans only one day's files. Good partition keys are:

  • High cardinality along the filter dimension (date for time-series, region for geo).
  • Low cardinality across rows (the number of distinct values, not per-row uniqueness).
  • Aligned with query patterns: partition by what you filter on, not by what you aggregate.

Athena and the Glue Data Catalog: Athena reads S3 directly. The Glue Data Catalog holds the table schema and partition locations. You must either run a Glue crawler (expensive at scale) or call add_partition via msck repair table or batch_create_partition after writing new partitions. Failing to register new partitions means Athena silently returns no rows for the new date.

Immutable inputs plus re-processability: if a bug in your ETL is found after the fact, you can re-process from raw because raw was never mutated. Curated data is derived; raw is the source of truth.

Decision rule: always write raw before curated. Never let downstream jobs read from raw directly; curated should be the stable interface. Partition curated data by the query dimension, not by ingestion time. Register partitions immediately after writing, not in a daily batch.

Minimal example Partition path generator and cost estimator

PythonPartition key paths and Athena scan cost modelruns
from datetime import date, timedelta
import math

def partition_path(prefix, year, month, day):
    return f"{prefix}year={year}/month={month:02d}/day={day:02d}/"

def athena_scan_estimate(total_gb, days_in_table, query_days, bytes_per_tb=1024):
    """
    Estimate how many GB a date-filtered query scans.
    Without partitioning: scans everything.
    With daily partitioning: scans only the requested days.
    Athena charges per TB scanned; check current pricing.
    """
    gb_per_day = total_gb / days_in_table
    partitioned_gb = gb_per_day * query_days
    unpartitioned_gb = total_gb
    return unpartitioned_gb, partitioned_gb

# Partition paths for a week
start = date(2024, 1, 15)
for i in range(5):
    d = start + timedelta(days=i)
    print(partition_path("curated/orders/", d.year, d.month, d.day))

# Cost comparison: 1 TB table, 365 days, query 7 days
total_gb = 1024
unpart, part = athena_scan_estimate(total_gb, 365, 7)
print(f"\nUnpartitioned scan: {unpart:,.0f} GB")
print(f"Partitioned scan (7 days): {part:.1f} GB")
print(f"Scan reduction: {(1 - part/unpart)*100:.1f}%")
Output
curated/orders/year=2024/month=01/day=15/
curated/orders/year=2024/month=01/day=16/
curated/orders/year=2024/month=01/day=17/
curated/orders/year=2024/month=01/day=18/
curated/orders/year=2024/month=01/day=19/

Unpartitioned scan: 1,024 GB
Partitioned scan (7 days): 19.6 GB
Scan reduction: 98.1%

In production Write curated Parquet and register partition

This is the shape of a real ETL output step; boto3 and pyarrow replace the stubs.

PythonWrite partitioned Parquet and register with Glue (needs boto3 + pyarrow + Glue catalog)
import boto3
from datetime import date

BUCKET = "my-data-lake"
DATABASE = "etl_db"
TABLE = "orders"

def write_partition(df, partition_date: date):
    prefix = (
        f"curated/{TABLE}/"
        f"year={partition_date.year}/"
        f"month={partition_date.month:02d}/"
        f"day={partition_date.day:02d}/"
    )
    s3_key = f"{prefix}part-0001.parquet"

    # Write Parquet to S3
    buf = df.to_parquet(index=False)
    boto3.client("s3").put_object(Bucket=BUCKET, Key=s3_key, Body=buf)

    # Register the new partition with Glue so Athena sees it immediately
    glue = boto3.client("glue")
    glue.batch_create_partition(
        DatabaseName=DATABASE,
        TableName=TABLE,
        PartitionInputList=[{
            "Values": [str(partition_date.year),
                       f"{partition_date.month:02d}",
                       f"{partition_date.day:02d}"],
            "StorageDescriptor": {
                "Location": f"s3://{BUCKET}/{prefix}",
                "InputFormat": "org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat",
                "OutputFormat": "org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat",
                "SerdeInfo": {"SerializationLibrary": "org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe"},
            }
        }]
    )
    print(f"Written and registered: s3://{BUCKET}/{s3_key}")

See it happen Raw to curated pipeline zones

write onceETL jobregister partitionlifecycle rule (90d)re-read on bug fixSource(Postgres / API)raw/(immutable CSV/JSON)curated/(Parquet, partitioned)archive/(Glacier after 90d)Athena / GluecatalogueRe-process(bug fix)
  1. Source data lands in raw/ as-is. No transformation; immutability is enforced by versioning.
  2. The ETL job reads raw/ and writes partitioned Parquet to curated/.
  3. The ETL job registers the new partition with Glue immediately. Athena can now query today's data.
  4. After 90 days, lifecycle transitions curated/ to Glacier. Archive is for retention, not active query.
  5. A bug is found in the ETL. Raw/ is still intact; re-run the ETL from raw/ to regenerate curated/.
Notice: raw is the source of truth. curated is derived. If curated is wrong you re-run from raw; if raw is gone you have nothing to re-run from.
Scenario A

In your world

The situation

Your team writes daily settlement CSV files directly to s3://etl-bucket/settlements/{date}/input.csv, where the ETL job reads them. No raw/curated split: the ETL writes its output to s3://etl-bucket/processed/{date}/output.csv over the same key each day.

What actually breaks

When a schema change in the source last month introduced a new column, the ETL failed silently (it just ignored the column) for three days before someone noticed. You want to re-run those three days from scratch, but the input files for those dates were overwritten by subsequent runs using the corrected ETL, and the source system does not keep a re-extract API.

What you'd do on Monday

Introduce a raw zone by writing the input file to raw/{date}/input.csv before any transformation, and making the raw zone immutable: enable Object Lock in governance mode with a 30-day retention period on the raw/ prefix. The ETL reads from raw, writes to curated. Re-running the ETL for any past date is safe because raw was never changed.

PythonRaw key vs processed key naming conventionruns
from datetime import date

def raw_key(source, event_date: date):
    return f"raw/{source}/year={event_date.year}/month={event_date.month:02d}/day={event_date.day:02d}/input.csv"

def curated_key(source, event_date: date, part: int = 1):
    return f"curated/{source}/year={event_date.year}/month={event_date.month:02d}/day={event_date.day:02d}/part-{part:04d}.parquet"

d = date(2024, 1, 15)
print(raw_key("settlements", d))
print(curated_key("settlements", d))
Output
raw/settlements/year=2024/month=01/day=15/input.csv
curated/settlements/year=2024/month=01/day=15/part-0001.parquet
Scenario B

Imagine you work at ShareChat

2 TB raw video metadata uploaded per day

The numbers

Illustrative: 2 TB/day raw video metadata, cleaned and deduplicated down to ~500 GB/day of curated Parquet, across ~100M daily events. At year/month/day partitioning, each day's curated partition holds the full ~500 GB written that day. A 7-day Athena query on 7 partitions scans 7 × 500 GB ≈ 3.5 TB. Without partitioning, a year of history is 365 × 500 GB ≈ 182 TB, and an unfiltered or badly-pruned query scans all of it.

What the Scenario-A answer does here

The Scenario A fix (raw/curated split plus daily partitions) works, but at 100M events/day the curated Parquet files may be fragmented: if each micro-batch of 10K events writes its own Parquet file, you get 10,000 small files per day, averaging 500 GB ÷ 10,000 ≈ 50 MB each. Athena reads one file per thread, so a query against that one partition still issues 10,000 separate S3 GET calls even though the total bytes match the partition size exactly; the per-request latency overhead dominates scan time, not the byte count.

What flips at this scale

At this scale, partition layout must include a compaction step. You still write small files during ingest (fast, low-memory), then run a daily Spark or Glue job that reads the day's partition and rewrites it as roughly 500 files of ~1 GB each, a size that keeps files splittable while cutting the GET count twentyfold. This collapses 10,000 GET calls into about 500 and drops per-query latency from minutes to seconds. The flip: compaction is a write-amplification step that adds daily Glue compute cost, but reduces Athena query latency by roughly an order of magnitude.

The trade-off you'd defend

I would defend a small-files ingest path plus a nightly compaction job, accepting write amplification, because the Athena query SLA (< 10 seconds) is a product requirement that cannot be met with 10,000 small files per partition. If Athena is exploratory only, I would skip compaction and accept slower queries.

Illustrative figures for the arithmetic; not ShareChat-published data volumes.

Gotchas

New partitions not registered in the Glue catalogue are invisible to Athena

Symptom

You run an Athena query for yesterday's data and get zero rows despite the Parquet files existing in S3.

Cause

Athena uses the Glue Data Catalog for partition metadata. Writing new Parquet files to a new partition prefix does not automatically register the partition.

Fix

Call batch_create_partition after each write, or run MSCK REPAIR TABLE in Athena (expensive at scale). Use Glue crawlers for ad-hoc discovery but batch_create_partition for programmatic writes.

Raw zone overwritten destroys replay ability

Symptom

A reprocessing request reveals that the input files for three days are missing from S3.

Cause

The ETL writes the raw input and the processed output to the same key path, or a cleanup script deleted "old inputs".

Fix

Treat the raw zone as append-only: write to a deterministic key that includes a date (not just an environment name), and set an S3 lifecycle rule that only expires raw files after your required retention window (never before).

Partitioning by ingestion time instead of event time causes stale queries

Symptom

Athena queries filtered WHERE event_date = '2024-01-15' miss rows that arrived late (processed the next day).

Cause

If partitions are named by the ETL run date rather than the event date in the data, late-arriving events land in a different partition than the one queries expect.

Fix

Partition by the event timestamp in the data, not by the wall-clock time of ingestion. Accept that late arrivals write to past partitions and that downstream queries may need to re-scan recent past partitions.

Tricky questions

Why is it a mistake to read directly from the raw zone in production queries?
Most people say

Reading raw data is fine; it is the original source of truth.

Actually

Raw files are typically CSV or JSON with inconsistent schema, no type coercion and no null handling. Production queries on raw data are slow, brittle and expose the pipeline to schema changes in the source. The curated zone exists specifically to provide a stable, typed, columnar interface. Raw is for re-processing, not for serving.

Does Athena automatically pick up new Parquet files added to an existing partition prefix?
Most people say

Yes, Athena scans the prefix and sees all files.

Actually

Athena scans all files under a registered partition's location, so files added to an already-registered partition are picked up. But a new date-based partition (a new prefix) is not visible until registered with Glue, either via batch_create_partition or MSCK REPAIR TABLE.

Scenario question

You're handed a data lake where the entire two-year history of settlement data (20 TB) is stored as un-partitioned CSV files under s3://etl-bucket/curated/settlements.csv. Every Athena query takes 10 minutes and costs the team ~$100 at $5/TB scanned. Propose a migration plan.

Model answer

I would run a one-time migration job (Spark or Glue) that reads the flat CSV, parses the timestamp column, and writes Parquet files partitioned by year/month/day to a new prefix. The job should be idempotent (overwrite if a partition already exists) and checkpointed by month so a crash can resume. During migration I would not touch the existing CSV; it stays as the raw archive. Once all partitions are written and registered with Glue, I would update the Athena table to point to the new partitioned location. A 7-day query that scanned 20 TB of CSV would now scan ~380 GB of Parquet after partition pruning; with column pruning and predicate pushdown it may scan 10–50 GB, dropping cost from ~$100 to under $1. The trade-off is the migration job cost and the operational risk of the cutover; I would test with a 1-month sample before migrating the full history.

Interview corner

CommonWhy partition an S3 data lake by date?
“I'd say Athena charges per TB scanned. A date partition lets the query engine skip all other partitions when you filter by date. For a year of daily data, a 7-day query goes from scanning 100% of the table to 2% of it.”
TrickYou write Parquet files to a new date partition but Athena returns zero rows. What do you check?
“I'd check whether the partition is registered in the Glue Data Catalog. Athena uses Glue for partition discovery; new prefixes are invisible until you call batch_create_partition or MSCK REPAIR TABLE.”
Experience checkHow does the raw/curated split save you on the day you find a bug in your ETL?
“Raw holds the original inputs, immutable. Curated is derived. If I find a bug, I re-run the ETL from raw over the affected date range and regenerate curated. If I had only curated, or if raw was mutable, the original data would be gone and I'd have to re-fetch from the source.”
DesignA data lake has 10,000 small Parquet files per day partition. Athena queries on that partition are slow. What do you do?
“I'd add a nightly compaction job that reads the partition and rewrites it as a small number of larger files, say 4 × 500 MB. Athena parallelises across files; 4 files means 4 threads instead of 10,000, and the S3 GET overhead drops by 2,500x. The cost is write amplification in the compaction job.”

Memory hook

Raw is the locked-glass original; curated is the indexed reading copy; never let anyone write to raw twice.

If you remember only one thing:

Partition by event time, register every new partition immediately, and never mutate raw.

Self-check

  1. Why should new Glue partitions be registered immediately after writing, not at the end of the day?

    Athena queries between the write and the end-of-day registration will see zero rows for the new partition. For daily reporting with SLA requirements, a missing partition looks like missing data.

  2. A query filters on region = 'IN' and year = 2024. The table is partitioned only by year/month/day. What happens?

    Athena scans all partitions for 2024 (365 day partitions) and applies the region filter at read time on each file. The year partition reduces scanned data but the region predicate does not prune partitions. To use region for partition pruning, it must be a partition column.

  3. What does Object Lock in governance mode provide for the raw zone?

    It prevents deletion or overwrite of objects before their retention period expires, even by the bucket owner. This enforces immutability for compliance scenarios.

Prove it

Create a local directory tree that simulates a partitioned S3 data lake (use pathlib). Write 10 CSV files per partition for 5 date partitions. Then write a function query(root, year, month, day) that returns only the files under that exact partition path. Count how many files would be scanned with and without the partition filter (i.e., total vs matched). Print the scan savings percentage.

aws-04-06Card 6 of 6

EBS versus EFS versus S3

MediumP3 · Role-dependentTalk about it14 min
ebsefss3storage-comparisonec2block-storagefile-storage

Why you care

At a product company interview, "when would you not use S3?" separates candidates who have thought about storage access patterns from those who have only used one tool. The three AWS storage types answer three fundamentally different questions: "what block device does my EC2 instance boot from?" (EBS), "how do 20 Lambda functions share a config directory?" (EFS), and "where do I put objects that any service in the world can read?" (S3). Confusing them leads to architecture mistakes: trying to mount S3 like a filesystem, or burning money on EBS volumes for data that should be in S3.

Explain like I'm 5

EBS is like an external hard drive plugged into your computer. Only your computer uses it. If you shut down your computer, the drive is still there, but it is still only connected to your computer's slot. EFS is like a network-attached drive in your office: every computer on the network can mount and read the same folder at the same time. S3 is like a cloud storage service: no mounting, no POSIX paths, but anyone anywhere can download a file with a URL.

Core concept

EBS (Elastic Block Store):

  • Block-level storage attached to exactly one EC2 instance at a time (with the exception of io2 Block Express Multi-Attach for clustered databases; check documentation for limits).
  • Tied to a single Availability Zone. If you need the data in another AZ, you create a snapshot and copy it.
  • Provides persistent storage: data survives instance stop/terminate (unless Delete on Termination is set).
  • Use for: root volumes, databases that need low-latency random I/O, any workload that needs filesystem semantics on one instance.
  • Types include gp3 (general-purpose SSD, check current baseline IOPS/throughput), io2 (provisioned IOPS for databases), st1 (throughput-optimised HDD for sequential big data), sc1 (cold HDD, lowest cost).

EFS (Elastic File System):

  • POSIX-compatible shared file system (NFS v4.1/v4.2). Multiple EC2 instances, containers and Lambda functions can mount the same filesystem concurrently.
  • Spans all AZs in a region (Standard class); One Zone class is single-AZ.
  • Scales automatically; no capacity to provision.
  • Use for: shared configuration, web serving from multiple instances, home directories, shared ETL scratch space across Lambda functions.
  • EFS is significantly more expensive per GB than EBS or S3; it is priced per GB used per month. Do not use EFS as a data lake.

S3 (Simple Storage Service):

  • Object storage accessible via HTTP API. No mounting, no POSIX semantics, no in-place edits.
  • Globally accessible; durably replicated within a region.
  • Use for: data lake storage, static assets, backups, ETL input/output, large file distribution.
  • Lowest per-GB storage price of the three; additional charges for requests and data transfer (Transfer and S3 request traps).

Backup and persistence boundaries:

  • EBS: create snapshots to S3 (managed by AWS Backup). The snapshot is region-scoped; copy it to another region for DR.
  • EFS: AWS Backup or DataSync for cross-region replication.
  • S3: built-in durability (check current SLA). Cross-Region Replication is a separate configuration.

Decision rule: if you need a filesystem that one instance owns, use EBS. If you need a filesystem that multiple instances share, use EFS. If you need to store objects that any service or user can access via URL or SDK, use S3. S3 is almost always the right answer for ETL data; EBS is for databases and instance boot volumes; EFS is for shared configuration and cross-instance scratch.

Minimal example Storage type decision tree

PythonDecision tree for AWS storage type selectionruns
def choose_storage(
    needs_posix: bool,
    multiple_instances: bool,
    http_access_only: bool,
    single_az_ok: bool,
    size_gb: float,
):
    """
    Returns the recommended storage type with a brief reason.
    Does not account for all edge cases; treat as a first-pass guide.
    """
    if http_access_only or not needs_posix:
        return "S3", "Object access via URL/SDK; no mount needed; cheapest per-GB for bulk data"
    if needs_posix and multiple_instances:
        if single_az_ok:
            return "EFS One Zone", "POSIX shared FS for multiple instances; lower cost than multi-AZ EFS"
        return "EFS Standard", "POSIX shared FS spanning all AZs; use for shared config or scratch space"
    if needs_posix and not multiple_instances:
        return "EBS", "Block storage for a single instance; use for databases, root volume, low-latency I/O"
    return "S3", "Default: object storage for most ETL and data storage use cases"

cases = [
    (False, False, True,  True,  500),    # ETL output
    (True,  True,  False, False, 10),     # shared Lambda config
    (True,  False, False, False, 200),    # Postgres data volume
    (False, False, False, True,  1024),   # data lake
    (True,  True,  False, True,  5),      # shared scratch, single AZ OK
]
for args in cases:
    t, reason = choose_storage(*args)
    print(f"{t:<20} {reason[:60]}")
Output
S3                   Object access via URL/SDK; no mount needed; cheapest per-GB
EFS Standard         POSIX shared FS spanning all AZs; use for shared config or s
EBS                  Block storage for a single instance; use for databases, root
S3                   Object access via URL/SDK; no mount needed; cheapest per-GB
EFS One Zone         POSIX shared FS for multiple instances; lower cost than mult

In production Comparing backup paths for each storage type

PythonBackup strategy model per storage typeruns
import json

strategies = {
    "EBS": {
        "backup_mechanism": "Snapshot to S3 (managed by AWS Backup or manual CreateSnapshot API)",
        "recovery_target": "New EBS volume in same AZ; cross-AZ requires copy",
        "cross_region_dr": "Copy snapshot to target region, create volume from copy",
        "rto_note": "Volume restore takes minutes; instance must be stopped (for root) or detached",
        "durability": "Snapshot durability backed by S3; volume replication within AZ",
    },
    "EFS": {
        "backup_mechanism": "AWS Backup vault or EFS-to-EFS replication (check current feature status)",
        "recovery_target": "Restore to same or different EFS filesystem",
        "cross_region_dr": "EFS Replication to a read-only replica in another region",
        "rto_note": "File-level restore available; full filesystem restore can take hours for large FS",
        "durability": "Standard class: replicated across AZs in a region",
    },
    "S3": {
        "backup_mechanism": "Versioning + lifecycle rules; S3 Cross-Region Replication for DR",
        "recovery_target": "Restore previous version with GetObject(VersionId=...)",
        "cross_region_dr": "CRR replicates new objects; existing objects require a one-time sync",
        "rto_note": "Object restore is immediate (Standard/IA); Glacier adds minutes to 48 hours",
        "durability": "Check current S3 SLA; designed for eleven-nines annual durability",
    },
}
for svc, props in strategies.items():
    print(f"\n--- {svc} ---")
    for k, v in props.items():
        print(f"  {k}: {v}")
Output
--- EBS ---
  backup_mechanism: Snapshot to S3 (managed by AWS Backup or manual CreateSnapshot API)
  recovery_target: New EBS volume in same AZ; cross-AZ requires copy
  cross_region_dr: Copy snapshot to target region, create volume from copy
  rto_note: Volume restore takes minutes; instance must be stopped (for root) or detached
  durability: Snapshot durability backed by S3; volume replication within AZ

--- EFS ---
  backup_mechanism: AWS Backup vault or EFS-to-EFS replication (check current feature status)
  recovery_target: Restore to same or different EFS filesystem
  cross_region_dr: EFS Replication to a read-only replica in another region
  rto_note: File-level restore available; full filesystem restore can take hours for large FS
  durability: Standard class: replicated across AZs in a region

--- S3 ---
  backup_mechanism: Versioning + lifecycle rules; S3 Cross-Region Replication for DR
  recovery_target: Restore previous version with GetObject(VersionId=...)
  cross_region_dr: CRR replicates new objects; existing objects require a one-time sync
  rto_note: Object restore is immediate (Standard/IA); Glacier adds minutes to 48 hours
  durability: Check current S3 SLA; designed for eleven-nines annual durability
Scenario A

In your world

The situation

Your team runs 20 Lambda functions that all need to read a config/mappings.json file. Currently, each deployment package embeds the file. When a mapping changes, you redeploy all 20 Lambdas.

What actually breaks

The redeploy cycle takes 15 minutes and a config typo once caused all 20 Lambdas to fail simultaneously because they all deployed before the error was caught.

What you'd do on Monday

Move the config to S3 and have each Lambda read it at startup with get_object. For files that change rarely and need to be read on every invocation with sub-millisecond latency, EFS is the right fit: mount the EFS filesystem in the Lambda's VPC, and the file is available as a local path inside the Lambda. Changing the file on EFS is immediately visible to all running Lambdas without redeployment. For config that is large and rarely read, S3 with a local cache (read once, store in /tmp, check age) is cheaper than EFS.

PythonS3 config cache with TTL (stdlib, no boto3)runs
import json
import time

# Simulates get_object returning bytes
def fake_get_object(bucket, key):
    return json.dumps({"fee_rate": 0.02, "version": 2}).encode()

_cache = {}
_CACHE_TTL = 300  # seconds

def get_config(bucket, key):
    now = time.monotonic()
    if key in _cache and now - _cache[key]["ts"] < _CACHE_TTL:
        return _cache[key]["data"]
    raw = fake_get_object(bucket, key)
    data = json.loads(raw)
    _cache[key] = {"data": data, "ts": now}
    return data

cfg = get_config("my-bucket", "config/mappings.json")
print("fee_rate:", cfg["fee_rate"])
cfg2 = get_config("my-bucket", "config/mappings.json")  # from cache
print("cache hit, version:", cfg2["version"])
Output
fee_rate: 0.02
cache hit, version: 2
Scenario B

Imagine you work at Zepto

500 EC2 instances, 100 Lambda functions, shared ETL scratch

The numbers

Illustrative: 500 EC2 instances each with a 100 GB gp3 root EBS volume (50 TB EBS total) and 1 TB EBS data volume (500 TB total). A shared EFS filesystem for cross-instance scratch: 10 GB actively written/day × 30 days = 300 GB. S3 for the data lake: 5 TB/month raw + 2 TB/month curated.

What the Scenario-A answer does here

The Scenario A fix (S3 for config, EFS for shared config) works, but at 500 instances with 1 TB EBS data volumes, unattached volumes (from terminated instances) can accumulate silently. Each terminated instance leaves a 1 TB EBS volume if it was not set to delete on termination. At a few cents per GB-month, 50 unattached volumes cost hundreds per month.

What flips at this scale

At this scale, EBS volume lifecycle is a cost-engineering concern. All non-root EBS volumes must have DeleteOnTermination set to true or be explicitly named and tagged for audit. Use AWS Cost Explorer with a ResourceType: EBS filter and an age-of-attachment dimension to find orphaned volumes. S3 remains the right default for any data that is not live on an instance.

The trade-off you'd defend

I would defend DeleteOnTermination: true for all ephemeral data volumes and S3 for persistent ETL data, accepting that "reattach this volume later" is an antipattern, for predictable storage costs and no zombie volume accumulation.

Illustrative figures; not Zepto-published infrastructure metrics.

EBS vs EFS vs S3

AxisEBSEFSS3
Access modelBlock (mountable, POSIX)File (POSIX, NFS)Object (HTTP API)
Concurrency1 instance (Multi-Attach limited to io2 Block Express)Many instances, containers, LambdaAny client, any region
AZ scopeSingle AZMulti-AZ (Standard) or Single AZ (One Zone)Region-scoped; CRR for cross-region
Pricing modelProvisioned GB + IOPS/throughputPer-GB-used (no provisioning)Per-GB stored + per-request + transfer
Typical useRoot volume, database, low-latency I/OShared config, web content, Lambda layersData lake, backups, ETL output, assets
Max capacity64 TB per volume (check current limits)Petabyte-scale (automatic)Effectively unlimited
In-place editYes (filesystem write)Yes (filesystem write)No (full object replace)
Snapshot / backupManual snapshots or AWS BackupAWS Backup or EFS ReplicationVersioning + lifecycle + CRR

Gotchas

EBS volume is AZ-locked: cannot be attached to an instance in a different AZ

Symptom

You try to attach a volume to a new instance launched in a different AZ and get InvalidVolume.ZoneMismatch.

Cause

An EBS volume is provisioned in a single AZ and cannot be moved.

Fix

Create a snapshot of the volume, then create a new volume from the snapshot in the target AZ.

Mounting S3 as a filesystem (s3fs, goofys) is not equivalent to native POSIX

Symptom

Code that worked locally on a filesystem fails on an S3 mount with partial writes, append errors or high latency.

Cause

S3 is object storage; file-system shims fake POSIX semantics with expensive workarounds. Random writes are especially broken.

Fix

Design your code to use the S3 SDK directly (get_object, put_object, multipart). Reserve EFS for workloads that genuinely need POSIX.

EFS Standard pricing surprises at high storage volume

Symptom

The storage bill for the EFS filesystem is 10× higher than expected.

Cause

EFS Standard is priced per GB-month actively stored, and is much more expensive than EBS or S3 per GB. It is designed for shared access, not for storing large datasets.

Fix

Move bulk data (ETL output, logs, archives) to S3. Use EFS only for shared config, code or small scratch files where POSIX access from multiple instances is required.

Tricky questions

Can two EC2 instances share the same EBS volume simultaneously?
Most people say

No, EBS is always single-instance.

Actually

With io2 Block Express Multi-Attach, up to 16 Nitro instances in the same AZ can attach the same volume simultaneously, for use in clustered database applications that handle concurrent access at the application level. For general-purpose workloads, EBS remains single-instance and EFS is the correct choice for shared access.

Is S3 data replicated across AZs automatically?
Most people say

No, S3 is region-scoped so it is stored in only one AZ.

Actually

S3 Standard and most storage classes are designed to replicate data across a minimum of three AZs within a region. S3 One Zone-IA is the exception: it stores data in a single AZ and data is lost if that AZ is destroyed.

Scenario question

You're handed a batch ETL system where 10 EC2 instances each process a shard of the daily data and write intermediate results to /tmp/etl_scratch/. After a month, the system starts failing with "no space left on device". Diagnose and propose a redesign.

Model answer

The root cause is that each instance is writing intermediate results to its own EBS root volume's /tmp, which is sized for the original load. Three months of increasing data volume has filled the disk. I would redesign in two steps. First, add a dedicated EBS data volume (separate from the root) sized with headroom and set to delete on termination; mount it at /data/etl_scratch. This fixes the immediate space issue. Second, if the intermediate files need to be shared across instances (for a merge step), switch from EBS to EFS mounted at the same path: all 10 instances write to the same EFS namespace, and the merge step reads from EFS without a separate copy step. If the intermediate files do not need to be shared, EBS is cheaper and lower-latency. For the final outputs, always write to S3, not to either EBS or EFS, so results survive after instances are terminated.

Interview corner

CommonWhen would you use EFS instead of S3 for your ETL system?
“I'd use EFS when multiple instances or Lambda functions need to share files with POSIX semantics: a shared config directory, a model file that needs to be read as a local path, or scratch space where one worker writes and another reads without coordination. S3 is better for everything else because it's cheaper per GB and has no mount required.”
TrickCan you mount an S3 bucket as a filesystem and treat it like a local disk?
“You can with shims like s3fs, but I'd call it a bad idea for most workloads. Appends, random writes and POSIX locking are either broken or emulated with high latency and error rates. S3 is designed for get and put whole objects; design your code for the SDK, not a filesystem shim.”
CommonYou need to share a 10 MB config file across 50 Lambda functions in a VPC. EFS or S3?
“I'd lean toward S3 with a local cache in /tmp for a 10 MB file: it's free to store, no VPC configuration needed, and a cold start reads it once and caches it. EFS is worth it when the file changes frequently and you need all Lambdas to see the change immediately without a redeploy, because EFS mounts the file system live. The trade-off is that EFS requires VPC configuration and costs more per GB.”
DesignAn EC2-backed Postgres instance is being moved to a different AZ. What happens to its EBS volume?
“EBS volumes are AZ-locked. I'd need to create a snapshot of the data volume, then create a new volume from the snapshot in the target AZ, and attach it to the new instance. The snapshot itself is stored in S3 so the data is safe during the copy. On a multi-AZ RDS instance, AWS handles this automatically.”

Memory hook

EBS is a personal hard drive, EFS is an office network share, S3 is cloud storage with a URL.

If you remember only one thing:

S3 is for objects via HTTP; EBS is for a single instance's block device; EFS is for shared POSIX access. Mounting S3 as a filesystem is a trap.

Self-check

  1. Which EBS volume type would you use for a Postgres database with high random I/O?

    io2 Block Express for maximum provisioned IOPS, or gp3 for most workloads where the default baseline IOPS is sufficient. Check the current EBS documentation for provisioned IOPS and throughput limits per type.

  2. What is the key cost difference between EFS and S3 per GB stored?

    EFS (Standard) is significantly more expensive per GB-month than S3 Standard. EFS is designed for shared access, not bulk storage. For large datasets, S3 is far cheaper.

  3. An EC2 instance is terminated with its EBS root volume. Is the data gone by default?

    The root EBS volume has DeleteOnTermination: true set by default, so the volume and its data are deleted when the instance terminates. Data volumes you attach yourself default to DeleteOnTermination: false, so they persist after termination.

Prove it

Write a Python function recommend_storage(scenario) that takes a dictionary describing a workload (keys: access_from_multiple_instances, needs_posix, http_access_ok, data_size_gb, access_frequency) and returns a recommendation with rationale. Test it against at least five scenarios covering EBS, EFS and S3 choices. Print a summary table.