Object storage semantics
Why you care
Every ETL developer knows how to put_object and get_object. The gap that costs you in interviews and incidents is the three things S3 is not: it is not a file system, it has no true folders, and a deleted current version is not the same as deleted data when versioning is on. Interviewers use S3 semantics questions to separate engineers who upload to S3 from engineers who can design storage contracts around it. If you cannot explain what "strongly consistent" means in S3 or why listing an empty prefix still costs you API requests, you will lose the round.
Explain like I'm 5
Imagine a giant post office with a single room of numbered mailboxes. The room has no aisles or sections. Every mailbox has a full address printed on it, like raw/2024/01/orders.csv. The slashes are part of the name, not actual dividers. Asking for "everything in raw/2024/" means the clerk reads every label and hands you the matching ones. There are no folders to peek inside. And if you send a second parcel with the same address while versioning is on, both parcels are still in the box; you just get a sticker saying which one arrived last.
Core concept
Buckets are globally-named containers. A bucket lives in exactly one region; S3 does not replicate across regions unless you configure Cross-Region Replication. Bucket names must be globally unique across all AWS accounts.
Keys are the full name of an object. raw/2024/01/orders.csv is one atomic key, not a path through three directories. The slash is a convention that the S3 console uses to fake a folder view, and that Athena and Glue use for partition discovery, but S3 itself knows nothing about it.
Strong consistency (in effect since December 2020): every successful PUT or DELETE is immediately visible to all subsequent GETs and LISTs for that key. There are no eventual-consistency windows to work around for single-object operations. However, listing is not transactional across all keys: two concurrent PUTs may appear in any order in a subsequent LIST.
Versioning adds a VersionId to every PUT. When enabled:
- A
DELETEwithout aVersionIdadds a delete marker; it does not remove the data or stop billing. - All noncurrent versions continue to accrue storage charges until a lifecycle rule or an explicit
delete_object(VersionId=...)removes them. - Versioning cannot be turned off once enabled; it can only be suspended (new PUTs get a null version).
Object operations are atomic: a PUT either succeeds completely or fails. There is no partial write, no append and no in-place edit. To update a 10 MB CSV, you write the whole 10 MB under the same key (or a new key).
Decision rule: use S3 for objects you write once and read many times, or for objects you replace as a whole. Use EFS or a local filesystem for random reads, appends or POSIX semantics (EBS versus EFS versus S3). Link large object transfers to multipart (Multipart and presigned transfers).
Minimal example S3 is a flat key-value store; slashes are part of the key
import hashlib
store = {} # simulates one bucket's namespace: key -> (body, etag)
def put(key, body):
etag = hashlib.md5(body.encode()).hexdigest()[:8]
store[key] = (body, etag)
def list_prefix(prefix):
return sorted(k for k in store if k.startswith(prefix))
put("raw/2024/01/orders.csv", "id,amt\n1,100")
put("raw/2024/01/settlements.csv", "id,settled\n1,true")
put("raw/2024/02/orders.csv", "id,amt\n2,200")
put("curated/orders/year=2024/data.parquet", "<parquet>")
print(list_prefix("raw/"))
print(list_prefix("raw/2024/01/"))
# Deleting one key does NOT remove a placeholder for the prefix
del store["raw/2024/01/orders.csv"]
print(list_prefix("raw/2024/01/")) # settlements.csv is still there
print("raw/2024/01/ is not a key:", "raw/2024/01/" not in store)['raw/2024/01/orders.csv', 'raw/2024/01/settlements.csv', 'raw/2024/02/orders.csv']
['raw/2024/01/orders.csv', 'raw/2024/01/settlements.csv']
['raw/2024/01/settlements.csv']
raw/2024/01/ is not a key: TrueIn production Versioning semantics and delete markers
This code runs on boto3 and a real bucket; it illustrates the four-state version model that trips up ETL jobs that "delete" objects and expect them to be gone.
import boto3
s3 = boto3.client("s3")
BUCKET = "my-etl-bucket"
KEY = "raw/2024/01/orders.csv"
# 1. Upload: creates version V1
v1 = s3.put_object(Bucket=BUCKET, Key=KEY, Body=b"id,amt\n1,100")["VersionId"]
# 2. Overwrite: V1 becomes noncurrent, V2 is current
v2 = s3.put_object(Bucket=BUCKET, Key=KEY, Body=b"id,amt\n2,200")["VersionId"]
# 3. Soft-delete: adds a delete marker; V1 and V2 still exist and still bill
s3.delete_object(Bucket=BUCKET, Key=KEY)
# GET now raises NoSuchKey even though both versions are on disk
try:
s3.get_object(Bucket=BUCKET, Key=KEY)
except s3.exceptions.NoSuchKey:
print("Soft-deleted: versions still exist, GET raises NoSuchKey")
# 4. Recover by deleting the delete marker (get its version id first)
versions = s3.list_object_versions(Bucket=BUCKET, Prefix=KEY)
marker_id = versions["DeleteMarkers"][0]["VersionId"]
s3.delete_object(Bucket=BUCKET, Key=KEY, VersionId=marker_id)
print("Recovered:", s3.get_object(Bucket=BUCKET, Key=KEY)["Body"].read())See it happen Version states for one key
- Before any write: the key does not exist and costs nothing.
- First PUT creates V1 as the current version. GET returns V1.
- Second PUT makes V2 current and demotes V1 to noncurrent. Both versions are stored and billed.
- DELETE without VersionId adds a delete marker. GET now raises NoSuchKey. V1 and V2 are untouched on disk and still billed.
- To truly delete: remove the delete marker, then delete each version by VersionId. Only then does billing stop.
In your world
The situation
Your nightly job writes raw/settlements/{date}/input.csv to S3, then reads it back to process. You turned on versioning last quarter for audit compliance. The job worked for months.
What actually breaks
A runbook typo last week ran the job twice on the same date. The second run overwrote the file. The version from the first run is now noncurrent, still exists, and still costs storage. Worse, the previous quarter had a lifecycle policy that expired objects after 30 days, but that policy targets current versions. The 60 noncurrent versions from six months of double-runs have never been cleared, and your bill has quietly risen.
What you'd do on Monday
Add a lifecycle rule that expires noncurrent versions after N days (N = your audit retention window). Confirm it with a daily CloudWatch metric on NumberOfObjects with StorageType: AllStorageTypes versus StorageType: StandardStorage, and alert if they diverge by more than a threshold. The Monday code in a lifecycle rule JSON:
import json
def noncurrent_expiry_rule(rule_id, prefix, days):
return {
"ID": rule_id,
"Status": "Enabled",
"Filter": {"Prefix": prefix},
"NoncurrentVersionExpiration": {"NoncurrentDays": days},
}
policy = {"Rules": [noncurrent_expiry_rule("audit-noncurrent", "raw/settlements/", 90)]}
print(json.dumps(policy, indent=2)){
"Rules": [
{
"ID": "audit-noncurrent",
"Status": "Enabled",
"Filter": {
"Prefix": "raw/settlements/"
},
"NoncurrentVersionExpiration": {
"NoncurrentDays": 90
}
}
]
}Imagine you work at Myntra
The numbers
Illustrative: 50M products × 3 versions × ~400 KB average = 60 TB stored. If each product update creates a new version on a versioned bucket without a noncurrent expiry rule, and products are updated monthly, then after the initial upload plus 12 monthly updates (13 versions total: 1 current, 12 noncurrent), you hold 12 × 60 TB = 720 TB of noncurrent objects alongside the 60 TB current set. Storage charges scale linearly with total bytes; noncurrent versions bill at the same rate as current ones.
What the Scenario-A answer does here
The Scenario A fix (lifecycle rule for noncurrent expiry) still applies, but at 50M objects, listing all versions to audit the policy takes hundreds of thousands of LIST calls at $0.005 per 1,000 LIST requests (check current S3 pricing), making ad-hoc audits non-trivial.
What flips at this scale
At this scale the object naming convention becomes a contract enforced at write time. Each image variant gets a deterministic key (products/{id}/{variant}/{timestamp}.jpg), and a lifecycle rule per prefix with a tight noncurrent expiry keeps the version backlog bounded. The LIST cost pushes you to use S3 Inventory (a daily or weekly object manifest delivered as CSV or Parquet) instead of scanning with list_object_versions.
The trade-off you'd defend
I would defend prefix-per-variant with tight noncurrent expiry and S3 Inventory for audits, trading an extra key-design constraint for predictable storage costs. The alternative, a single key per image overwritten in place, is simpler but loses the version history the audit requirement demands.
Illustrative figures for the arithmetic; 3 variants and 400 KB per image are approximations, not Myntra-published values.
Gotchas
Deleting an object on a versioned bucket does not free storage
Your delete_object call returns 200 OK and the key disappears from a plain GET, but the S3 storage metric and billing do not drop.
Without a VersionId, DELETE creates a delete marker. All previous versions remain and continue to accrue charges.
Add lifecycle rules with NoncurrentVersionExpiration and ExpiredObjectDeleteMarker. For immediate cleanup, list versions and delete each by VersionId.
Prefix listing charges API requests even if the result is empty
A cron job that checks whether yesterday's file arrived costs money even on days the file is missing.
Every list_objects_v2 call is billed as a LIST request regardless of how many objects it returns.
Use S3 event notifications to receive a PUT event when the file arrives, rather than polling with LIST. If polling is unavoidable, increase the interval.
Concurrent PUTs to the same key with versioning suspended
Two parallel job shards write to the same key; one write appears to be silently lost.
Without versioning, two PUTs to the same key are last-writer-wins. With versioning suspended (not enabled), there is no VersionId and the second write overwrites the first.
Use unique, deterministic keys per shard (include the shard id or timestamp in the key), or enable versioning and accept both copies, then reconcile.
Tricky questions
Yes, S3 was eventually consistent and versioning does not change that.
Since December 2020, S3 provides strong read-after-write consistency for all GETs and LISTs, including versioned buckets. The old eventual-consistency window no longer exists for single-object operations.
Yes, no objects means no storage charge.
Only if versioning is disabled or you explicitly delete every version and delete marker. On a versioned bucket, delete_object without a VersionId adds a delete marker and leaves every prior version intact and billable.
Scenario question
You're handed an ETL job that writes output/report.csv to an S3 bucket every hour, overwriting the same key each time. A teammate enables versioning on the bucket for a compliance audit. The job still passes, but the storage bill triples over the next month. Why, and what do you change?
Model answer
Before versioning, each overwrite replaced the previous object; only the current version was billed. After versioning, each overwrite creates a new current version and demotes the previous one to noncurrent; 24 versions per day × 30 days = 720 versions accumulate, and all of them bill. I would add a lifecycle rule with NoncurrentVersionExpiration set to however many days the audit requires (say, 30 days), which caps the backlog at 720 × (retention / 30) versions. I would also add ExpiredObjectDeleteMarker: true to clean up orphaned markers. The trade-off is that versions older than the retention window are permanently deleted; if the compliance requirement is to keep them forever, I would move them to Glacier instead of expiring them.
Interview corner
CommonWhat is the difference between a bucket and an object in S3?
TrickYou delete an object in a versioned bucket and the GET starts returning NoSuchKey. Is the data gone?
CommonWhat does strong consistency mean for S3 since December 2020?
Experience checkWhere has S3 versioning surprised you in real work?
Memory hook
S3 buckets are post offices with one flat room; slashes are labels, not aisles.
A DELETE on a versioned bucket adds a marker; it does not delete data or stop billing.
Self-check
Why does a GET return NoSuchKey after a DELETE on a versioned bucket, even though the object is still stored?
The DELETE added a delete marker, which becomes the current version and makes GET return NoSuchKey. The previous versions are still stored.
What is the correct test to tell if versioning is in the way of your delete?
Check if
NumberOfObjectsdrops afterdelete_object. If not, list versions for that key; if you see versions or delete markers, the data is still there.Can you turn off versioning on a bucket that has it enabled?
No. You can suspend it, which stops creating new versions, but the existing versions remain and new PUTs get a null VersionId.
Create a Python dict-based VersionedBucket class that tracks the full version history (each PUT assigns a new VersionId, a DELETE without VersionId adds a delete marker, a GET returns NoSuchKey when the current version is a marker). Test: put V1, overwrite with V2, delete (marker), confirm GET raises, restore by removing the marker, confirm GET returns V2, then delete V2 by VersionId and confirm GET returns V1. Paste the test output.