Glue catalogue, crawlers and managed Spark
Why you care
Every Jenkins job you have written assumes a fixed set of columns because you wrote the schema into the script. Glue flips that: the Glue Data Catalog is a shared schema definition that Athena, Spark, Redshift Spectrum and every Glue job read from, so one table definition drives every downstream query. The catch is the crawler that builds that definition infers schema from a sample of files, and it will happily overwrite a column's type the moment the data drifts—silently, with no error, no code change on your side. Interviewers ask about Glue not to test trivia but to see whether you reach for a schema contract the moment a table has real consumers, instead of trusting inference forever.
Explain like I'm 5
The Glue Data Catalog is a shared library's index-card system. The crawler is a librarian who walks the shelves (S3) on a schedule and updates each index card with whatever title, author and genre it currently sees on the book's cover. Every patron—Athena, Spark, Redshift—trusts whatever the card says the next time they go looking. If one day a book's cover has a sticker over the author's name, the librarian updates the card to match. A drifting shelf becomes a drifting card, and every patron's search is now reading a definition nobody reviewed.
Core concept
- The Glue Data Catalog is a managed, Hive-compatible metastore: databases → tables → partitions, shared across Athena, Redshift Spectrum, EMR and Glue jobs. One definition, many readers.
- A crawler scans a data source (an S3 prefix, a JDBC connection) on a schedule or on demand, infers column names and types from a sample of files, and writes or updates the table definition and partitions in the catalog.
- Crawler inference is probabilistic, not authoritative: it looks at a sample, not every file. One unusual batch can shift the inferred schema for the whole table on the next crawl.
- A schema contract is the opposite approach: you (or an upstream producer) define the table explicitly—via the Glue API, Terraform or CloudFormation—and configure the crawler so it cannot silently rewrite existing columns, or stop crawling that table at all once the schema is set.
- Glue's managed Spark ETL runs PySpark or Scala on Spark clusters AWS provisions for the duration of the job; you write against the schema-flexible DynamicFrame API (handles mismatched or "choice" types) or a plain Spark
DataFrameonce the schema is fixed and trusted. - Job bookmarks track which input files or partitions a specific job has already processed—by S3 path/timestamp or a JDBC bookmark key—so a rerun only picks up new data. This is Glue's version of an incremental watermark, scoped to one job, not a global ledger.
- Bad-record handling: a
DynamicFrameread lets you resolve type mismatches explicitly (resolveChoice) or filter unexpected rows to a separate sink instead of crashing the whole job or silently nulling values out.
Use a crawler to bootstrap an unknown dataset's schema, or when the data's shape is genuinely expected to keep changing. Lock the schema—explicit DDL, and a crawler restricted to adding partitions or new columns only—the moment the table has downstream consumers who assumed stability, and route anything that doesn't match to quarantine rather than let the catalog silently redefine the table underneath them.
Under the hood
Glue Spark jobs are billed by worker-hour (or DPU-hour, depending on how you provision workers) for however long the managed Spark cluster is up, not just the time your code actively processes rows—cold start to provision Spark before your script's first line runs can be tens of seconds to a couple of minutes, which matters for small, frequent jobs more than large, infrequent ones. Job bookmarks are per-job state, not per-table: Glue records which S3 objects (by path and last-modified timestamp) or JDBC bookmark keys a specific job has already seen across its previous successful runs. That is why a backfill run under a renamed or newly created job, or files landing out of order relative to their timestamps, can both reprocess data and miss data—the bookmark compares against what this job has seen, not a universal "has this row been loaded" ledger. Current Glue major versions pin specific Spark and Python runtimes (for example, Glue 4.0 ships Spark 3.3/Python 3.10, Glue 5.0 ships Spark 3.5/Python 3.11, Glue 6.0 ships Spark 4.1/Python 3.13)—check the current Glue version table before writing a new job, since library availability and behavior change across major versions.
Minimal example Crawler inference drifts; a schema contract catches it
def infer_schema(rows):
types = {}
for row in rows:
for k, v in row.items():
types.setdefault(k, set()).add(type(v).__name__)
return {k: sorted(v) for k, v in types.items()}
batch1 = [{"amount": 499.0}, {"amount": 250.0}]
batch2 = batch1 + [{"amount": "499.00"}] # one bad batch mixes in a string
print("crawl 1:", infer_schema(batch1))
print("crawl 2:", infer_schema(batch2))
print("contract violation:", "str" in infer_schema(batch2)["amount"])crawl 1: {'amount': ['float']}
crawl 2: {'amount': ['float', 'str']}
contract violation: TrueIn production Job bookmark skips processed files; bad rows go to quarantine
import json
from pathlib import Path
raw = Path("raw/orders")
raw.mkdir(parents=True)
(raw / "part-001.json").write_text(json.dumps({"order_id": 1, "amount": 499.0}))
(raw / "part-002.json").write_text(
"\n".join([json.dumps({"order_id": 2, "amount": 120.0}),
"{not valid json",
json.dumps({"order_id": 3, "amount": -5.0})])
)
seen = set() # bookmark: S3 keys this job has already processed
def run_job(source_dir, quarantine):
todo = [f for f in sorted(source_dir.glob("*.json")) if str(f) not in seen]
loaded = 0
for f in todo:
for line in f.read_text().splitlines():
try:
row = json.loads(line)
except json.JSONDecodeError as e:
quarantine.append((f.name, f"bad-json: {e.msg}")); continue
if row.get("amount", 0) < 0:
quarantine.append((f.name, "bad-value: negative amount")); continue
loaded += 1
seen.add(str(f))
return loaded, [f.name for f in todo]
quarantine = []
loaded1, files1 = run_job(raw, quarantine)
print(f"run 1: loaded={loaded1} files={files1} quarantine={len(quarantine)}")
(raw / "part-003.json").write_text(json.dumps({"order_id": 4, "amount": 75.0}))
loaded2, files2 = run_job(raw, quarantine)
print(f"run 2 (bookmark skips old files, picks up new one): loaded={loaded2} files={files2}")
for fname, reason in quarantine:
print(f"quarantined [{fname}]: {reason}")run 1: loaded=2 files=['part-001.json', 'part-002.json'] quarantine=2
run 2 (bookmark skips old files, picks up new one): loaded=1 files=['part-003.json']
quarantined [part-002.json]: bad-json: Expecting property name enclosed in double quotes
quarantined [part-002.json]: bad-value: negative amountSee it happen Crawler, catalog and job in one pipeline
- Crawler scans a sample of the raw S3 prefix on its schedule.
- Crawler writes or updates the inferred schema in the Glue Data Catalog.
- The Spark job reads the catalog's current schema as its contract before touching new files.
- The job checks job-bookmark state so a rerun only picks up files it has not already processed.
- Rows that fail validation go to quarantine; everything else lands in curated Parquet.
- New partitions in curated get registered back into the same catalog table.
- Athena, Spark and Redshift Spectrum all query the one schema the crawler and the job agree on.
In your world
The situation
Your Jenkins job currently loads S3 CSV order exports into Postgres via pandas with a hardcoded column list. Your team now wants Athena access to the same raw data for analysts, so you point a Glue crawler at the same S3 prefix on a nightly schedule and let it build and maintain the Glue Data Catalog table, with the crawler free to update the table definition on every run.
What actually breaks
A supplier's export starts including a discount_pct column that is usually blank or a plain number, but for one bad batch arrives as the literal string "10%". The crawler re-samples on its next run, infers discount_pct as string because of that one batch, and overwrites the table definition that previously had it as double. Every Athena query doing discount_pct * amount for the revenue report starts failing with a cast error the next morning. The Jenkins pandas job, which never touched the catalog, keeps running green the whole time—so the first sign of trouble is a broken analyst dashboard, not an alert.
What you'd do on Monday
Stop letting the crawler own a column's type once a table has real consumers. Define the schema once as an explicit contract, and restrict the crawler so it can only add new columns, never change existing ones. At the file level, add validation before anything reaches the curated table: a numeric column containing "10%" gets caught and quarantined, not passed through to a table whose type nobody reviewed.
import boto3
glue = boto3.client("glue", region_name="ap-south-1")
# One-time: define the contract explicitly instead of trusting inference.
glue.create_table(
DatabaseName="orders_raw",
TableInput={
"Name": "orders",
"StorageDescriptor": {
"Columns": [
{"Name": "order_id", "Type": "bigint"},
{"Name": "amount", "Type": "double"},
{"Name": "discount_pct", "Type": "double"},
],
"Location": "s3://etl-raw/orders/",
},
"PartitionKeys": [{"Name": "day", "Type": "string"}],
},
)
# Re-point the crawler so it can only add new columns, never overwrite
# the type of a column that already exists.
glue.update_crawler(
Name="orders-crawler",
SchemaChangePolicy={"UpdateBehavior": "LOG", "DeleteBehavior": "LOG"},
Configuration='{"Version":1.0,"CrawlerOutput":'
'{"Tables":{"AddOrUpdateBehavior":"MergeNewColumns"}}}',
)Imagine you work at Flipkart
The numbers
Illustrative: 1,200 supplier feed prefixes, averaging 25 GB each, land roughly 30 TB in raw S3 every night. One crawler configured across all 1,200 prefixes samples files from each to infer schema; even at a modest 2 seconds of listing-and-sampling overhead per prefix, that is 1,200 × 2s = 2,400s (40 minutes) of crawler time before a single Spark job starts.
What the Scenario-A answer does here
The Scenario A fix—lock the schema, restrict the crawler to MergeNewColumns—still works per table, but at 1,200 tables nobody is reviewing 1,200 crawler change logs by hand every morning.
What flips at this scale
At this scale the flip is making schema-locking a deploy-time step for every new supplier table (columns defined in code, not clicked once in a console), and alerting on any crawler run that reports a nonzero TablesUpdated for a table that was supposed to be locked—that becomes a signal of a bug or an unreviewed change, not routine behavior.
The trade-off you'd defend
I would defend IaC-defined, locked schemas for every supplier table over "let the crawler figure it out and review changes," accepting the upfront cost of defining 1,200 schemas in code in exchange for catching exactly one failure mode automatically instead of relying on 1,200 manual reviews.
Illustrative arithmetic; not Flipkart-published figures.
Gotchas
Crawler overwrites a manually-fixed schema
A column that cast correctly yesterday throws a type error in Athena or a Glue job today, with no code change on your side.
The crawler re-ran, re-sampled a different batch, inferred a different type, and the default UpdateBehavior lets it overwrite the existing column definition.
Set SchemaChangePolicy.UpdateBehavior to LOG and configure CrawlerOutput.Tables.AddOrUpdateBehavior to MergeNewColumns, or stop crawling tables that already have established consumers.
Job bookmark state is per-job, not global
A backfill run (a cloned or renamed job) reprocesses files that were already loaded, duplicating rows downstream.
Bookmarks track what this specific job has seen, keyed by job identity; a new or renamed job starts with no history at all.
Reuse the original job for backfills instead of cloning it, or explicitly manage bookmark state via the API/console—and make the load idempotent regardless, so a reprocessed file can never corrupt the destination.
DynamicFrame silently nulls mismatched values
A column that should always have a value shows unexpected nulls after a run, with no error raised.
DynamicFrame's flexible "choice" typing can resolve a type mismatch by picking one type and nulling out rows that don't fit, unless you explicitly resolve the choice.
Call resolveChoice (or an explicit filter) and route anything that doesn't match the expected type to a quarantine sink instead of letting it become a silent null.
Tricky questions
Yes—it samples the data, so it reflects what's really there.
It samples a subset of files and infers from that sample. A non-representative sample—one bad batch, a one-off format—can shift the inferred schema for the whole table. The crawler reflects what it saw, not necessarily what every row that will ever land in that prefix looks like.
Yes, it's the same incremental-processing idea, just AWS's version of it.
It is Glue's own job-scoped state, tracked by file path/timestamp or a JDBC bookmark key, entirely separate from any watermark your own code maintains. The two can disagree, especially after a backfill or a renamed job; owning the full watermark/CDC story end to end is Incremental extraction: watermarks and CDC's job, not the bookmark's.
Scenario question
You're handed a Glue table an analyst says started returning wrong revenue numbers this morning. The crawler ran at its usual 2 a.m. schedule. What do you check first, and how do you prevent a repeat?
Model answer
I would check the catalog table's schema history first—specifically whether a numeric column's type changed after last night's crawl—because a silent type change is the most common cause of "numbers went wrong with no code change." If that's it, I'd compare the current schema against what the downstream query expects, fix the immediate query or revert the type, and then stop the bleeding structurally: lock that table's schema with an explicit contract and restrict the crawler to adding columns only. The trade-off is that the table stops auto-adapting to upstream changes, so a genuinely new column needs someone to notice and add it deliberately—which is exactly the trade I want once a table has real consumers.
Interview corner
CommonWhat is the Glue Data Catalog and why does it matter for an ETL pipeline?
TrickA crawler infers a column as string when it used to be double. Is the crawler broken?
Experience checkHow have you handled a schema that changed under you in an ETL job?
DesignHow would you add schema evolution support to a catalogued table without breaking existing Athena queries?
Memory hook
The catalog is the index card every engine trusts; a crawler that keeps rewriting it is a librarian who never asks before changing the card.
Lock a table's schema the moment it has consumers; let the crawler only add partitions or new columns from then on.
Self-check
What is the practical difference between letting a crawler infer a schema and defining a schema contract?
A crawler infers from a sample of files and can silently change column types or names on its next run; a schema contract is explicitly defined and does not change until someone deliberately updates it.
Why can a backfill job under a new Glue job name reprocess data that was already loaded?
Job bookmarks are tracked per job, based on job identity; a new or renamed job has no bookmark history, so it treats every file as new.
What does DynamicFrame's resolveChoice step protect against?
Rows whose field types don't match the expected schema being silently coerced to null instead of being explicitly handled or routed to quarantine.
Extend the production example: add a fourth file where amount arrives as the string "not-a-number" instead of a float. Update run_job to catch that as a third quarantine reason (bad-value: non-numeric amount) using a try/except (TypeError, ValueError) around a numeric comparison, and print the final quarantine list showing all three distinct reasons.