Catch the intended exception branch
Why you care
Over-broad exception catching is one of the most common silent-kill traps in Python ETL code. When you write except Exception in a processing loop, you catch ValueError, TypeError, AttributeError and every other error indiscriminately. That sounds safe until a library upgrade silently introduces a bug that raises AttributeError on every row — your job logs 48,213 "row skipped" warnings, writes zero rows to Postgres, and Jenkins stays green. The bugs bury themselves inside suppressed warnings. Interviewers ask about the exception hierarchy because the answer distinguishes engineers who debug by re-running from those who catch only what they mean to catch and let everything else surface immediately.
Explain like I'm 5
Think of a building's alarm panel. A smoke detector wiring fault (AttributeError) and an actual fire (ValueError in data) and the emergency-exit override (SystemExit) are three different alarm circuits. Silencing all alarms when you smell smoke fixes the immediate annoyance but also disables the emergency exit signal. You need different handlers for each circuit.
Core concept
The entire hierarchy has one root: BaseException. Two branches descend from it.
SystemExit,KeyboardInterrupt, andGeneratorExitinherit directly fromBaseException, not fromException.except Exceptionnever reaches them.- Everything else —
ValueError,TypeError,OSError,RuntimeError,AttributeErrorand all your custom exceptions — inherits fromException.
The practical rules:
- Catch the narrowest specific type you know how to handle:
except (ValueError, KeyError). - Never write
except Exceptionin a loop. A bug that slips past your catch disappears into aWARNINGlog entry. - Never write
except BaseExceptionunless you are a process supervisor that must clean up before exit. Even then, re-raise. - A bare
except:with no type catchesBaseException— worse, it gives you noas excreminder of what you swallowed.
Decision rule: if you cannot write a specific recovery action for the exception type, do not catch it. Let it propagate to the top-level handler that logs and aborts. Noisy failures beat silent data loss every time.
Minimal example BaseException vs Exception — what each catch reaches
exc_types = [
ValueError, KeyError, AttributeError, RuntimeError,
SystemExit, KeyboardInterrupt, GeneratorExit,
]
for t in exc_types:
print(f"{t.__name__:22s} sub-Exception: {issubclass(t, Exception)}")ValueError sub-Exception: True
KeyError sub-Exception: True
AttributeError sub-Exception: True
RuntimeError sub-Exception: True
SystemExit sub-Exception: False
KeyboardInterrupt sub-Exception: False
GeneratorExit sub-Exception: FalseIn production Wide catch swallows a library bug; narrow catch surfaces it
import csv
from pathlib import Path
Path("settlements.csv").write_text("id,amount\n1,250.0\n2,150.0\n3,310.0\n")
def transform(val):
"""Buggy after a library upgrade: tries to call .strip() on a float."""
return float(val).strip() # AttributeError: 'float' object has no attribute 'strip'
def load_wide(rows):
loaded = 0
for row in rows:
try:
_ = {"id": int(row["id"]), "amount": transform(row["amount"])}
loaded += 1
except Exception: # swallows AttributeError silently
pass
return loaded
def load_narrow(rows):
loaded = 0
for row in rows:
try:
_ = {"id": int(row["id"]), "amount": transform(row["amount"])}
loaded += 1
except (ValueError, KeyError): # data errors only
pass
return loaded
with open("settlements.csv", newline="") as f:
rows = list(csv.DictReader(f))
print("wide:", load_wide(rows), "loaded")
try:
print("narrow:", load_narrow(rows), "loaded")
except AttributeError as exc:
print(f"AttributeError: {exc}")
print("narrow: bug surfaces on row 1, job fails loudly")wide: 0 loaded
AttributeError: 'float' object has no attribute 'strip'
narrow: bug surfaces on row 1, job fails loudlyIn your world
The situation
Your Jenkins Python ETL job processes supplier payments from S3. The parsing loop has except Exception as e: logging.warning("row skipped: %s", e); continue to tolerate bad rows from dirty source files.
What actually breaks
A third-party library update changes an internal method. Its AttributeError is swallowed for every single row. The job logs 48,213 "row skipped" warnings and writes zero rows to Postgres. Jenkins is green. Finance notices twelve hours later.
What you'd do on Monday
Change the catch to except (ValueError, KeyError, TypeError) — the exact errors the loop knows how to handle. The AttributeError from the library update propagates, Jenkins turns red, and you get paged in ninety seconds instead of being blamed after half a day. Also add a skip-rate assertion: if more than 0.5% of rows are skipped, abort immediately.
skipped = 0
for row in reader:
try:
record = parse(row)
except (ValueError, KeyError, TypeError) as exc:
logging.warning("row skipped: %s", exc)
skipped += 1
continue
cursor.execute(INSERT_SQL, record)
loaded += 1
if loaded == 0:
raise RuntimeError("0 rows loaded — aborting before audit commit")
if skipped / (loaded + skipped) > 0.005:
raise RuntimeError(f"skip rate {skipped/(loaded+skipped):.1%} exceeds 0.5% threshold")Imagine you work at Airbnb
The numbers
Illustrative: 200M records/night split across 400 workers of 500K records each. One library change adds an AttributeError to 0.1% of records. 200M × 0.001 = 200,000 records silently dropped. Each worker processes 500K ÷ 1,000 (batch size) = 500 batches. With except Exception, all 200,000 drops are logged as WARNING with no skip-rate threshold. The data team sees an unexplained 0.1% gap the next morning and cannot pin the cause.
What the Scenario-A answer does here
The Scenario A fix (narrow the catch) still works per batch. What breaks at scale is observability: even with a narrow catch, without a per-batch skip-rate check, 200 drops per batch across 400 workers is a statistical bleed that aggregates monitoring does not catch.
What flips at this scale
At this scale you enforce a per-batch error budget: if skipped / total > 0.005: raise RuntimeError(f"batch {batch_id} skip rate {skipped/total:.1%}"). One bad batch fails loudly within seconds. The pipeline pauses, the alert fires, and the bad library version is rolled back before it infects the full 200M-record run.
The trade-off you'd defend
I would defend a skip-rate circuit-breaker over catch-all logging, trading a handful of early loud failures for the ability to catch systemic bugs before they corrupt weeks of data.
Illustrative arithmetic; not Airbnb-published figures.
Gotchas
except Exception in a loop swallows bugs permanently
The job logs hundreds of "row skipped" warnings but no traceback; loaded row count is far below the source file row count.
A non-data exception (AttributeError, NameError, TypeError) matches except Exception and is swallowed on every row instead of surfacing once.
Narrow the catch to the specific data-parsing errors you know how to skip. Let everything else propagate.
A bare except: also catches SystemExit and KeyboardInterrupt
Ctrl-C does not stop the job; the script keeps running or prints "skipped row" repeatedly.
except: with no type matches BaseException, including KeyboardInterrupt. The keyboard interrupt is swallowed as if it were a data error.
Never write a bare except:. At minimum write except Exception:, but prefer a specific type.
Catching a parent hides a child's specialised handler
except OSError runs for a PermissionError that had its own intended handler further down.
Python matches the first clause whose type is a superclass of the raised type. PermissionError is a subclass of OSError, so the OSError handler wins.
Always order except clauses from most specific to most general: except PermissionError before except OSError.
Tricky questions
except Exception catch SystemExit?Yes — SystemExit is an exception so it must be under Exception.
SystemExit inherits directly from BaseException, not Exception. except Exception never catches it, which is intentional: you want sys.exit() to actually exit the process.
except Exception safe if I always re-raise?Yes, because re-raising means nothing is swallowed.
Even except Exception: ...; raise is dangerous because the code inside the handler can itself raise, replacing the original exception. Use except Exception: log.exception("..."); raise and nothing else — resist the urge to add any logic.
Scenario question
You're handed a job that has run for six months. The team says it occasionally loads zero rows but they cannot reproduce it. The only error handling is except Exception as e: log.warning(e); continue. What do you change first, and why?
Model answer
I'd narrow the catch before anything else. My hypothesis is that some exception type — a library change, a schema drift, or an AttributeError on a renamed field — is being caught at WARNING level instead of surfacing. I grep the codebase for except Exception in any loop, list what errors each actually knows how to handle, and replace each one with a specific tuple. I'd also add a skip-rate assertion: if more than 0.5% of rows are skipped in a single batch, raise and abort. The trade-off is that some jobs may start failing loudly for the first time; that is the desired outcome — silent zero-row loads are worse than loud failures.
Interview corner
CommonWhat is the difference between Exception and BaseException?
TrickWhy is a bare except: dangerous even if you re-raise?
Experience checkHas a broad except ever cost you data?
CommonHow do you order multiple except clauses?
Memory hook
Exception hierarchy is an alarm panel: each circuit has a dedicated handler. Silencing all circuits when one trips is how zero-row loads stay green for twelve hours.
Catch only the exception types you know how to handle, and let everything else propagate.
Self-check
Name the three built-in types that are under BaseException but not under Exception.
SystemExit, KeyboardInterrupt, GeneratorExit.
Why does except Exception not catch SystemExit?
Because SystemExit inherits from BaseException directly, not from Exception.
If you have
except OSErrorandexcept PermissionError, which should come first?except PermissionError— it is more specific. Python matches the first superclass, so ifOSErrorcame first it would consume allPermissionErrorinstances.What is the risk of except Exception in a batch loop even if you log the error?
The logged message loses the traceback and type context. A systematic bug (like an AttributeError on every row) appears as thousands of WARNING log lines rather than one loud traceback, masking the root cause.
Write a function safe_parse(raw) that tries int(raw), catches only ValueError, and returns None on bad input. Show that safe_parse("42") returns 42, safe_parse("x") returns None, and safe_parse(None) raises TypeError (not caught). Paste all three outputs.