Shelf · Python · Chapter 6

Chapter 6: Errors & defensive Python

Every Python program fails eventually; this chapter turns failures from embarrassing surprises into predictable events you designed for. By the end you can name exactly which exception a block is meant to handle, wire up the four control-flow paths of a try statement, choose the right defensive style at a boundary, and decide when to log, propagate or recover.

Difficulty
Priority
py-06-01Card 1 of 6

Catch the intended exception branch

MediumP1 · Must knowKnow cold10 min
exceptionshierarchybaseexceptionnarrow-catchetl
Needs: nothing. Start here.

Why you care

Over-broad exception catching is one of the most common silent-kill traps in Python ETL code. When you write except Exception in a processing loop, you catch ValueError, TypeError, AttributeError and every other error indiscriminately. That sounds safe until a library upgrade silently introduces a bug that raises AttributeError on every row — your job logs 48,213 "row skipped" warnings, writes zero rows to Postgres, and Jenkins stays green. The bugs bury themselves inside suppressed warnings. Interviewers ask about the exception hierarchy because the answer distinguishes engineers who debug by re-running from those who catch only what they mean to catch and let everything else surface immediately.

Explain like I'm 5

Think of a building's alarm panel. A smoke detector wiring fault (AttributeError) and an actual fire (ValueError in data) and the emergency-exit override (SystemExit) are three different alarm circuits. Silencing all alarms when you smell smoke fixes the immediate annoyance but also disables the emergency exit signal. You need different handlers for each circuit.

Core concept

The entire hierarchy has one root: BaseException. Two branches descend from it.

  • SystemExit, KeyboardInterrupt, and GeneratorExit inherit directly from BaseException, not from Exception. except Exception never reaches them.
  • Everything else — ValueError, TypeError, OSError, RuntimeError, AttributeError and all your custom exceptions — inherits from Exception.

The practical rules:

  • Catch the narrowest specific type you know how to handle: except (ValueError, KeyError).
  • Never write except Exception in a loop. A bug that slips past your catch disappears into a WARNING log entry.
  • Never write except BaseException unless you are a process supervisor that must clean up before exit. Even then, re-raise.
  • A bare except: with no type catches BaseException — worse, it gives you no as exc reminder of what you swallowed.

Decision rule: if you cannot write a specific recovery action for the exception type, do not catch it. Let it propagate to the top-level handler that logs and aborts. Noisy failures beat silent data loss every time.

Minimal example BaseException vs Exception — what each catch reaches

Pythonruns
exc_types = [
    ValueError, KeyError, AttributeError, RuntimeError,
    SystemExit, KeyboardInterrupt, GeneratorExit,
]
for t in exc_types:
    print(f"{t.__name__:22s} sub-Exception: {issubclass(t, Exception)}")
Output
ValueError             sub-Exception: True
KeyError               sub-Exception: True
AttributeError         sub-Exception: True
RuntimeError           sub-Exception: True
SystemExit             sub-Exception: False
KeyboardInterrupt      sub-Exception: False
GeneratorExit          sub-Exception: False

In production Wide catch swallows a library bug; narrow catch surfaces it

Pythonruns
import csv
from pathlib import Path

Path("settlements.csv").write_text("id,amount\n1,250.0\n2,150.0\n3,310.0\n")


def transform(val):
    """Buggy after a library upgrade: tries to call .strip() on a float."""
    return float(val).strip()   # AttributeError: 'float' object has no attribute 'strip'


def load_wide(rows):
    loaded = 0
    for row in rows:
        try:
            _ = {"id": int(row["id"]), "amount": transform(row["amount"])}
            loaded += 1
        except Exception:   # swallows AttributeError silently
            pass
    return loaded


def load_narrow(rows):
    loaded = 0
    for row in rows:
        try:
            _ = {"id": int(row["id"]), "amount": transform(row["amount"])}
            loaded += 1
        except (ValueError, KeyError):   # data errors only
            pass
    return loaded


with open("settlements.csv", newline="") as f:
    rows = list(csv.DictReader(f))

print("wide:", load_wide(rows), "loaded")

try:
    print("narrow:", load_narrow(rows), "loaded")
except AttributeError as exc:
    print(f"AttributeError: {exc}")
    print("narrow: bug surfaces on row 1, job fails loudly")
Output
wide: 0 loaded
AttributeError: 'float' object has no attribute 'strip'
narrow: bug surfaces on row 1, job fails loudly
Scenario A

In your world

The situation

Your Jenkins Python ETL job processes supplier payments from S3. The parsing loop has except Exception as e: logging.warning("row skipped: %s", e); continue to tolerate bad rows from dirty source files.

What actually breaks

A third-party library update changes an internal method. Its AttributeError is swallowed for every single row. The job logs 48,213 "row skipped" warnings and writes zero rows to Postgres. Jenkins is green. Finance notices twelve hours later.

What you'd do on Monday

Change the catch to except (ValueError, KeyError, TypeError) — the exact errors the loop knows how to handle. The AttributeError from the library update propagates, Jenkins turns red, and you get paged in ninety seconds instead of being blamed after half a day. Also add a skip-rate assertion: if more than 0.5% of rows are skipped, abort immediately.

PythonNarrow catch plus skip-rate guard (needs psycopg + real connection)
skipped = 0
for row in reader:
    try:
        record = parse(row)
    except (ValueError, KeyError, TypeError) as exc:
        logging.warning("row skipped: %s", exc)
        skipped += 1
        continue
    cursor.execute(INSERT_SQL, record)
    loaded += 1

if loaded == 0:
    raise RuntimeError("0 rows loaded — aborting before audit commit")
if skipped / (loaded + skipped) > 0.005:
    raise RuntimeError(f"skip rate {skipped/(loaded+skipped):.1%} exceeds 0.5% threshold")
Scenario B

Imagine you work at Airbnb

200M listing-event records per night

The numbers

Illustrative: 200M records/night split across 400 workers of 500K records each. One library change adds an AttributeError to 0.1% of records. 200M × 0.001 = 200,000 records silently dropped. Each worker processes 500K ÷ 1,000 (batch size) = 500 batches. With except Exception, all 200,000 drops are logged as WARNING with no skip-rate threshold. The data team sees an unexplained 0.1% gap the next morning and cannot pin the cause.

What the Scenario-A answer does here

The Scenario A fix (narrow the catch) still works per batch. What breaks at scale is observability: even with a narrow catch, without a per-batch skip-rate check, 200 drops per batch across 400 workers is a statistical bleed that aggregates monitoring does not catch.

What flips at this scale

At this scale you enforce a per-batch error budget: if skipped / total > 0.005: raise RuntimeError(f"batch {batch_id} skip rate {skipped/total:.1%}"). One bad batch fails loudly within seconds. The pipeline pauses, the alert fires, and the bad library version is rolled back before it infects the full 200M-record run.

The trade-off you'd defend

I would defend a skip-rate circuit-breaker over catch-all logging, trading a handful of early loud failures for the ability to catch systemic bugs before they corrupt weeks of data.

Illustrative arithmetic; not Airbnb-published figures.

Gotchas

except Exception in a loop swallows bugs permanently

Symptom

The job logs hundreds of "row skipped" warnings but no traceback; loaded row count is far below the source file row count.

Cause

A non-data exception (AttributeError, NameError, TypeError) matches except Exception and is swallowed on every row instead of surfacing once.

Fix

Narrow the catch to the specific data-parsing errors you know how to skip. Let everything else propagate.

A bare except: also catches SystemExit and KeyboardInterrupt

Symptom

Ctrl-C does not stop the job; the script keeps running or prints "skipped row" repeatedly.

Cause

except: with no type matches BaseException, including KeyboardInterrupt. The keyboard interrupt is swallowed as if it were a data error.

Fix

Never write a bare except:. At minimum write except Exception:, but prefer a specific type.

Catching a parent hides a child's specialised handler

Symptom

except OSError runs for a PermissionError that had its own intended handler further down.

Cause

Python matches the first clause whose type is a superclass of the raised type. PermissionError is a subclass of OSError, so the OSError handler wins.

Fix

Always order except clauses from most specific to most general: except PermissionError before except OSError.

Tricky questions

Does except Exception catch SystemExit?
Most people say

Yes — SystemExit is an exception so it must be under Exception.

Actually

SystemExit inherits directly from BaseException, not Exception. except Exception never catches it, which is intentional: you want sys.exit() to actually exit the process.

Is a broad except Exception safe if I always re-raise?
Most people say

Yes, because re-raising means nothing is swallowed.

Actually

Even except Exception: ...; raise is dangerous because the code inside the handler can itself raise, replacing the original exception. Use except Exception: log.exception("..."); raise and nothing else — resist the urge to add any logic.

Scenario question

You're handed a job that has run for six months. The team says it occasionally loads zero rows but they cannot reproduce it. The only error handling is except Exception as e: log.warning(e); continue. What do you change first, and why?

Model answer

I'd narrow the catch before anything else. My hypothesis is that some exception type — a library change, a schema drift, or an AttributeError on a renamed field — is being caught at WARNING level instead of surfacing. I grep the codebase for except Exception in any loop, list what errors each actually knows how to handle, and replace each one with a specific tuple. I'd also add a skip-rate assertion: if more than 0.5% of rows are skipped in a single batch, raise and abort. The trade-off is that some jobs may start failing loudly for the first time; that is the desired outcome — silent zero-row loads are worse than loud failures.

Interview corner

CommonWhat is the difference between Exception and BaseException?
“I'd say Exception is the base for all errors you normally handle. BaseException is the root of the entire hierarchy, and three important types — SystemExit, KeyboardInterrupt and GeneratorExit — sit under BaseException but not under Exception. That's why except Exception doesn't swallow Ctrl-C.”
TrickWhy is a bare except: dangerous even if you re-raise?
“I'd say that even with a re-raise, any code inside the handler can raise a new exception that replaces the original. A bare except also catches KeyboardInterrupt, so Ctrl-C stops responding. My rule is to always write a specific type or at worst except Exception — never a bare except.”
Experience checkHas a broad except ever cost you data?
“I've seen this firsthand — an ETL job where the catch was except Exception and a library upgrade raised AttributeError on every row. The job logged 48K warnings, loaded zero rows, and Jenkins stayed green. I narrowed the catch to ValueError and KeyError; the AttributeError surfaced immediately on the next run and we found the library regression in minutes.”
CommonHow do you order multiple except clauses?
“I'd always put the most specific first. PermissionError before OSError, because Python matches the first clause whose type is a superclass of the raised type. If OSError comes first, PermissionError never reaches its own handler.”

Memory hook

Exception hierarchy is an alarm panel: each circuit has a dedicated handler. Silencing all circuits when one trips is how zero-row loads stay green for twelve hours.

If you remember only one thing:

Catch only the exception types you know how to handle, and let everything else propagate.

Self-check

  1. Name the three built-in types that are under BaseException but not under Exception.

    SystemExit, KeyboardInterrupt, GeneratorExit.

  2. Why does except Exception not catch SystemExit?

    Because SystemExit inherits from BaseException directly, not from Exception.

  3. If you have except OSError and except PermissionError, which should come first?

    except PermissionError — it is more specific. Python matches the first superclass, so if OSError came first it would consume all PermissionError instances.

  4. What is the risk of except Exception in a batch loop even if you log the error?

    The logged message loses the traceback and type context. A systematic bug (like an AttributeError on every row) appears as thousands of WARNING log lines rather than one loud traceback, masking the root cause.

Prove it

Write a function safe_parse(raw) that tries int(raw), catches only ValueError, and returns None on bad input. Show that safe_parse("42") returns 42, safe_parse("x") returns None, and safe_parse(None) raises TypeError (not caught). Paste all three outputs.

py-06-02Card 2 of 6

Represent domain failures explicitly

MediumP2 · Very likelyKnow cold10 min
custom-exceptionsexception-chainingraise-fromdomain-errors

Why you care

When a function raises KeyError because a required field is missing in a payment record, the caller cannot tell whether that is a programming error (wrong key name in code) or a data problem (missing column in source). Custom exceptions are contracts: they tell every caller, every logger, and every on-call engineer exactly what went wrong without reading source code. The raise X from Y pattern preserves the original cause so the chain survives serialisation to log files. Interviewers ask about exception chaining because it separates engineers who build diagnosable systems from those who lose the root cause at the first boundary.

Explain like I'm 5

When a delivery rider returns without the parcel and writes "delivery failed" on the form, the warehouse team cannot act. If the form says "PARCEL REJECTED: address unrecognised by courier system" and staples the courier's original refusal slip to it, the team knows exactly who to call. The custom exception is the filled-in form; raise X from Y is the stapled-on original slip.

Core concept

A custom exception is a class that inherits from Exception or a narrower base you control:

PythonMinimal custom exception hierarchy
class SettlementError(Exception):
    """Base for all settlement-pipeline failures."""

class MissingFieldError(SettlementError):
    """A required field is absent from the source record."""

class InvalidValueError(SettlementError):
    """A field is present but its value fails validation."""

Rules that matter on the job:

  • Inherit from a domain base (SettlementError), not directly from Exception. Callers can catch the base to handle any settlement error, or catch a subclass for a specific failure.
  • Attach context in __init__: the record id, the field name, the bad value. Logs and alerts can print e.record_id without parsing the message string.
  • Preserve causes with raise X from Y: when translating a low-level KeyError into a MissingFieldError, from original_exc stores the original in __cause__. Python prints both when the exception is unhandled, and structured loggers can record both.
  • Never raise DomainError(str(original_exc)) — that converts the cause into a string and loses its type.

Decision rule: if a caller would need to inspect the exception message to decide what to do, that information should be an attribute instead.

Minimal example Custom hierarchy plus raise-from chaining

Pythonruns
class PaymentError(Exception):
    def __init__(self, pid, reason):
        super().__init__(f"payment {pid}: {reason}")
        self.pid = pid


class MissingFieldError(PaymentError):
    pass


class InvalidValueError(PaymentError):
    pass


def parse_payment(raw):
    pid = raw.get("id", "?")
    try:
        amount = float(raw["amount"])
    except KeyError as exc:
        raise MissingFieldError(pid, "missing 'amount'") from exc
    except ValueError as exc:
        raise InvalidValueError(pid, f"bad amount {raw['amount']!r}") from exc
    if amount <= 0:
        raise InvalidValueError(pid, f"amount must be positive, got {amount}")
    return {"id": pid, "amount": amount}


records = [
    {"id": "P1", "amount": "99.50"},
    {"id": "P2"},
    {"id": "P3", "amount": "N/A"},
    {"id": "P4", "amount": "-10"},
]
for rec in records:
    try:
        print("ok:", parse_payment(rec))
    except MissingFieldError as e:
        print(f"MissingFieldError: {e}")
    except InvalidValueError as e:
        print(f"InvalidValueError: {e}")
Output
ok: {'id': 'P1', 'amount': 99.5}
MissingFieldError: payment P2: missing 'amount'
InvalidValueError: payment P3: bad amount 'N/A'
InvalidValueError: payment P4: amount must be positive, got -10.0

In production Domain exception hierarchy for a payment ETL pipeline

Pythonruns
from pathlib import Path
import csv


class PipelineError(Exception):
    """Root for all pipeline-domain errors. Carries record_id for routing."""
    def __init__(self, record_id, message):
        super().__init__(f"[{record_id}] {message}")
        self.record_id = record_id


class DataError(PipelineError):
    """Bad input data — send to dead-letter queue, do not retry."""


class ServiceError(PipelineError):
    """Downstream service unavailable — safe to retry."""


def validate_row(row: dict) -> dict:
    rid = row.get("id", "?")
    for field in ("id", "amount", "currency"):
        if field not in row:
            raise DataError(rid, f"missing required field {field!r}")
    try:
        amount = float(row["amount"])
    except ValueError as exc:
        raise DataError(rid, f"bad amount {row['amount']!r}") from exc
    if row["currency"] not in ("USD", "EUR", "INR"):
        raise DataError(rid, f"unknown currency {row['currency']!r}")
    return {"id": row["id"], "amount": amount, "currency": row["currency"]}


Path("payments.csv").write_text(
    "id,amount,currency\n"
    "T1,250.00,USD\n"
    "T2,bad,INR\n"
    "T3,300.00,XYZ\n"
    "T4,150.00,EUR\n"
)

good, dead_letter = [], []
with open("payments.csv", newline="") as f:
    for row in csv.DictReader(f):
        try:
            good.append(validate_row(row))
        except DataError as e:
            dead_letter.append({"record_id": e.record_id, "reason": str(e)})

print(f"valid: {len(good)}")
print(f"dead-letter: {len(dead_letter)}")
for item in dead_letter:
    print(f"  {item['reason']}")
Output
valid: 2
dead-letter: 2
  [T2] bad amount 'bad'
  [T3] unknown currency 'XYZ'
Scenario A

In your world

The situation

Your pipeline uploads transformed records to S3 using an internal utility. The utility raises botocore.exceptions.ClientError on a bad request. Your code catches it and re-raises as RuntimeError("upload failed").

What actually breaks

Three months later the ops team searches Splunk for "upload failed" and finds fourteen incidents but no root causes. All Splunk shows is RuntimeError: upload failed. Whether the original error was a credential expiry, a bucket permission denial, or a wrong key prefix — it is gone.

What you'd do on Monday

Use raise UploadError(path, "upload failed") from exc where UploadError is your own class that stores the path as an attribute. The __cause__ chain survives to the log file. Splunk can now filter exc_cause_type:ClientError to separate credential problems from bucket problems in seconds.

PythonPreserve the cause when translating (needs botocore)
class UploadError(PipelineError):
    """S3 upload failed; record_id is the S3 key."""

try:
    s3.put_object(Bucket=bucket, Key=key, Body=data)
except ClientError as exc:
    raise UploadError(key, f"S3 upload failed: {exc.response['Error']['Code']}") from exc
Scenario B

Imagine you work at Grab

3M rides completed per day

The numbers

Illustrative: 3M rides/day across driver, routing and payment microservices, each translating low-level errors into domain exceptions. At 0.5% fault rate that is 15,000 incidents/day. Without exception chaining, each PagerDuty alert says "domain failed" and the on-call engineer must correlate log timestamps across three services to find the original network error — averaging 22 minutes per incident.

What the Scenario-A answer does here

Adding from exc to every raise works locally. At this volume the problem shifts: different services wrap exceptions inconsistently. Some teams add from exc, others forget, and some convert the original to a string. The chain is present for 60% of incidents and absent for 40%.

What flips at this scale

You define a shared GrabServiceError(message, service_name, cause_code) base class in an internal library. All gateways import it. A middleware decorator automatically logs exc.__cause__ if present. The contract is enforced by a linter rule: raise DomainError(...) without from cause_var is a linting error in CI.

The trade-off you'd defend

I would defend a linted shared-base contract over per-team convention, trading a one-time library and CI investment for cutting mean incident diagnosis time by more than half.

Illustrative; not Grab-published figures.

Gotchas

raise DomainError(str(exc)) loses the cause type

Symptom

except DataError as e: print(e.__cause__) prints None; searching logs for ClientError finds nothing.

Cause

str(exc) converts the original exception to a string. The resulting DataError has no __cause__ attribute chain.

Fix

Always use raise DomainError(message) from exc. If you need the original message, store it as an attribute: self.original = str(exc), but also set __cause__.

Catching the base and missing a subclass's attributes

Symptom

except PipelineError as e: route(e.record_id) raises AttributeError for some errors.

Cause

Not every subclass initialises record_id. A direct PipelineError("plain message") is missing the attribute.

Fix

Put required attributes in the base __init__ and call super().__init__(record_id, message) in every subclass. Use dataclasses for the base if the attribute list grows.

Re-raising with raise in an except block vs raise exc

Symptom

The traceback shows the re-raise line, not the original raise site.

Cause

raise exc creates a new traceback from the raise line. A bare raise inside an except block re-raises with the original traceback intact.

Fix

Use bare raise to re-raise without modification. Use raise NewError(...) from exc to translate. Never raise exc when you mean to preserve the original location.

Tricky questions

Can you raise a built-in exception with from?
Most people say

No — from only works with custom exceptions.

Actually

raise ValueError("bad") from original_exc is perfectly valid. from works on any exception and any cause. from None explicitly suppresses the cause chain when you want to hide internal details from callers.

Is raise X from X the same as raise X?
Most people say

Yes, it's just redundant.

Actually

raise X from X sets __cause__ = X which also sets __suppress_context__ = True, suppressing the implicit __context__. Bare raise X in an except block sets __context__ but not __cause__. The printed output differs: from prints "The above exception was the direct cause of the following exception", while implicit context prints "During handling of the above exception, another exception occurred".

Scenario question

You're handed a payments microservice that raises RuntimeError("payment failed") on every error path. On-call says they cannot diagnose incidents fast enough. What do you change, and in what order?

Model answer

I'd start by introducing a domain exception hierarchy with a PaymentError base and subclasses for AuthorizationError, InsufficientFundsError, and ServiceUnavailableError. Then I'd audit every raise RuntimeError("payment failed") and replace it with the appropriate subclass, using from original_exc to preserve the cause. I'd do the highest-volume incident type first — usually ServiceUnavailableError — because that immediately cuts the noise for on-call. The trade-off is that this touches many files; an incremental approach by incident type is safer than a big-bang refactor.

Interview corner

CommonWhat does raise X from Y do?
“I use raise X from Y whenever I translate a low-level exception into a domain one — it raises X and stores Y as X.__cause__, so Python prints both when the exception is unhandled. It also sets __suppress_context__ to True, which hides the implicit context. The caller sees the domain error; the log shows the original cause.”
Experience checkWhen have you used custom exceptions on the job?
“I replaced all the bare RuntimeError raises in a payments ETL with a DataError and ServiceError hierarchy. DataError meant 'send to dead-letter queue', ServiceError meant 'safe to retry'. Before that, every error handler had to parse the message string to decide what to do. After, the routing logic became a two-line except chain.”
TrickWhat is the difference between __cause__ and __context__?
“I'd say __cause__ is set explicitly by raise X from Y and means 'this caused that'. __context__ is set implicitly by Python whenever an exception is raised inside an except block. If you want to hide the context — say you're raising a clean public error from an internal mess — you use raise X from None to set __cause__ to None and suppress the context.”

Memory hook

A custom exception is the filled-in form; raise X from Y is the refusal slip stapled to it. Without the slip the warehouse team can only guess what went wrong.

If you remember only one thing:

Use raise DomainError(id, reason) from original_exc — never convert the cause to a string.

Self-check

  1. What does raise X from None do to the exception chain?

    Sets __cause__ = None and __suppress_context__ = True. Python prints only X, hiding the implicit context exception.

  2. How do you attach a record id to a custom exception so callers can access it programmatically?

    Set self.record_id = record_id in __init__ after calling super().__init__(message).

  3. Why should a domain exception hierarchy have a base class?

    So callers can catch PipelineError to handle any pipeline failure, or catch a subclass for a specific one, without knowing every leaf type.

  4. What is wrong with raise DataError(str(exc)) from a diagnosability standpoint?

    It converts the original exception to a string, losing its type and attributes. exc.__cause__ will be None. Structured logs cannot filter by original exception type.

Prove it

Write parse_trade(raw) that raises MissingFieldError (with __cause__) for a missing trade_id key, InvalidValueError for a non-positive notional, and returns a dict on success. Verify with three test cases: a good row, a missing key, and a bad value. Print type(e).__name__, str(e), and type(e.__cause__).__name__ for each failure. Paste the output.

py-06-03Card 3 of 6

Trace try, else and finally paths

MediumP1 · Must knowKnow cold10 min
try-else-finallycontrol-flowcleanupreturn-masking

Why you care

The else block of a try statement is almost never taught and is genuinely unknown to most three-year Python developers. More critically, a return inside finally silently swallows any pending exception — the job reports success while the database row was never written. Both facts change how you write production-safe code. Interviewers ask about the four paths through a try statement because the else and finally combinations catch bugs in code that already uses try/except.

Explain like I'm 5

Think of baking bread with a dangerous oven. The try block is the attempt. The except block is what you do when the oven catches fire. The else block is toasting the bread only when there was no fire — success-only actions. The finally block is putting the fire extinguisher back regardless of what happened. The trap: if you write a return inside "put the extinguisher back", you override everything else — even a fire alarm someone raised — and report that the kitchen is fine.

Core concept

A try statement has four mutually exclusive paths depending on what happens in the try body:

  • No exception: try body finishes normally → else block runs → finally block runs.
  • Matched exception: exception raised → matching except clause runs → finally block runs. else is skipped.
  • Unmatched exception: exception raised, no clause matches → finally runs → exception propagates to the caller. else is skipped.
  • return or break in try: finally still runs before the return is honoured.

Rules that determine what you put where:

  • Put success-only actions in else, not in try. Code in try is only there to establish what you're guarding; extra logic in try widens the catch surface unnecessarily.
  • Put cleanup in finally: closing file handles, releasing locks, logging the completion status.
  • Never put a return in finally if any other path also returns or raises. A return in finally silently wins over all of them, including a pending unhandled exception.

Decision rule: if the action only makes sense after the try body succeeded, put it in else. If the action must happen regardless, put it in finally. If there is any doubt, prefer a context manager (with) over manual try/finally (Context managers and deterministic cleanup).

Minimal example return in finally swallows a pending exception

Pythonruns
def risky():
    try:
        raise ValueError("real error")
    finally:
        return "finally wins"   # overwrites the ValueError


try:
    result = risky()
    print("result:", result)
    print("no exception reached here")
except ValueError as exc:
    print("caught:", exc)   # never runs
Output
result: finally wins
no exception reached here

In production try/else/finally with correct path separation

Pythonruns
import csv
from pathlib import Path

Path("orders.csv").write_text("id,amount\n1,100\n2,200\n3,300\n")


def process_orders(path):
    loaded, skipped = 0, 0
    fh = None
    try:
        fh = open(path, newline="")
        for row in csv.DictReader(fh):
            try:
                float(row["amount"])
                loaded += 1
            except (ValueError, KeyError):
                skipped += 1
    except FileNotFoundError:
        print("FileNotFoundError: file not found")
        return None, None
    else:
        # only if outer try succeeded — success-only action
        print(f"success: {loaded} loaded, {skipped} skipped")
        return loaded, skipped
    finally:
        # always — cleanup only, no return
        if fh is not None and not fh.closed:
            fh.close()
        print("cleanup: file closed")


process_orders("orders.csv")
print("---")
process_orders("no_such_file.csv")
Output
success: 3 loaded, 0 skipped
cleanup: file closed
---
FileNotFoundError: file not found
cleanup: file closed

See it happen The four paths through try/except/else/finally

succeedsraisesmatchedunmatchedtry blockno exceptionexception raisedelse blockexcept blockfinally block
  1. Execution enters the try block. This is the only guarded section.
  2. If the try body completes without raising, the else block runs. This is where success-only actions belong.
  3. If an exception is raised and a matching except clause exists, it runs instead of else.
  4. If an exception is raised but no clause matches, finally still runs before the exception propagates to the caller.
  5. Finally always runs: after else, after except, and on unmatched propagation. A return inside finally overwrites any pending return or exception.
Notice: the else block is the success path — use it instead of extra code at the end of try. Finally is the cleanup guarantee — never put a return there.
Scenario A

In your world

The situation

Your batch job opens a binary file, processes its records, and has finally: file_handle.close(); return True to report success to the Jenkins scheduler plugin.

What actually breaks

A corrupt record raises struct.error mid-processing. The finally block runs, closes the handle, and then return True overwrites the pending exception. The scheduler records "success", writes zero rows, and the next run picks up the same corrupt file again. The job loops indefinitely.

What you'd do on Monday

Remove the return from finally. Put return True at the end of the try block after the processing loop, and return False in the except clause. Use finally only for file_handle.close() — no return, no raise, no value production of any kind.

Pythonreturn only in try or except, never in finally
def process_file(path):
    fh = None
    try:
        fh = open(path, "rb")
        process(fh)
        return True           # success: only here
    except (OSError, struct.error) as exc:
        logging.exception("processing failed: %s", path)
        return False          # known failure: explicit
    finally:
        if fh:
            fh.close()        # cleanup only — no return
Scenario B

Imagine you work at IRCTC

1.2M ticket bookings per day

The numbers

Illustrative: 1.2M bookings/day peaks at 800K in a 6-hour window, ≈ 37 transactions per second. A return True in a finally block inside the payment-confirmation handler means any IntegrityError from a duplicate booking attempt is reported as success. At the peak rate of 37 TPS with a 0.01% duplicate rate, that is roughly one false confirmation per 270 seconds during the peak window — and about 120 silent duplicates across the full day (1.2M × 0.01%).

What the Scenario-A answer does here

Removing return from finally fixes the immediate problem. At 37 TPS the scale problem is side-effects: a confirmed-but-not-persisted booking triggers a downstream email, an SMS notification and an inventory decrement — three irreversible side-effects on a transaction that was silently rolled back.

What flips at this scale

The side-effects must fire only after the database commit is confirmed. The else block becomes the coordinator: commit first, then send downstream events. The finally block handles only connection cleanup. If the commit raises, else never runs, and no downstream event fires.

The trade-off you'd defend

I would defend deferred side-effects gated on the commit result over optimistic firing, trading slightly more complex control flow for correctness guarantees the business relies on.

Illustrative; not IRCTC-published figures.

Gotchas

return in finally discards a pending exception silently

Symptom

The job reports success and a caller receives a return value; no exception is visible anywhere. The data was never written.

Cause

A return statement inside finally is executed before the pending exception propagates, and it discards the exception entirely.

Fix

Never put return in finally. Move all return statements to the try body or the except clause.

Code at the end of try runs even after an exception if no except clause matches

Symptom

Code after the failing line inside the try block never runs, but a line added accidentally at the very end of try seems to execute — or not — unpredictably.

Cause

Once an exception is raised, execution skips to the matching except or to finally. Lines after the raise site inside try do not run.

Fix

Use else for any code that should run only when the whole try body succeeded. Never rely on lines at the end of try.

finally hides exceptions raised inside itself

Symptom

The original exception message changes to something from the cleanup code; the original traceback is lost.

Cause

If the finally block itself raises an exception, it replaces the pending exception.

Fix

In finally, only perform operations that are unlikely to fail (close already-open handles, log). Wrap risky cleanup in its own try/except inside finally.

Tricky questions

Does else run if the try body has a return statement?
Most people say

No — the function has already returned.

Actually

The else block does not run if try has a return; finally does run. The else block runs only when try exits normally by reaching its end without a return, break, or continue. This is why putting a return inside try skips else.

Can finally re-raise the original exception?
Most people say

No — finally is for cleanup, not exception handling.

Actually

A bare raise inside finally re-raises the current exception. But this is rarely needed; if finally itself raises, the new exception replaces the original. Use finally for cleanup, and leave exception decisions to except.

Scenario question

You're handed a file processor that reports success but writes zero bytes. The code has finally: file_handle.flush(); return True. What happened and what do you fix?

Model answer

I'd say the return True in finally is swallowing a pending exception from the try block. When the try body raises — say an OSError on a full disk — finally runs, attempts the flush (which may also raise and replace the original), and then return True discards everything. The job reports success. I'd move return True to inside the try block after the flush, put return False in the except clause, and leave finally with only file_handle.close() and nothing else. The trade-off is that the job now fails loudly on disk-full, which is the correct behaviour.

Interview corner

CommonWhat is the difference between else and finally in a try statement?
“I'd say else runs only when the try body had no exception — it's the success path. Finally always runs regardless of what happened. I use else for success-only side-effects like committing a transaction or logging a count. I use finally for cleanup like closing file handles.”
TrickWhat happens if you put a return in finally?
“I never put return in finally — it overwrites any pending exception or return from try or except. The exception is silently discarded and the caller receives the finally return value as if nothing went wrong.”
Experience checkWhere has the try/else path saved you?
“I moved the audit-table insert from inside try to else in a batch job. Before that, if the insert raised an IntegrityError, the except clause caught it as if it were a processing error. After the move, an IntegrityError propagates correctly and the batch is marked failed rather than double-counted.”

Memory hook

else is toasting the bread only when the oven didn't catch fire; finally is putting the extinguisher back regardless. A return in finally reports the kitchen is fine even while it's burning.

If you remember only one thing:

Never put a return in finally — it silently wins over every exception.

Self-check

  1. When does the else block run?

    Only when the try body completes without raising any exception.

  2. Does finally run if the try body returns early?

    Yes. Finally always runs, even if try returns, breaks, or continues.

  3. What happens when finally itself raises an exception?

    The new exception from finally replaces the pending exception from try or except. The original is lost.

  4. If you need to log success and close a file, where does each action go?

    The success log goes in else; the file close goes in finally.

Prove it

Write def divide_and_log(a, b) with a try/except/else/finally structure. The try block computes a / b. The except catches ZeroDivisionError and returns None. The else block prints "success: {result}". The finally block prints "done". Run with (10, 2) and (5, 0). Show the printed output for both. Then add a return "override" to finally and show how it changes the (10, 2) output.

py-06-04Card 4 of 6

Choose EAFP or LBYL at the boundary

MediumP2 · Very likelyKnow cold10 min
eafplbylvalidationtoctoudefensive-style

Why you care

Python's culture is EAFP — Easier to Ask Forgiveness than Permission — but engineers who learned from Java or C backgrounds often default to LBYL (Look Before You Leap) everywhere. The difference is not just style: LBYL at an I/O boundary introduces a time-of-check-time-of-use (TOCTOU) race where the condition can change between the check and the act, and LBYL on a lazy ORM proxy triggers two round-trips instead of one. Both waste time and neither is safer than EAFP for those cases. Interviewers ask which to use because the wrong choice in a high-throughput service adds measurable latency or introduces correctness bugs.

Explain like I'm 5

EAFP is trying to open a door and dealing with being locked out only if it actually happens. LBYL is checking whether the door is locked before trying. Both work fine until someone changes the lock between your check and your try — which happens in concurrent systems constantly. EAFP eliminates the window between the check and the act because the check and the act are the same step.

Core concept

Two rules cover most cases:

  • Use EAFP for operations: file access, dict lookup, type coercion, attribute access. The failure is rare and the operation is atomic. try: value = d[key] is shorter and race-free; if key in d: value = d[key] checks then accesses, and the key can disappear between the two.
  • Use LBYL for validation boundaries: when you need to collect all errors before proceeding, and when the validation does not produce a side-effect. if field not in record: errors.append(...) is clearer and collects multiple problems in one pass; EAFP would exit on the first.

The critical decision rule: if you still need a try/except after the LBYL check (to handle a race or an edge case), the LBYL check adds noise without safety. Switch to EAFP. If you need to report all problems before proceeding, LBYL is cleaner.

Two common LBYL anti-patterns to recognise:

  • if hasattr(obj, "x"): use(obj.x) on an ORM proxy — hasattr triggers the query, obj.x triggers it again.
  • if os.path.exists(path): open(path) — the file can be deleted between the exists call and the open.

Minimal example EAFP and LBYL for a missing config key

Pythonruns
config = {"host": "db.prod.local", "port": "5432"}

# LBYL: check then act
if "timeout" in config:
    timeout_lbyl = int(config["timeout"])
else:
    timeout_lbyl = 30

# EAFP: act and recover
try:
    timeout_eafp = int(config["timeout"])
except KeyError:
    timeout_eafp = 30

# Python idiom for a simple default
timeout_get = int(config.get("timeout", 30))

print("lbyl:", timeout_lbyl)
print("eafp:", timeout_eafp)
print("get:", timeout_get)
print("all equal:", timeout_lbyl == timeout_eafp == timeout_get)
Output
lbyl: 30
eafp: 30
get: 30
all equal: True

In production LBYL at validation boundary, EAFP for I/O

Pythonruns
from pathlib import Path


# LBYL is right here: we want to collect ALL errors before touching any file
def validate_config(cfg: dict) -> list:
    errors = []
    for key in ("host", "port", "database"):
        if key not in cfg:
            errors.append(f"missing key: {key!r}")
    return errors


# EAFP is right here: the failure IS the interesting event, and it is atomic
def read_report(path: str):
    try:
        return Path(path).read_text()
    except FileNotFoundError:
        return None
    except PermissionError as exc:
        raise RuntimeError(f"cannot read {path}: permission denied") from exc


Path("report.csv").write_text("id,total\n1,500\n")

errs = validate_config({"host": "db.local"})
print("config errors:", errs)
print("report found:", read_report("report.csv") is not None)
print("missing:", read_report("gone.csv"))
Output
config errors: ["missing key: 'port'", "missing key: 'database'"]
report found: True
missing: None
Scenario A

In your world

The situation

Your job validates incoming trade records with if "trade_id" in record and "notional" in record: proceed(). The record object is a lazy-loading SQLAlchemy ORM proxy. The code works in unit tests, which use plain dicts.

What actually breaks

With the real ORM proxy, "trade_id" in record triggers a SQL query that materialises all column names and returns True. Then record["trade_id"] raises TypeError because the proxy does not support __getitem__. LBYL passed, and you still got an exception from the access.

What you'd do on Monday

Switch to EAFP. Use try: tid = record.trade_id; notional = record.notional; except AttributeError: skip(). The check and the access are now the same step, and you test on the actual attribute protocol the ORM proxy supports.

PythonEAFP on an ORM proxy (needs SQLAlchemy + DB session)
try:
    tid = record.trade_id
    notional = float(record.notional)
except AttributeError as exc:
    log.warning("trade skipped — missing attribute: %s", exc)
    skipped += 1
    continue
Scenario B

Imagine you work at ShareChat

10B feed impressions per day

The numbers

Illustrative: 10B impressions/day. Each feed item look-up checks a Redis-backed cache. A LBYL pattern (if r.exists(key): value = r.get(key)) makes two network round trips, and the key can expire between them. The single-step version (value = r.get(key), then treat None as a miss; redis-py returns None for a missing key rather than raising) makes one. At ~1 ms per in-region round trip, 10B × 1 ms = 10M extra seconds of request wait per day (≈ 116 days of cumulative latency), plus 10B extra commands/day of Redis load.

What the Scenario-A answer does here

EAFP for the cache hot path is the right call. The scale problem is enforcement: different teams own different feed generators, and LBYL patterns creep back in through code review gaps, adding latency per service.

What flips at this scale

You add a linter rule that flags if r.exists(key) followed by r.get(key) within ten lines. The linter runs in CI. LBYL is allowed only in the configuration-validation module, which is explicitly excluded from the rule.

The trade-off you'd defend

I would defend a CI linter rule over code-review convention, trading a one-time tooling investment for eliminating the entire class of latency regression from LBYL in hot paths.

Illustrative; not ShareChat-published figures.

EAFP vs LBYL

axisEAFPLBYL
race safetyatomic: check and act are one steprace window between check and act
redundancyone try/except covers both check and actoften still needs try/except for the race
best forI/O, dict access, attribute access, coercioninput validation where you collect all errors
readabilityconcise when failures are rareclearer intent when listing multiple validation errors
Python idiomtry/except; dict.get; getattr(obj, k, default)if k in d; isinstance; hasattr at a boundary

Gotchas

LBYL on a lazy ORM proxy triggers N+1 queries

Symptom

hasattr(record, "amount") returns True but record.amount raises DetachedInstanceError or triggers an unexpected SQL query.

Cause

hasattr calls __getattr__, which materialises the proxy. The subsequent attribute access can re-trigger the query or fail if the session is closed.

Fix

Access the attribute directly inside try/except AttributeError. One access, one round-trip.

if key in dict followed by dict[key] is not thread-safe

Symptom

KeyError on d[key] despite if key in d passing a line earlier.

Cause

Another thread deleted the key between the two operations.

Fix

EAFP: try: v = d[key]; except KeyError: .... Or protect with a lock if mutation is concurrent.

dict.get hides KeyError on nested access

Symptom

row.get("address", {}).get("city") returns None silently when address is None instead of a dict.

Cause

.get on a None object raises AttributeError, but if the outer .get returns a default dict that has no "city" key, it returns None silently.

Fix

Use EAFP: try: city = row["address"]["city"]; except (KeyError, TypeError): city = None.

Tricky questions

Is EAFP always faster than LBYL?
Most people say

Yes — EAFP avoids the extra check, so it is always faster.

Actually

EAFP is faster when the success case is common, because try with no exception has near-zero overhead. When the failure rate is high (for example input-validation code that rejects 30% of records), the exception machinery overhead makes LBYL faster per call. For I/O the difference is irrelevant; for tight loops validating dirty data, LBYL can be the right choice on performance grounds.

Does Python's for loop use EAFP or LBYL internally?
Most people say

LBYL — it checks if there is a next item before calling next().

Actually

EAFP — Python calls next() and catches StopIteration. This is why the iterator protocol does not need a has_next() method. Every for loop is EAFP under the hood (Iterator protocol, exhaustion and pull model).

Scenario question

You're handed a data-validation module that checks if hasattr(record, field) before accessing record.field on an ORM proxy. The team reports occasional AttributeError escaping despite the check. What is wrong and what do you fix?

Model answer

This is a TOCTOU issue specific to lazy-loading proxies. hasattr calls __getattr__, which may succeed by triggering a query, but by the time the direct access happens, the session may have expired or the attribute routing may differ. I'd switch to EAFP: try: val = getattr(record, field); except AttributeError: handle_missing(field). The check and the access become a single operation. The trade-off is that EAFP in validation-heavy code is slightly harder to read, but it is the only correct choice for lazy proxies and concurrent access.

Interview corner

CommonWhat do EAFP and LBYL mean, and when do you use each?
“I use EAFP — try then handle failure — for I/O and attribute access, where the check and act are a single atomic step. For input validation, where I need to collect all errors before proceeding rather than exit on the first, I switch to LBYL.”
TrickWhy is if key in d: value = d[key] potentially unsafe?
“I'd say in a threaded program another thread can delete the key between the membership test and the access, so you can still get a KeyError despite the check passing. EAFP eliminates the window: try and access are one step.”
Experience checkHas LBYL on an ORM proxy ever cost you in practice?
“I'd say yes — a validator I worked on used hasattr before attribute access on a SQLAlchemy lazy-load proxy. hasattr succeeded by triggering a SQL query, but the session expired before the direct access, so AttributeError still propagated. Switching to a try/except AttributeError wrapped the whole thing in one step.”
DesignHow would you enforce EAFP on hot paths in a team codebase?
“I'd write a pylint custom rule that detects the pattern if r.exists(key): r.get(key) and similar double-access sequences, and wire it into CI. Policy alone doesn't scale; the linter gives instant feedback.”

Memory hook

LBYL checks the lock and then tries the door — the lock can change in between. EAFP tries the door once: the check and the act are the same step.

If you remember only one thing:

At I/O and attribute boundaries, always use EAFP — the check and the act must be the same step.

Self-check

  1. What is a TOCTOU race?

    A time-of-check-time-of-use race: the condition you checked (key exists, file exists) changes between the check and the action, so the action can fail despite the check passing.

  2. When is LBYL the right choice?

    At input-validation boundaries where you need to collect all errors before proceeding, and where the check has no side-effects.

  3. Why is every Python for loop an example of EAFP?

    Python calls next() on the iterator and catches StopIteration internally — it does not check "is there a next item?" before calling.

  4. Name one case where LBYL is faster than EAFP.

    When the failure rate is high (for example a validation loop that rejects 30% of inputs), because exception raising and catching has overhead that LBYL avoids.

Prove it

Write a function get_nested(data, *keys) that retrieves a value from a nested dict using EAFP. It should return None for any missing key or non-dict intermediate. Show it working with ({"a": {"b": 1}}, "a", "b") → 1, ({"a": {}}, "a", "b") → None, and ({"a": None}, "a", "b") → None. Paste the output.

py-06-05Card 5 of 6

Inspect failures in exception groups

HardP3 · Role-dependentTalk about it11 min
exception-groupexcept-starconcurrent-errorspython311task-group

Why you care

Before Python 3.11, a function could raise exactly one exception. When you launched ten async tasks and three failed, you either got the first failure (and cancelled the other nine) or you collected results with return_exceptions=True and wrote custom routing code. ExceptionGroup changes the contract: a function can now raise a structured bag of multiple simultaneous failures, and except* lets each handler pull out only the types it can handle, leaving the rest to propagate. In ETL terms: a batch of 500 API calls that has three data errors and one network error no longer has to cancel all 500 or collapse to one error message. Interviewers ask about ExceptionGroup because it predicts whether a candidate can work with the async and concurrent APIs added in 3.11.

Explain like I'm 5

When you send ten riders out to make deliveries and three return with failures, you need to know which three failed and why — not just "a delivery failed, all ten re-queued". ExceptionGroup is the dispatcher's report that lists all ten outcomes at once. except* is the warehouse manager who sorts the report: "I'll handle damaged parcels, someone else handles address errors, and anything else escalates to the supervisor."

Core concept

An ExceptionGroup is a real exception that carries a tuple of sub-exceptions:

PythonCreating an ExceptionGroup — Python 3.11+
# Requires Python 3.11 or later
raise ExceptionGroup("batch errors", [
    ValueError("record 2: negative amount"),
    TypeError("record 5: missing currency"),
])

The except* clause receives a sub-group — always an ExceptionGroup, never a bare exception — containing only the matched members:

  • Each except* clause sees a fresh sub-group of matching exceptions; unmatched members continue propagating.
  • Multiple except* clauses can match in the same try/except* block; unlike except, they do not short-circuit after the first match.
  • If all members are handled, execution continues after the block. If any remain unhandled, Python re-raises a new ExceptionGroup with just those members.

Decision rule: use ExceptionGroup when you run multiple independent operations (async tasks, batch validations) and the caller genuinely needs to know about all failures, not just the first one. Use a plain exception when the first failure makes all subsequent results meaningless.

Note: ExceptionGroup requires all members to be Exception subclasses. BaseExceptionGroup lifts that restriction and accepts KeyboardInterrupt and SystemExit — you will mostly encounter it inside TaskGroup cancellation.

Under the hood

ExceptionGroup was introduced in CPython 3.11 via PEP 654. The except* clause compiles to bytecode that walks the group's exceptions tuple and tests each member against the clause's type. Members that match are extracted into a new ExceptionGroup passed to the handler body — always a group, even with a single match, so eg.exceptions is always a tuple. After all except* clauses run, CPython collects every unhandled member and re-raises them as a new ExceptionGroup (or BaseExceptionGroup if any is a BaseException subclass). Nesting works one level at a time: except* flattens only the top level of a nested group; an ExceptionGroup inside an ExceptionGroup propagates as a single matched or unmatched unit. In 3.12, ExceptionGroup.__new__ raises TypeError if any member is not an Exception subclass; use BaseExceptionGroup explicitly for mixed content. Python 3.14 adds no changes to the group mechanics but does warn on control-flow statements (break, continue, return) inside finally blocks, which can also mask ExceptionGroup propagation.

Minimal example ExceptionGroup and except* — Python 3.11+

Pythonruns
def validate_records(records):
    errors = []
    for rec in records:
        if not isinstance(rec.get("amount"), (int, float)):
            errors.append(ValueError(f"id={rec['id']} amount must be numeric"))
        if rec.get("currency") not in ("USD", "EUR", "INR"):
            errors.append(TypeError(f"id={rec['id']} unknown currency"))
    if errors:
        raise ExceptionGroup("record validation", errors)


data = [
    {"id": 1, "amount": 100.0, "currency": "INR"},
    {"id": 2, "amount": "bad",  "currency": "INR"},
    {"id": 3, "amount": 200.0, "currency": "XYZ"},
]

try:
    validate_records(data)
    print("all valid")
except* ValueError as eg:
    print(f"value errors: {len(eg.exceptions)}")
    for e in eg.exceptions:
        print(f"  {e}")
except* TypeError as eg:
    print(f"type errors: {len(eg.exceptions)}")
    for e in eg.exceptions:
        print(f"  {e}")
Output
value errors: 1
  id=2 amount must be numeric
type errors: 1
  id=3 unknown currency

In production Concurrent batch validation with asyncio.gather — Python 3.11+

Pythonruns
import asyncio


async def validate_record(rec_id: int, amount: float) -> float:
    if amount < 0:
        raise ValueError(f"record {rec_id} negative amount {amount}")
    if amount > 1_000_000:
        raise OverflowError(f"record {rec_id} amount {amount} exceeds limit")
    return amount


async def validate_batch(records: list) -> list:
    coros = [validate_record(r["id"], r["amount"]) for r in records]
    results = await asyncio.gather(*coros, return_exceptions=True)
    errors = [r for r in results if isinstance(r, BaseException)]
    if errors:
        raise ExceptionGroup("batch validation", errors)
    return [r for r in results if not isinstance(r, BaseException)]


async def main():
    batch = [
        {"id": 1, "amount": 250.0},
        {"id": 2, "amount": -10.0},
        {"id": 3, "amount": 500.0},
        {"id": 4, "amount": 2_000_000.0},
    ]
    try:
        good = await validate_batch(batch)
        print(f"all valid: {good}")
    except* ValueError as eg:
        print(f"data errors ({len(eg.exceptions)}):")
        for e in eg.exceptions:
            print(f"  {e}")
    except* OverflowError as eg:
        print(f"limit errors ({len(eg.exceptions)}):")
        for e in eg.exceptions:
            print(f"  {e}")


asyncio.run(main())
Output
data errors (1):
  record 2 negative amount -10.0
limit errors (1):
  record 4 amount 2000000.0 exceeds limit

See it happen How except* splits an ExceptionGroup by type

ExceptionGroup (3 members)ValueErrorTypeErrorOverflowErrorexcept* ValueError seesValueErrorexcept* TypeError seesTypeErrorunhandled — re-raisedOverflowError
  1. An ExceptionGroup arrives containing three members of different types.
  2. except* ValueError matches and receives a sub-group containing only the ValueError member. TypeError and OverflowError remain in the original group.
  3. except* TypeError matches next, receiving its sub-group. Both clauses have now run; OverflowError has not been claimed.
  4. OverflowError matched no clause. Python re-raises a new ExceptionGroup containing only the unhandled member. The caller must handle it or it propagates further.
Notice: except* always receives a sub-group, never a bare exception. Unhandled members are automatically re-raised — you cannot accidentally swallow them by handling other types.
Scenario A

In your world

The situation

Your batch job calls an external scoring API for 500 records sequentially, wraps each call in try/except, and stops on the first non-retriable error.

What actually breaks

Record 7 returns a 400 Bad Request (bad data). The job raises and stops. Records 8-500 are never scored. The re-run picks up all 500 again. Record 7 fails again. The pipeline spins and on-call gets paged for a data issue, not an infrastructure issue.

What you'd do on Monday

Collect all errors in a list and raise ExceptionGroup at the end of the batch. The caller's except* ValueError handler routes bad-data errors to a dead-letter queue, except* OSError handler schedules a retry. Only genuinely transient errors get re-queued — not the 493 records that were fine.

PythonCollect errors, raise ExceptionGroup at end (needs aiohttp or requests)
errors = []
for rec in records:
    try:
        result = score(rec)
        good.append(result)
    except (ValueError, KeyError) as exc:
        errors.append(exc)
    except OSError as exc:
        errors.append(exc)   # network errors also collected
if errors:
    raise ExceptionGroup("scoring batch", errors)
Scenario B

Imagine you work at Rapido

5M bike rides processed per day

The numbers

Illustrative: 5M rides/day, validated concurrently in batches of 5,000 using asyncio. Each ride validation does 3 checks (GPS, payment, identity). At 0.5% failure rate per check, a batch of 5,000 has ~75 failures spread across type. Without ExceptionGroup, the first failure aborts the batch and re-queues all 5,000 rides = 5,000 × 3 re-checks = 15,000 redundant check calls per aborted batch.

What the Scenario-A answer does here

Raising ExceptionGroup at the end of the batch eliminates the abort-and-requeue loop. The scale problem shifts: 75 failures per batch × 1,000 batches/day = 75,000 ExceptionGroup members per day. Without typed sub-exceptions carrying ride_id attributes, the dead-letter consumer cannot route GPS failures versus payment failures.

What flips at this scale

Each custom exception stores self.ride_id. The dead-letter consumer unpacks except* by type: except* GPSError routes to the GPS-correction queue, except* PaymentError routes to payment retry, without any additional metadata or message parsing.

The trade-off you'd defend

I would defend typed sub-exceptions with ride_id attributes over a single exception class with a type field, trading slightly more exception classes for automatic queue routing at zero parsing cost.

Illustrative; not Rapido-published figures.

Gotchas

except* does not catch a bare exception — only an ExceptionGroup

Symptom

except* ValueError does not trigger when raise ValueError("bad") is called directly.

Cause

except* only matches ExceptionGroup or BaseExceptionGroup instances; a bare ValueError is not a group.

Fix

Use except* only where the function explicitly raises ExceptionGroup. Use ordinary except ValueError for bare exceptions.

except* always gives you a sub-group, never the bare exception

Symptom

except* ValueError as e: e.message raises AttributeError — the code expected a bare ValueError.

Cause

except* always passes an ExceptionGroup to the handler, even if only one member matched. Use eg.exceptions[0] or iterate.

Fix

In the handler, iterate: for e in eg.exceptions: handle(e). Do not assume a single exception.

ExceptionGroup raises TypeError if any member is a BaseException subclass

Symptom

ExceptionGroup("name", [KeyboardInterrupt()]) raises TypeError: Cannot nest BaseExceptions in an ExceptionGroup.

Cause

ExceptionGroup requires all members to subclass Exception. KeyboardInterrupt subclasses BaseException directly.

Fix

Use BaseExceptionGroup("name", [KeyboardInterrupt()]) when members include non-Exception types.

Tricky questions

Can you use except alongside except* in the same try block?
Most people say

Yes — you can mix except and except* freely.

Actually

You cannot mix except and except* in the same try block. Python raises SyntaxError. Use except for non-group code paths and except* for ExceptionGroup paths, in separate try blocks.

Does except* short-circuit like except after the first match?
Most people say

Yes — the first matching except* clause wins.

Actually

All except* clauses in the same block can match. Python splits the group, routes each sub-group to its clause, and re-raises any unmatched members. This is a fundamental difference from except.

Scenario question

You're handed a batch processor using asyncio.gather(return_exceptions=False) for 1,000 records. Any single failure cancels the whole batch and re-queues all 1,000. A 3% failure rate means most records are processed ten or more times. What do you change?

Model answer

I'd switch to return_exceptions=True, collect the results, separate BaseException instances (failures) from normal results, and raise an ExceptionGroup with only the failures. The caller's except* block routes data errors to a dead-letter queue and transient errors to a retry queue. That cuts re-processing from 1,000 records to roughly 30 transient failures per batch. The trade-off is that I now own the exception-routing logic rather than relying on catch-all retry; that is the right place to own it, because routing decisions require knowing what failed.

Interview corner

CommonWhat is an ExceptionGroup and when would you use one?
“I'd reach for ExceptionGroup when I run independent concurrent operations and the caller needs to know about all failures, not just the first. It's a Python 3.11+ exception that carries multiple sub-exceptions in a single raise. The classic case is async batch processing: gather the results, collect all errors, raise one ExceptionGroup so the caller can route data errors to dead-letter and transient errors to retry.”
TrickWhat does except* give you in the handler — the bare exception or something else?
“I always get a sub-group, never the bare exception. Even if only one member matched, except* passes an ExceptionGroup with one item — so I iterate over eg.exceptions to get the individual errors.”
DesignHow does ExceptionGroup change error routing in a high-volume batch pipeline?
“I'd say before ExceptionGroup one failure aborted the entire batch and re-queued everything. Now I collect all failures into an ExceptionGroup, unpack by type in the caller, and route each type independently — data errors to a dead-letter queue, network errors to retry. The reprocessing volume drops from the whole batch to just the truly transient subset.”

Memory hook

ExceptionGroup is the dispatcher's full report: all failures listed by type, each routable independently.

If you remember only one thing:

except* always gives you a sub-group — iterate eg.exceptions, never assume a single member.

Self-check

  1. What Python version introduced ExceptionGroup and except*?

    Python 3.11, via PEP 654.

  2. Why can you not mix except and except* in the same try block?

    SyntaxError — Python does not allow mixing the two forms in one statement.

  3. If an ExceptionGroup has three members and only two are handled, what happens?

    Python re-raises a new ExceptionGroup containing the one unhandled member.

  4. What is the difference between ExceptionGroup and BaseExceptionGroup?

    ExceptionGroup requires all members to be Exception subclasses. BaseExceptionGroup accepts any BaseException subclass, including KeyboardInterrupt and SystemExit.

Prove it

Collect three deliberately failing calls — int("a"), int("b"), float("c") — and raise an ExceptionGroup("parse errors", [...]) containing all three ValueErrors. Handle it with except* and print the count and each message. Then add a TypeError("not a number") to the group and show that the unhandled TypeError propagates (catch it with a plain except TypeError outside the except* block). Paste both outputs.

py-06-06Card 6 of 6

Log, propagate or recover deliberately

MediumP2 · Very likelyKnow cold11 min
loggingpropagationrecoverystructured-loggingexception-context

Why you care

print(f"Error: {e}") in an exception handler is not logging. It loses the call stack, the exception type, the file name, the line number, and the timestamp. When a Jenkins job fails at 3 AM and the only evidence is "Error: 'amount'" in a 50,000-line stdout file, the developer spends 45 minutes adding print statements, re-running, and guessing. One change — logging.exception("msg") — gives you all of that for free. The other half of this card is about deciding what to swallow, what to propagate and what to recover: swallowing too much hides bugs; propagating too much makes every caller handle the same error. Interviewers ask about logging and propagation together because the two decisions are made at the same place in the code and the wrong answer to either one degrades debuggability for the entire team.

Explain like I'm 5

A ship's black box records everything, including the exact moment each system failed, the file it was in, and the full sequence of events. print(e) is a sticky note on the captain's desk that says "something broke". logging.exception(e) is the black box entry: timestamp, module, line, type, message, and the full call stack. When the ship sinks, you want the black box.

Core concept

Three logging methods and when to use each:

  • log.exception("msg"): always inside an except block; logs at ERROR level and appends the full traceback automatically. Use for unexpected failures you are catching to do cleanup before re-raising, or for errors you are converting to a dead-letter record.
  • log.error("msg"): logs at ERROR without a traceback. Use when you caught an expected error (e.g., a validation error) and the type and message are enough.
  • log.warning("msg"): for recoverable, expected errors like bad rows in a batch. Use it instead of printing.

The three decisions at every except clause:

  • Re-raise: when the current function cannot meaningfully handle the error. Use bare raise to preserve the original traceback, or raise NewError(...) from exc to translate. Never raise exc (creates a new traceback) (Represent domain failures explicitly).
  • Recover: when you have a safe fallback value and the caller does not need to know. Return the fallback. Only do this when the fallback is semantically correct — returning 0 for a missing amount is a semantic lie.
  • Swallow: the rarest case. Log at WARNING, count the skipped item, and only swallow if you have verified that the caller's invariants still hold after skipping.

Decision rule: if a function promises a side-effect (writing a row, sending a message), it must never swallow exceptions silently. Swallowing is only valid for optional enhancements — audit metrics, debug traces — whose absence does not break the caller.

Minimal example logging.exception captures the full traceback; print does not

Pythonruns
import logging
import io

buf = io.StringIO()
handler = logging.StreamHandler(buf)
handler.setFormatter(logging.Formatter("%(levelname)s %(message)s"))
logger = logging.getLogger("etl_demo")
logger.addHandler(handler)
logger.setLevel(logging.DEBUG)
logger.propagate = False


def parse_amount_print(raw):
    try:
        return float(raw)
    except ValueError as exc:
        print(f"ERROR: {exc}")   # loses call stack and location


def parse_amount_log(raw):
    try:
        return float(raw)
    except ValueError:
        logger.exception("amount parse failed")   # full traceback in log


parse_amount_print("N/A")
parse_amount_log("N/A")

captured = buf.getvalue()
print("--- log output ---")
print(f"level: {captured.split()[0]}")
print(f"message: {' '.join(captured.split()[1:4])}")
print(f"has traceback: {'Traceback' in captured}")
Output
ERROR: could not convert string to float: 'N/A'
--- log output ---
level: ERROR
message: amount parse failed
has traceback: True

In production Structured logging with severity tiers in an ETL loader

Pythonruns
import logging
import io
import csv
from pathlib import Path

buf = io.StringIO()
handler = logging.StreamHandler(buf)
handler.setFormatter(logging.Formatter("%(levelname)s %(name)s: %(message)s"))


def make_logger(name):
    log = logging.getLogger(name)
    log.addHandler(handler)
    log.setLevel(logging.INFO)
    log.propagate = False
    return log


log = make_logger("etl.loader")

Path("payments.csv").write_text("id,amount\n1,100.0\n2,bad_value\n3,300.0\n")


def parse_row(row, line_num):
    try:
        return {"id": int(row["id"]), "amount": float(row["amount"])}
    except (ValueError, KeyError) as exc:
        log.warning("line %d skipped: %s", line_num, exc)   # expected data error
        return None


def load_file(path):
    good, skipped = 0, 0
    try:
        with open(path, newline="") as f:
            for i, row in enumerate(csv.DictReader(f), 1):
                if parse_row(row, i) is not None:
                    good += 1
                else:
                    skipped += 1
    except FileNotFoundError:
        log.exception("file not found: %s", path)
        raise   # propagate: the scheduler must decide
    log.info("loaded %d rows, skipped %d", good, skipped)
    return good, skipped


good, skipped = load_file("payments.csv")
print(f"result: good={good} skipped={skipped}")
print("---")
for line in buf.getvalue().strip().split("\n"):
    print(line)
Output
result: good=2 skipped=1
---
WARNING etl.loader: line 2 skipped: could not convert string to float: 'bad_value'
INFO etl.loader: loaded 2 rows, skipped 1
Scenario A

In your world

The situation

Your Jenkins ETL job's top-level exception handler is except Exception as e: print(f"Job failed: {e}"); sys.exit(1).

What actually breaks

Jenkins logs show "Job failed: 'amount'" — a KeyError on a column that was renamed in the source schema. No file name, no line number, no traceback. You spend 45 minutes re-running with added print statements before you find the renamed column.

What you'd do on Monday

Replace the print with logging.exception("job failed") and add logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(name)s: %(message)s") at the top of the script. The next failure shows the exact file, line, exception type, and column name in the first log line. Diagnosis time drops from 45 minutes to under two minutes.

PythonTop-level logging setup — add to the module entry point
import logging
import sys

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(levelname)s %(name)s: %(message)s",
)
log = logging.getLogger(__name__)

try:
    main()
except Exception:
    log.exception("job failed — see traceback above")
    sys.exit(1)
Scenario B

Imagine you work at Spotify

600M listening events per day

The numbers

Illustrative: 600M events/day across Python workers. Each worker handles 50,000 events/hour. At a 0.1% error rate that is 50 errors/worker/hour, or 500 workers × 50 = 25,000 error events/hour total. Each logging.exception() call writes roughly 12 lines (one message line plus ~10 traceback lines) at ~175 bytes/line: 25,000 × 12 × 175 ≈ 52 MB/hour of log from error tracebacks alone — versus under 5 MB/hour if each error were one WARNING line, making search slow at volume.

What the Scenario-A answer does here

Switching from print() to logging.exception() is the right call. At this volume the new problem is log volume: full tracebacks for every ValueError on bad user data inflate log storage by 10×.

What flips at this scale

You apply severity tiers: WARNING for recoverable data errors (no traceback logged), ERROR with log.exception() for unexpected process errors, CRITICAL for fatal failures. Tracebacks are stored only for ERROR and above. Data-error WARNING lines are one per row. Log volume drops 90% while preserving full context for real bugs.

The trade-off you'd defend

I would defend severity-tiered logging over full tracebacks for every exception, trading slightly more careful classification at each except clause for dramatically faster incident search and lower storage cost.

Illustrative; not Spotify-published figures.

Gotchas

logging.exception outside an except block logs None for the traceback

Symptom

log.exception("msg") called outside a try/except shows the message but the traceback is NoneType: None.

Cause

logging.exception calls sys.exc_info(), which returns (None, None, None) when there is no active exception.

Fix

Call log.exception only inside an except block, where the current exception is set.

logging.error does not log the traceback

Symptom

The log shows the error message but no traceback, making it impossible to find the line that failed.

Cause

log.error("msg") logs at ERROR level without traceback. Only log.exception appends exc_info=True automatically.

Fix

Use log.exception when you want the traceback, or pass exc_info=True explicitly: log.error("msg", exc_info=True).

Swallowing an exception inside a context manager suppresses it from the caller

Symptom

A with MyContextManager() block raises, but the caller sees no exception and execution continues.

Cause

If __exit__ returns a truthy value, Python suppresses the exception.

Fix

In __exit__, only suppress exceptions you explicitly intend to suppress. Return False or None to let exceptions propagate (Context managers and deterministic cleanup).

Tricky questions

Is logging.warning the right call for a skipped row in an ETL batch?
Most people say

No — skipped rows are errors, so log.error is more appropriate.

Actually

A skipped row is a recoverable, expected event — bad input data that the job was designed to tolerate. log.warning is correct. log.error implies an unexpected condition that may require intervention; using it for every bad row triggers alert fatigue and trains operators to ignore ERROR-level messages.

Does adding logging.basicConfig in a library module affect the application?
Most people say

No — basicConfig only sets up the root logger for this module.

Actually

logging.basicConfig sets up the root logger, which affects every logger in the process unless they have propagate = False. Calling it in a library is an anti-pattern because it changes the application's logging configuration without the application author's consent. Libraries should add a NullHandler at the top level and let the application configure the root logger.

Scenario question

You're handed a Python ETL job where all errors are caught at the top level with except Exception as e: print(f"Error: {e}"). Incidents take 45 minutes to diagnose. What do you change first, and what do you change next?

Model answer

First I replace print(f"Error: {e}") with logging.exception("job failed") and add a basicConfig call at entry. That alone cuts diagnosis time dramatically because the traceback shows the exact file, line and exception type. Second I add per-module loggers with log = logging.getLogger(__name__), replacing all other print() calls with log.warning() or log.info(), so every message carries its source module. The trade-off on the second step is that it touches many files; I'd do it incrementally, starting with the highest-incident modules.

Interview corner

CommonWhat is the difference between logging.exception and logging.error?
“I'd say both log at ERROR level, but logging.exception also appends the current exception's traceback automatically. I use logging.exception inside except blocks when I want the full context. I use logging.error when the message is enough and I don't need the traceback — for example, a known validation failure.”
Experience checkWhat was the cost of print statements in a real incident?
“I lived this in a Jenkins job where the handler was print(f'Error: {e}'). The output was 'Error: amount' — a KeyError with no context. I spent 45 minutes adding print statements before I found the renamed column. After switching to logging.exception the next incident took under two minutes to diagnose.”
TrickWhy should a library never call logging.basicConfig?
“I'd say basicConfig sets up the root logger, which affects every logger in the entire process. If a library calls it, it overrides the application's logging configuration without the app author's consent. Libraries should add a NullHandler to their top-level logger and let the application owner decide how logs are formatted and where they go.”
DesignHow would you structure logging severity tiers for a high-volume batch ETL?
“I'd use WARNING for expected recoverable errors like bad rows — one line, no traceback. ERROR with log.exception for unexpected process failures where I need the stack. CRITICAL for fatal failures that require human intervention. That keeps log volume manageable and preserves full context exactly where it matters.”

Memory hook

print(e) is a sticky note; log.exception("msg") is the black box. You want the black box at 3 AM.

If you remember only one thing:

Inside an except block, always use log.exception("msg") — never print(e).

Self-check

  1. What does logging.exception add that logging.error does not?

    It appends the current exception's traceback automatically (exc_info=True).

  2. When is it correct to swallow an exception?

    Only for optional enhancements — metrics, debug traces — whose absence does not break the caller's invariants. Never swallow silently in a function that promises a side-effect.

  3. Why should a library module not call logging.basicConfig?

    It sets up the root logger and overrides the application's logging configuration for the entire process.

  4. What is the bare raise idiom and why does it matter?

    Inside an except block, a bare raise re-raises the current exception with its original traceback intact. raise exc creates a new traceback pointing to the re-raise line, losing the original location.

Prove it

Write a function safe_divide(a, b) that catches ZeroDivisionError, logs it with logging.exception("divide failed") into a StringIO buffer, and returns None. Show that safe_divide(10, 0) returns None, the log output contains "Traceback", and safe_divide(10, 2) returns 5.0 with no log output. Paste the outputs.