Shelf · Python · Chapter 7

Chapter 7: Typing & modern Python

Every Python script you have written already uses type hints as documentation; this chapter makes them a working safety net. By the end you will know where hints stop and runtime validation begins, how generics preserve type relationships across transformations, and which Python 3.14 changes will break introspection code that ran fine on 3.12.

Difficulty
Priority
py-07-01Card 1 of 6

Separate hints from runtime validation

EasyP2 · Very likelyKnow cold11 min
type-hintsmypyannotationsruntime-validationboundary
Needs: nothing. Start here.

Why you care

Half the "TypeError at 2 am on a Monday" incidents in Python ETL come from one invisible gap: Python's type hints are metadata, not enforcement. The runtime compiles and executes your code without reading a single annotation. A passing mypy run tells you the code is consistently annotated; it does not mean external data is safe. Interviewers ask "what is the difference between a type hint and runtime validation" because the answer reveals whether you know where type safety actually comes from — and where it stops.

Explain like I'm 5

A restaurant menu says "this dish is spicy". The kitchen does not refuse to cook it without chillies; the menu is a message to the waiter. Only a bouncer at the kitchen door — someone who checks every tray — actually stops the wrong ingredient. Annotations are menus; a boundary validator (Pydantic, isinstance, or a dataclass __post_init__) is the bouncer. The menu and the bouncer do different jobs. You need both.

Core concept

Three facts to carry:

  • x: int = 3 stores {'x': int} in __annotations__. Python never reads it during execution.
  • def f(a: int) -> str — call it with f("hello") and it runs fine, returning whatever the body computes.
  • typing.get_type_hints(f) retrieves annotations at runtime. This is how validators and frameworks introspect them; Python itself does not call it automatically.

Static checkers — mypy, pyright, Pylance — read annotations before the code runs, flag mismatches, and exit non-zero if the check fails. They work on the text, not the running program.

Decision rule. Add hints everywhere for IDE feedback and CI gates. Add runtime validation only at trust boundaries: API request parsing, config file loading, CSV deserialisation, database row mapping (Pydantic v2 boundary validation and schemas). Inside a module where you control every caller, trust typed code and do not re-validate; the mypy gate is enough. When not to add isinstance guards: hot loops and pure internal helpers. Enforce at entry, trust thereafter.

Under the hood

In Python 3.14 (released 2025-10-07, PEP 649), annotations on functions and classes become lazily evaluated by default — Python defers evaluating them until something first reads __annotations__. The key change from 3.12: annotations are no longer evaluated at definition time. f.__annotations__["x"] still returns the real int class (not a string), so ann is int stays True. Three things change in 3.14 that worked in 3.12: (1) a class body no longer stores a ready-made __annotations__ dict in cls.__dict__ (it stores a function that builds it on demand), so code that reads cls.__dict__['__annotations__'] directly gets a KeyError, or None from .get(); (2) a forward reference to a name not yet defined no longer fails when the def runs, it raises NameError later, when something reads __annotations__; (3) code that assumed annotations are strings only gets strings under from __future__ import annotations (PEP 563), never from the 3.14 default. The new annotationlib module is the supported way to read them; inspect.get_annotations(cls) (3.10+) works on both versions. Libraries like dataclasses and Pydantic were updated for 3.14; your in-house introspection code is what needs auditing (Read the 3.11 to 3.14 migration map).

Minimal example Annotations do not change what Python accepts

PythonAnnotations are ignored at runtimeruns
def multiply(a: int, b: int) -> int:
    return a * b

result = multiply("row", 3)   # str × int is valid Python; no error
print(result)
print(type(result))
print(multiply.__annotations__)
Output
rowrowrow
<class 'str'>
{'a': <class 'int'>, 'b': <class 'int'>, 'return': <class 'int'>}

In production Config loaded from JSON carries the wrong types silently

PythonDataclass does not coerce; JSON strings slip through as-isruns
from dataclasses import dataclass
from pathlib import Path
import json


@dataclass
class JobConfig:
    batch_size: int
    source_key: str
    dry_run: bool


def load_config(raw: dict) -> JobConfig:
    """Annotations present, but no runtime check: wrong types accepted."""
    return JobConfig(
        batch_size=raw["batch_size"],
        source_key=raw["source_key"],
        dry_run=raw["dry_run"],
    )


Path("job_config.json").write_text(
    '{"batch_size": "500", "source_key": "s3://bucket/feed.csv", "dry_run": "false"}'
)
raw = json.loads(Path("job_config.json").read_text())
cfg = load_config(raw)
print(type(cfg.batch_size), cfg.batch_size)
print(type(cfg.dry_run), cfg.dry_run)
print(bool(cfg.dry_run))   # "false" is a non-empty string — truthy in Python
Output
<class 'str'> 500
<class 'str'> false
True

See it happen Where hints stop and validation begins

reads hintscompiles toacceptsvia boundaryvalidatorapproves shapesource code(annotations)mypy / pyrightCI gatePython runtimeno enforcementexternal dataAPI, CSV, configvalidatedinterior
  1. Annotations live in the source as metadata. Python stores them in __annotations__ and moves on.
  2. mypy reads those annotations at CI time and flags mismatches before the code ships.
  3. At runtime, Python compiles and executes the source. It does not read a single annotation.
  4. External data — an API payload, a CSV row, a config file — arrives with no guarantee it matches the declared types.
  5. A boundary validator checks and coerces data before it enters the typed interior, where mypy's guarantee holds.
Notice: mypy and the runtime run at different times and see different things. Only the boundary validator joins them.
Scenario A

In your world

The situation

Your nightly Jenkins job loads a batch_size from a JSON config file on S3 and later calls range(cfg.batch_size) to chunk the Postgres inserts.

What actually breaks

A platform team changes the config writer to emit "batch_size": "500" (a JSON string) instead of 500. The hint says int, mypy never ran on the config-loading module, and the job crashes with TypeError: 'str' object cannot be interpreted as an integer inside the chunking loop — three layers below the config load. The stack trace points nowhere useful.

What you'd do on Monday

Validate and coerce at the loading boundary, not at the call site:

PythonExplicit coercion at the config boundaryruns
from dataclasses import dataclass
from pathlib import Path
import json


@dataclass
class JobConfig:
    batch_size: int
    source_key: str
    dry_run: bool


def load_config(raw: dict) -> JobConfig:
    dry_raw = raw["dry_run"]
    dry_run = (dry_raw.lower() == "true") if isinstance(dry_raw, str) else bool(dry_raw)
    return JobConfig(
        batch_size=int(raw["batch_size"]),
        source_key=str(raw["source_key"]),
        dry_run=dry_run,
    )


Path("job_config.json").write_text(
    '{"batch_size": "500", "source_key": "s3://bucket/feed.csv", "dry_run": "false"}'
)
raw = json.loads(Path("job_config.json").read_text())
cfg = load_config(raw)
print(type(cfg.batch_size), cfg.batch_size)
print(type(cfg.dry_run), cfg.dry_run)
Output
<class 'int'> 500
<class 'bool'> False
Scenario B

Imagine you work at Booking.com

50M hotel search API requests per day

The numbers

Illustrative: 50M requests × ~10 config-object constructions per service restart wave = 500M constructions per deploy cycle. An isinstance check on every construction adds roughly 0.5 µs each — 250 s of CPU per wave — entirely avoidable if you validate once at the boundary and trust typed code downstream.

What the Scenario-A answer does here

The Scenario A fix (explicit coercion in load_config) works at this scale, but if CI never runs mypy, the same string-not-int slip recurs in the next config key that gets added. A single boundary model catches the full shape; re-checking at every call site is redundant and expensive.

What flips at this scale

At this scale the flip is a static-analysis gate in CI, not a per-call isinstance guard. mypy/pyright in CI catches type mismatches before deploy; a Pydantic model (Pydantic v2 boundary validation and schemas) validates each incoming API request once. Everything downstream trusts the types.

The trade-off you'd defend

I would defend a strict pyright gate in CI over scattered isinstance guards, trading two minutes of CI time for zero per-call overhead and a single, visible validation point.

Illustrative numbers to make the arithmetic visible; not Booking.com-published figures.

Gotchas

A passing mypy run does not mean external data is safe

Symptom

mypy exits 0, but the job crashes with TypeError or AttributeError on data loaded from S3 or a database.

Cause

mypy checks the annotation contract between your functions; it cannot see what arrives at runtime from outside the Python process.

Fix

Add runtime validation at every trust boundary — the point where data enters from a source you do not control.

cls.__dict__['__annotations__'] is gone in Python 3.14

Symptom

Code that reads cls.__dict__.get('__annotations__', {}) to build a type map gets an empty map on Python 3.14, so every field falls through to the default branch; the ['__annotations__'] form raises KeyError.

Cause

Under PEP 649 the class body stores a function that builds the annotations on first access, not the finished dict. cls.__annotations__ goes through the type's descriptor and still works; cls.__dict__ is just the raw namespace.

Fix

Use inspect.get_annotations(cls) (3.10+, own annotations only, same as the old __dict__ read) or typing.get_type_hints(cls) (also resolves strings and merges base classes). Cache the result per class on hot paths.

mypy only checks what it can reach

Symptom

A module with no annotations is silently ignored by mypy; errors there reach production.

Cause

mypy skips un-annotated modules in its default configuration.

Fix

Enable --strict or at minimum --check-untyped-defs in mypy.ini. Add annotations to new modules from day one.

Tricky questions

Does a passing mypy run guarantee no TypeError at runtime?
Most people say

Yes — that is the whole point of type hints.

Actually

mypy guarantees the contract between annotated functions is consistent. It cannot see what Python receives from json.loads, a database query or os.environ. A str variable annotated as int that arrives from JSON is invisible to the checker until you add a runtime validator at the entry point.

Is adding hints to existing code always safe?
Most people say

Yes, hints are just documentation — you can add them any time.

Actually

Adding from __future__ import annotations (PEP 563) makes all annotations compile-time strings. On Python 3.14, annotations are lazily evaluated but still return real types on access — the two mechanisms are different. Code that compares __annotations__ values to classes breaks under PEP 563, and code that reads cls.__dict__['__annotations__'] breaks on 3.14. Add hints one module at a time and test framework introspection after each batch.

Scenario question

You're handed a 3-year-old ETL script with no type hints. The team wants to add hints "for safety". What is the first real step, and what will and won't it do?

Model answer

I would add hints to the public functions first and run mypy in CI with --strict. That catches internal contract violations and gives the IDE autocomplete. Then I would identify every point where data arrives from outside the codebase: S3 file reads, Postgres query results, HTTP payloads. Those get a runtime validator, not just a hint. The trade-off is that hints without a CI gate are decorations, and a CI gate without boundary validators still lets bad data in. You need both layers. I would do the CI gate first because it costs nothing to run and immediately flags existing inconsistencies.

Interview corner

CommonWhat is the difference between a type hint and runtime validation?
“I'd say a hint is metadata stored in __annotations__ and read by mypy or the IDE, but Python never checks it during execution. Runtime validation is code that actually inspects the value and raises an exception if it is wrong. You need both: hints for internal contract checking, validators at the boundary where data arrives from outside your code.”
TrickIf mypy passes, is the code type-safe?
“I'd say only internally. mypy cannot see what Python receives from json.loads, a database row or os.environ. A string that arrives as an int annotation is invisible to the checker until you validate at the entry point. I think of mypy as auditing the plumbing between functions; a boundary validator is the filter on the pipe before it enters the building.”
Experience checkWhere has missing runtime validation caused a real incident?
“In my ETL work, a config key changed from an integer to a quoted string in a JSON file a platform team wrote. The dataclass annotation said int, mypy passed, but nobody had added int() at the load point. The job crashed three layers down with a TypeError that pointed at the chunking loop, not the config load. I now coerce at the boundary and assert types before any computation depends on them.”

Memory hook

Hints tell mypy what you mean; a boundary validator tells Python what you require.

If you remember only one thing:

If the data comes from outside your code, annotate it and validate it.

Self-check

  1. Does Python raise an error if you pass a string to a parameter annotated as int?

    No. Python stores the annotation but never checks it during execution. The function runs with whatever argument you pass.

  2. What does typing.get_type_hints(f) return that f.__annotations__ does not?

    get_type_hints resolves forward references and handles from __future__ import annotations (PEP 563) string annotations, always returning a dict of real type objects. In Python 3.12 without from __future__ import annotations, both return the same dict. In 3.14 (PEP 649), f.__annotations__ also returns real types; but cls.__dict__['__annotations__'] skips the lazy descriptor and breaks — get_type_hints is safe in all versions.

  3. Name two places in an ETL pipeline that qualify as trust boundaries.

    Anywhere external data enters: the JSON config loader, S3 CSV parser, Postgres query result mapper, HTTP request deserialiser. Any two of these count.

Prove it

Run the production example as-is, then change batch_size in the JSON to the integer 500 (no quotes). Run again. Confirm that the first run prints <class 'str'> 500 and the second prints <class 'int'> 500 — showing that the dataclass annotation does nothing; the JSON type is what arrives. Paste both outputs.

py-07-02Card 2 of 6

Keep relationships with generics

MediumP3 · Role-dependentTalk about it9 min
genericstypevargenerictype-relationshipsparameterised

Why you care

A function that accepts "a list of something" and returns "one of those things" loses the element type the moment you write list without a parameter. The caller gets list[Any], and the checker cannot catch a bug where someone treats the result as the wrong element type. This matters in ETL code that transforms typed collections — filters, paginators, deduplicators all have this shape. Interviewers ask about TypeVar because it separates engineers who annotate return types as Any from those who think about what the relationship between input and output actually is.

Explain like I'm 5

Imagine a vending machine labelled "put something in, get something out". A generic vending machine is more specific: "put quarters in, get exactly the item you pressed". If you press B3, you get a B3-item — not just "a vending-machine thing of unknown type". TypeVar is the label that pins the input type to the output type so the caller knows what comes back.

Core concept

Three building blocks:

  • T = TypeVar("T") names a placeholder. The checker fills it in at each call site.
  • def first(items: list[T]) -> T says "return something of the same type as the items". Without T, the signature is def first(items: list) -> Any, and first([1, 2, 3]) + "x" raises no mypy error.
  • Generic classes extend Generic[T]: class Batch(Generic[T]): items: list[T]. Each Batch[int] is a distinct static type; the runtime sees only one class.

The 3.12 shorthand skips the TypeVar declaration: def first[T](items: list[T]) -> T:. It is sugar; the semantics are identical.

Decision rule. Use TypeVar when a function's return type mirrors one of its input types (filter, first, last, identity). Use Any only when you explicitly intend to opt out of checking for a known reason. Use Generic[T] when a class wraps or produces typed items. When not to use: functions that deliberately convert between types have a fixed different return type and do not need TypeVar.

Under the hood

TypeVar is a checker fiction; Python never enforces it at runtime. A call keep([1, 2, 3], pred) fixes T = int in the checker for that call site; calling keep(["a", "b"], pred) on the same line fixes T = str. At runtime, both calls execute the same bytecode. You can pass a mixed list[int | str] and the checker infers a union. If a TypeVar is bound in one function but used in a second unrelated function, the checker treats each call site independently — two calls share nothing.

Minimal example TypeVar preserves the element type through a filter

PythonTypeVar keeps the return type concreteruns
from typing import TypeVar, Callable

T = TypeVar("T")


def keep(items: list[T], pred: Callable[[T], bool]) -> list[T]:
    return [x for x in items if pred(x)]


amounts = keep([100, 0, 250, 0, 75], lambda x: x > 0)
keys = keep(["order_1", "", "order_2", ""], bool)
print(amounts)
print(keys)
Output
[100, 250, 75]
['order_1', 'order_2']

In production Generic Page class for chunked ETL writes

PythonGeneric Page wrapper for typed chunkingruns
from typing import TypeVar, Generic, Iterator
from dataclasses import dataclass, field

T = TypeVar("T")


@dataclass
class Page(Generic[T]):
    """Fixed-size page of typed items, drained to a fresh list on flush."""
    size: int
    _buf: list[T] = field(default_factory=list)

    def push(self, item: T) -> bool:
        self._buf.append(item)
        return len(self._buf) >= self.size

    def drain(self) -> list[T]:
        out, self._buf = self._buf, []
        return out


def paginate(rows: list[T], page_size: int) -> Iterator[list[T]]:
    page: Page[T] = Page(size=page_size)
    for row in rows:
        if page.push(row):
            yield page.drain()
    tail = page.drain()
    if tail:
        yield tail


for chunk in paginate([10, 20, 30, 40, 50], 2):
    print(chunk)
Output
[10, 20]
[30, 40]
[50]
Scenario A

In your world

The situation

Your ETL has a filter_rows(rows: list, pred) -> list function used for both settlement rows and audit rows. After filtering, callers access attributes specific to each type.

What actually breaks

With list (unparameterised), mypy infers the output as list[Any]. It misses a bug where someone calls .settle() on an audit row returned by the filter. The error surfaces at runtime, not in CI.

What you'd do on Monday

Add TypeVar so mypy knows the output type mirrors the input type:

PythonTypeVar makes the filter type-awareruns
from typing import TypeVar, Callable
from dataclasses import dataclass

T = TypeVar("T")


def filter_rows(rows: list[T], pred: Callable[[T], bool]) -> list[T]:
    return [r for r in rows if pred(r)]


@dataclass
class Settlement:
    id: str
    amount: float


filtered = filter_rows(
    [Settlement("S1", 100.0), Settlement("S2", 0.0), Settlement("S3", 250.0)],
    lambda s: s.amount > 0,
)
print([s.id for s in filtered])
print(type(filtered[0]))
Output
['S1', 'S3']
<class '__main__.Settlement'>
Scenario B

Imagine you work at InMobi

2B ad events per day

The numbers

Illustrative: 2B events ÷ 86 400 s/day ≈ 23 000 events/s. A generic Page[AdEvent] with page_size 5 000 produces ≈ 400 000 commits per day. Without TypeVar, a shape mismatch between AdEvent and BidRow in a shared filter produces Any in the checker and AttributeError at runtime at peak traffic — no stack trace points to the filter itself.

What the Scenario-A answer does here

The Scenario A fix (filter_rows with explicit domain type) works, but "use list everywhere and cast downstream" is the pattern that emerges when TypeVar is unfamiliar. Each cast is a silent opt-out of checking; at 23 000 events/s you cannot afford to discover a cast was wrong in production.

What flips at this scale

At this scale the flip is disciplined generics at shared utility boundaries. filter_rows, paginate, dedupe are used by every pipeline team; a TypeVar on each ensures that each team's downstream code stays type-checked even when the shared utility is written by someone else.

The trade-off you'd defend

I would defend TypeVar on all shared utilities over per-team casts, trading slightly more complex function signatures for CI catching cross-team type mismatches before they reach 2B events per day.

Illustrative numbers to make the arithmetic visible; not InMobi-published figures.

Gotchas

TypeVar solved in one call site does not constrain another

Symptom

You expect mypy to infer T = int from one call, but a second call on the same function with str raises no error.

Cause

Each call site is solved independently. T has no memory between calls.

Fix

This is correct behaviour. If you need the two calls to agree on a type, they must share the same object (e.g., a Generic[T] class instance where T was fixed at construction).

Returning Any accidentally by dropping the TypeVar

Symptom

mypy shows list[Any] for the return type of your filter or transform function.

Cause

The return annotation uses list without a parameter, or the TypeVar was omitted from the signature.

Fix

Add T = TypeVar("T") and parameterise both the input and the return type.

Tricky questions

Can I reuse the same TypeVar across two unrelated functions?
Most people say

Yes, once declared at module level, a TypeVar constrains all functions that use it.

Actually

Each call site is solved independently. Sharing T = TypeVar("T") between two functions is fine; the checker solves T separately for each call. The TypeVar object itself is just a label.

Is Generic[T] enforced at runtime?
Most people say

Yes — you cannot pass the wrong type to a Generic[T] class.

Actually

Generic[T] is a static-checker fiction. At runtime, Page[int] and Page[str] are the same class. Python never checks the element type when you call page.push(item). The checker catches mismatches; the runtime does not.

Scenario question

You're handed a class Cache that stores values by string key. The team uses it for both SupplierRecord objects and plain dicts. Design it so mypy catches when a caller retrieves a value and tries to use it as the wrong type.

Model answer

I would make Cache generic: class Cache(Generic[V]) where V = TypeVar("V"). get(self, key: str) -> V | None returns the right type for each instantiation. A caller writes Cache[SupplierRecord]() and mypy knows every get returns SupplierRecord | None. The trade-off is that a single cache cannot hold both types at once — that would need Cache[SupplierRecord | dict], which loses specificity. If both types genuinely coexist in one cache, a Union is honest; if they are always separate, two typed caches are cleaner and give better downstream checking.

Interview corner

CommonWhat is TypeVar used for?
“I'd say TypeVar is a placeholder that the type checker fills in at each call site. It lets you say 'the return type is the same as the input type' without committing to a specific type in the function definition. Without it, you either over-restrict to one concrete type or under-restrict to Any.”
TrickWhy is def first(items: list) -> Any worse than using TypeVar?
“I'd say with Any, the checker treats the return as unchecked and stops warning about how callers use it. A caller can do first([1, 2, 3]) + 'x' and mypy says nothing. With TypeVar, the checker knows the return is int at that call site and flags the string concatenation. Any is an explicit opt-out; TypeVar is a precise opt-in.”
Experience checkWhen would you use Generic[T] on a class?
“I'd use it when the class wraps or produces items whose type should be tracked across method calls. A pagination buffer, a typed queue, a cache — anywhere the class holds items and callers need to know what type comes out. I would not use it for a class that works on a fixed domain type; that would just be a concrete annotation on the attribute.”

Memory hook

TypeVar is the label that pins "what goes in" to "what comes out". Without it, the checker loses the thread.

If you remember only one thing:

Use TypeVar whenever the return type mirrors an input type.

Self-check

  1. What does a TypeVar solve at runtime?

    Nothing. TypeVar is a static-checker concept only. Python executes the same code regardless of what type was inferred.

  2. What annotation does mypy assign to keep([1, 2], lambda x: x > 0) if keep uses TypeVar T?

    list[int], because the checker fixes T = int from the input list.

  3. When would you use Generic[T] on a class instead of TypeVar only on a function?

    When the class stores or produces items of a type that callers need to track across multiple method calls — e.g., a cache, a buffer, a typed queue.

Prove it

Write a class Stack(Generic[T]) with push(item: T) and pop() -> T | None. Instantiate as Stack[int]() and push three integers. Call s.pop() three times, printing each result. Then annotate a second variable as Stack[str] and push a string. Run it — Python won't stop you mixing types at runtime. Paste the output and note that the checker would have flagged the mismatch even though the code ran.

py-07-03Card 3 of 6

Model capabilities and missing values

MediumP2 · Very likelyKnow cold11 min
protocoloptionalunionnullablestructural-typingruntime-checkable

Why you care

"Duck typing is fine until the duck quacks wrong at 3 am". Python's Protocol gives you structural contracts without inheritance: the type checker verifies an object has the right methods before the code ships, not when it runs. Optional catches nullable fields that silently produce AttributeError: 'NoneType' object has no attribute ... — arguably the most common ETL crash. Interviewers ask about Protocol because it shows whether you understand duck typing formally, not just "it works if it has the right methods".

Explain like I'm 5

A power socket does not care who made the plug — only whether it has the right prongs. Protocol is the socket specification: it describes the shape, not the brand. If your object has an amount() method, it fits the HasAmount socket without inheriting from anything. Optional[str] is a socket that accepts either a string or nothing (the value called None). The type checker shouts if you try to use it as a plain string without first testing whether it is None.

Core concept

Four rules to carry:

  • Protocol (3.8+) marks a class as a structural interface. Any class with the required method signatures matches — no isinstance check needed at the call site, no inheritance required.
  • Optional[X] is exactly Union[X, None]. The 3.10+ union syntax is X | None; both are equivalent.
  • Nullable (value can be None) is different from omitted (key absent). name: str | None = None is present-but-null; name: str | None with no default is still required, even though it accepts None (Pydantic v2 boundary validation and schemas).
  • typing.runtime_checkable lets you use isinstance(obj, MyProtocol) at runtime. Without it, the Protocol is static-only and isinstance raises TypeError.

Decision rule. Use Protocol for functions that accept "anything with method X" — it beats an ABC when you do not control the implementing classes (third-party libraries, multiple teams). Use Optional[X] / X | None whenever a value can legitimately be absent; use X | None in 3.10+ code. When not to use Protocol: inside a module where you own every class — a proper ABC gives you @abstractmethod enforcement that Protocol does not.

ProtocolABCDuck typing
Inheritance requiredNoYesNo
Static checkingYes (mypy)Yes (mypy)No
isinstance() at runtimeWith runtime_checkableYesNo
Implementer must know interfaceNoYesNo
@abstractmethod enforcementNoYesNo
Best forThird-party types, library boundariesInternal hierarchies where you control implementersQuick scripts, small modules

Under the hood

Protocol stores its required members in __protocol_attrs__. runtime_checkable wraps isinstance to check only that those attribute names exist — it does not check argument types or return types. So isinstance(obj, HasAmount) passes if obj has a callable named amount, even if it returns str instead of float. The structural guarantee is full only at static-check time, not at runtime.

Minimal example Protocol checks shape, not class

PythonProtocol matches any class with the right methodruns
from typing import Protocol, runtime_checkable


@runtime_checkable
class HasAmount(Protocol):
    def amount(self) -> float: ...


class Order:
    def __init__(self, total: float) -> None:
        self._total = total

    def amount(self) -> float:
        return self._total


class Refund:
    def __init__(self, value: float) -> None:
        self._value = value

    def amount(self) -> float:
        return -self._value


def net_flow(items: list[HasAmount]) -> float:
    return sum(i.amount() for i in items)


events = [Order(500.0), Refund(75.0), Order(200.0)]
print(net_flow(events))
print(isinstance(Order(0), HasAmount))
Output
625.0
True

In production Nullable fields with an explicit None guard

PythonOptional[str] requires a None guard before attribute accessruns
from typing import Optional


def format_reference(order_id: str, notes: Optional[str]) -> str:
    """notes is nullable: caller must pass None explicitly if absent."""
    tag = f" ({notes})" if notes is not None else ""
    return f"REF-{order_id}{tag}"


# Settlement rows from a Postgres query (simulated as tuples)
rows = [
    ("ORD-001", "SWIFT transfer"),
    ("ORD-002", None),
    ("ORD-003", "NEFT"),
]

for oid, note in rows:
    print(format_reference(oid, note))
Output
REF-ORD-001 (SWIFT transfer)
REF-ORD-002
REF-ORD-003 (NEFT)

See it happen Structural matching — Protocol versus class hierarchy

type checkerHasAmount ProtocolOrder classRefund classreads protocolhas amount() -> float?yeshas amount() -> float?yes
  1. mypy reads the Protocol definition and records the required method signatures.
  2. mypy inspects Order's method signatures. No inheritance required.
  3. Order satisfies HasAmount structurally. mypy accepts it wherever HasAmount is expected.
  4. mypy inspects Refund. Same check, same result.
  5. Both Order and Refund satisfy HasAmount with no shared parent class.
Notice: the Protocol never appears in the MRO of Order or Refund. The relationship is checked once by the type checker, not stored in the class hierarchy.
Scenario A

In your world

The situation

Your ETL function def process(event) accepts Any. One day a teammate passes a raw dict from an S3 JSON parse instead of an Order object.

What actually breaks

event.amount() crashes with AttributeError: 'dict' object has no attribute 'amount'. The error is three frames deep and the message does not show what event was or where it came from.

What you'd do on Monday

Annotate process with event: HasAmount. mypy then flags the wrong call site at CI time. Add runtime_checkable and an early guard to give a better error message at runtime:

Pythonruntime_checkable gives a better error at the boundaryruns
from typing import Protocol, runtime_checkable


@runtime_checkable
class HasAmount(Protocol):
    def amount(self) -> float: ...


class Order:
    def __init__(self, v: float) -> None:
        self._v = v

    def amount(self) -> float:
        return self._v


def process(event: HasAmount) -> float:
    if not isinstance(event, HasAmount):
        raise TypeError(f"expected HasAmount, got {type(event).__name__}")
    return event.amount() * 1.02


try:
    process({"type": "order", "amount": 500.0})
except TypeError as e:
    print(f"TypeError: {e}")

print(process(Order(500.0)))
Output
TypeError: expected HasAmount, got dict
510.0
Scenario B

Imagine you work at HDFC Bank

30M transaction events per day

The numbers

Illustrative: 30M events/day ÷ 86 400 s/day ≈ 347 events/s. Each event passes through three processing stages (classify, enrich, persist). Without Protocol, a wrong type in a single stage propagates through all three before crashing. With Protocol, mypy flags the mismatch at the ingestion boundary; the other 29 999 999 events are unaffected.

What the Scenario-A answer does here

The Scenario A fix (isinstance guard in process) catches the wrong type at runtime but only in process. If the same event object is passed to enrich and persist before process, the crash happens later and the guard never fires. A static Protocol annotation catches the mismatch at the earliest call site, wherever that is.

What flips at this scale

At this scale the flip is annotating every shared interface with Protocol so that mypy finds cross-team mismatches at CI time, not at 347 events/s in production. The isinstance guard becomes a last-resort defence-in-depth check, not the primary catch.

The trade-off you'd defend

I would defend static Protocol annotations plus a single runtime_checkable guard at the ingestion boundary over per-function isinstance checks, trading slightly more upfront annotation work for CI catching cross-team errors automatically.

Illustrative numbers to make the arithmetic visible; not HDFC Bank-published figures.

Gotchas

Forgetting runtime_checkable and calling isinstance

Symptom

isinstance(obj, MyProtocol) raises TypeError: Instance and class checks can only be used with @runtime_checkable protocols.

Cause

Without @runtime_checkable, Protocol-based isinstance is forbidden at runtime.

Fix

Add @runtime_checkable to the Protocol. Note that runtime isinstance only checks attribute existence, not argument or return types.

Optional[str] with no default is still a required field

Symptom

mypy (or Pydantic) raises an error saying a field is required even though it was annotated Optional[str].

Cause

Optional[str] means "this value can be None". It says nothing about whether the caller must supply it. Without = None, the field has no default and is required.

Fix

Add = None if the field is genuinely optional at the call site: name: Optional[str] = None.

Using Optional when you mean "omitted" versus "null"

Symptom

A query returns rows where the column was NULL and rows where it was simply absent from the projection; your code treats both as None and loses the distinction.

Cause

Optional[str] collapses both to None. There is no way to distinguish "present-and-null" from "absent" once both are None.

Fix

Use a sentinel (NOT_SET = object()) or a dict with missing keys to represent truly absent fields. Reserve None for explicitly-null values.

Tricky questions

Does Protocol use isinstance under the hood at static-check time?
Most people say

Yes, it checks the class hierarchy at runtime.

Actually

No. mypy performs a structural check at static analysis time — it reads the class body and compares method signatures to the Protocol's requirements. At runtime, Python does not check Protocol conformance unless you call isinstance explicitly with a runtime_checkable Protocol.

What is the difference between Optional[str] and str | None?
Most people say

Optional[str] is older syntax that wraps the type; str | None is newer and slightly different.

Actually

They are exactly equivalent. Optional[X] is defined as Union[X, None], and X | None is the 3.10+ shorthand for the same union. Use X | None in 3.10+ code for readability; use Optional[X] when targeting older codebases.

Scenario question

You're handed a CSV parser that reads settlement files from S3. Some rows have a reference column; others do not. The downstream audit step needs to distinguish "reference was in the row but empty" from "reference column was absent". How do you model this?

Model answer

I would use a three-value model: str for a present non-empty reference, "" (empty string) for present-but-blank, and a sentinel or None for absent. If using Pydantic, reference: str | None with default=None collapses both absent and null to None. To preserve the distinction, I would parse to a dataclass with a missing: bool flag, or use a dict where absent fields are simply not present. The trade-off is that a flag adds a second attribute to track; the dict approach means no static type safety on individual fields. For an audit step where the distinction matters financially, I would use the explicit flag and document the intent.

Interview corner

CommonWhat is a Protocol?
“I'd say a Protocol is a structural interface definition. Any class that has the required method signatures satisfies it — no inheritance needed. It lets you write a function that accepts 'anything with an amount() method' and mypy checks that callers pass objects with that method.”
TrickWhat does isinstance check when used with a runtime_checkable Protocol?
“I'd say it checks that the required attribute names exist on the object, but it does not check their argument types or return types. So an object with a callable amount that returns a string still passes the isinstance check, even if the Protocol says amount() -> float. The full type check only happens at static analysis time.”
Experience checkWhere would you use Protocol over ABC in an ETL pipeline?
“I'd use Protocol for any function that processes events from multiple sources — when I cannot force every source to inherit from a common base. In my ETL work, Postgres rows, S3 CSV rows and API payloads all have similar shapes but come from different libraries. A Protocol lets me write one typed processor and have mypy verify each caller, without touching the library code.”
DesignHow do you model a field that can be null in the database but must be present in the row?
“I'd annotate it as str | None with no default, meaning the caller must pass None explicitly if the column was null. That is different from str | None = None, which allows the caller to omit the field entirely. The distinction matters in Pydantic v2 where omitting a required field raises ValidationError, but passing None does not.”

Memory hook

Protocol is the socket; your class is the plug. If the prongs fit, it works — no family tree required.

If you remember only one thing:

Optional[X] means nullable; it does not mean optional-to-provide. Add = None to make the field optional.

Self-check

  1. What is the minimum a class needs to satisfy a Protocol?

    The class must have all the attribute names and method signatures listed in the Protocol. No inheritance from the Protocol is required.

  2. Why does isinstance(obj, MyProtocol) work for one Protocol but raise TypeError for another?

    The Protocol without @runtime_checkable does not support isinstance. Only Protocols decorated with @runtime_checkable allow runtime isinstance checks.

  3. What is the difference between notes: Optional[str] = None and notes: Optional[str]?

    The first has a default: callers can omit notes and it defaults to None. The second has no default: callers must pass notes explicitly (either a string or None).

Prove it

Write a function def total(items: list[HasAmount]) -> float where HasAmount is a runtime_checkable Protocol with amount() -> float. Call it with a mixed list of two different classes that both have amount() but share no parent. Confirm the result is correct. Then call it with a plain dict and print the TypeError. Paste both outputs.

py-07-04Card 4 of 6

Pydantic v2 boundary validation and schemas

MediumP3 · Role-dependentKnow cold11 min
pydanticvalidationmodel-validatemodel-dumpv2boundaryfrom-attributes

Why you care

Pydantic v2 is the most common Python runtime validation library in product-company codebases, and it changed enough from v1 that copying a v1 tutorial will produce subtle bugs. Two failures are widespread: treating Optional[str] without a default as optional (it is required), and raising TypeError inside a validator (it escapes instead of becoming a ValidationError). Interviewers ask about Pydantic because S10's job description explicitly requires it, and the difference between v1 and v2 behaviour is a live filter between candidates who have actually used it and those who memorised a tutorial.

Explain like I'm 5

Think of Pydantic as a customs officer at the airport. You hand over a bag of raw data — a dict from an API, a row from a CSV. The officer checks each item against a declared manifest (your model). If everything matches, they wave you through with a stamped model instance you can trust. If anything is wrong — missing field, wrong type, invalid value — they hand you a ValidationError listing every problem at once, not one at a time. You deal with the error once at the gate; downstream code never sees untrustworthy data.

Core concept

Five things the v2 API changes from v1:

  • model_validate(dict) replaces Model(**dict) — the new entry point for external data.
  • model_dump() replaces .dict() — returns a plain Python dict from the model.
  • model_dump_json() replaces .json() — returns a JSON string.
  • from_attributes=True replaces orm_mode=True in model_config — lets Pydantic read from ORM row objects.
  • Coercion is stricter by default. In v1, "true" was coerced to True for a bool field. In v2, that raises ValidationError unless you add a field_validator or use model_config = {"coerce_numbers_to_str": True} etc.

Optional[str] with no default is a required field that accepts None. Add = None to make it genuinely optional (Model capabilities and missing values).

Decision rule. Put Pydantic models at every trust boundary. Do not use them for pure-Python-to-Python function calls between typed modules; that is mypy's job. Use model_validate for external input; use model_dump or model_dump_json for serialisation out.

Under the hood

Inside a v2 @field_validator, Pydantic catches ValueError and wraps it as a ValidationError field error. It does not catch TypeError. A TypeError propagates up the call stack past the validator and past the model_validate call, crashing the caller with an unhandled exception instead of a tidy ValidationError. This changed from v1, where both were caught. The fix is always to raise ValueError (not TypeError) from inside a validator, or to catch the TypeError and re-raise as ValueError.

Minimal example Dataclass without validation accepts wrong types silently

PythonDataclass does not coerce; wrong types enter undetectedruns
from dataclasses import dataclass


@dataclass
class SettlementRow:
    order_id: str
    amount: float
    settled: bool


raw = {"order_id": "ORD-042", "amount": "1500.50", "settled": "true"}
row = SettlementRow(**raw)

print(repr(row.amount))
print(repr(row.settled))

try:
    total = row.amount + 100.0
except TypeError as e:
    print(f"TypeError: {e}")

print(bool(row.settled))   # "true" is truthy — logic error
Output
'1500.50'
'true'
TypeError: can only concatenate str (not "float") to str
True

In production Pydantic v2 boundary model for S3 settlement rows

PythonPydantic v2 model_validate and field_validator (needs pydantic>=2.0)
from typing import Optional
from pydantic import BaseModel, field_validator, ValidationError


class SettlementRow(BaseModel):
    order_id: str
    amount: float
    settled: bool
    notes: Optional[str] = None   # = None makes it genuinely optional

    @field_validator("amount", mode="before")
    @classmethod
    def coerce_amount(cls, v):
        try:
            return float(v)
        except (TypeError, ValueError) as e:
            raise ValueError(f"amount must be numeric, got {v!r}") from e
        # Never raise TypeError here — it escapes ValidationError in v2


# --- model_validate: dict → typed model ---
row = SettlementRow.model_validate(
    {"order_id": "ORD-042", "amount": "1500.50", "settled": True}
)
print(row.amount, type(row.amount))      # 1500.5 <class 'float'>
print(row.model_dump())

# --- from_attributes: ORM row → model ---
class FakeOrmRow:
    order_id = "ORD-043"
    amount = 200.0
    settled = True
    notes = "NEFT"

class SettlementRowOrm(SettlementRow):
    model_config = {"from_attributes": True}

orm_row = SettlementRowOrm.model_validate(FakeOrmRow())
print(orm_row.model_dump())

# --- ValidationError: missing required field ---
try:
    SettlementRow.model_validate({"order_id": "ORD-044", "settled": True})
except ValidationError as e:
    print(e.error_count(), "error(s)")
    print(e.errors()[0]["type"])          # missing

See it happen Pydantic v2 as the boundary gate

untrusted inputpassesinvalidraw dictAPI / S3BaseModel.model_validate()typed modelsafe interiorValidationErrorcaller handles once
  1. Raw data from an API response or S3 CSV arrives as a plain dict. Types may be wrong or fields may be missing.
  2. model_validate() passes the dict through Pydantic's validation engine, applying field types and validators in order.
  3. Valid data emerges as a typed BaseModel instance. Downstream code can trust the types and call model_dump() freely.
  4. Invalid data raises ValidationError listing every field that failed. One handler at the boundary beats guarding every field access downstream.
Notice: the ValidationError lists all failures at once, not just the first one. Catching it once at the boundary replaces per-field None guards throughout the pipeline.
Scenario A

In your world

The situation

You migrate a Pydantic v1 model to v2. The validator used @validator (v1 API) converted to @field_validator. Inside it, you raise TypeError("expected numeric value") when the input cannot be converted.

What actually breaks

In v1, Pydantic wrapped that TypeError in a ValidationError. In v2, TypeError escapes the validator, propagates past model_validate, and crashes the calling request handler with a raw TypeError instead of a tidy 422 Unprocessable Entity.

What you'd do on Monday

Raise ValueError (not TypeError) from every validator. Pydantic v2 catches ValueError and wraps it; TypeError propagates up:

PythonRaise ValueError, never TypeError, inside a v2 validator (needs pydantic>=2.0)
from pydantic import BaseModel, field_validator


class PaymentRow(BaseModel):
    amount: float

    @field_validator("amount", mode="before")
    @classmethod
    def parse_amount(cls, v):
        try:
            return float(v)
        except (TypeError, ValueError) as e:
            # V2 catches ValueError and wraps it. TypeError escapes.
            raise ValueError(f"amount must be numeric, got {v!r}") from e
Scenario B

Imagine you work at Instamojo

1M payment events per day

The numbers

Illustrative: 1M events/day × ~20 fields per model = 20M field validations/day. A stray TypeError in a validator that fires on 0.01% of field validations = 2 000 unhandled exceptions/day. On average, that is roughly one crash per 43 s — enough to trigger PagerDuty but subtle enough to look like a flaky dependency.

What the Scenario-A answer does here

The Scenario A fix (raise ValueError) resolves the v2 TypeError escape. But the deeper v1-to-v2 migration issue is coercion: in v1, "true" became True for bool fields automatically; in v2 it raises ValidationError. A migration that changes only the decorator name without auditing coercion assumptions will silently start rejecting valid-looking payloads.

What flips at this scale

At this scale the flip is treating the v1-to-v2 migration as a data contract change, not a code change. Every field that relied on v1's liberal coercion (strings-to-bool, strings-to-int) needs an explicit field_validator in v2. Run both versions against a week of production payloads before cutting over.

The trade-off you'd defend

I would defend explicit field_validator coercion over implicit v2 strictness, trading more lines of code for a data contract that is visible in the source rather than discovered at 3 am on a Friday.

Illustrative numbers to make the arithmetic visible; not Instamojo-published figures.

Gotchas

Optional[str] without a default is still required in Pydantic v2

Symptom

ValidationError: 1 validation error for MyModel\nnotes\n Field required.

Cause

Optional[str] means the value can be None; it does not mean the field can be omitted. Without = None, Pydantic requires the caller to supply the field (even if they supply None).

Fix

Add = None: notes: Optional[str] = None. If the field is genuinely required but nullable, keep it without a default and require callers to pass None explicitly.

TypeError inside a v2 validator escapes as an unhandled exception

Symptom

A request crashes with bare TypeError: ... instead of 422 ValidationError.

Cause

Pydantic v2 only catches ValueError (and AssertionError) inside validators and wraps them as field errors. TypeError propagates up.

Fix

Inside every @field_validator, catch any TypeError and re-raise as ValueError. Never let TypeError reach the validator's outer frame.

v2 model_dump() drops None values by default only with exclude_none=True

Symptom

Serialised output includes "notes": null even though you expected it to be absent.

Cause

model_dump() includes all fields by default, including those set to None.

Fix

Call model_dump(exclude_none=True) to omit null fields from the output dict.

Tricky questions

Does Pydantic v2 coerce "true" to True for a bool field?
Most people say

Yes, it always has.

Actually

Pydantic v2 is stricter. "true" does not coerce to True by default; it raises ValidationError. In v1 it did. You need a field_validator with mode="before" to coerce strings to bool explicitly.

What does model_config = {"from_attributes": True} do?
Most people say

It allows extra fields in the input dict.

Actually

It tells Pydantic to read attributes (e.g., row.amount) instead of dict keys (e.g., row["amount"]). This is how you feed an SQLAlchemy ORM row into a Pydantic model. It replaces v1's orm_mode = True.

Scenario question

You're handed a FastAPI endpoint that receives a JSON payload and needs to write to Postgres using SQLAlchemy 2.0. The ORM model and the API schema are different shapes. How do you structure the validation and mapping?

Model answer

I would define two Pydantic models: a request model (SettlementIn) that validates the incoming JSON with model_validate, and a separate schema for the ORM row if I need to read back from the database using from_attributes=True. The SQLAlchemy ORM model stays as a plain mapped class — no Pydantic in the ORM layer. The mapping between SettlementIn and the ORM model happens in a service function that takes the Pydantic model and constructs the ORM object. The trade-off is two models instead of one; the benefit is that the API contract and the database schema can evolve independently without breaking each other.

Interview corner

CommonWhat changed between Pydantic v1 and v2?
“I'd say the main API changes are: model_validate() replaces direct construction, model_dump() replaces .dict(), and orm_mode becomes from_attributes in model_config. The bigger semantic change is coercion: v1 was lenient (strings-to-bool, for example); v2 is stricter and raises ValidationError for type mismatches by default. Any migration needs to audit every coercion assumption, not just rename decorators.”
TrickWhat happens if you raise TypeError inside a v2 field validator?
“I'd say it escapes. Pydantic v2 only catches ValueError and AssertionError inside validators and wraps them as field errors. A TypeError propagates straight up past model_validate and crashes the caller with an unhandled exception. I caught this the hard way migrating a v1 codebase — the fix is to catch TypeError inside the validator and re-raise as ValueError.”
Experience checkHow do you handle a SQLAlchemy ORM row in Pydantic?
“I add model_config = {'from_attributes': True} to the Pydantic model and call model_validate(orm_row). Pydantic reads attributes instead of dict keys. I keep a separate API schema and a separate ORM model; the mapping function in between keeps the two contracts independent.”
DesignWhere do you put Pydantic validation in a pipeline that reads from S3 and writes to Postgres?
“I'd place it at the boundary where the data enters the pipeline — immediately after parsing the S3 CSV row into a dict. That is the one place where the data is untrusted. Once I have a validated model, every downstream function receives a typed object and I do not re-validate. On the way out, model_dump() serialises for the Postgres insert. The trade-off is one validation cost per row at ingest; the benefit is that no downstream function needs to guard for None or wrong types.”

Memory hook

A Pydantic model is a customs officer: it checks every field at the door and issues a single report. Downstream code never sees untrustworthy data.

If you remember only one thing:

In v2, raise ValueError from validators — never TypeError. Optional[str] without a default is still required.

Self-check

  1. When do you use Model.model_validate(raw) instead of Model(**raw)?

    When the input is a dict or object you received rather than keyword arguments you wrote: both run full validation, but model_validate takes the data as one argument (so non-string keys and non-dict inputs work) and has a model_validate_json twin. Only model_construct skips validation.

  2. What does from_attributes=True enable?

    It lets Pydantic read object attributes (row.amount) instead of dict keys (row["amount"]). Required when feeding SQLAlchemy ORM rows into a Pydantic model.

  3. Why does Pydantic v2 raise ValidationError for "true" in a bool field?

    V2 does not coerce strings to bool by default. V1 did. You need a field_validator with mode="before" to convert the string before the type check runs.

  4. What is the difference between notes: Optional[str] = None and notes: Optional[str]?

    The first has a default and can be omitted by callers. The second is required — callers must pass either a string or None explicitly.

Prove it

Write a Python script (no pydantic — stdlib only) that parses five settlement dicts into the SettlementRow dataclass from the minimal example. Print the type of amount for each row. Then add explicit coercion inside a load_row function and print the corrected types. Confirm that without coercion every row has amount as str and with coercion every row has amount as float. Paste both output blocks.

py-07-05Card 5 of 6

Use match and walrus without hiding flow

MediumP3 · Role-dependentTalk about it11 min
matchwalrusstructural-pattern-matchingpy-310walrus-operator
Needs: nothing. Start here.

Why you care

Both match (3.10+) and := (3.8+) can make code shorter or longer depending on how you use them. The risk is writing a match block with six patterns where a simple if/elif chain is clearer, or using := to save one line but making the reader hunt for where a variable was assigned. The concrete failure is a match without case _:: new event types are silently dropped with no error and no log. Interviewers ask about walrus because it appears in list comprehensions and while loops and its scope rules are a known pitfall for developers who use it without reading PEP 572.

Explain like I'm 5

match is like a switch that also opens the box and looks inside: it not only checks which type of parcel arrived but also unpacks it and labels the contents in one step. while chunk := f.read(4096): is a post-it note on a door — "read a chunk; if empty, stop" — stuck there so you don't need a separate line to test the result. Both save steps. The problem comes when you stack too many post-it notes, and nobody can see the door anymore.

Core concept

Four rules to carry:

  • match (3.10+) uses structural pattern matching. case {"event": "order", "id": int(oid)} matches a dict with those keys and binds oid as an int in one step. It is not just a switch statement.
  • case _: is the wildcard — "anything else". Omitting it means unmatched values pass through silently, which is the most common match bug.
  • Walrus := binds a name while evaluating an expression. Valid in while, if and comprehensions. (x := f()) is a walrus; x := f() at statement level is a SyntaxError.
  • The readability test for walrus: if the reader needs to scroll back to find out what := bound, replace it with a regular assignment on the preceding line.

Decision rule. Use match when you have 3+ patterns on structured data (dicts, tuples, dataclasses) and there is a case _: fallback. Prefer if/elif for 2 branches or simple scalar comparisons. Use walrus in while loops that read chunks or poll for data. Avoid walrus in complex nested expressions.

Minimal example match dispatches on shape; walrus drives a chunk reader

Pythonmatch event type; walrus reads fixed chunksruns
import io


def handle(event: dict) -> str:
    match event:
        case {"type": "order", "id": int(oid)}:
            return f"order {oid}"
        case {"type": "refund", "id": int(oid)}:
            return f"refund {oid}"
        case _:
            return "unknown"


print(handle({"type": "order", "id": 42}))
print(handle({"type": "refund", "id": 7}))
print(handle({"type": "unknown_event"}))

data = io.BytesIO(b"ABCDEFGH")
chunks = []
while chunk := data.read(3):
    chunks.append(chunk)
print(chunks)
Output
order 42
refund 7
unknown
[b'ABC', b'DEF', b'GH']

In production ETL event router with match and walrus readline

Pythonmatch-based event router over a line streamruns
import json
import io
from typing import NamedTuple


class Metric(NamedTuple):
    orders: int = 0
    refunds: int = 0
    skipped: int = 0


def process_events(stream: io.TextIOBase) -> Metric:
    orders = refunds = skipped = 0
    while line := stream.readline():
        line = line.strip()
        if not line:
            continue
        try:
            event = json.loads(line)
        except json.JSONDecodeError:
            skipped += 1
            continue
        match event:
            case {"type": "order", "amount": float(amt)} if amt > 0:
                orders += 1
            case {"type": "refund", "amount": float(amt)} if amt > 0:
                refunds += 1
            case _:
                skipped += 1
    return Metric(orders=orders, refunds=refunds, skipped=skipped)


payload = "\n".join([
    '{"type": "order", "amount": 250.0}',
    '{"type": "refund", "amount": 75.0}',
    '{"type": "order", "amount": 400.0}',
    '{"type": "unknown"}',
    'not-json',
])
result = process_events(io.StringIO(payload))
print(result)
Output
Metric(orders=2, refunds=1, skipped=2)

See it happen match evaluates patterns top-to-bottom until one wins

entertry firsttry secondno matchevent dictmatchcase order+ guardcase refund+ guardcase _(wildcard)
  1. An event dict enters the match statement. Python evaluates the subject once.
  2. Python tries the first case. If the dict has type=order, a positive float amount, and the guard passes, execution jumps here.
  3. If the first case does not match, Python tries the second. Refund events with a positive amount land here.
  4. Anything unmatched falls to case _. Without this wildcard, unmatched events leave the match silently — zero processing, zero error.
  5. A zero-amount order does not pass the guard (amt > 0). It skips both typed cases and reaches case _, where it is counted as skipped.
Notice: case _: is not optional. Every match block that dispatches events must have it or unrecognised types disappear silently.
Scenario A

In your world

The situation

Your event router has eight if/elif branches. A new "cancel" event type is added to the upstream system. Nobody adds an elif event["type"] == "cancel": branch.

What actually breaks

Cancel events are silently counted as nothing — zero records processed, zero error. Finance notices missing cancellations three days later when reconciling.

What you'd do on Monday

Rewrite with match and a mandatory case _: that at minimum logs the unhandled type:

Pythonmatch with mandatory case _ fallbackruns
def handle_event(event: dict) -> str:
    match event:
        case {"type": "order", "id": int(oid)}:
            return f"processed order {oid}"
        case {"type": "refund", "id": int(oid)}:
            return f"processed refund {oid}"
        case {"type": "cancel", "id": int(oid)}:
            return f"processed cancel {oid}"
        case _:
            return f"unhandled: {event.get('type', 'unknown')}"


tests = [
    {"type": "order", "id": 1},
    {"type": "cancel", "id": 2},
    {"type": "mystery", "id": 3},
]
for ev in tests:
    print(handle_event(ev))
Output
processed order 1
processed cancel 2
unhandled: mystery
Scenario B

Imagine you work at Delhivery

5M shipment events per hour

The numbers

Illustrative: 5M events/h ÷ 3 600 s/h ≈ 1 389 events/s. A missing case _: at this rate means 1 389 unhandled events per second from a new event type. With no error and no log, the loss accumulates silently before anyone notices. At 1 389/s it takes fewer than 10 minutes to lose a full hour's worth of one event class.

What the Scenario-A answer does here

The Scenario A fix (add case _: return "unhandled: ...") stops the silent drop. But a case _ that only returns a string is also silent if the result is never checked. At this scale the fix needs a side-effect: a metric counter, a DLQ write, or a log line that triggers an alert.

What flips at this scale

At this scale the flip is treating case _: as a dead-letter handler — not just a fallback but an observable signal. Every unhandled event type increments a metric and triggers an alert if the rate exceeds a threshold. The match block is the classification gate; the wildcard is the circuit breaker.

The trade-off you'd defend

I would defend a case _: that writes to a dead-letter queue over one that silently logs, trading a slightly more complex match body for guaranteed recoverability of any unrecognised event type.

Illustrative numbers to make the arithmetic visible; not Delhivery-published figures.

Gotchas

Missing case _: silently drops unmatched events

Symptom

A new event type is introduced upstream and your processor simply stops recording it. No exception, no log line.

Cause

match without a wildcard is not an error. Unmatched subjects exit the block without executing any branch.

Fix

Always end a match block that dispatches events with case _: and at minimum log or count the unhandled type.

Walrus variable leaks into the enclosing scope

Symptom

A variable bound by := inside a comprehension or while is accessible and holds its last value after the loop ends.

Cause

Unlike a for-loop variable (which also leaks), := is designed to bind in the enclosing scope intentionally. The binding is visible after the expression completes.

Fix

Treat walrus-bound names like any other variable: name them clearly and do not rely on them being undefined outside the loop.

A guard failure does not stop pattern matching

Symptom

You expect case {"type": "order"} if amt > 0: to raise an error for negative amounts, but they fall through to case _: instead.

Cause

A guard failure means the case does not match, not that an exception is raised. Python tries the next case.

Fix

This is correct behaviour. Put a negative-amount check inside the case _: handler or add an explicit case {"type": "order", "amount": float(amt)} if amt <= 0: branch.

Tricky questions

Does match evaluate every case?
Most people say

Yes, like a series of if/elif statements.

Actually

match tries cases top-to-bottom and stops at the first one that matches (structure + guard). It does not evaluate every branch. This means case order matters: a more specific case must come before a broader one that would also match it.

What is the scope of a walrus variable inside a list comprehension?
Most people say

The variable is local to the comprehension, like a loop variable in a generator expression.

Actually

:= binds in the enclosing scope, not inside the comprehension. After result = [y := f(x) for x in data], y holds the last value f returned and is accessible outside the comprehension.

Scenario question

You're handed a 200-line function that dispatches on event["type"] with if/elif chains and no else clause. You've been asked to refactor it. What would you do and what would you warn about?

Model answer

I would refactor to a match block with one case per event type and a mandatory case _: at the end. The structural patterns also let me destructure each event in the case line itself, removing repeated event["key"] accesses. The warning I would give first: the existing code silently drops unrecognised event types — I would quantify how many events are being silently skipped by adding a counter to the current else-less chain before the refactor, so the team knows the scope of the problem. The trade-off is that match is not available before 3.10; if the service runs on 3.9, I would add case _: to the if/elif chain first and do the structural refactor only after the Python upgrade.

Interview corner

CommonWhat does structural pattern matching do differently from a switch statement?
“I'd say a switch checks equality on a scalar. match checks the shape of the object: it can verify dict keys exist, check the type of a value, and bind parts of it to names, all in one case line. case {'type': 'order', 'id': int(oid)} matches, type-checks, and names the id in a single expression. That would be three lines with if/isinstance/subscript.”
TrickWhat happens if you forget case _: in a match block?
“I'd say unmatched subjects exit the block silently — no exception, no log. In an event router that means new or unexpected event types are dropped with zero visibility. I treat case _: as mandatory on any match that dispatches events; the body at minimum increments a counter or writes to a dead-letter queue.”
Experience checkWhen would you replace walrus with a regular assignment?
“My rule is: if the reader needs to look backwards to understand what the variable holds, replace it. Walrus in while chunk := f.read(4096): is self-contained — the name and the value are on the same line. Walrus inside a nested expression, like [y := transform(x) for x in data if y is not None], forces the reader to track two binding points at once. I would split that into a generator with an explicit intermediate variable.”

Memory hook

match opens the box and labels the contents in one step. Without case _:, unlabelled boxes pass through invisibly.

If you remember only one thing:

Always end a match block with case _:. Silent drops are worse than crashes.

Self-check

  1. What is the difference between match and a chain of if/elif?

    match checks the shape, type, and keys of a value simultaneously and binds names from within it in a single pattern. A chain of if/elif accesses keys and checks types with separate expressions.

  2. Where does a walrus-bound variable live after the expression completes?

    In the enclosing function scope, not inside the comprehension or loop body. It holds the last value the expression produced.

  3. What does a guard clause do in a match case?

    A guard (if condition) is evaluated after the structural pattern matches. If the guard is False, the case does not match and Python tries the next one. No exception is raised.

Prove it

Write a function that reads lines from io.StringIO using while line := stream.readline() and routes each JSON line through a match block with three cases plus case _:. Feed it ten lines: three orders, two refunds, one cancel, two with zero amounts and two with bad JSON. Confirm the Metric totals are correct. Paste the output.

py-07-06Card 6 of 6

Read the 3.11 to 3.14 migration map

MediumP5 · SkippableKnow it exists12 min
migrationpy-311py-312py-313py-314annotationsfree-threadingtaskgroupexception-groups

Why you care

Python releases a new minor version every October. Each version removes deprecated APIs and changes behaviour. An engineer who last read "what's new" in 2022 will copy a 3.9-era blog post and hit a DeprecationWarning they do not understand, or miss a feature that replaces thirty lines of boilerplate. The concrete 3.14 risk is direct: code that accesses __annotations__ without going through typing.get_type_hints silently produces wrong results after upgrading. Interviewers flag migration awareness because the skill to upgrade safely under time pressure distinguishes a senior engineer from one who avoids version bumps.

Explain like I'm 5

Imagine a word processor that adds keyboard shortcuts every year. Some old shortcuts are removed; new ones do more in fewer keystrokes. If you have not read the "what's new" page since version 3.9, you are working the slow old way and you will be confused when a shortcut that used to work does nothing. The Python migration map is that "what's new" page — reading it takes an hour; not reading it costs a production incident.

Core concept

A version-by-version map:

3.11 (2022-10-24):

  • asyncio.TaskGroup for structured cancellation (Choose gather or a task group).
  • ExceptionGroup and except* for handling multiple concurrent exceptions.
  • tomllib in stdlib — no more third-party TOML parsers.
  • typing.Self for return-type annotations in methods.

3.12 (2023-10-02):

  • type Point = tuple[int, int] — the type statement for clean type aliases.
  • F-strings: backslashes and quotes now allowed inside {}.
  • @override in typing to mark intentional method overrides.
  • TypeVar syntax sugar: def first[T](items: list[T]) -> T: (no separate TypeVar declaration).

3.13 (2024-10-07):

  • Free-threading (GIL-optional) build available as an experimental option. On that build, importing an extension that has not declared support turns the GIL back on, with a RuntimeWarning (Runtime concurrency: threads, processes, async and GIL).
  • Improved locals() semantics (frames no longer aliased).
  • copy.replace() for shallow-copying dataclasses with field overrides.

3.14 (2025-10-07):

  • Annotations lazy by default (PEP 649). Annotations are no longer evaluated at definition time — they resolve on first __annotations__ access and return real types. The breaks: cls.__dict__ no longer holds an __annotations__ dict, and a forward-reference NameError moves from definition time to access time. The new annotationlib module, or inspect.get_annotations() on any 3.10+ version, is the safe way to read them (Separate hints from runtime validation).
  • Free-threading build promoted from experimental to supported (PEP 779), still a separate optional build.

Decision rule. Target 3.12 for new projects (LTS-grade, widely adopted). Know the 3.11 safety features (TaskGroup, ExceptionGroup). Test __annotations__ access and validator introspection before upgrading to 3.14. Do not upgrade early if a dependency has not declared 3.14 support — free-threading extension caveats may produce subtle concurrency bugs.

Minimal example 3.11 ExceptionGroup and 3.12 TypeAlias in one script

PythonExceptionGroup (3.11+) and TypeAlias (3.12 type statement requires 3.12)runs
from typing import TypeAlias

Row: TypeAlias = dict[str, str]


def parse_row(raw: list[str], headers: list[str]) -> Row:
    return dict(zip(headers, raw))


row = parse_row(["ORD-1", "500.0", "2026-01-15"], ["id", "amount", "date"])
print(row)

try:
    raise ExceptionGroup("load errors", [
        ValueError("bad amount on row 3"),
        ValueError("missing date on row 7"),
    ])
except* ValueError as eg:
    for exc in eg.exceptions:
        print(f"Caught: {exc}")
Output
{'id': 'ORD-1', 'amount': '500.0', 'date': '2026-01-15'}
Caught: bad amount on row 3
Caught: missing date on row 7

In production Safe annotation introspection across 3.12 and 3.14

Pythonget_type_hints is safe on both 3.12 and 3.14runs
import typing
import sys
from dataclasses import dataclass


@dataclass
class Trade:
    symbol: str
    quantity: int
    price: float


# cls.__annotations__ returns real types on 3.12 and 3.14; cls.__dict__['__annotations__'] is what 3.14 removes
anns = Trade.__annotations__
print(anns)
print(type(anns["quantity"]))

# get_type_hints always evaluates and returns real types
resolved = typing.get_type_hints(Trade)
print(resolved)

print(f"Python {sys.version_info.major}.{sys.version_info.minor}")
Output
{'symbol': <class 'str'>, 'quantity': <class 'int'>, 'price': <class 'float'>}
<class 'type'>
{'symbol': <class 'str'>, 'quantity': <class 'int'>, 'price': <class 'float'>}
Python 3.12
Scenario A

In your world

The situation

Your service has a custom serialiser that builds a type-dispatch map from ModelClass.__dict__['__annotations__'] — a pattern copied from a Python cookbook snippet that noted it "avoids inheriting parent-class annotations". It works on Python 3.12 and passes all tests.

What actually breaks

After upgrading to Python 3.14, the class __dict__ no longer contains an __annotations__ entry, so cls.__dict__.get('__annotations__', {}) returns {}. The serialiser's type map is empty and every field falls through to the str() branch, so prices go out as strings where the downstream service expects bare floats. No exception is raised; the failure surfaces three days later in a financial reconciliation report showing zeroes.

What you'd do on Monday

Replace cls.__dict__.get('__annotations__', {}) with inspect.get_annotations(cls). It returns the class's own annotations as real types on 3.10+ and 3.14, which keeps the cookbook's "no parent annotations" behaviour; typing.get_type_hints(cls) would also work but merges base-class fields, which changes the output for subclasses. Cache the result per class on hot paths:

Pythoninspect.get_annotations gives real types in 3.12 and 3.14runs
import inspect
from dataclasses import dataclass


@dataclass
class Trade:
    symbol: str
    quantity: int
    price: float


def serialise(obj) -> dict:
    result = {}
    hints = inspect.get_annotations(type(obj))   # own fields, real types, 3.10+ and 3.14
    for name, ann_type in hints.items():
        val = getattr(obj, name)
        if ann_type is float or ann_type is int:
            result[name] = val
        else:
            result[name] = str(val)
    return result


trade = Trade(symbol="HDFC", quantity=100, price=1750.50)
print(serialise(trade))
Output
{'symbol': 'HDFC', 'quantity': 100, 'price': 1750.5}
Scenario B

Imagine you work at PolicyBazaar

2M policy records per quarterly migration

The numbers

2M policy records per migration run, each on a ~15-field model, and the serialiser introspects the class once per row. Measured on one laptop with CPython 3.12 (timings vary by machine): inspect.get_annotations ≈ 8 µs per call and typing.get_type_hints ≈ 70 µs per call for a 15-field dataclass; a dict lookup in a per-class cache is well under 1 µs. Per run: 2M × 8 µs = 16 s, or 2M × 70 µs = 140 s with get_type_hints.

What the Scenario-A answer does here

The Scenario A fix (read annotations with inspect.get_annotations) is correct, and 16 s is noise in a quarterly batch. The same serialiser reused in a live event consumer at 1,000 events/s spends 1,000 × 8 µs = 8 ms of every second re-reading the same annotations, or 70 ms per second if a teammate swaps in get_type_hints.

What flips at this scale

At this scale the flip is caching: introspect each model class once, at import or on first use, store the map in a module-level dict keyed by class, and look up types from the cache on every row.

The trade-off you'd defend

I would defend a per-class cache over per-row introspection, trading a few lines of cache code (and a restart when a model changes) for removing introspection from the hot path entirely.

Illustrative numbers to make the arithmetic visible; not PolicyBazaar-published figures.

Gotchas

Reading annotations from cls.__dict__ stops working in Python 3.14

Symptom

cls.__dict__.get('__annotations__', {}) returns {} on Python 3.14, so a type map built from it is silently empty; cls.__dict__['__annotations__'] raises KeyError.

Cause

PEP 649 stores a function that builds the annotations on first access instead of storing the finished dict in the class namespace. cls.__annotations__ still works because it goes through the type's descriptor.

Fix

Use inspect.get_annotations(cls) (own annotations only) or typing.get_type_hints(cls) (also merges base classes and resolves strings). Cache the result per class if called on a hot path.

ExceptionGroup and except* require Python 3.11+

Symptom

SyntaxError: invalid syntax or NameError: name 'ExceptionGroup' is not defined on Python 3.10 and below.

Cause

Both were added in 3.11.

Fix

Guard with sys.version_info >= (3, 11) or use ExceptionGroup only in services confirmed to run 3.11+.

asyncio.TaskGroup is 3.11+ only

Symptom

AttributeError: module 'asyncio' has no attribute 'TaskGroup' on 3.10 and below.

Cause

TaskGroup was added in 3.11. Older code uses asyncio.gather instead (Choose gather or a task group).

Fix

Check your minimum supported version before using TaskGroup. asyncio.gather is available on 3.8+ and is the drop-in for older codebases.

Tricky questions

Does from __future__ import annotations behave the same as Python 3.14 lazy annotations?
Most people say

Yes — both make annotations lazy strings.

Actually

No. PEP 563 (from __future__ import annotations) stores annotations as literal strings at compile time; __annotations__["x"] returns "int". PEP 649 (3.14 default) defers evaluation to first access time but returns real type objects — __annotations__["x"] still returns int, not "int". The risks in 3.14 are not string annotations; they are cls.__dict__ no longer holding an __annotations__ dict, and forward-reference NameError shifting from definition time to access time. get_type_hints handles both PEP 563 strings and PEP 649 lazy annotations correctly.

Does upgrading Python require updating all dependencies?
Most people say

No — Python is backward-compatible; old packages work on new versions.

Actually

Not always. C extensions need wheels built for the new version, and on the free-threaded build an extension that has not declared support re-enables the GIL at import (with a RuntimeWarning). Libraries that introspect annotations, like dataclasses and Pydantic, shipped 3.14 updates; your own in-house introspection code has no one updating it but you.

Scenario question

You're handed a Pydantic-based service running Python 3.11. Engineering wants to upgrade to 3.14 before the end of the quarter. What do you check, in what order, and what would stop you?

Model answer

I would check in this order. First, confirm every direct dependency declares 3.14 support — Pydantic v2, SQLAlchemy 2.0, and any C-extension library. Second, search the codebase for annotation introspection (grep -rn "__annotations__") and replace reads from cls.__dict__ with inspect.get_annotations, caching per class on hot paths. Third, run the full test suite under Python 3.14 in CI on a branch before merging to main. What would stop me: a C extension with no 3.14 wheel, because building it from source in production is a risk I would not take for a version bump. If a key dependency has not published a 3.14 wheel yet, I would wait one minor release cycle rather than build from source in production.

Interview corner

CommonName one feature added in each of 3.11, 3.12, and 3.13.
“I'd say 3.11 added asyncio.TaskGroup and ExceptionGroup for structured concurrency. 3.12 added the type statement for clean type aliases and lifted f-string restrictions. 3.13 made the free-threaded (GIL-optional) build available as an experimental option.”
TrickWhat code that runs cleanly on 3.12 will break on 3.14?
“I'd look first for code that reads annotations out of cls.__dict__: in 3.14 the class namespace holds a function that builds them lazily, so .get('__annotations__', {}) quietly returns an empty dict. Then I'd look for forward references that used to fail at definition time, because they now raise NameError only when something reads the annotations. The fix is inspect.get_annotations(cls), which works on 3.10 through 3.14.”
Experience checkWhen should you not upgrade to a new Python version immediately?
“My rule: never before the key dependencies — especially C extensions — have declared support. The usual blocker is a C extension with no wheel for the new version; on the free-threaded build, an extension that has not declared support also turns the GIL back on at import. I wait until the top five dependencies in our requirements.txt publish 3.14 wheels before scheduling the upgrade.”

Memory hook

Each Python version is a migration, not just an upgrade. Read the "What's New" page before pip install python.

If you remember only one thing:

Never read annotations from cls.__dict__; use inspect.get_annotations() or typing.get_type_hints(), which work from 3.10 through 3.14.

Self-check

  1. What Python version introduced ExceptionGroup and except*?

    Python 3.11.

  2. What does the type statement in Python 3.12 replace?

    typing.TypeAlias assignments: type Point = tuple[int, int] replaces Point: TypeAlias = tuple[int, int].

  3. Why does typing.get_type_hints() work in both 3.12 and 3.14 for annotation introspection?

    It reads annotations through the supported API rather than cls.__dict__, evaluates PEP 563 strings (from __future__ import annotations) and forward references, and returns real type objects on both versions. It also merges base-class annotations, which inspect.get_annotations does not.

  4. Where does a forward-reference typo surface on 3.14 compared with 3.12?

    On 3.12 the NameError is raised when the def or class statement runs; on 3.14 the definition succeeds and the NameError is raised later, by whatever first reads the annotations.

Prove it

On Python 3.12, run the production example. Copy the output. Then add from __future__ import annotations at the top of the file (demonstrating PEP 563 compile-time string annotations — a distinct mechanism from Python 3.14's lazy evaluation) and run again. Confirm that print(anns) changes from showing <class 'str'> etc. to showing 'str' etc., but print(resolved) from get_type_hints stays the same. Paste both outputs.