Shelf · Python · Chapter 2

Chapter 2: Functions, scope & closures

Every function you write has hidden rules about how arguments land, where names come from and what state survives the call. Getting those rules wrong produces the kind of bug where the job completes successfully and the database is unchanged — and you only find out when someone asks why yesterday's data never arrived.

Difficulty
Priority
py-02-01Card 1 of 6

Bind and forward function arguments

EasyP1 · Must knowKnow cold9 min
argskwargspositional-onlykeyword-onlyforwarding

Why you care

Every wrapper function in an ETL codebase — retry adapters, logging decorators, generic loaders — must accept arguments it does not understand and forward them without modification. If you cannot distinguish positional-only from keyword-only parameters, adding one to a shared utility will silently break every caller that passes them in the wrong form, and the error only surfaces at runtime with a TypeError that names the parameter but not which caller is wrong. Interviewers ask about *args and **kwargs because the answer separates engineers who copy patterns from engineers who can reason about what a function signature actually enforces.

Explain like I'm 5

Think of a restaurant order form. The first three boxes are numbered: they expect item, quantity, table — in that order and in those positions; you cannot write "quantity" in box one. A second section has labelled fields: notes:, spice level:. These can appear in any order because the label tells the waiter which is which. *args is a pocket for any extra numbered items; **kwargs is a pocket for any extra labelled ones. A wrapper who takes your whole form and hands it to the kitchen untouched never needs to know what is on it.

Core concept

Python function parameters fall into five categories, controlled by two separators:

  • Positional-only (before /): callers must pass by position; they cannot say table="orders".
  • Regular (between / and *): callers may pass by position or by name.
  • Keyword-only (after a bare * or after *args): callers must name them; dry_run=True, not just True.
  • *args: collects all extra positional arguments into a tuple.
  • **kwargs: collects all extra keyword arguments into a dict.

Forwarding rule: func(*args, **kwargs) unpacks a tuple into positional slots and a dict into keyword slots. A thin wrapper that takes (*args, **kwargs) and passes them straight through is compatible with any callee signature.

Use positional-only (/) when the parameter name is an implementation detail you might rename (e.g. open(file, mode, /)). Use keyword-only (*) when passing a value by position would be ambiguous — every boolean flag qualifies. load(path, True) conveys no intent; load(path, dry_run=True) does.

When you cannot know what arguments a callee will need in the future, accept **opts and forward it. When the signature stabilises, replace **opts with explicit keyword-only parameters so callers get IDE autocomplete and early TypeError on typos.

Minimal example Separators and forwarding

PythonPositional-only, keyword-only and forwarding in one functionruns
def load(table, count, /, *, dry_run=False, batch=500):
    return (table, count, dry_run, batch)

print(load("orders", 200))
print(load("orders", 200, dry_run=True, batch=100))

def wrapper(*args, **kwargs):
    return load(*args, **kwargs)

print(wrapper("trades", 50, dry_run=False))
Output
('orders', 200, False, 500)
('orders', 200, True, 100)
('trades', 50, False, 500)

In production Generic ETL pipe with forwarded writer options

The caller controls CSV formatting; etl_pipe does not need to know what options exist.

PythonForwarding writer options through an ETL pipelineruns
import csv
from pathlib import Path


def write_rows(path, rows, /, *, delimiter=","):
    """path and rows are positional-only; delimiter is keyword-only."""
    Path(path).write_text(
        "\n".join(delimiter.join(str(c) for c in r) for r in rows)
    )


def etl_pipe(src, dst, transform, /, **writer_opts):
    """Caller controls writer options; etl_pipe does not need to know them."""
    with open(src, newline="") as f:
        rows = [transform(r) for r in csv.DictReader(f)]
    write_rows(dst, rows, **writer_opts)
    return len(rows)


Path("orders.csv").write_text("id,amount\n1,250\n2,90\n3,410\n")

n = etl_pipe("orders.csv", "doubled.csv",
             lambda r: [r["id"], int(r["amount"]) * 2])
print(f"{n} rows written")
print(Path("doubled.csv").read_text())
Output
3 rows written
1,500
2,180
3,820
Scenario A

In your world

The situation

Your team has a shared run_loader(table, rows, batch_size=1000) called in 40 Jenkins jobs. A migration adds strict_mode: bool = False to the underlying Postgres writer. The first engineer to need it adds it as a positional parameter. A second engineer, six months later, decides to swap the order of batch_size and strict_mode.

What actually breaks

Every job that passes either value positionally breaks with a TypeError. The test suite mocks the writer, so none of the tests catch it. Eight jobs fail in production before anyone notices. The post-mortem takes longer than the fix.

What you'd do on Monday

Make every configuration flag keyword-only from the start. Then run_loader("orders", rows, batch_size=1000) is explicit and order-immune. New options can be added without a flag day.

PythonKeyword-only flags prevent positional confusion (needs real DB connection)
def run_loader(table, rows, /, *, strict_mode=False, dry_run=False, batch_size=1000):
    # table and rows are positional-only: no accidental name coupling.
    # new keyword-only options can be added without touching any caller.
    if dry_run:
        print(f"[dry] would load {len(list(rows))} rows into {table}")
        return 0
    return _write_to_postgres(table, rows,
                              strict_mode=strict_mode,
                              batch_size=batch_size)
Scenario B

Imagine you work at Paytm

500M payment events/day

The numbers

Illustrative: 500M events ÷ 86,400 s ≈ 5,800 events/s peak, running across ~200 worker processes each calling a shared serialisation helper. If that helper gains a new positional parameter and 40 of 200 workers are deployed before the rest, those 40 crash on every call that passes the parameter positionally.

What the Scenario-A answer does here

A single-process ETL fixes argument errors at startup. At 200 workers, a partial deploy means partial breakage: 40 workers crash with TypeError, 160 continue, and the error rate in the monitoring dashboard climbs for twenty minutes before an on-call engineer investigates.

What flips at this scale

At this scale, argument contracts are versioned. You freeze a positional-only interface for the stable core (table name, payload), accept everything else as **opts, and validate opts keys at startup with an explicit allowlist. New options roll out without a flag day because callers that do not pass the option still work.

The trade-off you'd defend

I would defend strict positional-only for the first two parameters that will never change, plus **opts with an allowlist for everything else, trading some IDE autocomplete clarity for deployment independence.

Illustrative numbers chosen to make the arithmetic visible; not Paytm-published figures.

Gotchas

Passing a keyword argument by position past a /

Symptom

TypeError: load() got some positional-only arguments passed as keyword arguments.

Cause

A parameter declared before / cannot be named by the caller; the name is the function's private concern.

Fix

Drop the name; pass by position only. Or remove / if the name is stable and callers knowing it is fine.

Unknown kwargs silently ignored by a wrapper that does not forward

Symptom

dry_run=True is accepted without error, but the data is written anyway and no warning appears.

Cause

The wrapper accepts **kwargs but never passes it to the inner call. The extra keyword disappears.

Fix

Always forward: inner(*args, **kwargs). If you only want specific extras, check them explicitly and raise on unrecognised keys.

Passing the same argument positionally and by name

Symptom

TypeError: load() got multiple values for argument 'table'.

Cause

load("orders", table="orders") fills the table slot twice.

Fix

Pass either by position or by name, never both.

Tricky questions

Can *args appear before a positional-only parameter?
Most people say

No; *args always comes last.

Actually

*args can appear after /. For example def f(a, b, /, *args, kw=True) is valid and common. What you cannot do is declare a positional-only parameter after *args, because everything after *args is already keyword-only.

Does **kwargs capture every keyword argument the caller passes?
Most people say

Yes, every keyword argument.

Actually

Only keyword arguments that do not match an explicit parameter. In def f(a, **kw), calling f(1, a=2) raises TypeError: got multiple values for argument 'a' — a is not in kw because it is already a named parameter.

Scenario question

You're handed send_alert(team, message, level="ERROR") used in 40 places. You need to add a channel parameter that defaults to "slack", but three callers should be able to set it to "pagerduty". How do you change the signature while touching the fewest callers?

Model answer

I'd add channel="slack" as a keyword-only parameter after a *: def send_alert(team, message, *, level="ERROR", channel="slack"). The 40 existing callers pass team and message positionally and never mention level or channel, so they are unchanged. The three new callers add channel="pagerduty" explicitly. The trade-off is that any caller currently passing level positionally — send_alert("data", "failed", "WARN") — will break, so I'd grep for that pattern first.

Interview corner

CommonWhat is the difference between *args and **kwargs?
“I'd say *args collects extra positional arguments into a tuple and **kwargs collects extra keyword arguments into a dict. Together they let a function accept anything and forward it — useful in wrappers and decorators where the outer function does not need to know the inner function's signature.”
TrickCan you call a positional-only parameter by name?
“I'd say no — parameters before / are positional-only: Python raises a TypeError if you try to name them. That lets library authors rename internal parameters in a future version without breaking callers.”
Experience checkWhere have you used **kwargs forwarding in real ETL work?
“I use it in retry wrappers and logging decorators. The wrapper takes (*args, **kwargs) and passes them through unchanged, so it does not need updating every time the underlying loader gains a new option.”

Memory hook

Numbered order-form boxes cannot be labelled; labelled fields cannot be positional — the / is the line between them, and * starts the labelled section.

If you remember only one thing:

Use keyword-only flags for every boolean parameter so the call site reads as intent.

Self-check

  1. What does the / separator mean in a function signature?

    Every parameter before / is positional-only: callers cannot name it. The function can rename it in a future version without breaking callers.

  2. What happens if you call f(1, a=1) when f(a, **kw) is the signature?

    TypeError: f() got multiple values for argument 'a' — 1 fills a positionally and a=1 tries to fill it again by name.

  3. How do you forward all arguments a wrapper receives to an inner function?

    return inner(*args, **kwargs) inside a wrapper defined as def wrapper(*args, **kwargs).

Prove it

Write def strict_load(table, rows, /, *, batch_size=500, dry_run=False). Call it four ways: (a) correctly with strict_load("orders", [], batch_size=100); (b) with table= as keyword — catch the TypeError and print it; (c) with dry_run passed as the third positional — catch the TypeError and print it; (d) through a def relay(*a, **kw): return strict_load(*a, **kw) using the same args as (a). Paste the four results.

py-02-02Card 2 of 6

Replace shared mutable defaults

EasyP1 · Must knowKnow cold9 min
defaultsmutable-defaultdefinition-timeper-call-allocation

Why you care

The mutable-default bug is the most common Python correctness trap that a linter will not catch for you and that unit tests often miss — because tests typically call the function once per test case and never see state leaking from one call to the next. In an ETL job that processes batches in a loop, or in a Jenkins run that retries on failure, the function is called multiple times in the same process, and a shared default list grows silently. The job reports success; the audit numbers do not add up. Interviewers ask about mutable defaults because it tests whether you understand when Python evaluates expressions — definition time versus call time.

Explain like I'm 5

Imagine a whiteboard on the wall of your office. The first time you call the function, you write on the whiteboard. The next call looks at the same whiteboard and sees what you wrote last time. Mutable defaults are that whiteboard: one object, shared forever. The fix is to bring a fresh notepad each time you walk in: create the list inside the function body, after the call has already started.

Core concept

Python evaluates a function's default values exactly once, at the moment the def statement runs. The result is stored inside the function object itself, in __defaults__. Every call that does not pass that argument receives the same object — not a fresh copy.

  • Immutable defaults (integers, strings, tuples, None) are safe because you cannot mutate them in-place. count=0 is fine.
  • Mutable defaults (lists, dicts, sets, custom objects) are dangerous because every call that appends, extends or assigns to the default modifies the one shared object.

The fix is always the same: use None as the default, test for it inside the body, and allocate a fresh object there.

PythonTemplate for any mutable default fix
def process(row, errors=None, config=None):
    if errors is None:
        errors = []
    if config is None:
        config = {}
    ...

Use a mutable default deliberately only when you want per-function-object state — a simple call counter or cache. Document it, because every reader will assume it is a bug.

Minimal example The shared list in action

PythonOne default list shared across every callruns
def add_error(msg, errors=[]):
    errors.append(msg)
    return errors

print(add_error("missing id"))
print(add_error("null amount"))
print(add_error.__defaults__)
Output
['missing id']
['missing id', 'null amount']
(['missing id', 'null amount'],)

In production Batch validator — broken vs fixed

PythonMutable default accumulates across rows; None-default allocates freshruns
import csv
from pathlib import Path


def validate_row_broken(row, errors=[]):
    if not row.get("amount"):
        errors.append(f"row {row['id']}: missing amount")
    return errors


def validate_row(row, errors=None):
    if errors is None:
        errors = []
    if not row.get("amount"):
        errors.append(f"row {row['id']}: missing amount")
    return errors


Path("payments.csv").write_text("id,amount\n1,\n2,500\n3,\n")

with open("payments.csv", newline="") as f:
    rows = list(csv.DictReader(f))

print("broken run 1:", validate_row_broken(rows[0]))
print("broken run 2:", validate_row_broken(rows[1]))
print("broken run 3:", validate_row_broken(rows[2]))

print()

print("ok run 1:", validate_row(rows[0]))
print("ok run 2:", validate_row(rows[1]))
print("ok run 3:", validate_row(rows[2]))
Output
broken run 1: ['row 1: missing amount']
broken run 2: ['row 1: missing amount']
broken run 3: ['row 1: missing amount', 'row 3: missing amount']

ok run 1: ['row 1: missing amount']
ok run 2: []
ok run 3: ['row 3: missing amount']

See it happen The shared default list across calls

errors = [] (created once at def time)missing idnull amount
  1. Module loads: Python evaluates the default list exactly once and stores it inside the function object. No call has been made yet.
  2. Call 1 — row has a missing id. The list appends 'missing id'. One item so far.
  3. Call 2 — row is valid. No append. But the list still carries Call 1's error. Callers that did not reset it see stale data.
  4. Call 3 — another bad row. Appends a second error. The two errors came from different rows but look like one batch failure.
Notice: the list is not reset between calls. Every new call inherits whatever previous calls left behind.
Scenario A

In your world

The situation

A nightly Jenkins job calls validate_settlement(row) for each of the 50,000 rows in a CSV pulled from S3. The function accumulates errors in a default list. In staging the job runs fine: small dataset, one run per invocation.

What actually breaks

Jenkins retries the job on transient S3 errors. On the second attempt, the errors list from the first attempt is still in memory. A settlement row with zero errors is flagged as invalid because it inherits the previous run's errors. Finance receives duplicate alerts; the support queue fills up overnight.

What you'd do on Monday

Change the default to None and allocate inside the body. The fix takes thirty seconds; restoring trust in the alert system takes a week.

PythonFixed validator — fresh list every call (called in a loop, no boto3 needed here)
def validate_settlement(row, errors=None):
    if errors is None:
        errors = []
    if not row.get("amount"):
        errors.append(f"row {row['id']}: missing amount")
    if float(row.get("amount") or 0) < 0:
        errors.append(f"row {row['id']}: negative amount")
    return errors
Scenario B

Imagine you work at Myntra

45M product rows/day across 8 transform jobs

The numbers

Illustrative: 45M rows ÷ 8 jobs ≈ 5.6M rows per job, each job calling a validator in a loop. If the validator has a mutable default list, errors from early rows bleed into later rows in the same job run. At 5.6M rows, a single bad row early in the file contaminates every subsequent clean row's error report for that entire run.

What the Scenario-A answer does here

The Scenario A fix (checking errors is None) works on a single worker. But across 8 parallel jobs sharing the same Python module in memory, the default object is per-function-object, not per-process — each worker process has its own copy, so the default is not shared across workers. The bug is still per-run, not cross-process.

What flips at this scale

At this scale, the fix that matters is immutability by design: validators return a list of error objects rather than accumulating into a shared container. Each caller owns its result list. Functional style (return, do not append) eliminates the entire class of shared-state bugs.

The trade-off you'd defend

I would defend return-a-new-list over accumulate-in-place for any validator used in a batch loop, trading a small allocation cost per row for zero shared-state risk.

Illustrative numbers; not Myntra-published figures.

Gotchas

A dict default shared across calls

Symptom

A function that builds a row dict using a default {} accumulates keys from previous calls. The second call to build_row("id", cache={}) returns a dict with fields from the first call.

Cause

Same as the list case: the empty dict is created once at def time.

Fix

cache=None; cache = {} if cache is None else cache.

Passing None explicitly breaks the sentinel check

Symptom

A caller passes validate(row, errors=None) intending to start fresh, but the function has if errors is None: errors = [] — which is correct. The bug is the caller who passes the shared list directly: validate(row, errors=shared_list) — now the fix does not help.

Cause

The sentinel only protects against not passing the argument. An explicit mutable object bypasses it.

Fix

Document whether the function mutates errors in-place or returns a new list. Prefer returning a new list.

Using a mutable default as an intentional cache without documentation

Symptom

A hits={} default acts as an in-memory LRU. Works in development. Fails in tests when test isolation requires a fresh cache.

Cause

The cache is tied to the function object, which is module-level and survives between test cases unless explicitly cleared.

Fix

Expose fn.__defaults__ reset in test teardown, or use functools.lru_cache which has a documented .cache_clear().

Tricky questions

Is def f(x=[]) always a bug?
Most people say

Yes; never use mutable defaults.

Actually

It is a bug when the intent is "fresh list each call". It is intentional when the intent is "shared list that persists across calls" — a simple memoisation table or call counter. The issue is that the intent is invisible from the signature; document it with a comment.

Does def f(x=list()) avoid the problem?
Most people say

Yes, list() creates a new list each time.

Actually

No. list() is evaluated once, at def time, exactly like []. The result — one empty list — is stored in __defaults__. Calling list() in the default expression does not make it call-time.

Scenario question

You're handed a function def load_batch(rows, seen=set()) that is supposed to deduplicate rows by id. It works correctly in the first Jenkins run but produces wrong results in every subsequent run. What is the problem and how do you fix it without changing callers?

Model answer

seen is a mutable default: the same set object is reused across calls. By the second Jenkins run, the set already contains every id from the first run, so every row looks like a duplicate and nothing is loaded. I'd change the default to None and add if seen is None: seen = set() as the first line of the body. Callers that omit seen get a fresh set each call; callers that pass an explicit set still work. The trade-off is a tiny branch on every call, which is negligible compared to the row processing cost.

Interview corner

CommonWhy is def f(x=[]) dangerous?
“I'd say the list is created once, when the def statement runs, and stored in the function object. Every call that omits x gets the same list. Appending to it in one call means the next call sees those items. The fix is x=None and if x is None: x = [] at the top of the body.”
TrickDoes def f(x=list()) avoid the mutable-default problem?
“I'd say no — list() is evaluated at definition time, just like []. Python evaluates all default expressions once when the def runs. The result is one empty list stored in __defaults__, shared forever.”
Experience checkWhere has this bitten you?
“I hit it in an error accumulator for a batch validator. The function had a default list, the Jenkins job ran twice after a retry, and the second run flagged clean rows because the error list from the first run was still there. The fix took a minute; explaining it to finance took longer.”

Memory hook

Defaults are born when def runs, not when the function is called. A mutable default is a whiteboard, not a notepad.

If you remember only one thing:

Use None as the default for any mutable argument; allocate inside the body.

Self-check

  1. When does Python evaluate a function's default argument values?

    At definition time, once, when the def statement runs — not on each call.

  2. What is the idiomatic fix for a mutable default?

    Use None as the default and allocate a fresh object in the function body: if errors is None: errors = [].

  3. How can you verify that two calls are sharing the same default list?

    Call f.__defaults__ — it is a tuple containing the live default objects. After two calls that append to it, the list inside __defaults__ will have grown.

Prove it

Write def accumulate(item, bucket=[]) that appends item to bucket and returns bucket. Call it five times with different items. Print the list after each call — show it grows across calls. Then rewrite with bucket=None, call five times again, and show that each call returns a single-item list. Paste both output sequences.

py-02-03Card 3 of 6

Resolve names with LEGB

MediumP1 · Must knowKnow cold11 min
legbscopeglobalnonlocalunboundlocalerror

Why you care

Every time you read a name in Python, the interpreter follows a fixed lookup order. Most of the time you do not notice this because the name you want is right where you expect it. The moment it is not — when you try to increment a module-level counter inside a function, or when a nested helper tries to update its parent's variable — Python either gives you a stale read, raises UnboundLocalError, or mutates global state you did not intend to touch. In an ETL job with multiple helpers sharing configuration, getting scope wrong produces either silent miscounts or crashes at 3 AM. Interviewers ask about LEGB and UnboundLocalError because the answer reveals whether you know how Python assigns scope at compile time, not at runtime.

Explain like I'm 5

Think of rooms in a building. You are in your office (Local). To find a stapler, you look first on your desk. Not there? You check the shared supply room on your floor (Enclosing function, if any). Not there? You go to the main supply room for the whole building (Global, the module). Still not there? You ask the building manager, who has everything pre-installed (Built-in, like len and print). If you do not have a stapler and you want to buy one, you have to explicitly say "I am buying one for the whole building" (global), or "I am buying one for my floor" (nonlocal). Otherwise the stapler you buy is just yours.

Core concept

Python resolves names in this order — LEGB:

  1. Local — names assigned in the current function body (including parameters).
  2. Enclosing — names in any enclosing def or lambda, searched inside-out.
  3. Global — names at the module top level.
  4. Built-in — names in the builtins module (len, print, None, …).

Search stops at the first match. If no match is found anywhere, Python raises NameError.

The critical rule: Python's compiler determines a name's scope before execution. If a name appears on the left of any assignment anywhere in the function body — even if that assignment is never reached — the compiler marks it as local to that function. Reading it before the assignment runs produces UnboundLocalError, not a fallback to the outer scope.

global x and nonlocal x are compiler directives: they change the scope decision before the function runs. global x makes all reads and writes in the function go to module scope. nonlocal x makes them go to the nearest enclosing function scope that has x. Without either, any assignment in the function body makes x local.

Use global sparingly — it makes function behaviour depend on module-level state, which is hard to test and easy to corrupt. Prefer returning values and letting the caller manage state. Use nonlocal when a factory function needs a counter or accumulator in the enclosing scope (closures, Capture state in closures deliberately).

Under the hood

At compile time, Python's bytecode compiler scans each function body and assigns every name to a scope category stored in code object attributes. Names assigned anywhere in the body (including +=, for x in …, with … as x, import x) go into co_varnames (locals). Names that appear in an inner function but are assigned in an outer function go into co_cellvars (in the outer) and co_freevars (in the inner). Names that are neither local nor free are looked up in globals and builtins at runtime. This classification happens before the first bytecode is executed. That is why the assignment count = 0 six lines below a print(count) is enough to make the print raise UnboundLocalError — the compiler already decided count is local before the function ran.

Minimal example UnboundLocalError and the global fix

PythonAugmented assignment makes a name local; global lets it reach the moduleruns
count = 0

def bad_increment():
    try:
        count += 1          # Python sees assignment: count is local but unbound
    except UnboundLocalError as e:
        print("UnboundLocalError:", e)

bad_increment()

def good_increment():
    global count
    count += 1

good_increment()
print("count:", count)
Output
UnboundLocalError: cannot access local variable 'count' where it is not associated with a value
count: 1

In production Nested job runner with nonlocal row counter

nonlocal lets the inner run function update the enclosing row_count without touching module state. The global CONFIG snapshot is taken once at make_job time, so later mutations to CONFIG do not affect already-built jobs.

Pythonnonlocal accumulates across calls inside a closureruns
from pathlib import Path

CONFIG = {"batch_size": 500, "dry_run": False}


def make_job(table, *, overrides=None):
    """Build a job function. Snapshots CONFIG at definition time."""
    effective = {**CONFIG, **(overrides or {})}
    row_count = 0

    def run(rows):
        nonlocal row_count
        if not effective["dry_run"]:
            row_count += len(rows)
        return row_count

    return run


job = make_job("orders")
print(job([1, 2, 3]))
print(job([4, 5]))

CONFIG["dry_run"] = True        # mutate global AFTER job is built
dry_job = make_job("payments")  # new job sees the updated CONFIG
print(dry_job([1, 2, 3]))
print(dry_job([4, 5]))
Output
3
5
0
0

See it happen LEGB lookup order

not herenot herenot hereLocalEnclosingGlobalBuilt-in
  1. Python always starts in the Local scope. Any name assigned anywhere in the function body is treated as local — even before the assignment runs.
  2. Not found locally: Python checks the nearest Enclosing function scope (if any). Applies only to nested functions.
  3. Not in Enclosing: Python checks the module-level Global scope.
  4. Not in Global: Python checks Built-in scope. If still not found, NameError is raised.
  5. global x and nonlocal x are compile-time directives. They skip Local and go directly to Global or Enclosing, letting the function assign to those scopes.
Notice: search stops at the first match and never backtracks. Once a name is classified as local by the compiler, Python will not fall back to Global, even if the local assignment has not run yet.
Scenario A

In your world

The situation

Your Jenkins ETL job has a module-level ROW_COUNT = 0 for an audit log. A helper function process_batch(rows) is supposed to increment it.

What actually breaks

ROW_COUNT += 1 inside process_batch raises UnboundLocalError: cannot access local variable 'ROW_COUNT' where it is not associated with a value. The job crashes. Jenkins marks it failed with a traceback that points to the increment line, not to the declaration.

What you'd do on Monday

The immediate fix is global ROW_COUNT at the top of the function. The better fix is to return the new count and let the caller manage state — that is easier to test and does not couple the function to module-level variables.

PythonTwo fixes: global (immediate) and return-value (better)
ROW_COUNT = 0

# immediate fix — works but creates a hidden dependency on module state
def process_batch_global(rows):
    global ROW_COUNT
    ROW_COUNT += len(rows)

# better fix — pure function, easy to test
def process_batch_pure(rows):
    return len(rows)

# caller owns the counter
ROW_COUNT += process_batch_pure(rows)
Scenario B

Imagine you work at Zepto

200k order events/hour

The numbers

Illustrative: 200k events/hour ≈ 56 events/s. A worker process runs three nested pipeline stages. Each stage is a closure that uses nonlocal to update a shared stage counter. Under load testing at 3× peak, two workers run in the same process via threading. Both threads call the same closure and race on the shared nonlocal variable.

What the Scenario-A answer does here

In Scenario A (single process, sequential batches), nonlocal works correctly. At scale with two threads sharing the same function object, row_count += n is not atomic: read, add and write are three separate operations. Thread A reads 50,000; thread B reads 50,000; both write 50,001; one increment is lost.

What flips at this scale

At this scale, stage counters live outside the closure — in a threading.Lock-protected counter or an atomic multiprocessing.Value. Closures carry configuration (thresholds, column names) but not mutable state. Mutable state lives in objects whose mutation is explicitly synchronised.

The trade-off you'd defend

I would defend moving mutable stage counters into thread-safe objects over accepting lost increments, trading slightly more verbose call sites for correct audit numbers.

Illustrative numbers; not Zepto-published figures.

Gotchas

Reading a variable before its local assignment

Symptom

UnboundLocalError: cannot access local variable 'x' where it is not associated with a value on a read that comes before an assignment later in the function.

Cause

The compiler saw the assignment and marked x local for the whole function. Reading it before the assignment runs fails.

Fix

Move the assignment before the read, or use global/nonlocal if the intent was to reach an outer scope.

Accidentally shadowing a built-in

Symptom

TypeError: 'int' object is not callable when you call len(rows) — but only inside a specific function.

Cause

The function has len = len(rows) somewhere, which shadows the built-in len with an integer.

Fix

Rename the local variable. Avoid names that shadow built-ins: len, list, id, type, min, max, sum, input, open.

global in a nested function reaches the module, not the enclosing function

Symptom

global x inside a doubly-nested function jumps past the middle function's x directly to module scope, which is usually not what you wanted.

Cause

global always means the module scope; there is no "two levels up" in global. Only nonlocal climbs the enclosing chain.

Fix

Use nonlocal x to reach the nearest enclosing function that has x.

Tricky questions

Does global x create x if it does not exist yet?
Most people say

No; it just says the variable is global.

Actually

global x followed by x = 1 creates x in the module's namespace if it was not there before. global x alone (with no assignment) just changes scope resolution for reads in that function; if x does not exist globally, reading it raises NameError.

Can two functions share a local variable?
Most people say

No; local variables are private to each call frame.

Actually

Not directly. But if both are closures defined inside the same outer function, they can share a cell object via nonlocal — each closure reads and writes through the same cell. That is deliberate shared state; Capture state in closures deliberately covers the mechanics.

Scenario question

You're handed a module where three functions at the same level all read a shared _cache = {} dict that is defined at module top level. One of the functions clears it with _cache = {} (reassignment), but callers of the other two functions still see the old data. What is happening and how do you fix it?

Model answer

The function that clears it does _cache = {}, which creates a new local variable named _cache and leaves the module-level one untouched — because without global _cache, any assignment makes the name local. The other two functions still read the module-level dict, which has not changed. The immediate fix is global _cache in the clearing function so the reassignment reaches module scope. A cleaner fix is _cache.clear() instead of reassignment — that mutates the existing dict in place, so all references see the change without needing global. The trade-off: clear() modifies in place (any other reference sees the change immediately, which is what we want); reassignment creates a new object (other references keep the old one).

Interview corner

CommonWhat is the LEGB rule?
“I'd describe LEGB as the order Python searches for a name: Local first, then Enclosing function scopes moving outward, then Global (module level), then Built-in. Search stops at the first match. If nothing matches, you get a NameError.”
TrickWhy does x += 1 raise UnboundLocalError when x is defined at module level?
“I'd say it's because += is an assignment, and any assignment in a function body makes that name local for the whole function at compile time. Python treats x as local everywhere in the function, but the read before the assignment runs finds it unbound. The fix is global x at the top.”
Experience checkWhen would you use nonlocal in real code?
“I'd reach for it in a closure that needs to count how many times it has been called, or in a factory that builds a stateful helper. nonlocal lets me update the enclosing function's variable rather than a module-level one — it keeps the state localised to the factory and avoids global coupling.”

Memory hook

Python searches for a name the way you'd search for a stapler: your desk first, then the floor supply room, then the building's main supplies — and stops the moment it finds one.

If you remember only one thing:

If count += 1 raises UnboundLocalError, add global count or nonlocal count at the top of the function.

Self-check

  1. What is the LEGB rule?

    Local → Enclosing → Global → Built-in. Python checks each scope in this order and stops at the first match.

  2. Why does a name assigned anywhere in a function body raise UnboundLocalError when read before the assignment?

    The compiler marks it as local for the whole function at compile time, so there is no fallback to an outer scope. Reading the local name before it is bound fails.

  3. What is the difference between global x and nonlocal x?

    global x makes x refer to the module-level namespace. nonlocal x makes x refer to the nearest enclosing function scope that has an x. nonlocal cannot reach module scope.

Prove it

Write a nested function pair where outer defines tally = 0 and inner uses nonlocal tally to increment it. Call inner five times, printing tally each time. Then remove nonlocal, call once, and catch the UnboundLocalError. Paste both outputs.

py-02-04Card 4 of 6

Capture state in closures deliberately

MediumP2 · Very likelyKnow cold11 min
closureslate-bindingcell-objectexplicit-capturefactory

Why you care

Closures are how Python functions carry state without a class. You use them every time you write a factory: make_validator(col), make_retrier(max_attempts), make_transformer(config). The gotcha that catches everyone once is late binding: closures capture a reference to a variable, not the value at the moment they are defined. In a loop, every closure you build captures the same loop variable — and when you call them later, they all see the final value the loop left behind. In an ETL job that builds per-column processors in a loop, every processor silently targets the last column. The build is green; the output is garbage. Interviewers ask about closures and late binding because it is the most common "this runs fine, it just does the wrong thing" bug in higher-order Python.

Explain like I'm 5

Imagine you give each of three assistants a note that says "look at the scoreboard and tell me the current number". You write all three notes before any game is played. When you ask them later, they all go to the scoreboard — which now shows the final score — and they all say the same thing. Late binding is that scoreboard: closures do not copy the value; they copy the direction to the scoreboard. The fix is to write the current score on the note itself before handing it out.

Core concept

A closure is an inner function that references a name from its enclosing function. Python implements this with a cell object: a small wrapper that both the outer and the inner function read and write through. The cell holds the current value of the variable.

Late binding follows naturally: the cell is not frozen when the inner function is created. If the enclosing scope keeps changing the variable — as a for loop does on each iteration — the cell updates, and all closures that reference it see the new value when called.

Three ways to capture the value early, not the variable:

  1. Default argument: lambda i=i: i binds the current value of i to a parameter. The parameter is local to the lambda; changing the outer i later has no effect.
  2. Factory function: def make(i): return lambda: i — each call to make creates a new scope with its own i.
  3. functools.partial: covers the case where you want to pre-fill an argument without defining a lambda at all (Pass behavior and bind partials).

Use the factory pattern over default arguments when the closure body is more than one expression. Use default arguments for short lambdas where the intent is obvious. Avoid functools.reduce or other approaches that obscure which variable is being captured.

Under the hood

When Python's compiler detects that an inner function references a name defined in an outer function, it promotes that name from a regular local variable to a cell variable in the outer function (co_cellvars) and a free variable in the inner function (co_freevars). Both functions access it through a PyCell object: outer.__code__.co_cellvars lists the names; inner.__closure__ is a tuple of cell objects; inner.__closure__[0].cell_contents gives the current value. The cell lives as long as any closure referencing it is alive — even after the outer function has returned. That is why a closure can hold state between calls: the cell persists beyond the function's lifetime.

Minimal example Late binding in a list comprehension

PythonAll closures share one cell; default arg captures the current valueruns
handlers = [lambda: i for i in range(3)]
print([h() for h in handlers])

handlers_fixed = [lambda i=i: i for i in range(3)]
print([h() for h in handlers_fixed])
Output
[2, 2, 2]
[0, 1, 2]

In production Per-column validators built in a loop

PythonFactory function avoids late binding; walrus operator collects non-None errorsruns
import csv
from pathlib import Path


def make_validator(column, min_val):
    """Each call creates a new scope: column and min_val are independent copies."""
    def validate(row):
        v = row.get(column)
        if v is None or v == "":
            return f"{column}: missing"
        try:
            if float(v) < min_val:
                return f"{column}: {v} < {min_val}"
        except ValueError:
            return f"{column}: not a number"
        return None
    return validate


rules = {"amount": 0.01, "tax": 0.0}
validators = {col: make_validator(col, mn) for col, mn in rules.items()}

Path("txns.csv").write_text("id,amount,tax\n1,250.0,12.5\n2,,0.0\n3,-5.0,1.0\n")

with open("txns.csv", newline="") as f:
    for row in csv.DictReader(f):
        errors = [e for v in validators.values() if (e := v(row))]
        if errors:
            print(f"row {row['id']}: {errors}")
        else:
            print(f"row {row['id']}: ok")
Output
row 1: ok
row 2: ['amount: missing']
row 3: ['amount: -5.0 < 0.01']

See it happen How late binding works through a cell object

list compcell (i)handler[0]handler[1]handler[2]i = 0i = 1i = 2read iread iread i
  1. Iteration 0: Python creates a cell object for i and sets it to 0. handler[0] is built and stores a reference to the cell — not the value.
  2. Iteration 1: the same cell is updated to 1. handler[1] is built, also pointing at the same cell.
  3. Iteration 2: the cell is updated to 2 and the loop ends. All three handlers share one cell, now holding 2.
  4. Calling handler[0]() reads the cell. The cell says 2, so handler[0] returns 2, not 0.
  5. handler[1]() reads the same cell: 2.
  6. handler[2]() also returns 2. The result is [2, 2, 2], not [0, 1, 2]. This is late binding.
Notice: the cell is updated on every iteration, and all closures reference the same cell. The fix — default arg or factory — creates a separate, frozen binding per closure.
Scenario A

In your world

The situation

You build per-column ETL validators in a loop: validators = [lambda row: row[col] for col in schema_columns]. You have eight columns. Each validator is supposed to read the corresponding column.

What actually breaks

Every validator reads col from the shared cell. By the time any validator is called (after the loop finishes), col is the last column name. All eight validators target the last column. Eight validators return the same column's value; seven columns are silently unchecked. Jenkins reports 0 validation errors on rows that are genuinely invalid.

What you'd do on Monday

Use make_validator(col) as a factory, or use a default argument: lambda row, c=col: row[c]. Either creates an independent binding for each validator.

PythonTwo fixes — factory (preferred) and default-arg (quick)
# factory: clear intent, testable independently
def make_validator(col):
    def validate(row):
        return row.get(col)
    return validate

validators_ok = [make_validator(col) for col in schema_columns]

# default-arg: concise for simple lambdas
validators_quick = [lambda row, c=col: row.get(c) for col in schema_columns]
Scenario B

Imagine you work at Postman

500k test collection runs/day

The numbers

Illustrative: 500k collection runs/day, each collection containing ~50 request validators built dynamically from a JSON schema at parse time. If validators are built with late binding in a loop, all 50 per collection target the last schema rule. At 500k runs/day, every run silently skips 49 of 50 validations. A schema regression that would have been caught in testing reaches production.

What the Scenario-A answer does here

In Scenario A (a handful of validators, manually tested), the late-binding bug might be caught in code review. At 500k runs/day with hundreds of collection schemas, the only check is automated. A bug that passes all validators silently is a correctness failure at scale — not a performance one.

What flips at this scale

At this scale, schema validation logic is not built at runtime in a loop. It is compiled from the schema into a registry of named, unit-tested validator functions. The registry maps rule names to callables. No lambdas in loops; no closures capturing loop variables.

The trade-off you'd defend

I would defend a pre-compiled validator registry over dynamic closure construction, trading startup cost for correctness guarantees that survive schema changes.

Illustrative numbers; not Postman-published figures.

Gotchas

All loop closures return the same value

Symptom

A list of functions built in a for loop all return the same result — the last value the loop variable held.

Cause

Late binding: every closure shares one cell object, which holds the loop variable's final value.

Fix

Use a factory function or a default argument to capture the value at definition time.

A closure mutates the enclosing scope through a cell — surprising to callers

Symptom

After calling an inner function, a variable in the outer function has changed unexpectedly.

Cause

The inner function used nonlocal x and assigned to x, modifying the shared cell.

Fix

Document that the inner function mutates the outer scope, or redesign so the inner function returns a value rather than assigning to a nonlocal.

__closure__ is None for a function that does not close over anything

Symptom

f.__closure__ returns None, not an empty tuple.

Cause

A function that references no free variables has no cell objects and no __closure__.

Fix

This is not a bug; just check f.__closure__ is not None before indexing.

Tricky questions

Does a closure capture the value or the variable?
Most people say

It captures the value at the time the closure is created.

Actually

It captures a reference to the variable (via a cell object). The value it reads at call time is whatever the cell holds when the closure is called — which may be different from what it held when the closure was created.

Is lambda i=i: i creating a closure?
Most people say

Yes, it captures i from the enclosing scope.

Actually

No. The i=i default argument binds the current value of the outer i to a local parameter also named i. The lambda body reads its own local parameter, not the outer variable. You can verify: f.__closure__ is None.

Scenario question

You're handed code that builds a list of retry handlers: handlers = [lambda: retry(endpoint) for endpoint in endpoints]. Callers report that all handlers retry the same endpoint regardless of which one they call. What do you change and why?

Model answer

Late binding: all lambdas capture the endpoint cell, which holds the last value after the loop. I'd fix it with a factory: def make_handler(ep): return lambda: retry(ep) and handlers = [make_handler(ep) for ep in endpoints]. Each make_handler call creates a new scope with its own ep parameter. The trade-off versus a default argument: the factory is more readable when the body grows, and it gives the inner function a meaningful name in tracebacks. A default arg — lambda ep=endpoint: retry(ep) — works too and is fine for a one-liner.

Interview corner

CommonWhat is late binding in closures?
“I'd say a closure captures a reference to the variable, not the value at the time the closure was defined. If the variable changes after the closure is created, the closure sees the new value. In a loop, all closures share one cell object that holds the final loop value.”
TrickWhy does [lambda: i for i in range(3)] produce [2, 2, 2] and not [0, 1, 2]?
“I'd explain it as: all three lambdas reference the same cell object for i. The loop updates that cell on each iteration. By the time you call any of the lambdas, the loop has finished and the cell holds 2. The fix is a default arg — lambda i=i: i — which binds the current value to a local parameter.”
Experience checkWhere has late binding caused a real bug in your work?
“I hit it building per-column validators in a loop for a CSV schema. Each validator was a lambda reading col from the loop variable, so all of them ended up targeting the last column. The ETL job reported zero validation errors even though several columns were unchecked — I switched to a factory function.”
DesignWhen would you choose a closure over a class for stateful logic?
“I'd use a closure for simple retained state — a counter, a cache, a retry budget — because it is less overhead than a class. I'd switch to a class when the state has multiple attributes, when I need to inspect or serialise it, or when the logic is complex enough that named methods improve readability.”

Memory hook

A closure carries a reference to the scoreboard, not a photograph of it. The scoreboard changes; so does the closure's answer.

If you remember only one thing:

If loop closures all return the same value, they share a cell — fix with a factory def make(x): return lambda: x or lambda x=x: x.

Self-check

  1. What is a cell object?

    A thin wrapper object that both an outer function and its inner closures read and write through. It lets the outer scope and all closures share the same variable.

  2. How do you capture the current value of a loop variable in a closure?

    Either use a factory — def make(x): return lambda: x — so each factory call creates an independent scope; or use a default argument — lambda x=x: x — to bind the value to a local parameter at definition time.

  3. How do you inspect what a closure has captured?

    fn.__closure__ is a tuple of cell objects; fn.__closure__[0].cell_contents gives the current value held by the first cell.

Prove it

Build a list of five lambdas in a for i in range(5) loop with no capture fix. Print [h() for h in handlers] — all five should return 4. Then fix them with the default-argument trick and print again — you should get [0, 1, 2, 3, 4]. Finally, print handlers_fixed[2].__closure__ — it should be None, confirming the fixed lambda has no closure. Paste all three outputs.

py-02-05Card 5 of 6

Pass behavior and bind partials

MediumP2 · Very likelyKnow cold10 min
first-class-functionspartiallambdacallbacksregistry

Why you care

Python functions are objects. You can store them in a list, pass them to another function, return them from a function and put them in a dict. That makes an ETL pipeline configurable without a single if/elif chain: you keep a registry of transform callables and look one up by name. When a new transform type arrives, you add one line to the registry instead of touching the dispatch logic. Interviewers ask about first-class functions and functools.partial because they reveal whether a candidate thinks in terms of composable parts or monolithic switch statements — the former is what product companies expect at senior level.

Explain like I'm 5

Imagine a preset coffee machine where each button is labelled: "espresso", "latte", "oat flat white". Someone programmed each button to run a specific recipe. You press a button and the machine runs the recipe — you do not need to know what is in it. functools.partial is how you program a button: you take a general recipe function and pre-fill some of its knobs, then hand the resulting preset to the machine. The machine presses it at the right moment without knowing what recipe is inside.

Core concept

A first-class function is a function that can be stored in a variable, passed as an argument and returned as a value — like any other object. In Python every function is first-class.

functools.partial(f, *args, **kwargs) returns a new callable that, when called with additional arguments, invokes f(*pre_args, *args, **pre_kwargs, **kwargs). Use it to pre-fill positional or keyword arguments without writing a wrapper function or a lambda.

lambda creates an anonymous function inline. Use it for expressions too short to name, passed once and never reused. A lambda is a function object; it has all the same properties as a def-created function except a meaningful __name__.

Decision rule:

  • Pre-filling one or two arguments: partial is cleaner and picklable.
  • More complex logic or multiple expressions: a named function with def.
  • Truly one-shot, single-expression: lambda, but only if it fits on one line and names nothing.
  • Never use lambda where you would need to chain three operations or handle exceptions; write a named function.

partial objects have a readable repr, carry func, args and keywords attributes you can inspect, and are picklable if the wrapped function is. A lambda is named <lambda> in tracebacks. When a production job dies at 3 AM, <lambda> in the traceback is not helpful.

Minimal example Partial pre-fills arguments left to right

Pythonpartial creates a new callable with arguments pre-boundruns
from functools import partial


def send_alert(level, team, message):
    return f"[{level}] {team}: {message}"


warn_data = partial(send_alert, "WARN", "data-eng")
print(warn_data("null rows in settlements"))
print(warn_data("schema drift detected"))
Output
[WARN] data-eng: null rows in settlements
[WARN] data-eng: schema drift detected

In production Configurable transform pipeline with a registry

Named functions and partials replace an if/elif chain. Adding a transform is one line in the registry.

PythonTransform registry with partial; no if/elif neededruns
from functools import partial


def scale_field(factor, field, row):
    try:
        return {**row, field: round(float(row[field]) * factor, 2)}
    except (KeyError, ValueError):
        return row


def rename_field(old, new, row):
    if old in row:
        result = dict(row)
        result[new] = result.pop(old)
        return result
    return row


def run_pipeline(rows, transforms):
    for row in rows:
        for t in transforms:
            row = t(row)
        yield row


pipeline = [
    partial(scale_field, 1.18, "amount"),    # add 18% GST
    partial(rename_field, "amount", "gross"),
]

data = [
    {"id": "1", "amount": "250.00"},
    {"id": "2", "amount": "90.00"},
]

for r in run_pipeline(data, pipeline):
    print(r)
Output
{'id': '1', 'gross': 295.0}
{'id': '2', 'gross': 106.2}
Scenario A

In your world

The situation

Your ETL pipeline applies one of six transforms to each column depending on a transform_type field in the schema config. The current code is a 60-line if/elif chain inside apply_transform(row, col, transform_type). Three developers hit merge conflicts on that function in the same month.

What actually breaks

Every new transform type — and the business requests them frequently — requires editing the central function, re-reading 60 lines to find the right place, and coordinating with whoever else is also adding a transform. One developer merges the wrong branch; a transform is silently dropped and data goes through without being normalised.

What you'd do on Monday

Replace the chain with a dict registry. Each entry maps a name to a partial or a named callable. Adding a transform is adding one line to a dict that lives at the top of the module — no conflicts, no logic to touch.

PythonTransform registry replaces if/elif (no external dependencies)
from functools import partial

TRANSFORMS = {
    "scale_gst":    partial(scale_field, 1.18, "amount"),
    "scale_tds":    partial(scale_field, 0.90, "amount"),
    "rename_gross": partial(rename_field, "amount", "gross"),
}

def apply_transform(row, transform_type):
    fn = TRANSFORMS.get(transform_type)
    if fn is None:
        raise ValueError(f"unknown transform: {transform_type!r}")
    return fn(row)
Scenario B

Imagine you work at Freshworks

15k helpdesk tickets/day per tenant, 50k tenants

The numbers

Illustrative: 15k × 50k = 750M ticket events/day across tenants. Each tenant has a custom transform pipeline (field renames, SLA escalation rules, sentiment thresholds). At 50k tenants, a per-tenant if/elif block is impossible; the registry is the only sensible model.

What the Scenario-A answer does here

The Scenario A fix (a registry) works fine for one tenant. At 50k tenants, the registry is loaded from a database per tenant on first use. The problem is not the structure but the initialisation cost: building 50k partial objects at startup is ~50ms each × 50k = 2,500 s. You need lazy loading.

What flips at this scale

At this scale, the registry is not in memory for every tenant simultaneously. It is loaded on demand and cached with a TTL. The partial pre-binding is unchanged; only the cache and eviction policy are new. The composable unit (the named function or partial) stays simple; the orchestration around it grows.

The trade-off you'd defend

I would defend a lazily-loaded, TTL-cached registry over a static in-memory one, trading slightly more complex initialisation code for bounded memory usage across 50k tenants.

Illustrative numbers; not Freshworks-published figures.

lambda vs partial vs named function

Readable repr / __name__PicklableInspectable argsMultiple expressions
lambda<lambda>NoNoNo
partialfunctools.partial(f, ...)If f isYes (func, args, keywords)No (wraps one callable)
Named function (def)Function nameYesVia signatureYes

Use partial when you want partial application without a wrapper; use a named function when the transform logic is non-trivial or when picklability matters (multiprocessing queues).

Gotchas

Partial fills from the left; you cannot skip positional arguments

Symptom

partial(f, third_arg) does not pre-fill the third argument — it pre-fills the first.

Cause

partial fills positional arguments in order from the left.

Fix

Use keyword arguments if the function accepts them: partial(f, field="amount"). Or redesign the function signature so the argument you want to pre-fill comes first (factory-friendly order).

Lambda in a loop creates closures with late binding

Symptom

A list of lambdas built in a loop all call the wrong function or use the wrong parameter.

Cause

This is the closure late-binding bug covered in Capture state in closures deliberately.

Fix

Use partial or a factory function instead of a lambda in a loop.

partial on a method does not bind self

Symptom

partial(MyClass.load, "orders") passes "orders" as self when called.

Cause

MyClass.load is an unbound function; the first positional argument is self.

Fix

Bind to an instance first: partial(instance.load, "orders"), or use functools.partialmethod for class-level binding.

Tricky questions

Is partial(f, a) the same as lambda *args, **kwargs: f(a, *args, **kwargs)?
Most people say

Yes, they behave identically.

Actually

Mostly, but partial has a readable repr, carries .func, .args and .keywords attributes, and is picklable if f is. The lambda is named <lambda> in tracebacks and is not picklable. For a multiprocessing queue, partial works; the lambda does not.

Does partial evaluate its pre-filled arguments immediately?
Most people say

No, it evaluates them when the partial is called.

Actually

Yes. partial(f, expensive_call()) calls expensive_call() at the moment partial(...) runs, not when the resulting callable is invoked. If you want lazy evaluation, wrap in a factory: def make(arg): return partial(f, arg).

Scenario question

You're handed a data cleaning pipeline where each column's cleaning step is a different lambda stored in a list. A colleague says "replace all the lambdas with partial — they're cleaner". You agree for most but not all. Which lambdas would you keep and which would you replace, and why?

Model answer

I'd replace any lambda that just pre-fills arguments: lambda row: strip_field("name", row) becomes partial(strip_field, "name") — clearer, picklable, inspectable. I'd keep a lambda only if it combines two operations that don't have a single named function: lambda row: validate(normalise(row)) — here a small named function is actually better than both. The trade-off: partial wins on traceability and picklability; a named function wins on complexity and expressiveness; a lambda wins only on brevity, which matters less as soon as the codebase grows.

Interview corner

CommonWhat does functools.partial do?
“I'd say it creates a new callable with some arguments pre-filled. partial(send_alert, 'WARN', 'data-eng') returns a callable that only needs a message. When called, it invokes send_alert('WARN', 'data-eng', message).”
TrickWhy prefer partial over a lambda for argument binding in a registry?
“I'd choose partial because it has a readable repr that names the function and its pre-filled args, it is picklable if the wrapped function is, and it exposes .func, .args and .keywords for inspection. A lambda shows up as <lambda> in tracebacks, which is not useful at 3 AM.”
DesignWhen would you use a registry of callables instead of a class hierarchy?
“I'd reach for a registry when the variation is in behaviour, not data structure — a registry maps a key to a callable, adding a new variant is one line, and there is no subclass boilerplate. I use classes when variants carry different state; a registry when they carry different logic applied to the same data.”

Memory hook

A function is an object. A registry is just a dict. partial is a preset.

If you remember only one thing:

Replace if/elif dispatch on a type string with a dict of callables; adding a variant is one registry entry.

Self-check

  1. What does partial(f, a, b) return?

    A new callable that, when called with additional arguments *args, **kwargs, invokes f(a, b, *args, **kwargs).

  2. Why prefer partial over a lambda when storing a callable in a registry?

    partial objects have a readable repr, carry .func/.args/.keywords for inspection, and are picklable if the wrapped function is. A lambda is named <lambda> in tracebacks and is not picklable.

  3. When should you use a named function instead of a lambda?

    When the function body has more than one expression, when you need a useful name in tracebacks, or when the function is reused in more than one place.

Prove it

Create three partial transforms from scale_field and rename_field. Store them in a list called pipeline. Apply the pipeline to one row dict. Then print pipeline[0].func.__name__, pipeline[0].args, and the result of the transform. Paste the output.

py-02-06Card 6 of 6

Recognize recursion boundaries

EasyP4 · DifferentiatorTalk about it10 min
recursionrecursion-limititerativestacktail-call

Why you care

Python's default call stack is 1,000 frames deep. Recursive functions that process arbitrarily deep data structures — category trees, org hierarchies, file system trees, nested JSON — will crash with RecursionError on any input deeper than roughly 900 calls (leaving headroom for the interpreter itself). Unlike many languages, CPython does not perform tail-call optimisation: even a tail-recursive function consumes one frame per call. In a Jenkins job that processes a Postgres adjacency-list hierarchy, a data migration that creates a chain of 1,200 parent-child pairs is enough to crash the export. The fix is always available: every recursive algorithm has an iterative equivalent using an explicit stack. Interviewers at the conversational level ask about recursion limits to see whether you know Python's practical boundaries before deploying recursive logic against production data.

Explain like I'm 5

Recursion is like stacking plates: you add one plate every time you make a recursive call. Python allows you to stack up to 1,000 plates. When you try to add plate 1,001, the stack collapses and you get a RecursionError. An iterative approach uses a tray on the side (an explicit list) instead of stacking: you put nodes on the tray, take one off, process it, and put its children on the tray. The tray can hold as many items as your memory allows.

Core concept

Key facts about Python recursion:

  • Default limit: sys.getrecursionlimit() returns 1,000. You can raise it with sys.setrecursionlimit(n), but this does not increase the OS stack size. On most systems, setting it above ~10,000 risks a C-level stack overflow that bypasses Python's error handling.
  • No tail-call optimisation: CPython's bytecode interpreter does not detect or collapse tail calls. A tail-recursive factorial(n) consumes n frames, the same as a body-recursive one.
  • RecursionError is a subclass of Exception (since Python 3.5). You can catch it, but doing so usually masks the real problem.
  • Iterative equivalents: any recursive traversal can be rewritten using an explicit list as a stack (depth-first) or a collections.deque as a queue (breadth-first). The logic is identical; the depth limit is gone.

Use recursion when the call depth is genuinely bounded and small — parsing a fixed-depth config schema, traversing a known-shallow tree. Use an iterative stack when the data comes from user input, a database, or any source that can produce arbitrary depth.

Python 3.14 has no tail-call optimisation and the 3.14 release notes (S5) do not announce one. Treat the 1,000-frame limit as a permanent constraint, not a temporary one.

Minimal example RecursionError and the iterative equivalent

PythonDefault limit, RecursionError, and an iterative replacementruns
import sys

print(sys.getrecursionlimit())


def recursive_depth(n):
    if n == 0:
        return 0
    return 1 + recursive_depth(n - 1)


try:
    recursive_depth(2000)
except RecursionError as e:
    print("RecursionError:", e)


def iterative_depth(n):
    total = 0
    while n > 0:
        total += 1
        n -= 1
    return total


print(iterative_depth(2000))
Output
1000
RecursionError: maximum recursion depth exceeded
2000

In production Iterative org hierarchy flattener

The recursive version works for shallow trees. The iterative version works for any depth.

PythonRecursive and iterative tree flattening produce the same orderruns
ORG = {
    "name": "VP-Data", "reports": [
        {"name": "Lead-ETL", "reports": [
            {"name": "Dev-A", "reports": []},
            {"name": "Dev-B", "reports": []},
        ]},
        {"name": "Lead-ML", "reports": [
            {"name": "DS-C", "reports": []},
        ]},
    ]
}


def flatten_recursive(node, depth=0):
    yield (node["name"], depth)
    for child in node["reports"]:
        yield from flatten_recursive(child, depth + 1)


def flatten_iterative(root):
    stack = [(root, 0)]
    while stack:
        node, depth = stack.pop()
        yield (node["name"], depth)
        for child in reversed(node["reports"]):
            stack.append((child, depth + 1))


print("recursive:")
for name, depth in flatten_recursive(ORG):
    print("  " * depth + name)

print("iterative:")
for name, depth in flatten_iterative(ORG):
    print("  " * depth + name)
Output
recursive:
VP-Data
  Lead-ETL
    Dev-A
    Dev-B
  Lead-ML
    DS-C
iterative:
VP-Data
  Lead-ETL
    Dev-A
    Dev-B
  Lead-ML
    DS-C
Scenario A

In your world

The situation

A Jenkins job exports a category tree from Postgres. Categories are stored as an adjacency list: each row has id and parent_id. The recursive traversal works fine in staging, where categories are five levels deep at most.

What actually breaks

A data migration creates a promotional bundle hierarchy 1,200 nodes deep (unusual but valid for the business model). The export job crashes at 3 AM with RecursionError: maximum recursion depth exceeded. The default limit of 1,000 frames is hit before the deepest node is reached. The entire export fails; downstream services have stale data until the morning.

What you'd do on Monday

Convert the recursive traversal to an iterative stack-based traversal. The algorithm is identical — push children, pop one, process, push its children — but it uses a Python list as the stack instead of the call stack. There is no depth limit.

PythonIterative category export from Postgres adjacency list (needs psycopg)
def export_categories_iterative(conn):
    cur = conn.cursor()
    cur.execute("SELECT id, parent_id, name FROM categories")
    rows = {r[0]: {"id": r[0], "parent_id": r[1], "name": r[2], "children": []}
            for r in cur.fetchall()}

    roots = [r for r in rows.values() if r["parent_id"] is None]
    result = []
    stack = [(r, 0) for r in roots]

    while stack:
        node, depth = stack.pop()
        result.append({"id": node["id"], "name": node["name"], "depth": depth})
        children = [rows[r["id"]] for r in rows.values()
                    if r["parent_id"] == node["id"]]
        for child in reversed(children):
            stack.append((child, depth + 1))

    return result
Scenario B

Imagine you work at Juspay

1.2B payment transactions/year

The numbers

Illustrative: 1.2B ÷ 365 ≈ 3.3M transactions/day. Juspay processes routing graphs for payment orchestration — graphs where a payment request can fan out to multiple acquirers, each of which may have sub-routes. A routing graph with 50 nodes and average depth 8 is fine recursively; a misconfigured graph with depth 1,500 (a cycle or a degenerate chain) crashes the routing process.

What the Scenario-A answer does here

A sys.setrecursionlimit(5000) patch lets the code handle deeper graphs without fixing the root issue. At scale, a malicious or corrupt routing graph that exceeds even the raised limit brings down the payment routing worker. One worker crash equals payment failures for every transaction routed through it.

What flips at this scale

At this scale, the routing graph traversal uses an iterative DFS with explicit cycle detection (a visited set). Depth is bounded by node count, not by the Python call stack. The same worker process handles a graph of any depth without crashing.

The trade-off you'd defend

I would defend iterative traversal with cycle detection over raised recursion limits, trading slightly more code for a worker that cannot be crashed by input data.

Illustrative numbers; not Juspay-published figures.

Gotchas

sys.setrecursionlimit does not raise the OS stack size

Symptom

Setting sys.setrecursionlimit(50000) appears to work for moderate depths but causes a segmentation fault or silent process death on very deep calls.

Cause

The OS allocates a fixed stack size per thread (typically 1–8 MB). At very high limits, CPython runs out of OS stack space before Python's limit is hit, and the result is a C-level crash, not a Python exception.

Fix

Keep the limit at the default unless you have measured the OS stack size and confirmed headroom. Prefer iterative algorithms.

Catching RecursionError and retrying does not free stack frames

Symptom

You catch RecursionError, reduce the input size and retry recursively. The retry also crashes, even though the input is smaller.

Cause

Catching the error does not unwind the frames from the failed attempt. The stack is still near the limit when the retry starts.

Fix

Do not retry recursively after a RecursionError. Convert to iterative, or call sys.setrecursionlimit before the function runs.

Mutual recursion doubles the effective depth per "logical" call

Symptom

Two functions is_even(n) and is_odd(n) that call each other hit the limit at n = 500, not n = 1000.

Cause

Each logical step consumes two frames — one for is_even and one for is_odd.

Fix

Iterative equivalent: n % 2 == 0. In general, count actual frames, not logical steps.

Tricky questions

Does Python optimise tail calls?
Most people say

Yes, if the recursive call is the last statement in the function.

Actually

No. CPython does not optimise tail calls in any version, including 3.14. Guido van Rossum has explicitly said tail-call optimisation will not be added to CPython because it breaks stack traces. Every recursive call adds a frame regardless of position.

Is RecursionError catchable?
Most people say

No; it is a fatal error like SystemExit.

Actually

RecursionError is a subclass of Exception (since Python 3.5) and is catchable. Catching it is useful for graceful degradation — logging the depth and returning a partial result — but it does not fix the underlying algorithm. The stack is still near the limit when the handler runs, which limits what you can safely do inside the except block.

Scenario question

You're handed a function that parses nested JSON config files by recursing into each sub-object. The configs are generated by a tool that can theoretically nest to any depth. The function crashes on a test fixture with 200 levels of nesting. What is your approach?

Model answer

I'd convert to an iterative traversal using an explicit stack — a list of (node, path) tuples. Pop one, process it, push its children with their updated path. This handles any depth and requires no change to the logic, just to the control flow. I'd also add a maximum-depth check — if depth exceeds 500, log a warning and skip the subtree rather than crashing — so a corrupt config does not take down the whole parser. The trade-off is that the iterative version is more verbose than the recursive one; I'd add a comment explaining why recursion was not used.

Interview corner

CommonWhat is Python's default recursion limit?
“I'd say 1,000 frames by default, from sys.getrecursionlimit(). You can raise it with sys.setrecursionlimit(n), but that doesn't increase the OS stack — raising it past about 10,000 risks a C-level crash on most systems.”
TrickDoes Python optimise tail calls?
“I'd say no — CPython does not perform tail-call optimisation and has no plans to do so. Every recursive call adds a frame to the call stack regardless of where in the function body the call appears.”
Experience checkWhen have you switched from recursion to iteration in production code?
“I switched when processing a Postgres adjacency-list category hierarchy of unknown depth. In staging the tree was five levels deep; in production a data migration created a chain 1,200 nodes deep and the recursive traversal crashed at 3 AM. I rewrote it with an explicit stack — same DFS logic, no depth limit.”

Memory hook

Python gives you 1,000 plates. An explicit stack uses the heap instead.

If you remember only one thing:

Any recursive function that processes user-supplied or database-driven depth must be iterative.

Self-check

  1. What is Python's default recursion limit?

    1,000 frames, from sys.getrecursionlimit(). Set in CPython; may vary in other implementations.

  2. Does Python support tail-call optimisation?

    No. CPython does not optimise tail calls. Each recursive call adds a frame regardless of call position.

  3. How do you convert a recursive DFS to iterative?

    Use a list as an explicit stack. Push the root node, then loop: pop one node, process it, push its children (in reversed order to maintain left-to-right traversal order). No depth limit.

Prove it

Write a recursive factorial(n) and confirm it raises RecursionError at n=1500. Then write an iterative version. Confirm the iterative version returns a result for n=1500 (you only need to check isinstance(result, int) and result > 0). Print the recursion limit, the error message for the recursive attempt, and "iterative ok" for the iterative one. Paste all three outputs.