Shelf · Python · Chapter 1

Chapter 1: The data model & types

Every assignment in Python creates a reference, not a copy. This chapter makes that visible — and shows how identity, mutation, hashing and copying interact to produce the silent, build-green, data-corrupting bugs that trip up Python developers most often.

Difficulty
Priority
py-01-01Card 1 of 6

Names refer to objects

EasyP1 · Must knowKnow cold10 min
bindingreferencesobjectsidentityrebinding
Needs: nothing. Start here.

Why you care

Every "the job succeeded but the target table is empty" bug in Python ETL eventually traces to one mechanic: a variable name is not a box that holds a value, it is a label stuck to an object in memory. When you write rows = data you attach a second label to the same object, not copy the data. If you cannot tell the difference between rebinding a name and mutating an object, you will corrupt shared state, get no error, and spend your evening re-running a job that succeeded. Interviewers ask this on first-round screens because it filters out engineers who write defensive copy() calls everywhere from those who understand when a copy is actually needed.

Explain like I'm 5

Think of Python objects as balloons and variable names as strings tied to them. a = [1, 2, 3] inflates a balloon (the list) and ties the string a to it. b = a ties a second string, b, to the same balloon. Poking the balloon — appending to the list — is visible through both strings. Cutting string a and tying it to a new balloon (a = [99]) does not pop the first balloon; b still floats there, holding the original list.

Core concept

Every object in CPython has an identity (its memory address, returned by id()), a type, and a value. A name is a binding from a string key to an object reference inside the current namespace dictionary. Assignment evaluates the right side, finds or creates the object, and binds the name. No data moves.

  • a = expr creates the object (if new) and binds the name a to it.
  • b = a binds b to the same object. Two names, one object. a is b returns True.
  • Mutating the object via either name (a.append(x), b["k"] = v) changes the single underlying object. The change is immediately visible through every name bound to it.
  • Rebinding a name (a = something_else) only moves the a label; every other name is unaffected and still points at the original object.

The practical test: after b = a, ask a is b. If True, both names share one object. Appending to b also appends to a. The decision rule: use a second name for read-only access to a shared object; reach for an explicit copy (Choose shallow or deep copy) only when you need an independent value. Always document whether a function that receives a mutable container is allowed to mutate it, or must leave it intact.

Minimal example One list, two names, one mutation

PythonLabels and mutation are separate actionsruns
a = [10, 20, 30]
b = a                   # b is a second label on the same list
b.append(40)
print(a)                # mutation via b is visible through a
print(a is b)           # same object in memory

a = [99]                # rebind a to a new list; b is unaffected
print(b)                # b still holds the original list
print(a is b)           # now different objects
Output
[10, 20, 30, 40]
True
[10, 20, 30, 40]
False

In production Trade loader that tags each row in-place

A common ETL pattern adds audit fields to each dict as it is inserted. Any code that held a reference to the original list before loading will see those tags appear — useful when intentional, a surprise when not.

PythonETL loader tagging rows in-place (sqlite3 stands in for Postgres)runs
import csv, sqlite3
from pathlib import Path

Path("trades.csv").write_text("id,symbol,qty\n1,INFY,100\n2,TCS,50\n3,HDFC,200\n")

con = sqlite3.connect(":memory:")
con.execute("CREATE TABLE trades (id INT, symbol TEXT, qty INT)")

with open("trades.csv", newline="") as f:
    rows = list(csv.DictReader(f))

original_first = rows[0]   # hold a reference to the first dict before loading

def load_row(con, row):
    con.execute("INSERT INTO trades VALUES (?,?,?)",
                (int(row["id"]), row["symbol"], int(row["qty"])))
    row["_loaded"] = True   # mutates the shared dict object in-place

for row in rows:
    load_row(con, row)

# original_first is rows[0] — same dict, so the tag is visible
print(original_first.get("_loaded"))
count = con.execute("SELECT COUNT(*) FROM trades").fetchone()[0]
print("rows in db:", count)
print(original_first is rows[0])
Output
True
rows in db: 3
True
Scenario A

In your world

The situation

Your Jenkins job reads settlements.csv from S3 into a list of dicts, passes the list to a validate() helper, and then calls load() on the same variable. The validate() function removes bad rows using list.remove() in a loop.

What actually breaks

Rows that fail validation are silently deleted from settlements before load() sees them. The audit table says 1,247 rows read; Postgres gets 1,201. Jenkins is green. Finance's overnight reconciliation fails three days later with a 46-row discrepancy.

What you'd do on Monday

The root cause is in-place deletion on a shared list. Either make validate() return a new list and rebind before passing it to load(), or assert count consistency between stages:

PythonAssert row counts between pipeline stages (requires S3 and Postgres access)
raw      = read_from_s3(key)
valid    = [r for r in raw if is_valid(r)]      # new list, raw is untouched
rejected = len(raw) - len(valid)
log.info("validated: %d accepted, %d rejected", len(valid), rejected)
load(conn, valid)
assert_loaded_count(conn, expected=len(valid))
Scenario B

Imagine you work at PhonePe

200 M wallet transactions per day

The numbers

Illustrative: 200 M transactions ÷ 86,400 s ≈ 2,300 txn/s average, with intraday peaks 5–8× that. A batch ETL holding 1 M dicts — each ~500 bytes in CPython with a small key set — uses roughly 500 MB of RAM per batch just for Python objects.

What the Scenario-A answer does here

Passing the same batch list through multiple pipeline stages and letting each stage mutate it in-place works at 10 k rows. At 1 M rows the first mutating stage corrupts data that later stages expect intact, and the failures are non-deterministic depending on stage execution order.

What flips at this scale

At this scale each pipeline stage receives an immutable view — a tuple of frozen dataclasses, or dicts that convention (or a type checker) forbids mutating — and produces a new collection. The no-mutation contract is enforced by type, not by code review. Any state that must persist across stages is written to a checkpoint store, not threaded through a shared Python object.

The trade-off you'd defend

I would defend immutable pipeline objects over in-place mutation, trading one extra allocation per stage for a bug class that disappears entirely. At 2,300 txn/s the allocation cost is negligible; the debugging cost of shared-state corruption at scale is not.

Illustrative numbers scaled to make the arithmetic visible; not PhonePe-published figures.

Gotchas

Reassigning a parameter inside a function does not change the caller's variable

Symptom

After calling def reset(lst): lst = [], the caller's list is unchanged.

Cause

lst = [] rebinds the local name lst to a new empty list. The caller's name still points at the original object.

Fix

Either return the new value and rebind in the caller (rows = reset(rows)), or mutate in place (lst.clear()) and document which contract the function follows.

Appending inside a loop that uses two names for the same list doubles entries

Symptom

The list grows twice as fast as expected, or an assert len(a) == len(b) passes unexpectedly.

Cause

Both names point at the same list. Appending through either name affects the shared object.

Fix

Confirm with a is b. Use list(a) or copy.copy(a) to make a separate object (Choose shallow or deep copy).

Tricky questions

Does a = b copy the value?
Most people say

Yes — after a = b, changing b won't affect a.

Actually

a = b makes both names point at the same object. Mutating the object through b (e.g., b.append(1)) is immediately visible through a. Only rebinding b to a different object leaves a pointing at the original.

Does passing a list to a function let the function resize it?
Most people say

No — functions receive copies.

Actually

Functions receive a reference to the same object. If the function calls lst.append() or lst.remove(), the caller's list is changed. Only rebinding the parameter name (lst = []) leaves the caller's variable unaffected.

Scenario question

You're handed an ETL helper def clean(rows) that trims whitespace from string fields and returns nothing. In tests with 10 dicts it works. In production you occasionally see untrimmed values reaching the loader. The function is called before the load step.

Model answer

I'd check whether a second code path appends new rows to the same batch list after clean() runs — those rows never go through the cleaner. I'd also verify that clean() mutates each dict in-place and is not accidentally creating new dicts. The fix is to ensure all rows are assembled before the clean step, and add an assertion inside the loader that rejects any row with leading or trailing whitespace. The trade-off of asserting inside the loader is that it couples two concerns, but it catches the bug even if the call order changes again.

Interview corner

CommonWhat is the difference between rebinding a name and mutating an object?
“I'd say rebinding changes what a variable points at; mutation changes the object itself. After b = a, rebinding a = [] leaves b on the original list. But a.append(1) adds to the same list both names share.”
TrickYou pass a list to a function. The function does lst = []. Does the caller's list change?
“I'd say no — that line rebinds the local name lst to a new empty list; the caller's variable still points at the original. If the intent was to empty the caller's list, you call lst.clear() or del lst[:], which mutates the object rather than rebinding the name.”
Experience checkHas this caused a real bug for you?
“Yes. We had a validate step that removed bad rows in-place from the same list we later passed to load. The load saw a shorter list with no indication why. We fixed it by making validate return a new list and asserting counts at each stage.”
DesignHow do you enforce that a pipeline stage does not mutate its input?
“I'd type the parameter as a tuple or a frozen dataclass and let the type checker enforce it. For untyped code, I'd have the stage receive a shallow copy at the call site. The trade-off is an extra allocation; for large batches I'd design the pipeline so no stage needs to mutate its input at all.”

Memory hook

A variable name is a label tied to a balloon. b = a ties two strings to the same balloon; poking the balloon through either string pops it for everyone.

If you remember only one thing:

Assignment never copies. If you need independence, copy explicitly.

Self-check

  1. After b = a; a = [], what does b contain?

    The original list. Rebinding a only moves the a label; b still points at the original object.

  2. Name two ways to get an independent shallow copy of a flat list.

    list(a) or a[:]. For nested data use copy.deepcopy(a).

  3. What does id() return in CPython?

    The memory address of the object. Two names with the same id() share one object.

  4. What is the difference between lst.clear() and lst = []?

    lst.clear() empties the original object in-place; every name bound to it sees an empty list. lst = [] rebinds only lst; other names retain the original contents.

Prove it

In a script: create a = [[1], [2], [3]]. Bind b = a. Append 99 to a[0]. Print b[0] and b[0] is a[0]. Then set a = [[10], [20], [30]] and print b[0] again and a is b. Paste both outputs, then in one sentence explain why b[0] changed in the first step but b kept the original list after the second step.

py-01-02Card 2 of 6

Equality is not identity

EasyP1 · Must knowKnow cold10 min
identityequalityis-vs-eqnonesentinel

Why you care

Half the "it worked in the REPL but fails in production" None-handling bugs come from one confusion: == tests value equality; is tests object identity. For ordinary objects the difference rarely matters. For None, True and False it matters every time, because any class can override __eq__ to return something truthy when compared to None — SQLAlchemy column expressions and Pandas Series do exactly this. Using == None where you meant is None is silent on small test fixtures and loudly wrong in production. Interviewers ask this to distinguish developers who have memorised the rule from those who have been bitten by it.

Explain like I'm 5

Imagine two identical photocopies of a contract. They are equal in content (== is True) but they are physically different sheets of paper (is is False). Now imagine a single original contract. Any two people holding it up are holding the same sheet: == and is are both True. Python guarantees that None, True and False each exist as exactly one object in the process. That is why x is None will never lie to you, even if some third-party object has a broken __eq__.

Core concept

== calls the left operand's __eq__ method. Any class can override it to return whatever it likes. is skips __eq__ entirely and compares raw memory addresses: it is True if and only if both names refer to the same object.

  • Use is for None: if x is None: and if x is not None:. Never == None.
  • Use is for True and False only when you mean exactly that boolean, not any truthy value. Most conditional checks belong in a plain if x: or if not x:.
  • Use == for all value comparisons: numbers, strings, lists, custom objects.
  • Sentinels: when you need a value that means "no argument supplied" (different from None, which a caller might legitimately pass), create a module-level MISSING = object() and compare with if arg is MISSING:. A fresh object() is guaranteed unique; no other object will ever be is MISSING.

The decision rule: reach for is only with singletons (None, True, False) and explicit sentinel objects. Use == for everything else. Swapping them silently corrupts data when the object being compared has a non-trivial __eq__.

Minimal example Identity versus equality on lists and None

Pythonis checks the address; == checks the valueruns
a = [1, 2, 3]
b = [1, 2, 3]       # same value, different object
print(a == b)        # True — same content
print(a is b)        # False — different objects in memory

result = None
print(result is None)    # correct sentinel check
print(result == None)    # also True here, but unsafe with custom __eq__

MISSING = object()       # unique sentinel
value = MISSING
print(value is MISSING)  # True
print(value is None)     # False — MISSING and None are different objects
Output
True
False
True
True
True
False

In production Distinguishing NULL from empty string in a settlement table

A settlements table uses SQL NULL for rows not yet reconciled and "" (empty string) for rows reconciled with no reference number. Treating them identically loses both pieces of information.

PythonNULL vs empty string in a settlement query (sqlite3 stands in for Postgres)runs
import sqlite3

con = sqlite3.connect(":memory:")
con.execute("CREATE TABLE settlements (id INT, ref TEXT)")
con.executemany("INSERT INTO settlements VALUES (?,?)",
                [(1, "S100"), (2, None), (3, ""), (4, "S200")])

rows = list(con.execute("SELECT id, ref FROM settlements"))

# Wrong: treats NULL and empty string alike (both are falsy)
bad = [r[0] for r in rows if not r[1]]

# Right: distinguish NULL (not yet reconciled) from empty (reconciled, no ref)
not_null  = [r[0] for r in rows if r[1] is not None]
non_empty = [r[0] for r in rows if r[1] is not None and r[1] != ""]

print("falsy filter:", bad)
print("not null:    ", not_null)
print("non-empty ref:", non_empty)
Output
falsy filter: [2, 3]
not null:     [1, 3, 4]
non-empty ref: [1, 4]
Scenario A

In your world

The situation

Your ETL reads a column settlement_ref from a Postgres view. The view returns NULL for pending settlements and an empty string for settled-with-no-ref records. Your code filters with if row["settlement_ref"]: before writing to the audit table.

What actually breaks

Both NULL and empty-string rows are silently skipped. The audit table shows 8,412 records loaded. Finance says 11,304 settlements were processed. The discrepancy is exactly the pending + no-ref count, but the job never errored.

What you'd do on Monday

Replace the truthiness check with an explicit is not None check, and handle the empty-string case separately:

PythonExplicit None check in the settlement filter (requires real Postgres + psycopg)
for row in cur.fetchall():
    if row["settlement_ref"] is None:
        continue          # genuinely pending — skip for now
    ref = row["settlement_ref"] or "NO_REF"   # normalise empty string
    write_audit(row["id"], ref)
Scenario B

Imagine you work at Zomato

8 M food orders per day

The numbers

Illustrative: 8 M orders ÷ 86,400 s ≈ 93 orders/s, with dinnertime bursts to ~500/s. An order pipeline that tracks status transitions stores eta, delivered_at and cancelled_at as nullable timestamps. With three nullable fields per order, there are 24 M None-vs-value comparisons per day in a naive implementation that re-checks each field on every pipeline step.

What the Scenario-A answer does here

Using == None throughout works fine with bare Python dicts. It breaks when the same helper starts receiving pandas columns or NumPy arrays from a new analytics step: col == None returns an element-wise mask instead of a bool, and if col == None: raises ValueError: The truth value of a Series is ambiguous mid-run.

What flips at this scale

At this scale the team standardises on a typed layer (dataclasses or Pydantic models) that represents "not yet set" with Optional[datetime] and enforces is None checks through a linter rule. No magic truthiness: every nullable field has an explicit if field is None branch.

The trade-off you'd defend

I would defend is None over == None unconditionally, trading zero performance and zero expressiveness for immunity to the class of bugs that arise when third-party objects override __eq__. The linter rule costs one CI check; the debugging time saved is unbounded.

Illustrative numbers scaled to make the arithmetic visible; not Zomato-published figures.

Gotchas

SQLAlchemy column expressions and Pandas Series override __eq__

Symptom

if column == None: raises ValueError: The truth value of a Series is ambiguous, or silently returns a boolean mask instead of True/False.

Cause

Both libraries override __eq__ to return element-wise comparison objects, not a simple boolean.

Fix

Always use is None for Python-level None checks and .isna() for pandas. Inside a SQLAlchemy .filter() the rule flips: Model.col is None is plain False and silently drops the condition, so write Model.col.is_(None) (or == None, which SQLAlchemy turns into IS NULL).

not x is not the same as x is None

Symptom

A column containing 0 or "" is treated as missing, skipping valid rows.

Cause

not x tests truthiness, not presence. 0, "", [] and {} are all falsy but are not None.

Fix

Use x is None to mean "this field was absent". Use not x only when you explicitly mean "this field is falsy, whatever its type".

Tricky questions

Is 0 == False True?
Most people say

No — they are different types.

Actually

Yes. bool is a subclass of int in Python; False == 0 and True == 1. Use x is False if you need to distinguish the boolean from the integer.

Can x == None ever raise an exception?
Most people say

No — it just returns False if x is not None.

Actually

If x is a pandas Series or a NumPy array, x == None returns an element-wise array, and putting that in an if raises ValueError because the truth value is ambiguous. x is None never calls __eq__ and always gives a plain bool.

Scenario question

You're handed a function def is_settled(row) that returns True for settled rows. It uses return row["ref"] != None. A colleague flags it in review. What's wrong and what exactly do you change?

Model answer

!= None calls row["ref"].__ne__(None) (or __eq__ via negation). For a bare Python string that is fine, but if row["ref"] ever becomes a pandas or NumPy value (a row from df.itertuples() holding NaN, or a whole column), != either returns an array or treats NaN as "not None", so missing refs count as settled. I'd change it to return row["ref"] is not None. It is not just style: is not None is unambiguous about intent, costs nothing, and is immune to __eq__ overrides. The trade-off is purely one of explicitness versus relying on default behaviour.

Interview corner

CommonWhen should you use is versus ==?
“I use is only for singletons — None, True, False — and for explicit sentinel objects. Everything else gets ==. The reason is that is bypasses __eq__ entirely, so it never lies, but it only makes semantic sense when I mean 'literally the same object', not 'the same value'.”
TrickCan x == None ever fail even if x is a normal value?
“I'd say yes — if x is a Pandas Series or a SQLAlchemy column expression. Both override __eq__ to return something that is not a boolean, so if x == None: raises ValueError or evaluates the wrong thing. x is None never calls __eq__, so it always returns a plain bool.”
Experience checkWhere has the ==-versus-is distinction caused a real problem for you?
“I had a validation helper with if value == None: that worked for months on plain dict rows, then crashed with 'truth value of a Series is ambiguous' when someone passed it a pandas column. I switched to is None for scalars and .isna() for columns. I also watch for the reverse in SQLAlchemy filters, where is None silently becomes False.”
CommonWhat is a sentinel and why would you use one?
“A sentinel is a module-level object() that means 'this argument was not supplied'. I use it when None is a valid caller-supplied value. I compare with if arg is MISSING: and it is reliable because a fresh object() is guaranteed to be unique — no other object in the process will ever be identical to it.”

Memory hook

is None checks the address; == None asks the object's opinion. Some objects lie.

If you remember only one thing:

Use is for None, True, False, and explicit sentinels. Use == for everything else.

Self-check

  1. Why is x is None safer than x == None?

    is compares memory addresses and cannot be overridden; == calls __eq__, which third-party objects (SQLAlchemy, Pandas) override to return non-boolean results.

  2. What is a sentinel object and how do you create one?

    MISSING = object() at module level. Any value compared with is MISSING will return False except the sentinel itself, because object() creates a unique instance every time.

  3. What does bool inherit from in Python?

    int. False == 0 and True == 1 are both True.

  4. Can two objects that are not is-equal still be ==-equal?

    Yes. [1, 2] == [1, 2] is True; [1, 2] is [1, 2] is False (two different list objects with the same content).

Prove it

Write class Loud: def __eq__(self, other): print(f"__eq__ called with {other!r}"); return True. Create x = Loud(). Run x == None and x is None and paste both outputs. Then try x is None with a plain None value and confirm it returns False with no __eq__ call.

py-01-03Card 3 of 6

Trace mutation through aliases

MediumP1 · Must knowKnow cold10 min
mutationaliasingnestedshared-referencein-place

Why you care

Aliasing a flat list is surprising once. Aliasing a nested container — a list of dicts, a dict of lists — is the bug that survives code review because each individual assignment looks harmless. Every ETL pipeline that builds a batch as a list of dicts and passes it through multiple transforms is one accidental in-place mutation away from corrupting every row downstream. No exception is raised; the data is just wrong. Interviewers ask mutation-through-aliases questions because the answer shows whether you reason about memory topology or just run code and see what happens.

Explain like I'm 5

Imagine a filing cabinet (the outer list) full of folders (the dicts). You hand a colleague the same cabinet. When your colleague opens folder number 1 and crosses out a field, you both see the crossed-out field the next time you open that folder — because you were sharing one physical cabinet with one physical folder, not each holding a photocopy.

Core concept

Python containers — lists, dicts, sets — store references to their elements, not copies of them. When you assign snapshot = raw, both names share the list object. The list object itself holds references to the dict objects inside it. Any code that modifies a nested dict modifies the original object regardless of which outer name was used to reach it.

  • snapshot = raw → one list, two names. snapshot[0] and raw[0] are is-equal.
  • Deleting a key from snapshot[0] deletes it from raw[0]. There is one dict.
  • Appending to snapshot appends to raw. There is one list.
  • Reassigning snapshot = [] only rebinds the name; raw is untouched.
  • Nested dicts compound the effect: data["cfg"]["timeout"] = 60 propagates to every name that holds a reference to data["cfg"].

The decision rule: when a function receives a container it should not modify, either document the contract ("caller must not call with shared objects") or take a copy at the function boundary. For flat containers copy.copy() is enough; for nested containers you need copy.deepcopy() — covered in Choose shallow or deep copy.

Minimal example Nested mutation propagates through aliases

PythonA shared nested structure changes for all names at onceruns
config = {"timeout": 30, "retries": 3}
worker_cfg = config
worker_cfg["timeout"] = 60       # mutates the shared dict
print(config["timeout"])         # 60 — surprising if you expected a copy

batches = [[1, 2], [3, 4]]
first = batches[0]               # first is batches[0] — same inner list
first.append(99)
print(batches)                   # [[1, 2, 99], [3, 4]]
Output
60
[[1, 2, 99], [3, 4]]

In production Row normaliser that mutates records in-place

A normaliser that removes empty-string fields from each row dict mutates the dicts it receives. A caller holding a reference to the original list will see those deletions as if they happened to its own data.

PythonNormalise rows in-place — mutation is visible through every alias (stdlib only)runs
import csv
from pathlib import Path

Path("feed.csv").write_text("id,symbol,note\n1,INFY,\n2,TCS,block\n3,HDFC,\n")

def normalize(records):
    """Remove empty-string values from each record dict in-place."""
    for rec in records:
        for key in list(rec.keys()):
            if rec[key] == "":
                del rec[key]

with open("feed.csv", newline="") as f:
    raw = list(csv.DictReader(f))

snapshot = raw           # same list, same inner dicts
normalize(raw)

print(snapshot[0])       # note key is gone — mutation visible through alias
print(raw is snapshot)   # True
print(len(snapshot[0]))  # 2 keys remain
Output
{'id': '1', 'symbol': 'INFY'}
True
2

See it happen How two names share one nested structure

rawsnapshotlist [ref0, ref1]dict 0dict 1
  1. raw and snapshot are two names; both hold a reference to the same list object in memory.
  2. The list holds references to two dict objects; no data is copied into the list itself.
  3. normalize(raw) deletes the empty note key from dict 0 in-place.
  4. snapshot[0] is dict 0 — the same object — so snapshot sees the mutation through its own reference chain.
Notice: the list object never moves. Only the inner dict changes. Every name that can reach that dict — through any outer list, under any variable name — sees the change.
Scenario A

In your world

The situation

Your nightly ETL reads vendor_catalogue.csv into a list of dicts. A clean() helper normalises string fields. A separate enrich() helper then adds computed columns. Both helpers mutate dicts in-place. A QA team member added snapshot = catalogue before clean() for debugging.

What actually breaks

After enrich() runs, snapshot also has all the enrichment columns and the cleaned strings — because snapshot is the same list with the same dicts. The QA member's "before" snapshot is the same as the "after". The diff always shows no changes, even when something went wrong.

What you'd do on Monday

Either take an explicit deep copy for the snapshot (import copy; snapshot = copy.deepcopy(catalogue)), or make clean() and enrich() return new lists instead of mutating in-place. The Monday fix is one line:

PythonTake a real snapshot before mutations (no network or DB needed; stdlib only)
import copy
snapshot = copy.deepcopy(catalogue)   # independent copy; mutations to catalogue do not reach snapshot
clean(catalogue)
enrich(catalogue)
diff = [r for r in catalogue if r not in snapshot]
Scenario B

Imagine you work at CRED

15 M member accounts

The numbers

Illustrative: 15 M members each with a profile dict containing 20 fields ≈ 300 M field references. A nightly enrichment job that passes the member list through 6 transform stages, each mutating in-place, has 6 × 300 M = 1.8 B field-access-with-mutation operations on shared objects.

What the Scenario-A answer does here

At 15 M rows passing the same list through 6 stages looks efficient — no copies, constant memory. The hidden cost is that any stage bug corrupts every downstream stage silently, and a rollback or replay requires re-reading all 15 M rows from storage because the in-memory state is untrusted.

What flips at this scale

At this scale the team freezes the pipeline contract: each stage takes a list of immutable objects (frozen dataclasses or tuples) and returns a new list. The total memory cost is one allocation per stage per row — at 300 bytes/object × 15 M rows × 6 stages ≈ 27 GB peak RAM, which is traded for reliable re-playability and independently testable stages. In practice, streaming per-row transforms replace the full-batch model and keep peak memory near one batch size.

The trade-off you'd defend

I would defend immutable per-stage objects over in-place mutation, trading memory for correctness guarantees. For a job that runs nightly on a managed cluster, the memory is cheap; the debugging time for a shared-mutation bug at 15 M rows is not.

Illustrative numbers scaled to make the arithmetic visible; not CRED-published figures.

Gotchas

A function that "copies" a list but returns dicts from the original

Symptom

Modifying a key in a returned dict also changes the original list's dicts.

Cause

new_list = list(old_list) copies the outer list but not the dict objects inside it. The dicts are shared.

Fix

Use [dict(r) for r in old_list] for a shallow dict copy, or copy.deepcopy(old_list) for full isolation.

+= on a list mutates in-place; + creates a new list

Symptom

Two names both grow when only one was extended.

Cause

a += [4] calls list.__iadd__, which extends the original object. a = a + [4] creates a new list and rebinds only a.

Fix

Use a = a + [4] when you need the original unchanged. Use += only when you explicitly mean to grow the shared object.

Sorting a list that another variable depends on

Symptom

A sorted copy of a list changes the original order.

Cause

a.sort() sorts in-place. b = a; b.sort() sorts the shared list.

Fix

Use sorted(a) to get a new list; a.sort() to sort in-place with intent.

Tricky questions

Does new = list(old) protect against mutation?
Most people say

Yes — it creates a copy.

Actually

list(old) creates a new outer list but reuses the same inner objects. Mutating new[0]["k"] = v also mutates old[0]["k"] because new[0] is old[0].

Is dict(d) a deep copy?
Most people say

Yes — it creates a new dict with new values.

Actually

dict(d) creates a new dict with new keys, but the values are the same objects. If a value is itself a list or dict, mutations to it propagate back to the original.

Scenario question

You're handed a merge(base, overrides) function that updates base with values from overrides and returns base. A caller stores the result and later finds that the original base dict was modified. What is happening and how do you fix merge without changing callers?

Model answer

merge is returning the same dict it received after mutating it. Every caller that stored base before calling merge now sees the merged values in their variable. I'd change merge to create a new dict: result = {**base, **overrides}; return result. This leaves base intact and gives each caller an independent merged dict. The trade-off is one extra allocation per call; at typical config-dict sizes that is negligible.

Interview corner

CommonWhat is the difference between shallow and deep mutation?
“I'd describe it this way: shallow mutation means changing the container itself — appending, deleting a key. Deep mutation means reaching into a nested object and changing it. With shallow copies, deep mutations still propagate because the inner objects are shared. That is why list(rows) is not enough when rows is a list of dicts.”
Experience checkHas in-place mutation on shared data caused a real bug for you?
“I've been caught by this. A QA snapshot that was supposed to capture the state before a transform was set with snapshot = batch instead of deepcopy. Every 'diff' between snapshot and the final result showed nothing, which masked a bug in our enrichment step for months.”
Tricka += [1] versus a = a + [1] — which one is safe to use when another variable also points at the same list?
“Neither is 'safe' universally. a += [1] mutates the shared list, so every name bound to it sees the new element. a = a + [1] creates a new list and rebinds a, so the other names keep the original. I'd use += only when every owner of the list knows it can grow.”
DesignHow do you design a multi-stage pipeline that avoids shared-mutation bugs?
“I'd define each stage as a pure function: it receives an immutable input and returns a new output. Frozen dataclasses or NamedTuples enforce this at the type level. For high-throughput pipelines I'd use generators that transform one row at a time, which sidesteps the need to hold the full batch in memory at all.”

Memory hook

A list of dicts is a cabinet of folders. Handing someone the cabinet does not make them a photocopy; they reach into the same folders.

If you remember only one thing:

Containers store references. Mutating a nested object changes it for every name that can reach it.

Self-check

  1. Why does new = list(old) not protect against mutations to inner dicts?

    Because list(old) copies only the outer list object. The inner dict objects are the same as in old; new[0] is old[0] is True.

  2. Name one built-in that sorts a list in-place and one that returns a new list.

    list.sort() mutates in-place. sorted() returns a new list.

  3. What is the simplest way to make a fully independent copy of a nested structure?

    copy.deepcopy() from the copy module.

  4. What does a += [x] do that a = a + [x] does not?

    += calls list.__iadd__, which extends the original object and returns it, so all aliases see the new element. a = a + [x] creates a new list and rebinds only a.

Prove it

Write three assignments: a = {"x": [1, 2]}, b = dict(a), c = __import__("copy").deepcopy(a). Append 99 to a["x"]. Print b["x"] and c["x"]. Paste the output and explain why b["x"] changed but c["x"] did not.

py-01-04Card 4 of 6

Design safe dictionary keys

MediumP2 · Very likelyKnow cold11 min
hashingdictionarykeyshash-protocoleq-protocolhashable

Why you care

A TypeError: unhashable type at runtime is the polite failure. The dangerous failure is a custom object that is accepted as a dict key but gives wrong lookup results because __hash__ and __eq__ are inconsistent. Either way, the job fails or silently miscounts. In ETL pipelines that use dicts as in-memory caches — deduplicate by (account_id, trade_date), group by (symbol, currency) — a bad key design means cache misses on every lookup for objects that look equal but hash differently. Interviewers ask about hashability and the hash/equality contract to see whether you know what makes a key safe, not just whether you have seen the TypeError.

Explain like I'm 5

A dictionary is a filing room with numbered drawers. When you store something, Python calls hash(key) to decide which drawer number to use, then puts the item there. When you look something up, Python hashes the key again, goes to that drawer, and checks equality (==) on everything inside. The contract is: two things that are equal must always hash to the same drawer number. If you violate that — equal objects hash differently — Python goes to the wrong drawer and says "not found", even though the item is there.

Core concept

The hash/equality contract has two rules enforced by Python's data model:

  1. If a == b then hash(a) == hash(b). (The converse is not required; hash collisions are allowed.)
  2. An object's hash must not change over its lifetime. That is why mutable containers — list, dict, set — are not hashable by default.

In Python 3, defining __eq__ without __hash__ automatically sets __hash__ = None, making the class unhashable. This is intentional: if two objects compare equal, they must produce the same hash, and CPython cannot guarantee that without your involvement.

  • Safe keys: immutable built-ins (int, str, float, bool, bytes, frozenset), and tuples of hashable objects. A tuple is only hashable if all its elements are hashable.
  • Unsafe keys: list, dict, set, and any custom class that defines __eq__ without __hash__.
  • Custom hashable class: define both __hash__ and __eq__, ensure they are consistent, and make the object immutable (or at least stable on the hashed fields).

Decision rule: use a tuple of immutable scalars as a compound key ((user_id, date_str) rather than {"user": user_id, "date": date_str}). If you need a custom object as a key, use @dataclass(frozen=True), which generates consistent __eq__ and __hash__ from all fields and blocks attribute reassignment.

Under the hood

CPython's dict lookup is open-addressing hash table. hash(key) calls the object's tp_hash slot (or __hash__ in Python). The hash is used to find a slot; then == (the tp_richcompare slot with Py_EQ) is called to confirm the match. If __eq__ is defined in a class but __hash__ is not, Python 3 sets tp_hash to PyObject_HashNotImplemented, which raises TypeError the moment you try to hash the instance. This is why you cannot silence the error by ignoring __hash__: the slot itself becomes a no-op that always raises. To restore default identity-based hashing when __eq__ IS overridden (and you deliberately want identity-based hash), set __hash__ = object.__hash__ explicitly.

Minimal example Hashable and unhashable keys

PythonHashable tuples, unhashable lists, and a class that breaks the contractruns
d = {}
coord = (19.0760, 72.8777)       # tuple of floats: hashable
d[coord] = "Mumbai"
print(d[coord])

try:
    d[[1, 2]] = "bad"            # list: not hashable
except TypeError as e:
    print(f"TypeError: {e}")

class OrderId:
    def __init__(self, val):
        self.val = val
    def __eq__(self, other):
        return isinstance(other, OrderId) and self.val == other.val
    # __hash__ is now None — Python 3 sets it automatically

try:
    d[OrderId("O1")] = "order"
except TypeError as e:
    print(f"TypeError: {e}")
Output
Mumbai
TypeError: unhashable type: 'list'
TypeError: unhashable type: 'OrderId'

In production Portfolio cache keyed by (user_id, symbol)

A position cache avoids repeated Postgres lookups during intraday reconciliation. The key is a two-element tuple of ints and strings — both hashable — so lookups are O(1).

PythonPosition cache with tuple keys (sqlite3 stands in for Postgres)runs
import sqlite3

con = sqlite3.connect(":memory:")
con.execute("CREATE TABLE positions (user_id INT, symbol TEXT, qty INT)")
con.executemany("INSERT INTO positions VALUES (?,?,?)",
                [(101, "INFY", 50), (101, "TCS", 30), (102, "INFY", 100)])

cache = {}
for uid, sym, qty in con.execute("SELECT user_id, symbol, qty FROM positions"):
    cache[(uid, sym)] = qty      # (int, str) tuple: safe compound key

print(cache[(101, "INFY")])      # 50
print(cache.get((102, "TCS")))   # None — not in DB

# A tuple containing a list is NOT hashable
try:
    bad = ([101], "INFY")
    cache[bad] = 50
except TypeError as e:
    print(f"TypeError: {e}")
Output
50
None
TypeError: unhashable type: 'list'
Scenario A

In your world

The situation

Your deduplication step caches processed (account_id, trade_date) pairs in a set. A colleague refactors the pipeline to use a Trade dataclass. The new dataclass defines __eq__ to compare field-by-field, but does not define __hash__.

What actually breaks

Adding Trade(101, "2025-01-15") to the set raises TypeError: unhashable type: 'Trade'. The job crashes. If the colleague added __hash__ = None by accident, the crash is immediate and obvious. If they relied on the default (which Python 3 removes when __eq__ is defined), the crash appears only the first time a Trade is used as a key, which might be in a rarely-hit code path.

What you'd do on Monday

Add __hash__ consistently with __eq__, or use @dataclass(frozen=True), which generates both a consistent __hash__ and makes the fields immutable:

PythonSafe hashable Trade dataclass (stdlib only; no network needed)
from dataclasses import dataclass

@dataclass(frozen=True)
class Trade:
    account_id: int
    trade_date: str        # use a string; datetime is also hashable

seen: set[Trade] = set()
seen.add(Trade(101, "2025-01-15"))
print(Trade(101, "2025-01-15") in seen)   # True — consistent hash and equality
Scenario B

Imagine you work at Groww

10 M active stock portfolios

The numbers

Illustrative: 10 M portfolios × average 12 holdings each = 120 M (user_id, symbol) position pairs. A nightly reconciliation that builds an in-memory cache of all positions holds 120 M tuple keys; at ~56 bytes per (int, str) tuple plus dict overhead ≈ 8–12 GB of RAM for the cache alone.

What the Scenario-A answer does here

Using frozen dataclasses as keys at 120 M entries works correctly. The memory cost for a full-reconciliation in-memory cache becomes prohibitive: Python object overhead per entry is roughly 200–400 bytes, easily 24–48 GB for the key set alone.

What flips at this scale

At this scale the team partitions the reconciliation: process one user shard at a time, holding at most ~1 M positions in RAM, and write results to a staging table in Postgres rather than accumulating in a Python dict. The key type stays the same; only the scope of what is held in memory at once changes.

The trade-off you'd defend

I would defend the tuple key design over a custom hashable class, trading a small loss of type safety for zero risk of hash/equality inconsistency. For the memory problem I would defend shard-and-write over full-in-memory accumulation, trading I/O round-trips for bounded RAM.

Illustrative numbers scaled to make the arithmetic visible; not Groww-published figures.

Gotchas

Two instances that compare equal but hash differently cause silent cache misses

Symptom

key in cache returns False even though the same logical key was stored before.

Cause

__hash__ was not defined, so it uses the default identity-based hash (id(obj) // 16). Two different objects with the same field values hash to different slots.

Fix

Define both __hash__ and __eq__ consistently, or use @dataclass(frozen=True).

A tuple that contains a list is not hashable

Symptom

TypeError: unhashable type: 'list' when using a tuple as a dict key.

Cause

hash() is recursive on tuples: Python calls hash() on every element. A list has no __hash__.

Fix

Replace the inner list with a tuple or frozenset. Prefer flat scalar tuples for dict keys.

Mutating an object after using it as a key makes it unreachable

Symptom

key in d returns False after modifying key, even though key was used to insert the value.

Cause

Mutation changed the hash. The value is still in the slot where the old hash pointed, but the new hash points to a different slot.

Fix

Only use immutable objects (or frozen dataclasses) as dict or set keys.

Tricky questions

Can two objects have the same hash but not be equal?
Most people say

No — same hash means same object.

Actually

Yes. Hash collisions are allowed and expected. Python requires equal objects to have the same hash, but two unequal objects may also share a hash. A collision causes Python to fall back to __eq__ for disambiguation.

If I define __hash__ to always return 42, is the dict correct?
Most people say

No — all keys would collide, breaking lookup.

Actually

Functionally correct but extremely slow: every lookup degenerates to a linear scan of all values in the single collision chain. The dict is correct; it is just O(n) instead of O(1).

Scenario question

You're handed a deduplication function that stores records in a set using a custom RecordKey class. The class defines __eq__ based on three fields, but __hash__ returns id(self) // 16. Deduplication is not working. What is wrong?

Model answer

id(self) // 16 gives each instance a unique hash based on memory address, so two RecordKey objects with identical fields hash to different slots and == is never called to detect the duplicate. The set treats them as distinct. I'd fix __hash__ to combine hashes of the three fields: return hash((self.field1, self.field2, self.field3)). The trade-off is that the hash is now field-based; if the fields can change, the object must be frozen to avoid the "mutated key becomes unreachable" bug.

Interview corner

CommonWhat makes an object hashable in Python?
“I'd say it needs a __hash__ method that returns an integer, and the hash must be consistent with __eq__: if two objects compare equal they must produce the same hash. In Python 3, defining __eq__ without __hash__ automatically makes the class unhashable.”
TrickIs a tuple always hashable?
“I'd say no — a tuple is hashable only if all its elements are hashable. A tuple containing a list raises TypeError: unhashable type: 'list' when you try to hash it.”
Experience checkWhen have you used a tuple as a compound dict key?
“I cached position lookups by (user_id, symbol) in a nightly reconciliation job. Tuples of ints and strings are hashable, the lookup is O(1), and there is no risk of hash/equality inconsistency that you would get with a custom class that defines __eq__ without __hash__.”
DesignHow do you design a hashable domain object for a high-volume cache?
“I'd use @dataclass(frozen=True) with scalar fields. Frozen means immutable, which guarantees the hash stays stable. Python generates a consistent __hash__ from all fields. The trade-off is that you cannot update the object; you create a new instance instead — which is fine for cache keys.”

Memory hook

A dict is a filing room with numbered drawers. Hash says which drawer; equality confirms the match. Equal objects must agree on the drawer number, or lookups return nothing.

If you remember only one thing:

If a == b, then hash(a) == hash(b) must also be true — every time, for every object.

Self-check

  1. What happens in Python 3 when you define __eq__ but not __hash__?

    Python sets __hash__ = None, making the class unhashable. Any attempt to hash an instance raises TypeError.

  2. Why are lists not hashable?

    Lists are mutable. Their hash would change when the list changes, breaking the "stable hash over the object's lifetime" rule.

  3. How do you create a custom hashable class with field-based equality?

    Use @dataclass(frozen=True), which generates both __eq__ and __hash__ from the declared fields and makes the object immutable.

  4. What is a hash collision, and how does Python handle it?

    Two unequal objects produce the same hash. Python's dict stores them in the same hash bucket and uses __eq__ to distinguish them during lookup. Performance degrades from O(1) toward O(n) as collisions accumulate.

Prove it

Write a @dataclass(frozen=True) class Key with two fields a: int and b: str. Create k1 = Key(1, "x") and k2 = Key(1, "x"). Check k1 == k2, k1 is k2, hash(k1) == hash(k2), and {k1: "found"}[k2]. Paste all four outputs. Then create class BadKey with __eq__ returning True always but __hash__ returning id(self). Show that BadKey() in {BadKey()} returns False.

py-01-05Card 5 of 6

Separate empty, missing and interned

EasyP2 · Very likelyKnow cold10 min
truthinessnonefalsyinterningmissingempty

Why you care

The if row["amount"]: pattern is one of the most common sources of data loss in Python ETL: it silently skips rows where amount is 0, which is a valid financial value. The same pattern deletes "" (settled with no reference), False (a boolean flag), and [] (an empty list of tags) along with genuinely absent None values. Every time you write a truthiness check to guard against a missing value, you are making an implicit claim about what "falsy" means for that field — and in a wide Postgres result set that claim is almost always wrong for at least one column. Interviewers ask this to see whether you distinguish truthiness from presence.

Explain like I'm 5

Imagine a warehouse checklist. "Is the box empty?" and "Did the box arrive?" are two different questions. A box that arrived empty is not the same as a box that never came. Python's if value: answers the first question (is it empty or zero?). if value is None: answers the second (did it arrive at all?). Calling an arrived-empty box "missing" makes you lose valid shipments.

Core concept

Truthiness and presence are independent concepts. An object's truthiness is determined by __bool__ (falling back to __len__): if the method returns False or 0, the object is falsy. Presence is whether a value was ever supplied — None in Python convention, NULL in SQL.

Falsy values in Python:

  • None, False, 0, 0.0, 0j (complex zero)
  • "", b"", [], (), {}, set(), frozenset()
  • Any object whose __bool__ returns False or __len__ returns 0

Truthy values: everything else, including "0", [None], {"": ""}, negative numbers.

The decision rule:

  • Use if x is None: to test for an absent value.
  • Use if x: to test that a value is truthy — only when 0, "" and [] are all legitimately falsy for that field.
  • Use if x is not None and x != "": (or whatever the field's empty-string semantics are) to distinguish absent from empty.

Interning is a separate concern: CPython caches small integers in the range -5 to 256, so 256 is 256 returns True in CPython — but this is an implementation detail, not a language guarantee. For integers outside that range, such as 1000, is may or may not return True depending on whether both literals share the same compiled code object. Never rely on is for integer or string value comparisons; use ==. The safe rule from Equality is not identity applies: is is only for None, True, False, and explicit sentinels.

Minimal example Falsy table and the cost of naive None checks

PythonFalsy values and the None/empty distinctionruns
falsy  = [None, "", 0, 0.0, [], {}, set(), False]
truthy = ["x", 1, -1, [0], {"a": 1}]
print([bool(v) for v in falsy])
print([bool(v) for v in truthy])

data = {"price": 0, "ref": "", "note": None}
# Naive: treats 0, "" and None all as missing
naive_missing = {k for k, v in data.items() if not v}
# Precise: only genuinely absent values
only_none = {k for k, v in data.items() if v is None}
print(sorted(naive_missing))
print(sorted(only_none))
Output
[False, False, False, False, False, False, False, False]
[True, True, True, True, True]
['note', 'price', 'ref']
['note']

In production ETL row filter that preserves zero-amount transactions

A payment pipeline must pass through zero-amount adjustment rows; the naive guard drops them.

PythonPayment filter distinguishing absent from zero (sqlite3 stands in for Postgres)runs
import sqlite3

con = sqlite3.connect(":memory:")
con.execute("CREATE TABLE payments (id INT, amount REAL, ref TEXT)")
con.executemany("INSERT INTO payments VALUES (?,?,?)",
                [(1, 500.0, "P100"),
                 (2, 0.0,   "ADJ-01"),   # valid zero-amount adjustment
                 (3, None,  None),        # incomplete record — genuinely absent
                 (4, 200.0, "")])         # paid, ref not yet assigned

rows = list(con.execute("SELECT id, amount, ref FROM payments"))

# Wrong: drops the zero-amount adjustment
wrong = [r for r in rows if r[1]]
# Right: only skip rows where amount is genuinely absent
right = [r for r in rows if r[1] is not None]

print("wrong filter ids:", [r[0] for r in wrong])
print("right filter ids:", [r[0] for r in right])
Output
wrong filter ids: [1, 4]
right filter ids: [1, 2, 4]
Scenario A

In your world

The situation

Your Jenkins job loads fx_transactions.csv into Postgres. Rows with a zero notional column represent netting adjustments. Your validation step includes if row["notional"]: before inserting.

What actually breaks

All netting adjustments (notional = 0.0) are silently skipped. The batch shows a green Jenkins run. The risk desk's end-of-day P&L is off. The discrepancy is exactly the sum of the netting rows: 47 rows, 0.0 notional each, but their presence is required for the audit trail.

What you'd do on Monday

Replace the truthiness guard with an explicit presence check:

PythonExplicit None guard for the notional column (requires real Postgres + psycopg)
for row in reader:
    if row["notional"] is None:
        log.warning("skipping row %s: notional is NULL", row["id"])
        continue
    insert(conn, row)
Scenario B

Imagine you work at Nykaa

3.5 M SKUs in catalogue

The numbers

Illustrative: 3.5 M SKUs × average 35 attributes each = 122.5 M attribute values. On any given day, roughly 5% of attributes are None (not yet filled), 3% are "" (filled but empty, as allowed by the supplier form), and 2% are 0 (valid numeric defaults like a zero-discount field). A naive if not v: filter over the full attribute set would discard ~12.25 M attribute values — about 10% of the catalogue.

What the Scenario-A answer does here

A data-quality pipeline that filters with if not v: continue runs fine on the small test dataset where all falsy fields are genuinely absent. At 122.5 M values, 10% silent skips means the Nykaa search index is built on 110 M attributes instead of 122.5 M, and the missing 12.5 M include legitimate zeros and empty strings that have business meaning.

What flips at this scale

At this scale the team defines per-attribute semantics in a schema registry: each attribute declares whether None means absent, whether "" is a valid value, and the expected type. The pipeline validates against the schema, and any attribute with a value that violates the schema's nullable or empty rules is routed to a quarantine table, not silently dropped.

The trade-off you'd defend

I would defend schema-driven validation with a quarantine lane over boolean guards, trading the up-front cost of schema maintenance for complete visibility into data quality at scale. Silent drops at 3.5 M SKUs are business risk, not a cleanup task.

Illustrative numbers scaled to make the arithmetic visible; not Nykaa-published figures.

Gotchas

if row["amount"]: drops valid zero-amount rows

Symptom

Zero-value adjustment records are absent from the target table with no error.

Cause

0, 0.0 are falsy. The condition skips them.

Fix

Use if row["amount"] is not None: for a presence check. If the intent was truly to skip zeros, document it explicitly: if row["amount"] != 0:.

Interning makes is unreliable for integers outside -5..256

Symptom

count is 0 returns True in development (small count, CPython caches it) and unpredictably True or False in production (large count, no cache).

Cause

CPython caches small integers; large integers may or may not be interned depending on context.

Fix

Always use count == 0 for value comparisons. Reserve is for None, True, False, and explicit sentinels (Equality is not identity).

bool is a subclass of int, so True == 1 and False == 0

Symptom

A column storing 0 or 1 as integers passes an if col == True: check for 1 and an if not col: check for 0.

Cause

True == 1 is True; False == 0 is True because bool inherits from int.

Fix

Be explicit: use if col is True: only when you need exactly the boolean True, not any truthy value or the integer 1.

Tricky questions

Is "" equal to None?
Most people say

They're both "empty" so they might behave the same way.

Actually

"" == None is False. "" is an empty string; None is the absence of a value. bool("") is False and bool(None) is False, but "" is None is False and they are not equal.

Can you always use if x: as a guard for a string that should not be blank?
Most people say

Yes — empty strings are falsy so if x: catches them.

Actually

Only if you are certain the field will never legitimately contain 0, False, or an empty list. For a string column that should not be blank, if x: is usually correct. For a numeric column, it is wrong as soon as valid zero values appear.

Scenario question

You're handed a data-quality check def has_all_fields(row, required) that returns True when all required fields are present. It uses return all(row.get(k) for k in required). A row {"amount": 0, "currency": "INR"} fails the check even though both fields are present. What is wrong and how do you fix it?

Model answer

row.get(k) returns the value, which for amount is 0. bool(0) is False, so all(...) fails. The check conflates presence with truthiness. I'd fix it to: return all(row.get(k) is not None for k in required). If None is a valid stored value and "present" means "the key exists", use all(k in row for k in required) instead. The trade-off is that the first fix uses None as the absence sentinel; if callers store explicit None to mean "blank", use the k in row form.

Interview corner

CommonName five falsy values in Python.
“I'd list None, False, 0, empty string, empty list — and also empty dict, empty set, 0.0, and any object whose __bool__ returns False.”
TrickWhy is if row['qty']: dangerous in an inventory ETL?
“I'd point out that a quantity of 0 is a valid inventory level — it means out of stock. bool(0) is False, so the check silently skips zero-quantity rows. I'd use if row['qty'] is not None: to mean 'the field was supplied', and handle the zero case explicitly downstream.”
Experience checkWhen has a truthiness check caused a real data loss?
“I've been bitten by this. We had a settlement pipeline that filtered with if row['amount']:. Zero-amount netting adjustments were required for regulatory reporting but they were all dropped silently. The batch succeeded and the discrepancy only surfaced during audit.”
DesignHow do you distinguish between an absent value, an empty value, and a zero in a wide Postgres result set?
“I define per-column semantics in a configuration or schema file: nullable yes/no, and what an empty string or zero means for that column. The validator applies column-specific rules rather than a blanket if not v: guard. It costs more upfront but eliminates an entire class of silent data-quality failures.”

Memory hook

if value: asks whether the box is full. if value is None: asks whether the box arrived. An arrived-empty box is not a missing box.

If you remember only one thing:

Use is None for presence; use truthiness only when you can accept all falsy values as equivalent.

Self-check

  1. Is "0" truthy or falsy?

    Truthy. It is a non-empty string. bool("0") is True.

  2. What does row.get("key") return when the key exists but its value is None?

    None. dict.get returns None both when the key is absent and when its value is None. To distinguish, use "key" in row.

  3. Name one case where if x: is correct and one where it is dangerous.

    Correct: checking that a list of results is non-empty (if results:). Dangerous: checking that a numeric amount was supplied, when 0 is a valid value.

  4. Why should you not use is to compare integers outside -5 to 256?

    CPython caches small integers, so is happens to work for them. For larger integers the caching is not guaranteed, and is compares memory addresses, not values. Use == for all integer value comparisons.

Prove it

Create a list of payment dicts including one with {"amount": 0.0, "ref": "ADJ"}. Apply three filters: if r["amount"]:, if r["amount"] is not None:, and if r["amount"] == 0:. Print the IDs that pass each filter. Paste the output and explain which filter is correct for "process all records that have a supplied amount, including zero-value adjustments".

py-01-06Card 6 of 6

Choose shallow or deep copy

MediumP2 · Very likelyKnow cold12 min
copyshallow-copydeep-copycopy-modulenested

Why you care

Every ETL pipeline that passes a configuration dict or a batch of row dicts through multiple stages needs to know when a copy is necessary and which kind. copy.copy() looks like the safe option but silently shares nested objects, so a downstream stage that modifies a nested list or dict corrupts the "original" you thought you had preserved. copy.deepcopy() is fully independent but pays a cost that grows with depth and cycles. Using the wrong one means either wasted CPU or silent data corruption. Interviewers ask this to see whether you understand what exactly is copied — the outer container, or everything recursively.

Explain like I'm 5

b = a is like taping a second name card to a box. copy.copy(a) puts the same items into a new outer box but does not copy the items themselves — they are still shared. copy.deepcopy(a) photocopies the box and every item inside, recursively, so no object is shared at any depth. Modifying an item in a shallow copy modifies the same item in the original; modifying an item in a deep copy has no effect on the original.

Core concept

Three levels of "copy" exist in Python:

  • Binding (b = a): one object, two names. No copy at any level.
  • Shallow copy (copy.copy(a), list(a), a[:], dict(a), {**a}): a new outer container, but the elements inside are the same objects as in the original. Mutating an element (appending to an inner list, updating a nested dict) changes it in both the copy and the original.
  • Deep copy (copy.deepcopy(a)): a fully independent clone. Every object at every depth is recursively copied. Mutating anything in the deep copy cannot affect the original.

Decision rule:

  • Use a shallow copy when the container is flat (its elements are immutable scalars: strings, ints, floats, tuples of scalars). A shallow copy of a list of integers is fully independent.
  • Use a deep copy when the container has mutable nested objects (list of dicts, dict of lists, config dicts with nested sub-dicts). If you are not certain, use deepcopy.
  • Be aware that deepcopy handles cycles (objects that reference themselves) via a memo dict. It raises TypeError on objects that explicitly forbid copying (open file handles, database connections, locks). For those, copy only the data, not the resource.

The three levels compared:

b = acopy.copy(a)copy.deepcopy(a)
New outer container?NoYesYes
New inner objects?NoNoYes
Inner mutation propagates?YesYesNo
Handles circular references?n/aNoYes
SpeedInstantO(n elements)O(all nodes)
When to useRead-only aliasFlat, immutable-element containersNested mutable containers

Under the hood

copy.copy(obj) looks for __copy__ on the type; if absent it falls back to __reduce_ex__ or __reduce__ (the pickle protocol). Slicing a list (a[:]) directly calls list_subscript in C, which is faster for lists. copy.deepcopy(obj) uses a memo dict keyed by id(obj) to detect already-copied objects and break cycles. Each object is dispatched via _deepcopy_dispatch, a type-to-function registry. You can customise copying by defining __copy__ (return a shallow copy) or __deepcopy__(self, memo) (return a deep copy using memo). This is useful when an object holds a database connection that should not be copied: define __deepcopy__ to copy only the data fields and re-use the shared connection.

Minimal example Shallow shares inner objects; deep does not

PythonShallow copy versus deep copy on nested listsruns
import copy

original = [[1, 2], [3, 4]]
shallow  = copy.copy(original)
deep     = copy.deepcopy(original)

original[0].append(99)          # mutate the first inner list

print("original:", original)
print("shallow: ", shallow)     # shares inner list — sees the mutation
print("deep:    ", deep)        # independent — does not see the mutation
Output
original: [[1, 2, 99], [3, 4]]
shallow:  [[1, 2, 99], [3, 4]]
deep:     [[1, 2], [3, 4]]

In production ETL config that is modified per pipeline stage

A base config dict is shared across pipeline stages. A shallow copy lets a stage corrupt the nested filters dict for subsequent stages; a deep copy gives each stage independence.

PythonShallow vs deep copy on a nested config dictruns
import copy

base_config = {
    "pipeline": "settlements",
    "filters": {"currency": "INR"},   # nested dict
    "tags":    ["batch", "daily"],    # nested list
}

def stage_shallow(cfg):
    """Takes a shallow copy — modifies nested objects."""
    local = copy.copy(cfg)
    local["filters"]["currency"] = "ALL"   # mutates shared nested dict
    local["tags"].append("modified")       # mutates shared nested list

def stage_deep(cfg):
    """Takes a deep copy — fully independent."""
    local = copy.deepcopy(cfg)
    local["filters"]["currency"] = "ALL"
    local["tags"].append("modified")

# Shallow: base_config is corrupted after the stage runs
stage_shallow(base_config)
print("after shallow — base currency:", base_config["filters"]["currency"])
print("after shallow — base tags:    ", base_config["tags"])

# Deep: base_config2 is intact after the stage runs
base_config2 = {
    "pipeline": "settlements",
    "filters": {"currency": "INR"},
    "tags":    ["batch", "daily"],
}
stage_deep(base_config2)
print("after deep    — base currency:", base_config2["filters"]["currency"])
print("after deep    — base tags:    ", base_config2["tags"])
Output
after shallow — base currency: ALL
after shallow — base tags:     ['batch', 'daily', 'modified']
after deep    — base currency: INR
after deep    — base tags:     ['batch', 'daily']

See it happen What each copy level shares

samesameoriginalcopy.copydeepcopyinner [1,2,99]inner [3,4]copy [1,2]copy [3,4]
  1. The original list holds references to two inner lists.
  2. copy.copy creates a new outer list but points at the same inner objects as the original.
  3. copy.deepcopy creates a new outer list and new copies of every inner list — no sharing.
  4. Appending 99 to original[0] also changes shallow[0] because they share the inner list; deepcopy[0] is unaffected.
Notice: the shallow copy has a new outer box but the same inner objects. Only deepcopy gives every level its own object.
Scenario A

In your world

The situation

Your ETL pipeline loads a YAML config once at startup into a Python dict. Each nightly job run passes the config to three transform stages. Stage 2 appends a "run_id" to config["audit"]["tags"] for tracking purposes.

What actually breaks

On the second nightly run, config["audit"]["tags"] already contains yesterday's run_id from Stage 2's in-place append. By the end of the year the list has 365 stale run IDs. The downstream audit query returns incorrect results because the tag filter matches multiple runs.

What you'd do on Monday

Each job run should receive an independent config. Use deepcopy at the start of each run, not a new YAML parse:

PythonPer-run deep copy of the shared config (no network or DB needed)
import copy

BASE_CONFIG = load_yaml("pipeline.yaml")   # loaded once at startup

for run_id in scheduled_runs():
    cfg = copy.deepcopy(BASE_CONFIG)       # fully independent per run
    cfg["audit"]["tags"].append(run_id)    # safe: does not affect BASE_CONFIG
    run_pipeline(cfg)
Scenario B

Imagine you work at Ola

2 M rides per day

The numbers

Illustrative: 2 M rides × ~40 fields per ride event = 80 M field values per day. A fare-adjustment pipeline that deep-copies each ride event dict before enrichment copies 40 Python object references per event. At ~8 bytes per reference × 40 fields × 2 M events ≈ 640 MB of reference overhead per full-day batch run — before the actual string and numeric objects.

What the Scenario-A answer does here

Deep-copying every event for isolation works at 10 k events. At 2 M events the full-batch deepcopy budget becomes significant. A 100-byte event dict × 2 M × 2 (original + copy) × 3 stages = roughly 1.2 GB of live dict objects at peak.

What flips at this scale

At this scale the team does not deep-copy batches. Instead each stage is a pure function that produces a new list of new objects from its input — effectively a map transform. The allocation cost is the same but the pattern makes testing easier (no shared state to poison), and immutable input objects (frozen dataclasses) eliminate the need for any copy at all.

The trade-off you'd defend

I would defend per-event frozen dataclasses over per-batch deepcopy, trading one more allocation per transform for eliminating the deepcopy overhead entirely. Frozen dataclasses are also typed and introspectable, which pays dividends during debugging at 2 M events.

Illustrative numbers scaled to make the arithmetic visible; not Ola-published figures.

Gotchas

dict(d) and {**d} are shallow copies

Symptom

Modifying copy["nested_list"].append(x) also adds x to original["nested_list"].

Cause

Both create a new outer dict with the same value objects. Nested mutable values are shared.

Fix

Use copy.deepcopy(d) when any value is itself mutable (a list, a dict, a set).

deepcopy raises TypeError on non-copyable objects

Symptom

TypeError: cannot pickle 'lock' object or similar when deep-copying a config that contains a database connection or a thread lock.

Cause

deepcopy internally uses the pickle protocol for some objects; locks, file handles and sockets are not picklable.

Fix

Define __deepcopy__ on the containing class to copy only the data fields and re-use the resource, or strip the resource out before copying.

Slicing a list (a[:]) is a shallow copy, not a deep copy

Symptom

After b = a[:], modifying a[0]["key"] also changes b[0]["key"].

Cause

a[:] copies the outer list only; inner objects are the same references.

Fix

Use copy.deepcopy(a) when the list contains mutable elements.

Tricky questions

Is copy.copy of a list of integers a deep copy?
Most people say

No — you always need deepcopy for full independence.

Actually

Effectively yes. Integers are immutable; there is no mutation you can perform on an integer that would propagate to the original. copy.copy of a list of integers gives you a list where every element is the same integer object, but since integers cannot be mutated, the two lists are functionally independent.

Does deepcopy copy everything?
Most people say

Yes — it copies every object reachable from the root.

Actually

Not quite. deepcopy skips objects whose types are registered as atomic (ints, strings, bytes, etc.) and returns them directly. It also skips objects that define __deepcopy__ to opt out of recursion. The result is still functionally independent at all mutable depths.

Scenario question

You're handed a config-loading function def get_config() that returns the module-level CONFIG dict on every call. Multiple pipeline stages call get_config() and modify the returned dict in-place with stage-specific settings. Bugs appear when stages run in different orders. How do you fix this without changing the call sites?

Model answer

I'd make get_config() return copy.deepcopy(CONFIG) instead of CONFIG itself. Each caller gets an independent dict, so any modification is stage-local. The trade-off is an allocation on every call; if CONFIG is large or called at high frequency, I'd consider a pattern where each stage declares its overrides as a separate argument rather than mutating the config. For a nightly batch ETL, the deepcopy cost is negligible and the isolation is worth it.

Interview corner

CommonWhat is the difference between copy.copy and copy.deepcopy?
“I'd describe it as: shallow copy creates a new outer container but reuses the same inner objects. Deep copy recursively creates new objects at every level. With a list of dicts, shallow copy means the dicts are shared; mutating one dict mutates it for both the original and the copy.”
TrickIs list(a) a deep copy of a?
“I'd say no — list(a) creates a new outer list but the elements are the same objects as in a. For a list of integers that is effectively independent, because integers are immutable. For a list of dicts it is a shallow copy — the dicts are shared.”
Experience checkWhen have you been surprised by a shallow copy in real code?
“I've been caught by this. We had a YAML config loaded once at startup. Pipeline stages would cfg = dict(base_config) for 'their own copy', then mutate cfg['filters']. Because the filters value was itself a dict, the mutation went through to base_config. We fixed it by deep-copying at the start of each run.”
DesignWhen is deepcopy too expensive and what do you use instead?
“When the object graph is large or contains many levels. At that point I'd switch to immutable data structures — frozen dataclasses or NamedTuples — so no copy is needed at all. The pipeline stages become pure functions that produce new objects; mutation never happens, so there is nothing to copy.”

Memory hook

Shallow copy is a new box with the same items inside. Deep copy is a new box with new items inside. Shake one box and only deepcopy's box stays still.

If you remember only one thing:

If any nested object is mutable, use deepcopy. copy.copy only protects the outermost container.

Self-check

  1. What does list(a) copy?

    The outer list. Inner objects are the same references as in a. Mutations to inner mutable objects propagate to both lists.

  2. Name one scenario where copy.copy is sufficient.

    Copying a list of integers, strings, or tuples of scalars, where all elements are immutable. No inner mutation can propagate because immutable objects cannot be mutated.

  3. Why does deepcopy sometimes raise TypeError?

    It uses the pickle protocol internally. Objects that cannot be pickled — open file handles, database connections, thread locks — raise TypeError. Define __deepcopy__ to handle these explicitly.

  4. What is the memo argument in __deepcopy__(self, memo)?

    A dict mapping id(original_object) to the already-copied object. It prevents infinite recursion when an object graph contains cycles.

Prove it

Create a = {"config": {"timeout": 30}, "tags": ["batch"]}. Make b = copy.copy(a) and c = copy.deepcopy(a). Set a["config"]["timeout"] = 99. Print b["config"]["timeout"] and c["config"]["timeout"]. Then append "new" to a["tags"] and print b["tags"] and c["tags"]. Paste all four outputs and explain which copies are independent at each level.