Shelf · Python · Chapter 3

Chapter 3: OOP done properly

A class is just a namespace until you understand where state actually lives. These six cards take you from the mutable-class-attribute trap through dunders, properties, composition, MRO and dataclasses — the design primitives every product-company engineer reaches for weekly.

Difficulty
Priority
py-03-01Card 1 of 6

Separate class and instance state

EasyP1 · Must knowKnow cold10 min
classinstancestateinitshadowing

Why you care

The most common OOP bug in Python ETL code is not a crash: it is a class-level list that silently accumulates data across every run because the list was put on the class, not on self. The job is green, the counts look plausible, and the error is invisible until someone compares two runs side by side. Interviewers ask "what is the difference between a class attribute and an instance attribute" because the answer predicts whether you have built that bug yet. It also gates everything else in OOP: properties, descriptors and __slots__ all depend on where Python looks for an attribute first.

Explain like I'm 5

Think of a company's HR policy document versus each employee's personal HR file. The policy document sits on a shared shelf: everyone in the company reads from the same copy, and if you write on it you are writing for everyone. Each employee's personal file lives in their own desk drawer: what one employee writes goes into their file alone. In Python, class attributes are the shared shelf. Instance attributes are the personal files created when each employee (object) joins (is __init__-ialised). Handing someone your bookmark on the shared shelf when they asked for their personal file is the bug.

Core concept

Python looks up an attribute on an object in two stages: first the instance's own namespace (obj.__dict__), then the class namespace. This gives class attributes their shared-by-default quality.

  • Class attributes are defined directly in the class body, outside any method. Every instance sees the same value unless it creates its own copy.
  • Instance attributes are created inside methods by assigning to self.attr. They live in the instance's __dict__ and shadow any class attribute with the same name.
  • self.x = value always writes to the instance's namespace. It never modifies the class attribute, even when the class has one named x. It creates a shadow.
  • self.items.append(value) when items is a class-level list does not create a shadow — it mutates the shared list in-place, because reading self.items found the class attribute and appended to it.
  • To modify shared class state intentionally, write ClassName.attr = new_value explicitly. The explicit class name signals intent.

Decision rule: Class attributes are correct for constants and intentional shared registries. Everything that must differ per instance goes in __init__ via self.

Minimal example The shared-list trap and its shadow

PythonClass vs instance namespaceruns
class Counter:
    count = 0        # class attribute — shared
    items = []       # class attribute — shared mutable: DANGER

    def __init__(self, name):
        self.name = name  # instance attribute

c1 = Counter("first")
c2 = Counter("second")

c1.count = 99          # creates instance attr on c1; class attr stays 0
Counter.count += 1     # modifies the class attr to 1
c1.items.append("a")   # mutates the shared list — NOT a shadow

print(c1.count, c2.count, Counter.count)
print(c1.items, c2.items)
print("count" in c1.__dict__, "count" in c2.__dict__)
Output
99 1 1
['a'] ['a']
True False

In production Per-pipeline error lists

Each EtlPipeline gets its own errors list because it is created in __init__. The class-level _registry is shared intentionally — it is the lookup table that all pipelines write into.

PythonTwo pipelines each track their own errorsruns
class EtlPipeline:
    _registry = {}       # intentional shared registry

    def __init__(self, name, batch_size=1000):
        self.name = name
        self.batch_size = batch_size
        self.errors = []   # per-instance; fresh every time

    @classmethod
    def register(cls, name, obj):
        cls._registry[name] = obj

    def log_error(self, msg):
        self.errors.append(msg)

    def __repr__(self):
        return f"EtlPipeline({self.name!r}, batch_size={self.batch_size})"


sales = EtlPipeline("sales", batch_size=500)
hr = EtlPipeline("hr", batch_size=2000)
EtlPipeline.register("sales", sales)
EtlPipeline.register("hr", hr)

sales.log_error("row 3: missing amount")
hr.log_error("row 1: invalid employee_id")

print(list(EtlPipeline._registry))
print(sales.errors)
print(hr.errors)
Output
['sales', 'hr']
['row 3: missing amount']
['row 1: invalid employee_id']

See it happen How Python resolves a name on an instance

step 1: check instancestep 2: not found → check classalways writes herec1.__dict__{name:'first', count:99}Counter.__dict__{count:0, items:[...]}read c1.countc1.count = 99
  1. Python reads c1.count: checks instance __dict__ first.
  2. 'count' is in c1.__dict__ (value 99) so the search stops. The class attribute is never seen.
  3. Now read c2.count: c2.__dict__ has no 'count', so Python falls through to the class.
  4. Writing self.count = 99 always lands in the instance __dict__. The class attribute is never modified.
Notice: the write arrow always points to the instance. Only the read path ever reaches the class, and only when the instance dict has no match.
Scenario A

In your world

The situation

Your nightly Jenkins job creates a PipelineRun per source file — sales.csv first, then hr.csv — and emails each object's error list as a separate report. You added errors = [] at the class level, expecting each object to start fresh.

What actually breaks

The report for hr.csv contains both the hr errors and the sales errors that were logged earlier in the same run. Finance sees duplicate and misattributed errors in the hr report and files a ticket. The job exits green because no exception was raised.

What you'd do on Monday

Move errors = [] into __init__ as self.errors = []. The sales object and the hr object each receive their own list at construction time. The hr report now contains only hr errors.

Scenario B

Imagine you work at Stripe

50M payment objects per day

The numbers

Illustrative: 50M Payment objects/day. If each object carries a class-level mutable counter and 0.01% trigger an error callback that appends to it, 5,000 appends/day accumulate in a structure pinned to the class for the process lifetime.

What the Scenario-A answer does here

Moving errors to __init__ fixes correctness. But 50M objects each with a full object-plus-dict footprint of ~344 bytes (48 B object + 296 B __dict__, CPython 3.12); 50M × 344 bytes ≈ 17 GB of object memory before the actual field values.

What flips at this scale

At this scale, high-volume row objects declare fields in __slots__ or use @dataclass(slots=True) (Choose dataclass and slots deliberately), eliminating __dict__ entirely. Errors go to a structured log or a DB column, not a list on any object.

The trade-off you'd defend

I would defend slots-only objects for ETL rows over dicts, trading the ability to attach arbitrary attributes at runtime for predictable, bounded per-object memory. Every field that might be needed is declared at design time.

Illustrative; not Stripe-published figures.

Gotchas

A class-level list shared across every instance

Symptom

Two pipeline objects accumulate each other's errors. Errors from previous runs appear in the current run's report.

Cause

errors = [] in the class body creates one list shared by every instance. self.errors.append(msg) finds and appends to the class-level list, not an instance-level one.

Fix

Move the definition into __init__: self.errors = []. Now each instance gets its own list at creation time.

Assigning to self shadows without updating shared state

Symptom

A class counter incremented via self.count += 1 inside a method never changes the class-level value.

Cause

self.count += 1 reads the class attribute (0), computes 1, then writes 1 to the instance's namespace, leaving the class attribute unchanged.

Fix

To modify shared class state, write ClassName.count += 1 explicitly. To track per-instance state, initialise in __init__.

Tricky questions

Does self.attr = value ever change the class attribute?
Most people say

Yes, if the class already has an attribute with that name, the assignment updates it.

Actually

No. self.attr = value always writes to the instance's __dict__, creating a shadow. The class attribute retains its value. Only ClassName.attr = value modifies the class attribute.

When you read self.attr, which namespace does Python check first?
Most people say

The class namespace, because attributes are defined on the class.

Actually

Python checks the instance __dict__ first. If not found there, it checks the class (and then its bases). This is why an instance attribute "shadows" a class attribute of the same name.

Scenario question

You're handed a report that says a nightly ETL job loaded 0 rows on Tuesday but 48,213 on Monday and Wednesday. The code uses a class-level rows_loaded = 0 counter incremented inside the load loop via self.rows_loaded += 1. How do you explain the zero-row Tuesday, and what do you change?

Model answer

I would check whether a second pipeline run on Tuesday read ClassName.rows_loaded as a guard condition, expecting the count from the first run's object. Because self.rows_loaded += 1 creates an instance shadow, ClassName.rows_loaded stays 0 forever. If the second run checked if ClassName.rows_loaded > 0: skip_load(), it would skip the entire load. The fix is to use instance-level counters throughout: read and log from self.rows_loaded, and reset by constructing a new instance per run. The trade-off: if you genuinely need cross-instance aggregation, use a classmethod to accumulate into cls.rows_loaded intentionally and document why.

Interview corner

CommonWhat is the difference between a class attribute and an instance attribute?
“I'd say a class attribute lives on the class itself — every instance sees the same value unless it creates its own shadow. An instance attribute is set on self in __init__ and lives in that instance's __dict__. The danger is a mutable class attribute like a list: every instance that appends to it is appending to the same list.”
TrickYou do self.count += 1 in a method. Does this change the class attribute?
“I'd say no — self.count += 1 reads the class attribute, adds 1, and writes the result to the instance's namespace. The class attribute is unchanged. To increment shared state I write ClassName.count += 1.”
Experience checkHave you hit a class-state bug in real code?
“I hit this in a Jenkins job where errors = [] was on the class. The second PipelineRun object created in the same process appended its errors to the same list, so the second report contained both files' errors. Moving it to self.errors = [] in __init__ fixed it immediately.”

Memory hook

The class attribute is the shared shelf; self.attr = value always writes to the employee's personal drawer, never back to the shelf.

If you remember only one thing:

If you want per-instance state, put it in __init__ via self. Anything defined in the class body is shared by default.

Self-check

  1. If you add c.x = 5 on an instance, does it modify the class?

    No. It creates a new entry in c.__dict__. ClassName.x is unchanged; reading c.x returns 5 because the instance namespace is checked first.

  2. How do you check whether an attribute is in the instance's own namespace?

    "attr" in obj.__dict__ is True if the instance has its own copy, False if it is only inherited from the class.

  3. What is the safe way to modify shared class state inside a method?

    Write ClassName.attr = new_value using the explicit class name, not self, to signal that you are intentionally writing to shared state.

Prove it

Write a class RunLog with a class-level runs = 0 counter and an instance-level errors = [] list. Create two instances. In the first, call a method that does self.runs += 1. In the second, call RunLog.runs += 1. Print RunLog.runs, instance1.__dict__["runs"], and show that "runs" is not in instance2.__dict__. Paste the three output lines.

py-03-02Card 2 of 6

Give objects coherent protocols

MediumP2 · Very likelyKnow cold10 min
dundersrepreqhashprotocols

Why you care

An object without __repr__ prints as <__main__.InvoiceRow object at 0x...> in every log line and debug session: useless. An object without __eq__ compares by identity, so two rows with identical fields are never equal and a set never deduplicates them. The silent version of that bug is worse: once you define __eq__, Python automatically sets __hash__ = None, making your objects un-hashable the moment you add the == check you wanted. The set that used to store them now raises TypeError: unhashable type. Interviewers ask about dunders because the chain of consequences — define __eq__, lose __hash__, break the deduplication set — is invisible until it hits production, and it predicts whether a developer has actually read the Python data model.

Explain like I'm 5

Think of customs paperwork at an international border. Every package must declare its contents, weight and country of origin — that is the protocol customs uses to process it. If your package has no label at all, customs either returns it or assigns it an arbitrary tracking number. Python objects have the same "customs forms": __repr__ is the shipping label (what does this thing say about itself?), __eq__ is the contents check (is this the same package?), and __hash__ is the tracking barcode (where does this package go in the sorted queue?). Forget one and customs makes something up, or refuses entry entirely.

Core concept

Python uses dunder (double-underscore) methods to make objects participate in built-in operations. The three that matter most for ETL data objects are:

  • __repr__: called by repr(), by the REPL and by %r in log format strings. Return a string that could recreate the object if possible; otherwise <ClassName field=value>. Always define this on every class you write.
  • __eq__: called by ==. Default: identity (self is other). Once you define __eq__, Python sets __hash__ = None unless you also define __hash__. An object with __hash__ = None raises TypeError: unhashable type when placed in a set.
  • __hash__: called by hash() and used internally by set and dict. Must be consistent with __eq__: equal objects must produce equal hashes. Hash only the fields you compare in __eq__.

When not to define __hash__: if your object is mutable, omit it (Python sets it to None when you define __eq__, which is intentionally correct). Only define __hash__ on logically immutable objects, or hash only the immutable identity fields and document the constraint.

Related dunders worth knowing: __lt__ enables sorted(), __len__ drives bool() and len(), __iter__ enables for loops (Iterator protocol, exhaustion and pull model).

Decision rule: Define __repr__ on every class. Define __eq__ when you compare by value. Define __hash__ alongside __eq__ only when the object needs to go into a set or dict key and it is logically immutable.

Minimal example repr, eq, hash — and the hash consequence

Pythonruns
class InvoiceRow:
    def __init__(self, invoice_id, amount):
        self.invoice_id = invoice_id
        self.amount = amount

    def __repr__(self):
        return f"InvoiceRow({self.invoice_id!r}, amount={self.amount})"

    def __eq__(self, other):
        if not isinstance(other, InvoiceRow):
            return NotImplemented
        return self.invoice_id == other.invoice_id and self.amount == other.amount

    def __hash__(self):
        return hash((self.invoice_id, self.amount))


r1 = InvoiceRow("INV-001", 250)
r2 = InvoiceRow("INV-001", 250)
r3 = InvoiceRow("INV-002", 90)

print(repr(r1))
print(r1 == r2, r1 == r3)
print(len({r1, r2, r3}))   # r1 and r2 collapse into one
Output
InvoiceRow('INV-001', amount=250)
True False
2

In production Deduplicating settlement rows before insert

S3 exports overlap by one hour each day; duplicate rows arrive in every batch. Without __eq__ and __hash__, the set deduplication does nothing.

PythonSettlement row deduplication via value-based equalityruns
class SettlementRow:
    def __init__(self, txn_id, amount, currency):
        self.txn_id = txn_id
        self.amount = amount
        self.currency = currency

    def __repr__(self):
        return f"SettlementRow({self.txn_id!r}, {self.amount} {self.currency})"

    def __eq__(self, other):
        if not isinstance(other, SettlementRow):
            return NotImplemented
        return (self.txn_id, self.amount, self.currency) == (
            other.txn_id, other.amount, other.currency
        )

    def __hash__(self):
        return hash((self.txn_id, self.amount, self.currency))


rows = [
    SettlementRow("T001", 1500, "INR"),
    SettlementRow("T002", 800, "INR"),
    SettlementRow("T001", 1500, "INR"),   # duplicate from overlapping export
    SettlementRow("T003", 300, "USD"),
]

unique = set(rows)
print(f"raw={len(rows)}, deduplicated={len(unique)}")
for r in sorted(unique, key=lambda r: r.txn_id):
    print(r)
Output
raw=4, deduplicated=3
SettlementRow('T001', 1500 INR)
SettlementRow('T002', 800 INR)
SettlementRow('T003', 300 USD)
Scenario A

In your world

The situation

Your settlement loader deduplicates rows using a Python set before the Postgres INSERT. Two S3 exports overlap by one hour, so the raw input always contains some duplicates. The class has no __eq__ or __hash__.

What actually breaks

The target table receives duplicate rows. The audit count matches the raw input count, not the expected unique count. Postgres does not raise an error because the table has no UNIQUE constraint. Finance finds duplicate settlement amounts in the daily report a week after the overlap was introduced.

What you'd do on Monday

Define __eq__ to compare by (txn_id, amount, currency) and __hash__ to hash those same fields. After the change, len(set(rows)) < len(rows) for inputs with overlapping windows.

Scenario B

Imagine you work at Shopify

100M order events per day

The numbers

Illustrative: 100M events/day, ~10% retried deliveries = 10M duplicates. In-memory set deduplication works per batch, but holding all 100M objects in one Python set costs ~27 bytes/entry in the hash table alone — 100M × 27 B ≈ 2.7 GB of hash-table overhead before the object data.

What the Scenario-A answer does here

__eq__/__hash__ correct the deduplication logic, but holding 100M objects in a single set is impractical. The set is lost on process restart with no persistence guarantee.

What flips at this scale

At this scale you push deduplication to Postgres: INSERT ... ON CONFLICT (txn_id) DO NOTHING, or a UPSERT. Python-level set deduplication applies only to bounded in-flight batches (50k rows per chunk), not the full day's stream.

The trade-off you'd defend

I would defend database-level deduplication via a unique index and ON CONFLICT DO NOTHING over Python-level set deduplication, trading the simplicity of a pure-Python check for correct behaviour even when the Python process restarts mid-stream and the in-memory set is lost.

Illustrative; not Shopify-published figures.

Gotchas

Defining __eq__ silently removes __hash__

Symptom

TypeError: unhashable type: 'SettlementRow' when rows are placed in a set or used as dict keys, even though the class worked as a dict key before you added __eq__.

Cause

Python sets __hash__ = None automatically when you define __eq__ without also defining __hash__. This is correct semantics, but it surprises developers who added __eq__ for == comparisons and did not yet need sets.

Fix

Define __hash__ alongside __eq__, hashing the same fields. Or use @dataclass(frozen=True) (Choose dataclass and slots deliberately) which generates both together.

__repr__ versus __str__

Symptom

print(row) shows a different format from what appears in the REPL or in logging.error("%r", row).

Cause

print() calls __str__; the REPL and %r call __repr__. If only one is defined, __str__ falls back to __repr__, but __repr__ does not fall back to __str__.

Fix

Define __repr__ to be unambiguous and machine-readable. Define __str__ only when you want a separate human-friendly display format.

Tricky questions

Does defining __eq__ affect how the object is stored in a dict?
Most people say

No; dict storage uses the id of the object.

Actually

Yes. dict uses hash() to find the bucket and == to confirm a match. If you define __eq__ but forget __hash__, the object becomes un-hashable and cannot be a dict key at all. Equal objects must have equal hashes; otherwise two value-equal objects land in different buckets and d[r1] and d[r2] become separate entries.

Should __eq__ return False or NotImplemented when compared with a different type?
Most people say

Return False — they are clearly not equal.

Actually

Return NotImplemented. Returning False prevents Python from trying the comparison in reverse (other.__eq__(self)). NotImplemented tells Python to try the other object's __eq__, which may know how to compare. Returning False is usually harmless but breaks symmetry in mixed-type comparisons.

Scenario question

You're handed a dedupe_settlements(rows) function that returns the correct unique count when called with a hand-built list but returns the full count (no deduplication) when called with objects deserialized from JSON. What is the most likely cause and how do you verify it?

Model answer

I would suspect the deserialized objects are a different class or have no __eq__, so every comparison falls back to identity. I verify with type(rows[0]), rows[0] == rows[1] for two rows that should match, and hash(rows[0]) == hash(rows[1]). If __eq__ is missing, == returns False for different objects regardless of field values, and the set never collapses duplicates. The fix is to define __eq__ and __hash__ on the deserialized class, or to use @dataclass(frozen=True) which generates them. The trade-off: hashing on mutable fields means that mutating a field after insertion corrupts the set — I would document the immutability constraint.

Interview corner

CommonWhat happens to __hash__ when you define __eq__?
“I'd say Python sets __hash__ = None, making the type un-hashable. I always define both together, hashing the same fields I compare in __eq__.”
TrickYou define __eq__ but not __hash__. The tests pass. What silent bug might exist?
“I'd say any code that puts instances in a set or uses them as dict keys will raise TypeError: unhashable type at runtime. The tests passed because they only used ==, not set membership. I'd add a test that inserts the object into a set and verifies deduplication.”
Experience checkWhen have dunders saved you debugging time in an ETL job?
“I added __repr__ to every row model once, and after that every log line showed actual field values instead of a memory address. That alone cut my average debug time from twenty minutes of print-and-re-run to reading one log line.”

Memory hook

__repr__ is the shipping label; __eq__ is the contents check; __hash__ is the tracking barcode. Forget one and customs either loses your package or refuses entry.

If you remember only one thing:

Define all three together on any data class that needs to log cleanly, compare by value, or go into a set.

Self-check

  1. What does Python do to __hash__ the moment you define __eq__?

    Sets it to None, making the object un-hashable unless you also define __hash__.

  2. Two SettlementRow objects have identical fields. Without __eq__, does == return True?

    No. == uses identity by default, so two different objects are never equal regardless of field values.

  3. Why return NotImplemented from __eq__ instead of False for a different type?

    Returning NotImplemented lets Python try other.__eq__(self), preserving the possibility that the other type knows how to compare. Returning False short-circuits that and breaks symmetry.

Prove it

Create a Payment class with txn_id and amount fields. Print Payment("T1", 100) == Payment("T1", 100) — confirm it is False. Add __eq__ and __hash__. Print the equality again — confirm it is True. Print len({Payment("T1", 100), Payment("T1", 100)}) — confirm it is 1. Paste the four output lines (two before, two after).

py-03-03Card 3 of 6

Expose properties and constructors

MediumP2 · Very likelyKnow cold10 min
propertyclassmethodstaticmethodvalidationconstructors

Why you care

A raw attribute lets callers set batch_size = -1 or amount = "oops" with no complaint at assignment time. The error surfaces three hours later when a loop runs forever or int("oops") raises in the middle of a load — and the stack trace points into the loop, not the config line. @property moves the error to the moment of assignment, so the traceback is immediately useful. Class methods are the second half: they provide a second constructor that does not require callers to know how to convert their format. Without them, callers either learn too much about your internals or you scatter format-conversion code across the codebase. Interviewers ask about @property specifically because it is the boundary between "Python that runs" and "Python that is maintainable without surprises".

Explain like I'm 5

Think of a hotel's front desk. The room number on a key card must be valid (1 to 500) before the card is issued — the desk validates it when the clerk types it, not when the guest tries an invalid door. If anyone could write any number on a key card directly, room 9999 would eventually be issued and the guest would wander the building. @property is the validation step built into the desk: every issued card is always valid. The @classmethod named check_in_from_booking_ref is a second desk window that accepts a booking reference and knows how to convert it to a room number — callers do not need to memorise the conversion.

Core concept

Three decorators give classes controlled access to their state:

  • @property turns a method into read-only attribute access. Add @name.setter to allow writes with validation. Callers write obj.amount = 500 and get validation for free; the property is transparent to them.
  • @classmethod receives the class (cls) as its first argument instead of the instance. Use it for alternative constructors that accept a different input format and return cls(...). Using cls instead of the class name means subclasses inherit the constructor correctly.
  • @staticmethod is a plain function attached to the class. It receives no implicit first argument — neither self nor cls. Use it for utility helpers that are logically related to the class but read and write nothing on the instance or class.

When not to use @property: never put slow computation behind a property. Callers expect attribute access to be instant; a property that queries a database will surprise them. Use a method with a verb in its name (fetch_balance()) for expensive operations.

Decision rule: Use @property when you need validation, a computed view of private data, or backward compatibility when changing a plain attribute. Use @classmethod when you need more than one constructor. Use @staticmethod when the function belongs conceptually to the class but needs neither self nor cls.

Minimal example Validated property and classmethod constructor

Pythonruns
class Transaction:
    def __init__(self, txn_id, amount_paise):
        self.txn_id = txn_id
        self._amount_paise = None
        self.amount_paise = amount_paise   # triggers the setter

    @property
    def amount_paise(self):
        return self._amount_paise

    @amount_paise.setter
    def amount_paise(self, value):
        if not isinstance(value, int) or value < 0:
            raise ValueError(
                f"amount_paise must be a non-negative int, got {value!r}"
            )
        self._amount_paise = value

    @classmethod
    def from_rupees(cls, txn_id, rupees):
        """Alternative constructor: converts rupees to paise."""
        return cls(txn_id, round(rupees * 100))

    @staticmethod
    def is_high_value(amount_paise, threshold=10_000_000):
        """True if amount >= 1 lakh rupees (10_000_000 paise)."""
        return amount_paise >= threshold


t = Transaction.from_rupees("T-001", 1250.50)
print(t.amount_paise)
print(Transaction.is_high_value(t.amount_paise))
print(Transaction.is_high_value(15_000_000))

try:
    t.amount_paise = -50
except ValueError as e:
    print(f"ValueError: {e}")
Output
125050
False
True
ValueError: amount_paise must be a non-negative int, got -50

In production ETL pipeline with validated settings and multiple constructors

PythonETL pipeline built from JSON config via classmethodruns
import json
import tempfile
from pathlib import Path


class EtlPipeline:
    def __init__(self, name, batch_size, source_bucket):
        self.name = name
        self._batch_size = None
        self.batch_size = batch_size       # validated by setter
        self.source_bucket = source_bucket

    @property
    def batch_size(self):
        return self._batch_size

    @batch_size.setter
    def batch_size(self, value):
        if not isinstance(value, int) or not (1 <= value <= 50_000):
            raise ValueError(f"batch_size must be 1..50000, got {value!r}")
        self._batch_size = value

    @classmethod
    def from_dict(cls, cfg):
        return cls(cfg["name"], cfg["batch_size"], cfg["source_bucket"])

    @classmethod
    def from_json_file(cls, path):
        data = json.loads(Path(path).read_text())
        return cls.from_dict(data)

    @staticmethod
    def s3_key(bucket, prefix, date_str):
        return f"s3://{bucket}/{prefix}/{date_str}.csv"

    def __repr__(self):
        return f"EtlPipeline({self.name!r}, batch_size={self.batch_size})"


with tempfile.TemporaryDirectory() as tmpdir:
    cfg_path = Path(tmpdir) / "pipeline.json"
    cfg_path.write_text(json.dumps({
        "name": "settlements",
        "batch_size": 2000,
        "source_bucket": "dg-finance-raw",
    }))

    p = EtlPipeline.from_json_file(str(cfg_path))
    print(repr(p))
    print(EtlPipeline.s3_key(p.source_bucket, "settlements", "2026-09-26"))

    try:
        p.batch_size = 0
    except ValueError as e:
        print(f"ValueError: {e}")
Output
EtlPipeline('settlements', batch_size=2000)
s3://dg-finance-raw/settlements/2026-09-26.csv
ValueError: batch_size must be 1..50000, got 0
Scenario A

In your world

The situation

Your ETL job reads batch_size from a Jenkins parameter dict and assigns it directly: self.batch_size = int(cfg["batch_size"]). A misconfigured run passes batch_size = 0.

What actually breaks

The load loop does for i in range(0, total, 0):, which raises ValueError: range() arg 3 must not be zero — but this fires in a worker thread, not the main thread. Jenkins logs the exception buried in 3000 lines of output. The job's exit code is 0 because the main thread never checked the worker's result. The target table receives zero rows.

What you'd do on Monday

Add a @batch_size.setter that raises ValueError immediately when the object is created with an invalid value. The stack trace points to the config-loading line, not the worker thread. Jenkins marks the build as failed within the first second.

Scenario B

Imagine you work at Atlassian

200M Jira events per month

The numbers

Illustrative: 200M events/month ÷ 30 days ≈ 6.7M events/day. A single misconfigured batch_size = 0 on a hot-deployed config reload causes every in-flight pipeline worker to raise ValueError on their next batch boundary — roughly 6.7M missed events before the alert fires.

What the Scenario-A answer does here

Property validation catches the error at object creation per-worker, which is fine for a single-tenant job. But config hot-reloads that construct new pipeline objects in 64 workers simultaneously mean 64 simultaneous ValueErrors and a thundering-herd error storm if the config is pushed to all workers at once.

What flips at this scale

At this scale, config changes go through a schema validator (Pydantic (Pydantic v2) or equivalent) before they reach any worker. The property remains as a last-resort guard inside the class, not the primary validation path. Invalid configs are rejected at the deploy boundary, not inside running workers.

The trade-off you'd defend

I would defend schema-validated config at the deploy boundary plus property guards in the class, trading two validation layers for defence-in-depth: misconfigured deploys fail early, and malformed runtime state is still caught.

Illustrative; not Atlassian-published figures.

Gotchas

Calling the setter in __init__ before the backing attribute exists

Symptom

AttributeError: '_amount_paise' referenced before assignment when __init__ calls self.amount_paise = value before self._amount_paise is created.

Cause

If the setter reads self._amount_paise as part of its logic, but __init__ has not yet created _amount_paise, the setter raises on the first line.

Fix

Initialise the backing attribute first: self._amount_paise = None, then self.amount_paise = value. The setter can now safely read and overwrite the backing attribute.

@classmethod inheritance with hardcoded class names

Symptom

A subclass's classmethod constructor returns the parent class, not the subclass.

Cause

The classmethod calls ParentClass(...) directly instead of cls(...).

Fix

Use cls(...) inside every @classmethod constructor. cls refers to whatever class the method is called on, including subclasses.

Tricky questions

Can a staticmethod access instance data?
Most people say

Yes, because it is defined inside the class.

Actually

No. A @staticmethod receives no implicit first argument — not self, not cls. It can only access data passed explicitly as arguments. If you need instance data, use a regular method. If you need class state, use @classmethod.

Does using @property slow down attribute access?
Most people say

Not noticeably — Python is already slow.

Actually

A property call goes through the descriptor protocol; it is measurably slower than a plain __dict__ lookup. For objects created millions of times per second, a heavy property getter can become a bottleneck. Use @property for boundaries and validation; avoid wrapping every attribute with one.

Scenario question

You're handed a PaymentRecord class that stores amount as a plain attribute. Callers have been writing record.amount = some_value for two years. You now need to add validation that rejects negative amounts. How do you add it without breaking any existing callers?

Model answer

I would change amount to a @property with a getter that returns self._amount and a setter that validates and raises ValueError on negative values. Callers use record.amount = value exactly as before — the property is transparent to callers. The one migration step is to rename all internal direct writes from self.amount = x to self._amount = x inside the class, or to route them through the setter via self.amount = initial_value in __init__. The trade-off: property access is slightly slower than direct attribute access; for high-volume ETL row objects I would consider @dataclass with __post_init__ validation (Choose dataclass and slots deliberately) instead.

Interview corner

CommonWhat does @property let you do that a plain attribute does not?
“I use it to intercept reads and writes so I can validate on assignment, compute derived values lazily, or change the internal storage without changing the caller API. The caller still writes obj.amount = 500 — they don't know a method is running.”
TrickWhat is the difference between @classmethod and @staticmethod?
“I'd say a classmethod receives the class as cls, so it can call cls(...) and return the right subclass. A staticmethod is a plain function attached to the class for organisation — no implicit argument. I use classmethod for alternative constructors and staticmethod for pure helper logic that belongs conceptually to the class.”
Experience checkWhen did you use a classmethod constructor in production?
“I used it when the same ETL pipeline needed to be configured from a YAML file, a dict, and a JSON API response. I wrote from_dict, from_yaml and from_json classmethods. Each one parsed its format and called cls(name, batch_size, ...). Adding a new input format meant adding one classmethod, not changing four callers.”

Memory hook

A @property is validation built into the doorstep: the invalid value never enters the house. A @classmethod is a second front door that accepts a different key format.

If you remember only one thing:

Use @property when you need to validate on write or compute on read; use @classmethod for every alternative constructor.

Self-check

  1. What does a @property setter receive as its argument?

    The new value being assigned. The setter validates it, raises if invalid, and stores the result in a backing attribute (by convention self._field).

  2. Why use cls(...) instead of ClassName(...) in a classmethod constructor?

    cls refers to whichever class the method is called on, including subclasses. ClassName(...) would always return the parent class even when a subclass calls the method.

  3. What happens if you put slow database logic in a @property?

    Callers who write obj.balance expect instant access. A database query in the getter surprises them and can cause performance problems. Use a method with a verb in its name (fetch_balance()) for expensive operations instead.

Prove it

Write a BatchConfig class with a chunk_size property that rejects values outside 1–10000 and a from_env classmethod that reads BATCH_CHUNK_SIZE from os.environ (default 500 if the variable is unset). Set the env var to "200", construct via from_env, print chunk_size, then try setting chunk_size = 0 and print the exception. Paste both output lines.

py-03-04Card 4 of 6

Prefer composition where substitution fails

MediumP2 · Very likelyTalk about it10 min
compositioninheritancedelegationlspdesign

Why you care

Inheritance is the first tool most developers reach for when they want to reuse code. It works until the parent class changes or the subclass cannot honour the parent's contract. ETL pipelines that inherit from a third-party base class break silently after a library upgrade when the parent adds a new required method signature. The failure happens at runtime during a retry, not at startup, and the stack trace points into the library, not your code. Interviewers ask "composition versus inheritance" because the design choice predicts how brittle your code is when requirements change — every product company has a story about inheriting a class from a library it later regretted.

Explain like I'm 5

Think of a Swiss Army knife versus a toolbelt. The Swiss Army knife inherits all tools into one object: great until the manufacturer changes the screwdriver mechanism and every tool that shares the same handle behaves differently. A toolbelt composes independent tools: you pick exactly what you need, and swapping the screwdriver out does not affect the hammer. In Python, inheritance is the Swiss Army knife. Composition is the toolbelt: you keep a reference to another object and ask it to do its job, replacing it at any time without touching the outer class.

Core concept

The key question before using inheritance is: can every subclass be used anywhere the parent is expected, without breaking callers?

  • Liskov Substitution Principle (LSP): a subclass must be substitutable for its parent. If you find yourself overriding a parent method to raise NotImplementedError, or to change its return type, that subclass is not a valid substitute — you are using inheritance for code reuse, not for substitutability.
  • Inheritance expresses "is-a". Use it when the subclass is genuinely a more specific kind of the parent and every parent method makes sense on the subclass.
  • Composition expresses "has-a". The outer object holds a reference to an inner object and delegates calls to it. No shared namespace; no super() required.
  • Delegation: the outer class forwards a call to the inner object, optionally transforming inputs or outputs. The inner object is swappable at construction time, which makes it mockable in tests.

Decision rule: Reach for inheritance when substitutability holds and you need polymorphism through a shared interface. Prefer composition when you want to reuse behaviour without the contractual cost. When in doubt, compose first.

Minimal example Inheritance that violates LSP versus composition that does not

Pythonruns
class BaseLoader:
    def load(self, rows):
        raise NotImplementedError


class S3Loader(BaseLoader):
    def load(self, rows):
        return f"loaded {len(rows)} rows to S3"


class CsvOnlyLoader(BaseLoader):
    """Violates LSP: cannot accept rows; only file paths."""
    def load(self, rows):
        raise TypeError("CsvOnlyLoader only accepts file paths, not rows")

    def load_from_path(self, path):
        return f"loaded from {path}"


# Composition: no false is-a promise
class Pipeline:
    def __init__(self, loader):
        self._loader = loader       # has-a, not is-a

    def run(self, rows):
        return self._loader.load(rows)


p = Pipeline(S3Loader())
print(p.run(["r1", "r2", "r3"]))

try:
    CsvOnlyLoader().load(["r1"])   # LSP violation: breaks the caller's assumption
except TypeError as e:
    print(f"TypeError: {e}")
Output
loaded 3 rows to S3
TypeError: CsvOnlyLoader only accepts file paths, not rows

In production Composable ETL pipeline with injected source and validator

PythonETL stages composed via constructor injectionruns
class S3Source:
    def __init__(self, bucket, prefix):
        self.bucket = bucket
        self.prefix = prefix

    def read(self, date_str):
        # In production: boto3. Here: stub with realistic fixture data.
        return [
            {"id": "T001", "amount": "1500"},
            {"id": "T002", "amount": "800"},
        ]


class Validator:
    def __init__(self, required_keys):
        self.required_keys = set(required_keys)

    def validate(self, row):
        missing = self.required_keys - row.keys()
        if missing:
            raise ValueError(f"Row missing keys: {missing}")
        if int(row["amount"]) <= 0:
            raise ValueError(f"Non-positive amount in row {row['id']!r}")
        return True


class SettlementPipeline:
    """Composes a source and a validator; neither inherits from the other."""

    def __init__(self, source, validator):
        self._source = source
        self._validator = validator

    def run(self, date_str):
        rows = self._source.read(date_str)
        valid, bad = [], []
        for row in rows:
            try:
                self._validator.validate(row)
                valid.append(row)
            except ValueError as e:
                bad.append((row["id"], str(e)))
        return valid, bad


src = S3Source("dg-finance-raw", "settlements")
val = Validator(["id", "amount"])
pipeline = SettlementPipeline(src, val)
good, errors = pipeline.run("2026-09-26")
print(f"valid={len(good)}, invalid={len(errors)}")
print(good[0]["id"], good[1]["id"])
Output
valid=2, invalid=0
T001 T002
Scenario A

In your world

The situation

Your ETL pipeline subclasses BasePipeline from a third-party library to inherit its connection-pool and retry logic. The library releases v2, which adds a required validate_schema() method to BasePipeline.

What actually breaks

Your subclass does not implement validate_schema(). The first retry after the upgrade calls the parent's retry wrapper, which calls validate_schema() on your subclass. Python raises NotImplementedError. The first 1000 rows load fine; rows 1001 onwards fail, and the job produces a partial load with no audit flag.

What you'd do on Monday

Extract the retry helper as a composed object: self._retry = library.RetryExecutor(max_retries=3). Call self._retry.run(self._load_batch) instead of inheriting. The library upgrade affects only RetryExecutor; your pipeline class is unchanged.

Scenario B

Imagine you work at Chargebee

50M subscription events per month

The numbers

Illustrative: 50M events/month, 8 pipeline variants (trials, invoices, renewals, cancellations, etc.). All 8 subclass BasePipeline. One breaking change in the parent class during a library upgrade hits all 8 variants in the same release — a 50M-events/month disruption on a single deploy.

What the Scenario-A answer does here

Composition isolates each variant, but if each composed pipeline creates its own database connection pool, the total pool size is 8× the intended maximum, exhausting the Postgres connection limit.

What flips at this scale

At this scale you inject shared resources as constructor arguments: SubscriptionPipeline(loader=shared_loader, validator=SpecificValidator()). The shared connection pool is created once and passed in; each pipeline composes it without owning it.

The trade-off you'd defend

I would defend constructor-injected composition over inheritance-with-shared-parent-state, trading a slightly more verbose construction call for predictable dependency lifetimes and no inherited side effects when the library changes.

Illustrative; not Chargebee-published figures.

Gotchas

Inheriting from a library class couples you to its internals

Symptom

After upgrading a dependency, your class raises AttributeError or TypeError at runtime on a method you did not touch.

Cause

The library changed or added a method that your subclass now shadows or conflicts with.

Fix

Wrap the library class as a composed object. Your class calls into the library through a thin adapter; the library's internal changes do not reach your code.

Deep inheritance chains hide where a method comes from

Symptom

Pipeline.run() does not behave as expected; checking the source does not explain why.

Cause

A method three levels up the chain was overridden two levels up, in a file no one has looked at in two years.

Fix

Keep inheritance shallow (one or two levels maximum). Use ClassName.__mro__ to trace where methods actually live. For anything deeper, composition is almost always clearer.

Tricky questions

If a subclass adds new methods without overriding any parent method, is LSP satisfied?
Most people say

No — a subclass must have exactly the same public interface as the parent, so any new method breaks the contract.

Actually

Adding methods without overriding is generally fine for LSP. The risk is in __init__ overrides that change invariants, or in new methods that rely on state the parent does not set. The cleanest test: replace the subclass with the parent everywhere in your tests and verify they still pass.

Is it ever correct to raise NotImplementedError in a subclass method?
Most people say

Yes — that's what abstract base classes are for.

Actually

Raising NotImplementedError in a method the parent declares as concrete violates LSP. The right tool is an ABC with @abstractmethod (Trace MRO and define ABC contracts), where the abstract method is declared at the class level and subclasses are rejected at instantiation time if they do not implement it.

Scenario question

You're handed a codebase where PostgresLoader extends S3Loader, and the comment says "we inherit because it has nice retry logic". PostgresLoader.load() completely overrides S3Loader.load() and raises NotImplementedError on the inherited upload_to_s3 method. What do you tell the author?

Model answer

I would say this is an inheritance-for-convenience pattern that violates LSP: PostgresLoader cannot substitute for S3Loader because it raises on a method the parent declares as working. Any caller that holds an S3Loader reference and receives a PostgresLoader at runtime will hit NotImplementedError. The fix is to extract the retry logic into a RetryExecutor class and compose both loaders with it: each loader holds self._retry = RetryExecutor() and calls it directly, with no "is-a" relationship between them. The trade-off: the extraction is a refactor, not a one-line fix; I would size that against the risk of an inheritance bug surfacing in production during the next library upgrade.

Interview corner

CommonWhen would you choose composition over inheritance?
“I default to composition unless the subclass truly substitutes for the parent in every context. If I am inheriting just to reuse a method, that's a sign I should extract that method into a separate class and inject it.”
TrickA class inherits from another and overrides load() to raise NotImplementedError. Is LSP satisfied?
“I'd say no. LSP requires a subclass to be substitutable for the parent. If load() raises where the parent's raises nothing, any caller that accepted a parent reference and receives this subclass at runtime will break. The right tool is an ABC with @abstractmethod, which catches the missing implementation at class creation time.”
DesignYour team is building 10 pipeline variants that share retry and rate-limiting logic. Inheritance or composition?
“I'd go with composition and constructor injection. I'd build RetryExecutor and RateLimiter as standalone classes, inject them into each pipeline via the constructor, and share the same instance across variants that share a resource pool. Inheritance would mean a breaking change in the base class ripples to all 10 variants simultaneously.”

Memory hook

Inheritance is the Swiss Army knife: all tools share one handle, and changing the manufacturer changes every tool at once. Composition is the toolbelt: each tool is independent and swappable without touching the others.

If you remember only one thing:

Prefer composition when you want code reuse without the "is-a" contract; reach for inheritance only when every subclass can substitute for the parent without surprising callers.

Self-check

  1. What does the Liskov Substitution Principle say in one sentence?

    A subclass must be usable anywhere its parent is expected, without breaking the caller's assumptions.

  2. In the composition pattern, what is "delegation"?

    The outer class forwards a call to the inner object and optionally transforms the result, rather than inheriting the inner object's method.

  3. If you need both retry logic and S3 write capability in a new pipeline class, which approach is safer?

    Composition: inject a RetryExecutor and an S3Writer as separate objects. This avoids a false "is-a" claim and keeps each piece independently replaceable and testable.

Prove it

Write CsvLoader and JsonLoader, both with a load(rows) method returning a distinct string. Write Pipeline(loader) that composes one. Create both loaders, run Pipeline(loader).run(rows) for each, and verify that swapping the loader changes the output without changing Pipeline. Paste the two output lines.

py-03-05Card 5 of 6

Trace MRO and define ABC contracts

HardP3 · Role-dependentTalk about it11 min
mrosuperabcabstractmethodmultiple-inheritance

Why you care

A Python class that cooperatively inherits from two parents sharing a grandparent will silently double-call the grandparent's __init__ if any one class in the chain omits super().__init__(). The symptom is a bug that only appears when two mixins are combined, which happens in production but not in the isolated unit tests that test each mixin separately. ABCs catch the symmetric failure in the other direction: a class that inherits a required interface but forgets one method raises TypeError at instantiation time, not when that method is eventually called at 2 AM. Interviewers ask about MRO and ABCs together because they are the two failure modes of multiple inheritance — one is about call order, the other is about enforced contracts.

Explain like I'm 5

Think of a relay race where each runner hands the baton to the next in a fixed queue. The Method Resolution Order is the queue. super() is the handoff: if you call it, the baton reaches the next runner; if you skip it, the baton stops and the runners after you never run. An ABC is the race-entry form: you must declare each leg you will run before the race starts. Fail to declare one leg and you are not allowed on the track. Skip one handoff in the relay and the race finishes correctly in tests (when you run one leg at a time), but silently drops the final runners in the full relay.

Core concept

MRO (Method Resolution Order): the ordered list of classes Python searches when looking up a method. ClassName.__mro__ exposes it. Python uses the C3 linearization algorithm, which guarantees local precedence order (left-to-right base order is preserved) and monotonicity (if A appears before B in any base's MRO, A appears before B in the derived MRO).

super() does not call the parent — it calls the next class in the MRO. In diamond inheritance, calling super() in every class in the chain is the only way to ensure each class's method is called exactly once.

ABCs (Abstract Base Classes): from abc import ABC, abstractmethod. A class that inherits ABC and marks methods with @abstractmethod cannot be instantiated until all abstract methods are implemented. The TypeError fires at instantiation, not at the method call site, which is always earlier and more useful.

You can also register a class as a virtual subclass of an ABC with MyABC.register(ThirdPartyClass). This affects isinstance checks without enforcing abstract method contracts.

Decision rule: Use super() consistently in every __init__ when multiple inheritance is in play. Use ABCs when you need an interface contract that fails loudly at class creation, not at call time. If your mixin hierarchy grows beyond three levels, favour composition (Prefer composition where substitution fails).

Under the hood

CPython's C3 algorithm merges several MRO sequences by repeatedly taking the head of a candidate list if that head does not appear in the tail of any other candidate. For class D(B, C) where B and C both inherit from A, the merge starts with [B, A, object], [C, A, object], [B, C]. Step 1: take B — B is not in any tail. Step 2: take C — A is in tails so skip; C is safe. Step 3: take A. Step 4: take object. Result: D → B → C → A → object. If the merge hits a conflict (every remaining head appears in a tail), Python raises TypeError: Cannot create a consistent method resolution order at class definition time, not at runtime. This means an inconsistent hierarchy fails immediately when the class is defined, which is the earliest possible signal.

Minimal example MRO, cooperative super, and ABC

Pythonruns
from abc import ABC, abstractmethod


class A:
    def process(self):
        print("A.process")


class B(A):
    def process(self):
        print("B.process")
        super().process()


class C(A):
    def process(self):
        print("C.process")
        super().process()


class D(B, C):
    def process(self):
        print("D.process")
        super().process()


print([cls.__name__ for cls in D.__mro__])
D().process()
Output
['D', 'B', 'C', 'A', 'object']
D.process
B.process
C.process
A.process

In production ABC-enforced loader interface

PythonABC catches a missing method at instantiation, not at call timeruns
from abc import ABC, abstractmethod


class DataLoader(ABC):
    @abstractmethod
    def read(self, date_str):
        ...

    @abstractmethod
    def close(self):
        ...

    def read_safe(self, date_str):
        """Template method: all subclasses inherit this for free."""
        try:
            return self.read(date_str)
        finally:
            self.close()


class CsvLoader(DataLoader):
    def read(self, date_str):
        return [{"id": "T001", "date": date_str}]

    def close(self):
        pass


class BrokenLoader(DataLoader):
    """Implements read() but not close()."""
    def read(self, date_str):
        return []
    # close() is missing


loader = CsvLoader()
print(loader.read_safe("2026-09-26"))

try:
    bad = BrokenLoader()
except TypeError as e:
    print(f"TypeError: {e}")
Output
[{'id': 'T001', 'date': '2026-09-26'}]
TypeError: Can't instantiate abstract class BrokenLoader without an implementation for abstract method 'close'

See it happen MRO resolves the diamond without double-calling A

DBCAsuper().process()super().process()super().process()returnreturnreturn
  1. D calls super(); the MRO says next is B, not A directly.
  2. B calls super(); the MRO says next is C, still not A.
  3. C calls super(); now the MRO reaches A.
  4. A.process() runs once and returns. No double-call of A.
  5. C returns to B after A completes.
  6. B returns to D. Every class in the MRO ran exactly once.
Notice: super() means "next in this object's MRO", not "parent". That is why B delegates to C rather than directly to A, even though B's only declared base is A.
Scenario A

In your world

The situation

Your team adds a TimestampMixin and an AuditMixin to a shared BasePipeline. All three implement __init__ and call super().__init__(). A new engineer adds a CacheMixin that initialises a cache dict in __init__ but does not call super().__init__().

What actually breaks

All pipelines that include CacheMixin skip every __init__ below it in the MRO, including BasePipeline.__init__(). The connection pool is never initialised. The first database call raises AttributeError: 'Pipeline' object has no attribute 'conn'. The unit test for CacheMixin alone passes because it tests the mixin in isolation, not in the combined class.

What you'd do on Monday

Add super().__init__(*args, **kwargs) to CacheMixin.__init__(). Check ClassName.__mro__ to verify the full chain. Write an integration test that constructs the combined class and asserts that conn is not None.

Scenario B

Imagine you work at Urban Company

10M service bookings per month

The numbers

Illustrative: 10M bookings/month, each object passing through 4 processing stages, each a class with 2 mixins = 8 classes to audit per deployment. A cooperative-super break that fires in 0.1% of bookings silently drops 10,000 bookings per month.

What the Scenario-A answer does here

Fixing the cooperative chain catches the break. But a hierarchy of 8 classes means any new mixin must be audited before deployment; the cost grows with team size.

What flips at this scale

At this scale you replace deep mixin hierarchies with ABCs plus composition: the ABC defines the interface, and behaviour lives in injected helper objects. There is no cooperative super chain to break; the ABC rejects incomplete implementations at class creation time.

The trade-off you'd defend

I would defend ABCs plus composition over deep cooperative-super hierarchies, trading the convenient code reuse of mixins for a design where a missing implementation raises at class creation, not in a 0.1% production corner case.

Illustrative; not Urban Company-published figures.

Gotchas

One class in the chain does not call super().__init__

Symptom

Attributes set in the classes below the break are never initialised. Code passes unit tests that test each class alone, but fails when the combined class is used.

Cause

super().__init__() passes control to the next class in the MRO. Skipping it terminates the chain at that point.

Fix

Add super().__init__(*args, **kwargs) in every class in the cooperative hierarchy. Pass **kwargs so unexpected arguments can flow through to object.__init__() without raising TypeError.

TypeError: Cannot create a consistent method resolution order

Symptom

The class definition itself raises TypeError: Cannot create a consistent method resolution order (MRO) for bases ....

Cause

The inheritance graph has a contradiction that C3 cannot resolve — for example, class D(A, B) and class E(B, A) combined in class F(D, E).

Fix

This is a design error; restructure the inheritance graph. Usually this means extracting one of the conflicting classes into a mixin and a separate base.

Tricky questions

Does super() always call the immediate parent?
Most people say

Yes — super() is short for "parent class".

Actually

super() calls the next class in the MRO, which is not necessarily the immediate parent. In class D(B, C) with MRO D, B, C, A, object, super() called inside B resolves to C, not A.

Can you use an ABC as a type check without inheriting from it?
Most people say

No — isinstance checks require actual inheritance.

Actually

Yes. MyABC.register(SomeClass) makes isinstance(obj, MyABC) return True even if SomeClass does not inherit from MyABC. Abstract method contracts are not enforced on registered classes — only on direct subclasses.

Scenario question

You're handed a mixin-heavy codebase with ReportPipeline(TimestampMixin, RetryMixin, BaseLoader). A type checker flags BaseLoader.connect as possibly None in 10% of calls, and logs confirm occasional AttributeError: 'NoneType' object has no attribute 'execute'. How do you diagnose and fix this?

Model answer

I would print ReportPipeline.__mro__ and add a log statement to each __init__ to trace the chain. The type checker flagging connect as possibly None means __init__ sometimes exits without setting it — the most likely cause is a mixin that does not call super().__init__(). I would find that mixin, add super().__init__(*args, **kwargs), and write an integration test that constructs ReportPipeline and asserts pipeline.connect is not None. The trade-off: this is the third time the chain has been broken if the error recurs; I would propose migrating to ABCs plus composition for this class, where a missing implementation raises at class creation rather than in a rare runtime path.

Interview corner

CommonWhat does super() return?
“I'd describe it as a proxy that delegates method calls to the next class in the MRO — not necessarily the direct parent. In a diamond inheritance, calling super() in each class is how every class in the chain gets called exactly once.”
TrickYou add a mixin to a class and suddenly one attribute is None. What do you check first?
“I'd print ClassName.__mro__ and add a print to each __init__ to see which order they fire. My first suspicion is a mixin that does not call super().__init__(), which breaks the cooperative chain and leaves attributes in classes below it uninitialised.”
DesignWhen would you use an ABC instead of duck typing?
“I use an ABC when I want a class-creation failure rather than a runtime AttributeError. If a team is writing five loader implementations and one forgets close(), I would rather that error fires when they define the class than when the first cleanup path is exercised in production at 2 AM.”

Memory hook

super() hands the baton to the next runner in the MRO queue. Skip one handoff and every runner after it stands still.

If you remember only one thing:

In a cooperative inheritance chain, every __init__ must call super().__init__(). Check ClassName.__mro__ to see the full queue.

Self-check

  1. What does ClassName.__mro__ contain?

    A tuple of classes in the order Python searches for methods — starting with the class itself and ending with object.

  2. Why does an ABC raise TypeError at instantiation rather than at call time?

    Python checks at instantiation whether all @abstractmethod methods are implemented. An unimplemented abstract method means the class is incomplete; Python refuses to create an instance.

  3. What is the difference between inheriting from an ABC and registering with it?

    Inheriting enforces implementation of all abstract methods. Registering with register() only affects isinstance checks; it does not enforce any method contracts.

Prove it

Create class Diamond(B, C) where B and C each inherit from A. Have each class print its name in __init__ and call super().__init__(). Print Diamond.__mro__ and then construct Diamond() — each class name should print once. Then remove super().__init__() from C and show that A.__init__ no longer prints. Paste both outputs.

py-03-06Card 6 of 6

Choose dataclass and slots deliberately

MediumP3 · Role-dependentTalk about it10 min
dataclassslotsfrozenstate-representation

Why you care

A plain Python class with six fields and a hand-written __init__, __repr__ and __eq__ is 40 lines. A @dataclass with three options reduces it to 8 and keeps every field in the class signature where the type checker can see it. The reduction is not cosmetic: every hand-written __repr__ that falls out of sync with a field change is a debugging session. The memory argument is more direct: a 2M-row ETL job that stores each row as a plain class object can consume 800 MB in __dict__ overhead before counting the actual field values. @dataclass(slots=True) removes that overhead. Interviewers ask about @dataclass to see whether you treat Python as a scripting language or as a typed, tool-assisted language; they ask about __slots__ to check whether you have thought about memory at all.

Explain like I'm 5

Think of house blueprints. A regular class is like a house where rooms can be added after construction by setting obj.new_field = value — the builder left a spare key under the mat. A class with __slots__ is built to fixed plans: the rooms are declared up front, no additions are allowed, and because the builder knew the floor plan the house uses less concrete. A @dataclass is a pre-printed blueprint form: you fill in the room names and the contractor auto-writes all the standard wiring — plumbing for __init__, electrical for __repr__, fire alarm for __eq__. @dataclass(slots=True) is both: the pre-printed form and the fixed floor plan.

Core concept

@dataclass generates __init__, __repr__ and __eq__ from annotated class variables. Key options:

  • frozen=True: makes the instance immutable (sets __setattr__ and __delattr__ to raise FrozenInstanceError) and adds __hash__. Use for value objects that go into sets and dict keys (Give objects coherent protocols).
  • slots=True (Python 3.10+, stable in 3.12): generates __slots__ automatically. Instances use a fixed-layout struct instead of a __dict__. No arbitrary attribute assignment after creation.
  • order=True: generates __lt__, __le__, __gt__, __ge__ from field order, enabling sorted().

The mutable default trap: tags: list = [] raises ValueError: mutable default <class 'list'> for field tags is not allowed: use default_factory at class definition time. Use field(default_factory=list) instead. This is the same root trap as mutable function defaults (Replace shared mutable defaults).

__slots__ on plain classes: define __slots__ = ('x', 'y') in the class body. Instances have no __dict__; only declared attributes can be set. Subclasses must also declare __slots__ or they silently reacquire __dict__ and lose the savings.

When not to use @dataclass: if you need coercion (accept a string, store an int), cross-field validation, or rejection of unknown fields from untrusted input, use Pydantic (Pydantic v2). @dataclass is a code-generation tool, not a validator.

Decision rule: Use @dataclass for any class that primarily holds data. Add frozen=True for immutable records you hash. Add slots=True when creating millions of instances. Use Pydantic when you need validation and coercion of external data.

Minimal example dataclass essentials and the mutable-default trap

Pythonruns
from dataclasses import dataclass, field


@dataclass
class Row:
    txn_id: str
    amount: int
    tags: list = field(default_factory=list)   # each instance gets its own list


r1 = Row("T001", 500)
r2 = Row("T002", 300)
r1.tags.append("high-value")
print(r1.tags, r2.tags)    # r2 has its own empty list

@dataclass(frozen=True)
class ImmutableRow:
    txn_id: str
    amount: int


ir = ImmutableRow("T001", 500)
try:
    ir.amount = 999
except Exception as e:
    print(f"{type(e).__name__}: {e}")

@dataclass(slots=True)
class SlottedRow:
    txn_id: str
    amount: int


sr = SlottedRow("T001", 500)
print(hasattr(r1, '__dict__'), hasattr(sr, '__dict__'))
Output
['high-value'] []
FrozenInstanceError: cannot assign to field 'amount'
True False

In production Memory-conscious ETL row model for a 2M-row job

PythonSlotted frozen dataclass avoids per-instance __dict__ overheadruns
from dataclasses import dataclass
import sys


@dataclass(slots=True, frozen=True)
class SettlementRow:
    txn_id: str
    amount_paise: int
    currency: str
    source: str


@dataclass
class NaiveRow:
    txn_id: str
    amount_paise: int
    currency: str
    source: str


slotted = SettlementRow("T001", 150000, "INR", "settlements")
naive = NaiveRow("T001", 150000, "INR", "settlements")

# Slotted: all storage is inline; no separate __dict__ allocation.
# Naive: __dict__ is a separate object — the dominant overhead at scale.
naive_total = sys.getsizeof(naive) + sys.getsizeof(naive.__dict__)
slotted_total = sys.getsizeof(slotted)

print(hasattr(slotted, '__dict__'), hasattr(naive, '__dict__'))
print(slotted_total < naive_total)
Output
False True
True
Scenario A

In your world

The situation

Your ETL job creates a RowModel object for each row in a 2M-row CSV. The class has 6 fields and a hand-written __init__. Peak RAM on the Jenkins box is 3.2 GB on a 4 GB machine.

What actually breaks

The job intermittently OOMs when another service runs in parallel. The CPU profiler shows 60% of time in __init__ and attribute lookup. The __dict__ for each of the 2M objects contributes roughly 296 bytes of overhead on CPython 3.12, independent of the field values: 2M × 296 bytes ≈ 592 MB consumed by dictionaries alone.

What you'd do on Monday

Convert to @dataclass(slots=True). Slots remove the per-instance __dict__, reducing each object's overhead to just the inline slot storage. Freed memory lowers peak RAM by 400–500 MB, keeping the job under 2.5 GB.

Scenario B

Imagine you work at MakeMyTrip

100M booking records per year

The numbers

Illustrative: 100M records/year ÷ 365 ≈ 274k records/day. A nightly reconciliation loads 10M records into memory for cross-matching. 10M × 296 bytes dict overhead (naive) ≈ 2.96 GB dict cost alone; 10M × 64 bytes slot overhead (slotted) ≈ 640 MB. The difference determines whether you need a 4 GB machine or can stay under 2 GB.

What the Scenario-A answer does here

@dataclass(slots=True) fixes the RAM spike for a single model. But constructing 10M objects still costs CPU time, and a naive in-memory load holds all 10M objects simultaneously.

What flips at this scale

At this scale you do not load 10M full objects into RAM at all. Reconciliation runs as a SQL join in Postgres, or as a streaming row-by-row compare using the iterator protocol (Stream a 20 GB file within a memory budget). Python objects appear only for the output discrepancy report, not the full comparison.

The trade-off you'd defend

I would defend database-level reconciliation over in-memory object loading for anything above 1M rows, trading Python control for set-based SQL that runs in constant memory and uses database indexes. Slots are the right call for sub-million batch windows that must stay in Python.

Illustrative; not MakeMyTrip-published figures.

Gotchas

Mutable default raises at class definition, not at runtime

Symptom

ValueError: mutable default <class 'list'> for field tags is not allowed: use default_factory when the module is imported — before any instance is created.

Cause

@dataclass detects the mutable default at class-body evaluation time and refuses to silently share it the way plain classes would.

Fix

Replace tags: list = [] with tags: list = field(default_factory=list). Each instance gets a fresh list from the factory.

slots=True and inheritance: subclasses must also declare __slots__

Symptom

A subclass of a slotted dataclass has a __dict__ even though the parent does not. The memory savings disappear for subclass instances.

Cause

If a subclass does not declare __slots__ (or use slots=True in its own @dataclass), Python adds __dict__ back to the subclass.

Fix

Every class in the inheritance chain must declare __slots__ or use @dataclass(slots=True). Check hasattr(SubclassInstance, '__dict__') after each change.

Tricky questions

Does @dataclass(frozen=True) make the object deeply immutable?
Most people say

Yes — you cannot change any value inside the object.

Actually

frozen=True only prevents reassignment of top-level fields. If a field holds a mutable object like a list, the list itself can still be mutated: row.tags.append("x") succeeds even on a frozen dataclass. Only the reference row.tags = new_list is blocked.

What is the difference between dataclass and Pydantic for ETL models?
Most people say

They are equivalent; use whichever has better syntax.

Actually

@dataclass is code generation — it creates __init__ and friends but does no validation or coercion. Pydantic validates types, coerces strings to int, and rejects unknown fields from untrusted input. Use @dataclass for internal models where you control the data; use Pydantic at the system boundary where external data arrives (Pydantic v2).

Scenario question

You're handed a Python ETL job that loads 5M customer records into memory each night for deduplication. Memory usage is 8 GB and the machine has 10 GB. Adding one more data source will push it over the limit. The model class is a plain class with 8 fields. What are your options and which do you recommend?

Model answer

I see three options. First, convert to @dataclass(slots=True, frozen=True): removes per-instance __dict__ overhead and adds __hash__ for set deduplication — likely saves 1–2 GB. Second, move deduplication to Postgres using ON CONFLICT DO NOTHING or a UNIQUE index, eliminating the in-memory model entirely. Third, stream records in bounded chunks using the iterator protocol and deduplicate per-chunk against a Bloom filter in the database. I would recommend option 2 if the Postgres schema allows it: it solves the problem at the data layer, scales without a memory ceiling, and handles process restarts gracefully. Option 1 is a quick win that buys time; option 3 is the right architecture for a streaming requirement. The trade-off for option 2 is a more complex SQL schema and a migration; I would size that against the risk of the next data source push crossing the memory limit.

Interview corner

CommonWhat does @dataclass generate for you?
“I'd say it generates __init__, __repr__ and __eq__ from the annotated fields. With frozen=True it also adds __hash__ and makes the instance immutable. With slots=True it removes __dict__ and reduces per-instance memory.”
TrickCan you use a list as a default field value in a dataclass?
“I'd say no — @dataclass raises ValueError at class definition time if you use a mutable default directly. You use field(default_factory=list) to give each instance its own fresh list. This is the same rule as mutable function defaults, just caught earlier.”
DesignA team stores 10M rows in memory for nightly deduplication. Memory is a problem. What do you suggest?
“I'd ask whether the deduplication can move to the database — a UNIQUE index and ON CONFLICT DO NOTHING handles it at the insert layer with no Python memory cost. If it must stay in Python, I'd convert the row model to @dataclass(slots=True, frozen=True) to remove __dict__ overhead, and then switch to streaming with bounded windows rather than loading all 10M at once.”

Memory hook

@dataclass is the blueprint form that auto-writes the boilerplate. slots=True is the fixed floor plan that eliminates the __dict__ concrete.

If you remember only one thing:

Use @dataclass for data-holding classes, frozen=True for immutable records, slots=True for millions of instances, and Pydantic when the data arrives from outside.

Self-check

  1. What does @dataclass(frozen=True) add that @dataclass alone does not?

    It prevents field reassignment by raising FrozenInstanceError and adds __hash__ so instances can go into sets and dict keys.

  2. Why does @dataclass raise ValueError on tags: list = []?

    It detects the mutable default at class-body evaluation time and refuses to share one list across all instances — it catches the same bug that mutable function defaults create, but immediately.

  3. What happens to __dict__ on a @dataclass(slots=True) instance?

    There is no __dict__. All attribute storage uses the declared slots. Arbitrary attribute assignment after creation raises AttributeError.

Prove it

Create @dataclass and @dataclass(slots=True) versions of the same 4-field row class. Construct one instance of each. Print hasattr(instance, '__dict__') for both. Then compute naive_total = sys.getsizeof(naive) + sys.getsizeof(naive.__dict__) and slotted_total = sys.getsizeof(slotted), and print slotted_total < naive_total. Paste the three output lines.