Separate class and instance state
Why you care
The most common OOP bug in Python ETL code is not a crash: it is a class-level list that silently accumulates data across every run because the list was put on the class, not on self. The job is green, the counts look plausible, and the error is invisible until someone compares two runs side by side. Interviewers ask "what is the difference between a class attribute and an instance attribute" because the answer predicts whether you have built that bug yet. It also gates everything else in OOP: properties, descriptors and __slots__ all depend on where Python looks for an attribute first.
Explain like I'm 5
Think of a company's HR policy document versus each employee's personal HR file. The policy document sits on a shared shelf: everyone in the company reads from the same copy, and if you write on it you are writing for everyone. Each employee's personal file lives in their own desk drawer: what one employee writes goes into their file alone. In Python, class attributes are the shared shelf. Instance attributes are the personal files created when each employee (object) joins (is __init__-ialised). Handing someone your bookmark on the shared shelf when they asked for their personal file is the bug.
Core concept
Python looks up an attribute on an object in two stages: first the instance's own namespace (obj.__dict__), then the class namespace. This gives class attributes their shared-by-default quality.
- Class attributes are defined directly in the class body, outside any method. Every instance sees the same value unless it creates its own copy.
- Instance attributes are created inside methods by assigning to
self.attr. They live in the instance's__dict__and shadow any class attribute with the same name. self.x = valuealways writes to the instance's namespace. It never modifies the class attribute, even when the class has one namedx. It creates a shadow.self.items.append(value)whenitemsis a class-level list does not create a shadow — it mutates the shared list in-place, because readingself.itemsfound the class attribute and appended to it.- To modify shared class state intentionally, write
ClassName.attr = new_valueexplicitly. The explicit class name signals intent.
Decision rule: Class attributes are correct for constants and intentional shared registries. Everything that must differ per instance goes in __init__ via self.
Minimal example The shared-list trap and its shadow
class Counter:
count = 0 # class attribute — shared
items = [] # class attribute — shared mutable: DANGER
def __init__(self, name):
self.name = name # instance attribute
c1 = Counter("first")
c2 = Counter("second")
c1.count = 99 # creates instance attr on c1; class attr stays 0
Counter.count += 1 # modifies the class attr to 1
c1.items.append("a") # mutates the shared list — NOT a shadow
print(c1.count, c2.count, Counter.count)
print(c1.items, c2.items)
print("count" in c1.__dict__, "count" in c2.__dict__)99 1 1
['a'] ['a']
True FalseIn production Per-pipeline error lists
Each EtlPipeline gets its own errors list because it is created in __init__. The class-level _registry is shared intentionally — it is the lookup table that all pipelines write into.
class EtlPipeline:
_registry = {} # intentional shared registry
def __init__(self, name, batch_size=1000):
self.name = name
self.batch_size = batch_size
self.errors = [] # per-instance; fresh every time
@classmethod
def register(cls, name, obj):
cls._registry[name] = obj
def log_error(self, msg):
self.errors.append(msg)
def __repr__(self):
return f"EtlPipeline({self.name!r}, batch_size={self.batch_size})"
sales = EtlPipeline("sales", batch_size=500)
hr = EtlPipeline("hr", batch_size=2000)
EtlPipeline.register("sales", sales)
EtlPipeline.register("hr", hr)
sales.log_error("row 3: missing amount")
hr.log_error("row 1: invalid employee_id")
print(list(EtlPipeline._registry))
print(sales.errors)
print(hr.errors)['sales', 'hr']
['row 3: missing amount']
['row 1: invalid employee_id']See it happen How Python resolves a name on an instance
- Python reads c1.count: checks instance __dict__ first.
- 'count' is in c1.__dict__ (value 99) so the search stops. The class attribute is never seen.
- Now read c2.count: c2.__dict__ has no 'count', so Python falls through to the class.
- Writing self.count = 99 always lands in the instance __dict__. The class attribute is never modified.
In your world
The situation
Your nightly Jenkins job creates a PipelineRun per source file — sales.csv first, then hr.csv — and emails each object's error list as a separate report. You added errors = [] at the class level, expecting each object to start fresh.
What actually breaks
The report for hr.csv contains both the hr errors and the sales errors that were logged earlier in the same run. Finance sees duplicate and misattributed errors in the hr report and files a ticket. The job exits green because no exception was raised.
What you'd do on Monday
Move errors = [] into __init__ as self.errors = []. The sales object and the hr object each receive their own list at construction time. The hr report now contains only hr errors.
Imagine you work at Stripe
The numbers
Illustrative: 50M Payment objects/day. If each object carries a class-level mutable counter and 0.01% trigger an error callback that appends to it, 5,000 appends/day accumulate in a structure pinned to the class for the process lifetime.
What the Scenario-A answer does here
Moving errors to __init__ fixes correctness. But 50M objects each with a full object-plus-dict footprint of ~344 bytes (48 B object + 296 B __dict__, CPython 3.12); 50M × 344 bytes ≈ 17 GB of object memory before the actual field values.
What flips at this scale
At this scale, high-volume row objects declare fields in __slots__ or use @dataclass(slots=True) (Choose dataclass and slots deliberately), eliminating __dict__ entirely. Errors go to a structured log or a DB column, not a list on any object.
The trade-off you'd defend
I would defend slots-only objects for ETL rows over dicts, trading the ability to attach arbitrary attributes at runtime for predictable, bounded per-object memory. Every field that might be needed is declared at design time.
Illustrative; not Stripe-published figures.
Gotchas
A class-level list shared across every instance
Two pipeline objects accumulate each other's errors. Errors from previous runs appear in the current run's report.
errors = [] in the class body creates one list shared by every instance. self.errors.append(msg) finds and appends to the class-level list, not an instance-level one.
Move the definition into __init__: self.errors = []. Now each instance gets its own list at creation time.
Assigning to self shadows without updating shared state
A class counter incremented via self.count += 1 inside a method never changes the class-level value.
self.count += 1 reads the class attribute (0), computes 1, then writes 1 to the instance's namespace, leaving the class attribute unchanged.
To modify shared class state, write ClassName.count += 1 explicitly. To track per-instance state, initialise in __init__.
Tricky questions
self.attr = value ever change the class attribute?Yes, if the class already has an attribute with that name, the assignment updates it.
No. self.attr = value always writes to the instance's __dict__, creating a shadow. The class attribute retains its value. Only ClassName.attr = value modifies the class attribute.
self.attr, which namespace does Python check first?The class namespace, because attributes are defined on the class.
Python checks the instance __dict__ first. If not found there, it checks the class (and then its bases). This is why an instance attribute "shadows" a class attribute of the same name.
Scenario question
You're handed a report that says a nightly ETL job loaded 0 rows on Tuesday but 48,213 on Monday and Wednesday. The code uses a class-level rows_loaded = 0 counter incremented inside the load loop via self.rows_loaded += 1. How do you explain the zero-row Tuesday, and what do you change?
Model answer
I would check whether a second pipeline run on Tuesday read ClassName.rows_loaded as a guard condition, expecting the count from the first run's object. Because self.rows_loaded += 1 creates an instance shadow, ClassName.rows_loaded stays 0 forever. If the second run checked if ClassName.rows_loaded > 0: skip_load(), it would skip the entire load. The fix is to use instance-level counters throughout: read and log from self.rows_loaded, and reset by constructing a new instance per run. The trade-off: if you genuinely need cross-instance aggregation, use a classmethod to accumulate into cls.rows_loaded intentionally and document why.
Interview corner
CommonWhat is the difference between a class attribute and an instance attribute?
self in __init__ and lives in that instance's __dict__. The danger is a mutable class attribute like a list: every instance that appends to it is appending to the same list.”TrickYou do self.count += 1 in a method. Does this change the class attribute?
self.count += 1 reads the class attribute, adds 1, and writes the result to the instance's namespace. The class attribute is unchanged. To increment shared state I write ClassName.count += 1.”Experience checkHave you hit a class-state bug in real code?
errors = [] was on the class. The second PipelineRun object created in the same process appended its errors to the same list, so the second report contained both files' errors. Moving it to self.errors = [] in __init__ fixed it immediately.”Memory hook
The class attribute is the shared shelf; self.attr = value always writes to the employee's personal drawer, never back to the shelf.
If you want per-instance state, put it in __init__ via self. Anything defined in the class body is shared by default.
Self-check
If you add
c.x = 5on an instance, does it modify the class?No. It creates a new entry in
c.__dict__.ClassName.xis unchanged; readingc.xreturns5because the instance namespace is checked first.How do you check whether an attribute is in the instance's own namespace?
"attr" in obj.__dict__isTrueif the instance has its own copy,Falseif it is only inherited from the class.What is the safe way to modify shared class state inside a method?
Write
ClassName.attr = new_valueusing the explicit class name, notself, to signal that you are intentionally writing to shared state.
Write a class RunLog with a class-level runs = 0 counter and an instance-level errors = [] list. Create two instances. In the first, call a method that does self.runs += 1. In the second, call RunLog.runs += 1. Print RunLog.runs, instance1.__dict__["runs"], and show that "runs" is not in instance2.__dict__. Paste the three output lines.