Shelf · Python · Chapter 9

Chapter 9: Memory, performance & profiling

You cannot fix what you have not measured. This chapter gives you three tools — refcount-aware object design, a profiler for CPU, and a tracer for allocations — and the rule that makes all three useful: fix the algorithm before you touch the constants.

Difficulty
Priority
py-09-01Card 1 of 6

Explain object retention and collection

MediumP3 · Role-dependentTalk about it13 min
memoryrefcountinggccyclesobject-lifetime

Why you care

Every Python process leaks memory at some point, and the most common cause is a name you believe you deleted that is still alive because something else holds a reference. In ETL jobs that run for hours, a steadily growing resident set eventually triggers the OOM killer, killing the build at 3 AM. Interviewers ask about refcounting and the GC because the answer reveals whether you understand why del x does not always free memory, and why deterministic cleanup belongs in a context manager rather than __del__. This card is the foundation; Trace allocation growth with tracemalloc shows you how to measure whether the pattern fixed the problem.

Explain like I'm 5

Imagine a library that keeps a loan counter on each book. When you borrow it, the counter goes up by one. When you return it, it goes down. When the counter hits zero, the librarian puts the book back on the shelf immediately — no waiting. But if you and your friend borrow copies that each cite the other book, neither counter ever reaches zero, even after both of you leave. The librarian hires a separate inspector (the cyclic GC) who comes by periodically, spots the pair, and reclaims both books.

Core concept

CPython uses reference counting as its primary memory strategy. Every object carries a reference count. When you do y = x, both names point at the same object and the count rises. When a name falls out of scope or is explicitly deleted, the count drops. When it hits zero, CPython frees the object immediately — no pause, no waiting for a GC cycle.

Three things increase a reference count you might not expect:

  • A container (list, dict, tuple) referencing the object.
  • A closure cell capturing a variable from an enclosing scope.
  • A frame on the call stack holding a local variable.

Reference counting alone cannot free cycles: a.next = b; b.prev = a; del a, b leaves both objects alive because each still holds a reference to the other. CPython's cyclic GC (gc module) collects these. It runs automatically in three generations and is the reason you cannot rely on __del__ for resource cleanup — the GC may call it seconds or minutes after the last name is gone. Use a context manager (with / __exit__) or contextlib.closing (see Context managers and deterministic cleanup) instead.

Decision rule. Use __slots__ on classes that will be instantiated millions of times (each measured instance without slots costs 344 bytes: 48 for the object + 296 for the __dict__; with __slots__, the same six attributes cost 80 bytes). Use context managers for every resource. Only call gc.collect() explicitly in tests or after bulk-deleting known cycles.

Note on builds: reference counting is a CPython implementation detail. Jython, PyPy and the free-threaded 3.13+ build all handle object reclamation differently. On CPython 3.12 (the target runtime here), refcount is the primary path.

Under the hood

CPython stores the reference count in the ob_refcnt field of PyObject. The interpreter increments it with Py_INCREF and decrements with Py_DECREF. When Py_DECREF drops the count to zero it calls the type's tp_dealloc slot immediately, in the same C stack frame — no scheduling. This is what makes CPython's with block the only reliable resource-cleanup primitive: __exit__ is called before the next Python instruction runs.

sys.getrefcount(x) always reports one more than you expect because passing x to the function itself creates a temporary reference in the call frame.

The cyclic GC (enabled by default, configurable with gc.disable()) tracks objects that can participate in cycles. User-defined class instances — with or without __slots__ — are tracked; gc.is_tracked(obj) returns True for both. Adding __slots__ does not remove the instance from GC tracking. What __slots__ does remove is the per-instance __dict__, which is itself a GC-tracked container: switching to __slots__ eliminates one tracked object per instance, not the instance itself. Objects that truly cannot hold references — int, str, float, bytes — are untracked by the GC.

Minimal example Delete a name; observe when memory is actually freed

PythonReference count and cycle demonstrationruns
import sys, gc

class Node:
    def __init__(self, v):
        self.val = v
        self.sibling = None

a = Node('a')
b = Node('b')

# Refcount on a: 1 from 'a', 1 inside getrefcount call = reports 2
print('a refcount:', sys.getrefcount(a))  # 2

a.sibling = b   # b's count rises
b.sibling = a   # a's count rises — cycle created

gc.collect()                     # run before to baseline
unreachable_before = gc.garbage  # should be empty

del a, b                         # names gone, but cycle holds both alive
collected = gc.collect()         # GC finds and frees the pair
print('GC collected cycle objects:', collected >= 2)
print('gc.garbage is empty:', gc.garbage == [])
Output
a refcount: 2
GC collected cycle objects: True
gc.garbage is empty: True

In production Slots versus __dict__ for a high-volume order class

PythonSix-attribute Order class: __dict__ vs __slots__ (stdlib only)runs
import sys

class OrderDict:
    """Baseline: attributes stored in a per-instance __dict__."""

class OrderSlots:
    """Optimised: no __dict__, no __weakref__."""
    __slots__ = ('order_id', 'amount', 'currency', 'status', 'created_at', 'user_id')


def make_dict(i):
    o = OrderDict()
    o.order_id = i
    o.amount = float(i * 10)
    o.currency = 'INR'
    o.status = 'SETTLED'
    o.created_at = 1_700_000_000 + i
    o.user_id = i % 10_000
    return o


def make_slots(i):
    o = OrderSlots()
    o.order_id = i
    o.amount = float(i * 10)
    o.currency = 'INR'
    o.status = 'SETTLED'
    o.created_at = 1_700_000_000 + i
    o.user_id = i % 10_000
    return o


d = make_dict(1)
s = make_slots(1)

dict_bytes = sys.getsizeof(d) + sys.getsizeof(d.__dict__)
slot_bytes = sys.getsizeof(s)
saving = dict_bytes - slot_bytes

print(f'__dict__ instance: {sys.getsizeof(d)} B + dict {sys.getsizeof(d.__dict__)} B = {dict_bytes} B')
print(f'__slots__ instance: {slot_bytes} B')
print(f'saving per object: {saving} B')
# For 5 000 000 orders: 264 * 5_000_000 = 1_320_000_000 B / 1_048_576 ≈ 1 258.9 MiB (floored below)
total_saving_mb = saving * 5_000_000 // (1024 * 1024)
print(f'saving for 5 M objects: {total_saving_mb} MB')
Output
__dict__ instance: 48 B + dict 296 B = 344 B
__slots__ instance: 80 B
saving per object: 264 B
saving for 5 M objects: 1258 MB

See it happen Refcount rises and falls through the object lifetime

codeCPython refcntcyclic GCa = Node('a')b = adel bdel aa.sibling = b; b.sibling = adel a, del bgc.collect()collected = 2
  1. Object created, refcount set to 1.
  2. Second name added; refcount rises to 2.
  3. One name removed; refcount drops to 1. Object stays alive.
  4. Last name removed; refcount drops to 0. tp_dealloc called immediately.
  5. Cycle created: a and b each hold a reference to the other.
  6. Both names deleted, but cycle keeps both alive — refcounts stay at 1.
  7. GC detects the unreachable cycle and schedules both for deallocation.
  8. Both objects freed. Without gc.collect(), they would wait for the next automatic generation sweep.
Notice: objects in a cycle are invisible to the refcount mechanism — only the cyclic GC can reclaim them.
Scenario A

In your world

The situation

Your hourly ETL job reads supplier quotes from S3, parses them into a list of plain Python objects, and appends each batch to a module-level cache dict for downstream correlation. The job has run for three months without a restart.

What actually breaks

Resident memory grows by ~80 MB per hour. After 18 hours the container is OOM-killed and the next run re-downloads the entire backlog. The cache dict holds a reference to every object ever parsed; del on the local variable does nothing because the dict still points at the same objects.

What you'd do on Monday

Bound the cache with a collections.OrderedDict + max-size eviction, or use weakref.WeakValueDictionary so entries are collected once nothing else holds them. For resources that must be closed in order, use a context manager so cleanup happens at a deterministic line, not whenever the GC runs.

PythonBounded cache and weakref pattern (stdlib only)runs
import weakref, sys

class Quote:
    __slots__ = ('supplier_id', 'price', '__weakref__')
    def __init__(self, sid, price):
        self.supplier_id = sid
        self.price = price

# WeakValueDictionary: entries disappear when nothing else holds the object
cache = weakref.WeakValueDictionary()

q = Quote('ACME', 9.99)
cache['ACME'] = q
print('in cache:', 'ACME' in cache)  # True — live reference in `q`

del q
print('after del:', 'ACME' in cache)  # False — entry silently evicted
Output
in cache: True
after del: False
Scenario B

Imagine you work at Netflix

200 M member sessions, ~5 M active objects per region

The numbers

Netflix serves around 200 M members globally. During peak, a single AWS region may hold session-state objects for ~5 M concurrent streams. A plain __dict__-backed session object with six fields costs 344 bytes (48 B object + 296 B __dict__; measured on CPython 3.12). Five million such objects occupy 344 × 5 000 000 = 1 720 000 000 bytes ≈ 1 640 MB of heap. Switching to __slots__ cuts each object to 80 bytes, totalling 80 × 5 000 000 = 400 000 000 bytes ≈ 381 MB — a reduction of 1 259 MB per region.

What the Scenario-A answer does here

Adding __slots__ to a single hot class looks like a micro-optimisation at 1 000 objects. At 5 M it saves over a gigabyte per process, which shifts the instance type from "fits in a t3.large" to "fits in a t3.small".

What flips at this scale

At this scale the design changes from "shrink the object" to "eliminate the object": session state moves to an external store (Redis, DynamoDB) so the Python process holds only a thin handle per active request, not the full state for every live session.

The trade-off you'd defend

I would defend moving hot session state to an external store over keeping it in-process, trading a network round-trip per lookup for a far smaller resident heap and the ability to scale horizontally without sticky sessions.

Illustrative; member and concurrency figures are widely reported public estimates, not Netflix-published engineering specs.

Gotchas

Deleting a name does not free the object

Symptom

Memory does not drop after del large_list or after a function returns.

Cause

Another reference — a global dict, a closure, a loop variable — keeps the object alive.

Fix

Use gc.get_referrers(obj) before the delete to find all live references. Move caches behind weakref.WeakValueDictionary so entries evict themselves.

__del__ is not a context manager

Symptom

A file or database connection stays open longer than expected; __del__ is defined but the resource is released minutes after the object becomes unreachable.

Cause

Objects in cycles are not freed by refcount — the GC calls __del__ later. CPython does not guarantee when.

Fix

Replace __del__ with a context manager (__enter__/__exit__) so cleanup happens at a known line. See Context managers and deterministic cleanup.

__slots__ breaks pickle and subclassing silently

Symptom

pickle.dumps(obj) raises TypeError: cannot pickle 'OrderSlots' object, or a subclass unexpectedly regains a __dict__.

Cause

__slots__ suppresses __dict__ only on the class that declares it. A subclass that adds no __slots__ gets a __dict__ back. Pickle needs __getstate__/__setstate__ when __dict__ is absent.

Fix

Add __slots__ = () in every subclass, and implement __getstate__/__setstate__ if pickling is needed.

Tricky questions

Does del x free the memory?
Most people say

Yes — del deallocates the object.

Actually

del removes the name binding and decrements the reference count by one. The object is freed only if that decrement brings the count to zero. If a list, a closure, or another name still holds a reference, the object stays alive and the memory is not returned.

Is the cyclic GC the main memory-management path in CPython?
Most people say

Yes — Python uses garbage collection like Java or Go.

Actually

Reference counting is the primary mechanism and handles the vast majority of objects. The cyclic GC is a fallback that only runs periodically and only processes objects that can form cycles (containers, instances with __dict__). Most objects are freed immediately when their last name is deleted.

Scenario question

You're handed a long-running Python ETL service whose resident memory grows 50 MB per hour. del calls are in place. What do you do first?

Model answer

I'd start by confirming the growth is real and not just RSS inflation from the Python allocator holding freed pages. I'd add a metric that logs tracemalloc.get_traced_memory() peak at the end of each batch — if peak grows monotonically, there is a genuine leak. Then I'd call gc.get_referrers() on a sample object I expect to be freed and trace which container holds it. The trade-off is that tracemalloc adds ~5–10% overhead per allocation, so I'd turn it on in staging, find the leak path, then turn it off in production once the fix is validated.

Interview corner

CommonWhat is the difference between reference counting and the cyclic GC in CPython?
“I'd explain that reference counting is the fast path — every object tracks how many names point at it, and when that count hits zero the object is freed immediately in the same C stack frame. The cyclic GC is the slow path that runs periodically to clean up reference cycles that refcounting can't see, like two objects pointing at each other.”
TrickWhy can't you rely on __del__ for resource cleanup?
“I wouldn't rely on __del__ because objects in cycles aren't freed by the refcount mechanism — they wait for the cyclic GC, which can run much later or not at all if the interpreter is shutting down. I'd use a context manager instead, where __exit__ is called at a predictable line.”
Experience checkYou suspect a module-level dict is causing a memory leak. How do you confirm it?
“I'd import gc and call gc.get_referrers(suspect_object) to list every container that holds a reference. If the module-level dict shows up, I know the object is alive. Then I'd decide whether to bound the dict, switch to a WeakValueDictionary, or shard it with an LRU eviction policy.”
CommonWhen should you use __slots__?
“I'd use __slots__ on classes where I'll create millions of instances and the attribute set is fixed — like rows parsed from an ETL file. The measured saving on CPython 3.12 is 264 bytes per six-attribute instance, which adds up to over a gigabyte for five million objects. I wouldn't use it on classes I need to subclass freely or pickle without extra work.”

Memory hook

The library keeps a loan counter on each book: when the counter hits zero the librarian shelves it immediately; if two books reference each other, neither counter ever reaches zero, and the inspector (GC) has to visit.

If you remember only one thing:

del x only decrements a counter; the object stays alive until every reference — names, containers, closures, and frames — is gone.

Self-check

  1. What does sys.getrefcount(x) report that is one higher than you expect?

    The function call itself creates a temporary reference in its own argument slot, so the reported count is always one more than the actual live references outside the call.

  2. An object has a reference count of 2 and you delete one name. Is it freed?

    No. The count drops to 1, which means one more reference still holds it alive. The object is freed only when the count reaches 0.

  3. Why does CPython's cyclic GC not run for every object?

    It tracks only objects that can participate in cycles. User-defined instances (with or without __slots__) are tracked. Objects that cannot hold references — int, str, float — are untracked. __slots__ removes the per-instance __dict__ (one fewer tracked container), but the instance itself remains tracked.

  4. What is the measured size difference between a six-attribute class with __dict__ versus __slots__ on CPython 3.12?

    344 bytes (48 B object + 296 B __dict__) versus 80 bytes with __slots__ — a saving of 264 bytes per instance.

  5. Why is __exit__ safer than __del__ for resource cleanup?

    __exit__ is called at a deterministic source line when the with block ends. __del__ is called whenever the refcount hits zero, which can be delayed by cycles, interpreter shutdown, or CPython internals.

Prove it

Write two Node classes: one with a plain attribute and one with __slots__ = ('child', '__weakref__'). For each, create a 10-node cycle (each node points to the next, the last to the first), delete all names, and call gc.collect(). Confirm that both cycles are collected (they will be, because user instances are GC-tracked regardless of __slots__). Then check gc.is_tracked(node) before deletion for each: both return True. Finally, check whether the __slots__ version has a __dict__ attribute: it does not. Paste the is_tracked results and the collection counts.

py-09-02Card 2 of 6

Locate hot paths with cProfile

MediumP2 · Very likelyKnow cold11 min
cprofileprofilinghot-pathcumulative-timetottime
Needs: nothing. Start here.

Why you care

Guessing which function is slow costs more engineering time than running a profiler. Every ETL job has at least one function that accounts for 70% of the CPU time, and until you measure it you are as likely to optimise the wrong place as the right one. Interviewers ask "how do you find a performance bottleneck" because the answer separates engineers who instrument from engineers who guess — and guessing is a career-limiting habit in production.

Explain like I'm 5

Imagine you run a restaurant kitchen and you want to know where time is lost. You put a stopwatch on every cook. At end of service you read the stopwatch totals. Two numbers matter: how long each cook spent actually cutting vegetables or stirring (local time), and how long the full recipe they were responsible for took from start to finish (cumulative time). A cook with low local time but high cumulative time is waiting on someone else — that someone else is your real bottleneck.

Core concept

cProfile is CPython's built-in deterministic profiler. It wraps every function call in the process and records two time figures:

  • tottime: time spent inside the function itself, excluding time in functions it called.
  • cumtime: total time from function entry to exit, including every nested call.

Sort by cumtime to find the slowest call chain. Sort by tottime to find the function doing the most actual work.

Text
ncalls  tottime  percall  cumtime  percall  function
     1    0.000    0.000    3.421    3.421  pipeline()
     1    3.419    3.419    3.419    3.419  _load_from_s3()
     1    0.002    0.002    0.002    0.002  _validate()

_load_from_s3 has high tottime — it is the actual work. pipeline has high cumtime because it called _load_from_s3.

Running cProfile adds overhead (each function call is instrumented). Use it in development and staging, not in a production hot path. For production sampling without full instrumentation, use py-spy (external) or sys.setprofile with selective hooks.

Decision rule. Run cProfile first, every time, before touching code. If the bottleneck is a built-in (list.sort, str.join, a C extension), the fix is usually algorithmic (see Fix algorithms before micro-optimising). If it is your own loop, look at the per-call count: a function called millions of times with low per-call cost is a different problem from one called once with high per-call cost.

Minimal example Profile a two-function pipeline and identify the hottest path

PythonFind hottest function by tottime using cProfileruns
import cProfile, pstats

def slow_path(rows):
    # O(n^2): each lookup scans a list
    blacklist = list(range(200))
    return [r for r in rows if r not in blacklist]

def fast_path(rows):
    # trivial pass-through used to contrast tottime
    return rows

def pipeline(data):
    fast_path(data)
    return slow_path(data)

data = list(range(10_000))
pr = cProfile.Profile()
pr.enable()
pipeline(data)
pr.disable()

# pstats.Stats.stats: {(file, lineno, name): (cc, nc, tottime, cumtime, callers)}
raw = pstats.Stats(pr).stats
by_tot = sorted(
    {name: tt for (_, _, name), (cc, nc, tt, ct, _) in raw.items()}.items(),
    key=lambda kv: kv[1],
    reverse=True,
)
print('hottest function:', by_tot[0][0])
print('slow_path is top:', by_tot[0][0] == 'slow_path')
Output
hottest function: slow_path
slow_path is top: True

In production Profile an ETL pipeline writing results to a fixture file

PythonRealistic profiling session (writes CSV fixture, prints stats to stdout)
# norun: writes timing output that varies per machine; run manually to see your numbers
import cProfile, pstats, io, csv
from pathlib import Path


def load_rows(path):
    with open(path, newline='') as f:
        return list(csv.DictReader(f))


def validate(rows):
    return [r for r in rows if r.get('status') == 'SETTLED']


def enrich(rows):
    blocklist = list(range(50_000))   # deliberate list; will be the hotspot
    return [r for r in rows if int(r['order_id']) not in blocklist]


def run_pipeline(path):
    rows = load_rows(path)
    valid = validate(rows)
    return enrich(valid)


# --- fixture ---
p = Path('orders_sample.csv')
p.write_text(
    'order_id,amount,status\n'
    + '\n'.join(f'{i},{i*10},SETTLED' for i in range(5_000))
)

pr = cProfile.Profile()
pr.enable()
result = run_pipeline(str(p))
pr.disable()

buf = io.StringIO()
pstats.Stats(pr, stream=buf).sort_stats('tottime').print_stats(5)
print(buf.getvalue())
print('rows out:', len(result))

See it happen cumtime versus tottime

callscallspipeline()cumtime=3.42sload_rows()tottime=3.40svalidate()tottime=0.01stottime = local workcumtime = chain total
  1. pipeline has high cumtime because it waits for everything it calls.
  2. load_rows has high tottime — it does the actual work.
  3. validate is fast: low tottime, low cumtime. Not the bottleneck.
  4. Sort by tottime to find actual work; sort by cumtime to find slow chains.
Notice: a function with high cumtime but low tottime is a coordinator, not the bottleneck — follow its callees.
Scenario A

In your world

The situation

Your Jenkins job runs a 30-minute nightly pipeline. Business wants it under 10 minutes. You added logging and guessed the S3 download was slow, so you parallelised it — the job is now 28 minutes.

What actually breaks

You optimised the wrong function. The actual bottleneck was a validation function that performed a linear-scan in check against a list of 200 000 blocked IDs for each of 5 million rows — a silent O(n²) loop that never appeared in logs.

What you'd do on Monday

Profile first, then act.

PythonSimulate ETL profiling: find the O(n^2) loop before optimisingruns
import cProfile, pstats


def download_rows(n):
    # Simulate fast network fetch
    return list(range(n))


def validate_rows(rows):
    # The real bottleneck: O(n*m) linear scan
    blocked = list(range(500))  # list, not set
    return [r for r in rows if r not in blocked]


def pipeline(n):
    rows = download_rows(n)
    return validate_rows(rows)


pr = cProfile.Profile()
pr.enable()
pipeline(10_000)
pr.disable()

raw = pstats.Stats(pr).stats
by_tot = sorted(
    {name: tt for (_, _, name), (cc, nc, tt, ct, _) in raw.items()}.items(),
    key=lambda kv: kv[1], reverse=True,
)
print('hottest:', by_tot[0][0])
print('validate_rows is hottest:', by_tot[0][0] == 'validate_rows')
Output
hottest: validate_rows
validate_rows is hottest: True
Scenario B

Imagine you work at Flipkart

50 M product catalogue queries/day at sale peaks

The numbers

Flipkart's Big Billion Days sale drives tens of millions of product-detail requests. Assume 50 M catalogue queries over a 12-hour sale window: 50 000 000 ÷ 43 200 s ≈ 1 157 queries/s sustained. A pipeline that spends 80% of its CPU in one validation loop (tottime 80 ms, cumtime 100 ms per request) blocks 1 157 × 0.08 s ≈ 92 request-seconds of CPU per second — the service cannot keep up without 92 cores just for that one function.

What the Scenario-A answer does here

Horizontally scaling the service adds cost linearly with the bottleneck. Spending a week profiling and fixing that one 80 ms function first is cheaper than buying 90 extra cores.

What flips at this scale

At this scale a profiler run is a prerequisite gate before any scale-out decision. The design shift is process: profiling becomes a required step in every performance review, not an ad-hoc action when an alert fires.

The trade-off you'd defend

I would defend a mandatory profiling step in the performance review process over reactive scaling, trading engineer time upfront for infrastructure cost savings and a reproducible baseline.

Illustrative; query volumes are public-domain estimates from news coverage, not Flipkart-published engineering metrics.

Gotchas

cProfile adds enough overhead to change which path is hottest

Symptom

The profile run points to a different function than the production bottleneck; fixing it changes nothing.

Cause

cProfile instruments every Python function call. Very short functions called millions of times spend a significant fraction of their time inside the profiler's own bookkeeping.

Fix

Use cProfile for development profiling to identify candidates, then confirm with a sampling profiler (e.g. py-spy) in staging before shipping a fix.

sort_stats('cumulative') hides local work

Symptom

pipeline() tops the cumulative list; you optimise pipeline's own body but speed changes nothing.

Cause

cumtime includes all time in callees. The actual work is one level down.

Fix

After sorting by cumulative to find the slow chain, switch to sort_stats('tottime') to find which function in that chain is doing the work.

The profiler captures a cold run

Symptom

The profiler output is dominated by import time, file-open overhead, or JIT warming, not the real hot path.

Cause

A single run includes one-time setup costs that do not represent steady-state load.

Fix

Run the pipeline twice; profile only the second call, or wrap just the hot section with pr.enable() / pr.disable() rather than the whole script.

Tricky questions

What does high cumtime with low tottime mean for a function?
Most people say

The function is slow and should be optimised.

Actually

The function is a coordinator — it calls other functions that do the real work. Its cumtime is high because it waits for those callees. The bottleneck is one of its callees with high tottime, not the function itself.

Does cProfile show time spent inside built-ins like list.sort?
Most people say

No — cProfile only measures Python-level functions.

Actually

cProfile does record time spent in built-ins; they appear as entries like {built-in method builtins.sorted} in the stats. Their tottime can reveal that a built-in, not your code, is the dominant cost.

Scenario question

You're handed an ETL pipeline that runs in 45 minutes. Your manager wants it under 15 minutes and suggests switching from Python to Go. What do you do first?

Model answer

I'd run cProfile before touching the language question. Rewriting in Go is a weeks-long project; if one Python function accounts for 80% of the runtime, fixing it takes a day and may bring the pipeline under 15 minutes. I'd profile the full run, sort by tottime, identify the top three functions, and present that data before recommending a language change. The trade-off I'd name: profiling adds one day of work; an unnecessary rewrite adds one month.

Interview corner

CommonHow do you find a performance bottleneck in a Python script?
“I'd run cProfile with sort_stats('tottime') to find which function spends the most time doing its own work, then sort_stats('cumulative') to see which chain is slowest end-to-end. I'd measure before touching code — guessing is slower than a five-minute profile run.”
TrickYou profile and find pipeline() at the top of the cumtime list. Is pipeline() the problem?
“I'd say not necessarily — high cumtime means the function waits for callees, not that its own body is slow. I'd check tottime: if that's low, the bottleneck is a callee. I'd drill down with sort_stats('tottime') to find which callee is doing the actual work.”
Experience checkA cProfile run in dev shows a different bottleneck than production. Why?
“I'd look at three causes: different data sizes in dev versus prod, cProfile's own overhead skewing results for very hot tight loops, and one-time startup costs dominating a short dev run. I'd profile with a prod-representative dataset and exclude import time with selective enable/disable brackets.”
DesignWhen would you not use cProfile in production?
“I wouldn't use cProfile in a production hot path because it instruments every function call and can add 10–50% overhead. I'd use a sampling profiler like py-spy that attaches to a running process without code changes and adds less than 1% overhead. cProfile stays in dev and staging.”

Memory hook

You put a stopwatch on every cook: the one with high local time is actually chopping; the one with high total time is waiting on the chopper. Find the chopper — that is your bottleneck.

If you remember only one thing:

Sort by tottime to find who does the real work; sort by cumtime to find who waits for them.

Self-check

  1. What is the difference between tottime and cumtime?

    tottime is time spent inside the function itself, excluding callees. cumtime is the total wall time from entry to exit, including all nested calls.

  2. Why does a function with high cumtime but low tottime not need to be optimised?

    Because it is a coordinator — it merely calls other functions. Its own body is fast; the bottleneck is in one of its callees.

  3. Why should you not run cProfile in a production hot path?

    cProfile instruments every Python function call, which can add 10–50% overhead and slow down the service noticeably.

  4. What does it mean when a built-in appears at the top of the cProfile report?

    A built-in function like list.sort or str.join is the dominant CPU cost. The next question is algorithmic: is this the right data structure or operation for the task?

Prove it

Write a function deduplicate(rows) that removes duplicates using a list-linear-scan approach (if r not in seen_list). Write a second version using a set. Profile both over 20 000 rows using cProfile. Paste the tottime for each function from pstats.Stats.stats. Verify that the set version has tottime at least 10x lower.

py-09-03Card 3 of 6

Design a useful timeit experiment

MediumP3 · Role-dependentTalk about it10 min
timeitmicrobenchmarksetupwarm-upexperiment-design
Needs: nothing. Start here.

Why you care

A timeit result that does not isolate the code under test is not a measurement — it is noise. ETL engineers run microbenchmarks to choose between a list comprehension and a generator, between str.join and concatenation, or between two serialisation libraries. If your setup is inside the timed code, if you run only one iteration, or if you forget to prove the results are equivalent, you may ship a "faster" version that is actually slower on real data. Interviewers ask about timeit because experiment design is a proxy for engineering rigour.

Explain like I'm 5

You are testing two bakers to see who makes a loaf faster. If you start the clock while they are still weighing flour, you are timing the prep, not the baking. A fair test: pre-weigh everything identically (setup), then say "go" (timer starts), both bakers bake their loaf (number runs), clock stops. You repeat 1 000 times, take the minimum (not the mean), and verify both loaves taste the same (result equivalence).

Core concept

timeit.timeit(stmt, setup, number) runs setup once, then stmt exactly number times, and returns the total wall time. Divide by number to get per-call time.

Three design rules that most engineers break:

  1. Keep setup outside the timer. Building a fixture list inside stmt measures construction, not the operation you care about. Move it to setup.
  2. Use enough repetitions for a stable total. A single run on a fast operation reports sub-microsecond times dominated by timing overhead. Aim for a total elapsed time of at least 0.2 s per trial; for microsecond operations that means thousands of repetitions, for slow operations (full ETL batch) tens may be enough. Use timeit.repeat(number=…, repeat=7) and take min().
  3. Verify result equivalence. A "faster" version that returns different data is wrong, not fast. Assert that both implementations produce the same output on the same input before comparing times.

Decision rule. Use timeit for code that runs millions of times per ETL job and where the measured difference is at least 2×. A 5% difference is noise; a 10× difference justifies a change. After timeit confirms the winner, re-run a full-pipeline profile (Locate hot paths with cProfile) to confirm the micro-improvement translates to an end-to-end gain.

Minimal example Compare list lookup versus set lookup with correct setup isolation

PythonTimeit with setup outside the timer; print ratio not raw timesruns
import timeit

setup = '''
import random
random.seed(42)
ids = list(range(1000))
ids_set = set(ids)
target = 999
'''

list_time = timeit.timeit('target in ids', setup=setup, number=100_000)
set_time  = timeit.timeit('target in ids_set', setup=setup, number=100_000)
ratio = round(list_time / set_time)
print(f'list lookup is ~{ratio}x slower than set for 1000 elements')
print(f'set is faster: {set_time < list_time}')
Output
list lookup is ~329x slower than set for 1000 elements
set is faster: True

In production Compare two row-deduplication strategies on ETL data

PythonDeduplication benchmark with fixture creation in setupruns
import timeit

# Create fixture once, outside the timer — 750 rows, 50% duplicates
rows = list(range(500)) + list(range(250))

setup = f'rows = {rows!r}'

list_stmt = '''
seen = []
result = []
for r in rows:
    if r not in seen:
        seen.append(r)
        result.append(r)
'''

set_stmt = '''
seen = set()
result = [r for r in rows if r not in seen and not seen.add(r)]
'''

n = 50
list_total = timeit.timeit(list_stmt, setup=setup, number=n)
set_total  = timeit.timeit(set_stmt,  setup=setup, number=n)

ratio = round(list_total / set_total)
print(f'list dedup is ~{ratio}x slower than set dedup over 750 rows')
print(f'set is faster: {set_total < list_total}')

# Verify equivalence
exec(compile(setup + '\n' + list_stmt, '<>', 'exec'), ns_list := {})
exec(compile(setup + '\n' + set_stmt,  '<>', 'exec'), ns_set  := {})
list_result = sorted(ns_list['result'])
set_result  = sorted(ns_set['result'])
print(f'results equal: {list_result == set_result}')
Output
list dedup is ~25x slower than set dedup over 750 rows
set is faster: True
results equal: True

See it happen What timeit measures versus what you think it measures

timeit runsetupstmt×1stmt×2stmt×Nwrong: setup inside stmtstmt+setupstmt+setupstmt+setupstmt+setup
  1. A correct timeit call runs setup exactly once before the loop.
  2. Then stmt runs N times; only these N runs are timed.
  3. If setup is inside stmt, every run includes construction overhead.
  4. The bad version measures setup N times — you are not timing what you think.
Notice: move anything that constructs input data into setup; the timer bracket covers only the operation you are comparing.
Scenario A

In your world

The situation

You benchmarked two CSV serialisation methods. Version A used str.join and Version B concatenated strings in a loop. Version B printed a lower timeit number.

What actually breaks

You put the row-building logic inside the stmt string for both. Version B's rows happened to be shorter (fewer columns) because you typed a shorter fixture. The benchmark compared different workloads.

What you'd do on Monday

Write one fixture in setup and import it into both stmt strings. After the timeit, assert that both return the same bytes. The measured rule: the setup string runs once and is shared by both branches being compared.

PythonCorrectly isolated serialisation comparisonruns
import timeit

rows = [('ORD-001', '250.00', 'SETTLED'), ('ORD-002', '90.00', 'PENDING')]
setup = f'rows = {rows!r}'

join_stmt  = 'result = [",".join(r) for r in rows]'
build_stmt = '''
result = []
for r in rows:
    s = ""
    for col in r:
        s = s + col + ","
    result.append(s.rstrip(","))
'''

n = 50_000
join_t  = timeit.timeit(join_stmt,  setup=setup, number=n)
build_t = timeit.timeit(build_stmt, setup=setup, number=n)
ratio = round(build_t / join_t)
print(f'str.join is ~{ratio}x faster than concat loop')
print(f'join is faster: {join_t < build_t}')
Output
str.join is ~3x faster than concat loop
join is faster: True
Scenario B

Imagine you work at Myntra

10 M product listings, batch enrichment at 3 AM

The numbers

Myntra processes roughly 10 M active listings. A nightly enrichment job reads each listing, applies 12 classification checks, and writes the enriched record. If each classification check is implemented as a list-scan of 500 candidate tags (O(n) per check), the total comparisons are 10 000 000 × 12 × 500 = 60 000 000 000 — sixty billion comparisons. Converting each candidate list to a set reduces per-check cost to O(1), dropping the total to 10 000 000 × 12 = 120 000 000. A timeit run on 1 000 rows confirmed the set version is ~329× faster on a 1 000-element membership test (measured on CPython 3.12). End-to-end improvement: at this scale the job drops from an estimated 10-hour run to under 2 minutes.

What the Scenario-A answer does here

A timeit run on a 10-element fixture showed only a 5× difference. At 10 M rows the 500-element production tag list reveals the O(n) cost in full.

What flips at this scale

At 10 M rows the bottleneck moves from serialisation to data-structure choice. The design change is: always benchmark with production-representative fixtures, not toy data.

The trade-off you'd defend

I would defend set-based membership over list-based for any fixed candidate list, trading a one-time construction cost for O(1) lookup on every row.

Illustrative; listing counts are from public press coverage, not Myntra-published engineering data.

Gotchas

Putting fixture construction inside stmt

Symptom

Both versions look similar in the timeit result even though one is algorithmically faster; or one is faster only because its fixture happens to be smaller.

Cause

The fixture is rebuilt on every iteration, inflating both times. Any difference in fixture construction time pollutes the comparison.

Fix

Move all data construction into the setup argument so it runs once.

Using the mean instead of the minimum

Symptom

Results vary by 3× between runs; the "faster" version sometimes loses.

Cause

OS scheduling, GC pauses, and cache effects create outliers. The mean includes them; the minimum represents the best achievable throughput under ideal conditions.

Fix

Use timeit.repeat(repeat=7) and take min(results).

Not verifying result equivalence

Symptom

The faster version ships and produces wrong output in production.

Cause

The optimised version had a subtle bug — it dropped duplicates, reordered rows, or changed encoding — that timeit did not catch.

Fix

After the benchmark, assert that both versions produce == output on the same input. Make this a prerequisite, not an afterthought.

Tricky questions

You run timeit with number=1. Is the result reliable?
Most people say

Yes — you can measure it once for a slow operation.

Actually

A single run includes OS scheduling jitter, Python interpreter warm-up, and cold-cache effects. Even for a slow operation, run at least 3 repetitions with timeit.repeat and take the minimum. For fast operations, number=1 measures the interpreter overhead more than the code.

Does the minimum time from timeit.repeat represent average production throughput?
Most people say

Yes — minimum is the representative time.

Actually

The minimum represents the best case — no GC pause, cache warm, no context switches. Production throughput is closer to the mean or 90th percentile. Use the minimum to compare two implementations, not to predict production latency.

Scenario question

You're handed a claim that function A is 40% faster than function B based on a timeit run of number=1. How do you verify it?

Model answer

I'd first check whether the setup is correctly isolated — anything that constructs input data must be in the setup string, not in the stmt. Then I'd rerun with number=10_000 and timeit.repeat(repeat=7), taking the minimum of the seven trials. A 40% difference that survives 70 000 iterations with min-of-seven is real; one that disappears with more iterations was noise. I'd also check result equivalence — if A and B don't produce the same output, the "faster" result is meaningless. The trade-off: rigorous benchmarking takes an hour; shipping on a number=1 result risks a production regression.

Interview corner

CommonWhat is the difference between timeit number and timeit repeat?
“I'd distinguish them like this: number is how many times the statement runs in one trial — it controls sample size and warms up the branch predictor. repeat is how many separate trials you run. I'd use repeat=7 and take the minimum, because any one trial can be skewed by a GC pause or OS interrupt.”
TrickWhy should you take the minimum rather than the mean of timeit.repeat results?
“I'd take the minimum because it represents the fastest achievable time with no interference. The mean includes outliers from GC, OS scheduling and cache misses that don't represent the code's intrinsic speed. For comparing two implementations, minimum-of-seven gives a fair head-to-head.”
Experience checkA colleague claims version A is 2x faster than B because of a timeit result. What questions do you ask?
“I'd ask three things: is the fixture in setup or inside the stmt, what number did they use, and did they verify both versions produce the same output? If setup is inside stmt, the result is invalid. If number is 1, it is noise. If the outputs differ, the speed comparison is meaningless.”

Memory hook

You are timing the baking, not the kitchen setup: pre-weigh the flour (setup), start the clock, bake a thousand loaves, stop — then take the fastest batch, not the average.

If you remember only one thing:

Put all fixture construction in setup, use repeat=7, and take min().

Self-check

  1. What does timeit.timeit(stmt, setup, number) actually measure?

    It measures the total wall time for running stmt exactly number times, after running setup once. Divide by number to get per-call time.

  2. Why should fixture data go in the setup string, not inside stmt?

    Because setup runs once before the loop. If fixture construction is inside stmt, you time construction on every iteration, which inflates both sides equally and hides real differences.

  3. Why take the minimum of timeit.repeat results rather than the mean?

    The minimum represents the cleanest run with no GC pause or OS interrupt. The mean is pulled up by rare bad runs that do not reflect the code's actual speed.

  4. What must you verify before declaring one implementation faster than another?

    That both produce the same output on the same input — result equivalence. A version that is fast but wrong is not faster; it is broken.

Prove it

Write two functions that check whether a 6-digit string is numeric: one using str.isdigit() and one using a try/except int() cast. Build a fixture of 50 000 strings (mix of valid and invalid) in setup. Run timeit.repeat(number=1000, repeat=7) on both. Take min() for each, print the ratio rounded to one decimal, and assert the two functions return the same results on the fixture.

py-09-04Card 4 of 6

Trace allocation growth with tracemalloc

MediumP3 · Role-dependentTalk about it11 min
tracemallocallocationmemory-profilingpeak-memory
Needs: nothing. Start here.

Why you care

sys.getsizeof tells you the shallow size of one object; it ignores everything it points to. tracemalloc tells you how much memory your Python code actually allocated, which function allocated it, and whether it is growing over time. ETL jobs that process large files frequently OOM on their third run but not their first, because each run accumulates objects that the developer assumed were freed. Interviewers ask about tracemalloc because it separates engineers who instrument memory from engineers who watch the process RSS in top and guess.

Explain like I'm 5

Your water company puts a meter on each tap in the house. Each month you see exactly which tap used how much water. Without meters, all you know is the total bill. tracemalloc is the per-tap meter: it tracks which line of code asked for how many bytes, so you can see "line 42 in transform.py allocated 18 MB of dicts this run and 36 MB last run" instead of "the process used 1 GB and I do not know why".

Core concept

Three operations cover 90% of profiling use:

  1. tracemalloc.start(nframe) — begin tracing; nframe is the call-stack depth recorded per allocation (default 1, increase to 10 for deeper attribution).
  2. tracemalloc.take_snapshot() — capture the current allocation table.
  3. snapshot.compare_to(earlier, 'lineno') — return a list of StatisticDiff objects sorted by size_diff (bytes added since the earlier snapshot).

Two things to know before trusting the numbers:

  • tracemalloc tracks only Python-level allocations (objects created via pymalloc). Memory allocated directly by C extensions (numpy, psycopg, lxml) does not appear.
  • tracemalloc.get_traced_memory() returns (current, peak) since start(). Peak is the watermark — it does not reset unless you call tracemalloc.reset_peak().

Decision rule. Use compare_to to find the growing allocator between two snapshots (taken at the start and end of a batch). Use get_traced_memory() peak to confirm a fix actually reduced the high-water mark, not just the steady-state footprint.

Minimal example Before/after snapshot to find the largest allocator

PythonTake two snapshots and print the top growing allocatorruns
import tracemalloc

tracemalloc.start(5)
snap1 = tracemalloc.take_snapshot()

# Allocate a large structure
rows = [{'id': i, 'amount': float(i)} for i in range(50_000)]

snap2 = tracemalloc.take_snapshot()
tracemalloc.stop()

stats = snap2.compare_to(snap1, 'lineno')
top = stats[0]
print(f'top allocator size_diff > 0: {top.size_diff > 0}')
print(f'top allocator count_diff > 0: {top.count_diff > 0}')
# size_diff is in bytes; confirm it is in the MB range for 50k dicts
print(f'size_diff > 1_000_000: {top.size_diff > 1_000_000}')
Output
top allocator size_diff > 0: True
top allocator count_diff > 0: True
size_diff > 1_000_000: True

In production Trace allocation growth across ETL batches

PythonBatch ETL allocation tracker (sqlite3 stands in for Postgres)runs
import tracemalloc, sqlite3
from pathlib import Path

# --- fixture: a small CSV standing in for S3 input ---
Path('settlements.csv').write_text(
    'id,amount,status\n'
    + '\n'.join(f'{i},{i*5},SETTLED' for i in range(5_000))
)


def load_batch(path, batch_size=1_000):
    """Yield rows one batch at a time (generator — constant memory per batch)."""
    import csv
    with open(path, newline='') as f:
        reader = csv.DictReader(f)
        batch = []
        for row in reader:
            batch.append(row)
            if len(batch) >= batch_size:
                yield batch
                batch = []
        if batch:
            yield batch


tracemalloc.start(3)
snap_start = tracemalloc.take_snapshot()

conn = sqlite3.connect(':memory:')
conn.execute('CREATE TABLE out (id INT, amount REAL)')

for batch in load_batch('settlements.csv'):
    conn.executemany('INSERT INTO out VALUES (?, ?)',
                     [(int(r['id']), float(r['amount'])) for r in batch])
conn.commit()

snap_end = tracemalloc.take_snapshot()
current, peak = tracemalloc.get_traced_memory()
tracemalloc.stop()

stats = snap_end.compare_to(snap_start, 'lineno')
total_added = sum(s.size_diff for s in stats if s.size_diff > 0)
row_count = conn.execute('SELECT COUNT(*) FROM out').fetchone()[0]

print(f'rows written: {row_count}')
print(f'net allocation > 0: {total_added > 0}')
# Peak should be well under 5 MB for a 5k-row batched load
print(f'peak < 5_000_000: {peak < 5_000_000}')
conn.close()
Output
rows written: 5000
net allocation > 0: True
peak < 5_000_000: True

See it happen Two snapshots locate the growing allocator

your codetracemallocPython heaptracemalloc.start(5)snap1 = take_snapshot()rows = [dict(...) for ...]snap2 = take_snapshot()snap2.compare_to(snap1)stats[0]: line 42, +18 MB
  1. tracemalloc attaches a hook to every Python allocation; nframe=5 records 5 levels of call stack per alloc.
  2. Snapshot 1 captures the current allocation table — a baseline before the batch.
  3. 50 000 dicts are created; each allocation is recorded by the hook.
  4. Snapshot 2 captures allocations after the batch.
  5. compare_to returns StatisticDiff objects sorted by size_diff — the largest growers first.
  6. The top result names the exact source line that allocated the most memory between the two snapshots.
Notice: compare_to shows growth between two points, not total heap size — a line that allocated and freed equally shows zero diff even if it allocated gigabytes.
Scenario A

In your world

The situation

Your hourly settlement job processes 2 million rows from S3. After 6 hours the container is OOM-killed. You added del batch inside the loop but memory still grows. Jenkins shows green.

What actually breaks

del batch removes the name but not the object, because the conn.executemany call holds an intermediate list of tuples that was never freed, and a module-level audit list appends every batch for end-of-job summary. Two separate retainers, invisible from a code scan.

What you'd do on Monday

Wrap the critical section in tracemalloc.start() / take_snapshot() calls. Run a 10 000-row sample, call snap2.compare_to(snap1), and look at the top two StatisticDiff entries. The audit list and the tuple buffer will both appear.

PythonShow that del on a name does not always show zero in tracemallocruns
import tracemalloc

tracemalloc.start(3)
snap1 = tracemalloc.take_snapshot()

retained = []                           # module-level retainer
big = [b'x' * 1024 for _ in range(500)] # 500 KB
retained.append(big)                    # big is now held by retained
del big                                 # name gone, but retained still holds it

snap2 = tracemalloc.take_snapshot()
tracemalloc.stop()

stats = snap2.compare_to(snap1, 'lineno')
retained_diff = sum(s.size_diff for s in stats if s.size_diff > 0)
print(f'net allocation after del big: {retained_diff > 0}')
print(f'retained still holds data: {len(retained[0]) == 500}')
Output
net allocation after del big: True
retained still holds data: True
Scenario B

Imagine you work at Groww

12 M active investors, real-time portfolio snapshots

The numbers

Groww serves roughly 12 M active investors. During market hours a portfolio recalculation job runs every 30 seconds, loading each investor's holdings into a dict for P&L calculation. A plain dict row for two fields ({'id': i, 'amount': float(i)}) measured 247 bytes peak allocation per row on CPython 3.12 (measured via tracemalloc.get_traced_memory() on 100 000 rows). For 12 M investors loaded at once: 247 × 12 000 000 = 2 964 000 000 bytes ÷ 1 048 576 ≈ 2 827 MiB. Processing in bounded batches of 100 000 keeps peak allocation at 247 × 100 000 = 24 700 000 bytes ÷ 1 048 576 ≈ 23.6 MiB per batch, independent of total investor count.

What the Scenario-A answer does here

Loading all 12 M rows before processing avoids database round-trips but allocates 2 827 MiB (≈ 2.76 GiB) per run. A small container handles one run; after the second concurrent process starts, the node OOMs.

What flips at this scale

At this scale the design constraint changes from "reduce object size" to "bound the working set". The batch size becomes a first-class parameter measured and tuned with tracemalloc.get_traced_memory() peak on each batch.

The trade-off you'd defend

I would defend a bounded batch size with a measured peak-memory SLO over loading all rows at once, trading database round-trips for a predictable memory footprint that survives concurrent runs.

Illustrative; 247 bytes/row is a measured CPython 3.12 figure (tracemalloc on 100k dicts with two numeric fields); investor count is from widely reported public coverage, not Groww-published metrics.

Gotchas

tracemalloc peak does not include C extension allocations

Symptom

tracemalloc shows 50 MB peak, but the process RSS is 800 MB.

Cause

NumPy arrays, psycopg result sets, and most C extensions allocate memory directly from malloc, not via pymalloc. tracemalloc only hooks Python-level allocations.

Fix

Use resource.getrusage(resource.RUSAGE_SELF).ru_maxrss (Linux) or psutil.Process().memory_info().rss to measure total process RSS alongside tracemalloc.

compare_to hides allocations that were created and freed between snapshots

Symptom

A function that allocates 500 MB of temporary objects shows zero diff in compare_to.

Cause

The objects were allocated and freed before the second snapshot. size_diff is a net figure.

Fix

Use get_traced_memory() peak (which is a high-water mark) to detect transient spikes, not just compare_to diffs.

Starting tracemalloc in production permanently

Symptom

Memory in production is 10% higher than staging; the service is slightly slower.

Cause

tracemalloc adds overhead per allocation and stores tracebacks for every live object.

Fix

Turn it on for a bounded investigation window (tracemalloc.start() / tracemalloc.stop()) and remove it after the fix is deployed.

Tricky questions

Does sys.getsizeof give you the total memory an object occupies?
Most people say

Yes — sys.getsizeof tells you how much memory an object uses.

Actually

sys.getsizeof returns only the shallow size of the container object itself. For a list of 1 000 dicts, it returns 88 bytes (the list header) — it does not include the dicts the list points to. tracemalloc accounts for the full allocation tree; sys.getsizeof does not.

tracemalloc peak is the same as tracemalloc current?
Most people say

Yes — both report the current usage.

Actually

current is bytes currently allocated; peak is the maximum reached since tracemalloc.start() was called. A job that allocated 500 MB mid-run and freed it will show current=50 MB but peak=500 MB. Peak is the number that predicts OOM risk.

Scenario question

You're handed a Python ETL job that OOMs on its second hourly run but not its first. tracemalloc shows 50 MB after run 1 and 200 MB after run 2. What do you investigate?

Model answer

I'd look for objects that accumulate between runs — module-level containers (caches, audit lists, session objects) that are appended on every run but never cleared. I'd take a snapshot at the end of run 1 and the end of run 2, then call compare_to on them. The top StatisticDiff entries by size_diff will point at the exact lines and containers that grew. If the top retainer is a cache, I'd bound it with an LRU limit or switch to WeakValueDictionary. The trade-off: turning on tracemalloc for production runs adds 5–10% overhead per allocation, so I'd time-box the investigation window.

Interview corner

CommonWhat is the difference between sys.getsizeof and tracemalloc?
“I'd explain that sys.getsizeof gives only the shallow size of one object — for a list it returns the header size, not the total including items. tracemalloc tracks every Python-level allocation and lets me compare snapshots to see which line allocated the most memory between two points.”
Tricktracemalloc shows small growth, but the process RSS keeps climbing. Why?
“I'd suspect C extension allocations — numpy, psycopg, or lxml bypassing Python's allocator. tracemalloc only hooks Python-level allocations. I'd cross-check with psutil.Process().memory_info().rss or resource.ru_maxrss to see whether the RSS growth exceeds what tracemalloc reports.”
Experience checkHow do you confirm a memory fix actually worked?
“I'd use tracemalloc.get_traced_memory() peak before and after the fix on the same input. Peak is the high-water mark — it catches transient spikes that compare_to between snapshots would miss. If peak drops by the expected amount, the fix is real.”
DesignWhen would you not run tracemalloc in production?
“I wouldn't leave tracemalloc always-on in production because it adds per-allocation overhead and stores tracebacks for every live object, increasing resident memory. I'd use it in a bounded investigation window — start it, capture the data, stop it — then remove it from the deployment after the issue is fixed.”

Memory hook

Your water meter tracks which tap is running, not just the total bill — tracemalloc is the per-line meter that tells you which line of code is consuming memory, not just the total process size.

If you remember only one thing:

Use get_traced_memory() peak for OOM risk, compare_to two snapshots to find the growing line.

Self-check

  1. What does compare_to(snap1, 'lineno') sort by?

    By size_diff — the net bytes added between snap1 and snap2 at each source line, largest first.

  2. Why might a function that allocates 500 MB show zero in compare_to?

    If it allocated and freed the memory between the two snapshots, the net diff is zero. Use get_traced_memory() peak to detect transient spikes.

  3. tracemalloc shows 40 MB but the process RSS is 800 MB. Why?

    tracemalloc only hooks Python-level allocations. C extensions (numpy, psycopg, etc.) allocate directly via malloc and are invisible to tracemalloc.

  4. What is the difference between tracemalloc current and peak?

    current is bytes currently alive; peak is the maximum since start() was called. Peak does not reset unless you call tracemalloc.reset_peak().

Prove it

Write an ETL function that reads 50 000 integer rows, converts each to a dict, appends them all to a module-level list, and returns the count. Run it twice without restarting Python. Use tracemalloc to take a snapshot before run 1 and after run 2, call compare_to, and print the size_diff of the top entry. Then clear the module-level list between runs and repeat. Paste both size_diff values.

py-09-05Card 5 of 6

Fix algorithms before micro-optimising

MediumP2 · Very likelyKnow cold11 min
algorithmic-optimisationdata-structuresbig-omicro-optimisationcomplexity
Needs: nothing. Start here.

Why you care

An O(n²) loop running on 10 million rows will still take hours after you add every CPython micro-optimisation available. Engineers waste months replacing list comprehensions with map(), rewriting pure Python with Cython, or switching to PyPy — and none of it matters if the algorithm is wrong. Interviewers ask "what did you do to optimise a slow script?" because the answer reveals whether you measure and reason about complexity, or whether you reach for micro-optimisations as a first instinct.

Explain like I'm 5

You are hand-washing 10 000 dishes. You can buy a gold-plated scrubbing brush (micro-optimisation) or you can build a dishwasher (algorithmic improvement). The gold brush saves you 2 seconds per dish — 5.5 hours total. The dishwasher cuts each dish to 2 seconds of loading — still 5.5 hours. But a dishwasher that loads 100 dishes at once and runs in 20 minutes? That is algorithmic: you changed how many dishes you touch per unit of time, not just how fast you scrub each one.

Core concept

The rule is non-negotiable: profile first, then fix the algorithm, then consider micro-optimisation. Skipping the first step is guessing. Skipping the second step while doing the third is polishing a slow algorithm.

What counts as an algorithmic fix:

  • Replacing a list membership test (O(n)) with a set or dict lookup (O(1)).
  • Using collections.Counter or collections.defaultdict to avoid repeated scanning.
  • Sorting once before a multi-pass scan instead of scanning unsorted on every pass.
  • Using bisect for repeated range lookups on a sorted list instead of linear search.
  • Streaming data with generators (Iterator protocol, exhaustion and pull model) instead of materialising the full dataset.

What counts as a micro-optimisation (apply only after the algorithm is right):

  • List comprehension instead of a for loop with .append.
  • str.join instead of string concatenation in a loop.
  • local = some_global inside a hot loop to avoid global-scope lookup overhead.
  • Moving to array or numpy for numeric work that stays in C.

Decision rule. If cProfile (Locate hot paths with cProfile) shows a function called O(n) times with O(n) internal work, the complexity is O(n²). Fix that first. Micro-optimisations belong to the last 10% of performance work, not the first.

Minimal example Replace O(n) list scan with O(1) set lookup

PythonAlgorithmic fix: list vs set membership in a loopruns
import timeit

setup = '''
import random
random.seed(0)
blocked_list = list(range(200))
blocked_set  = set(blocked_list)
rows = list(range(1_000))
'''

list_stmt = 'result = [r for r in rows if r not in blocked_list]'
set_stmt  = 'result = [r for r in rows if r not in blocked_set]'

n = 200
list_t = timeit.timeit(list_stmt, setup=setup, number=n)
set_t  = timeit.timeit(set_stmt,  setup=setup, number=n)
ratio  = round(list_t / set_t)
print(f'list scan is ~{ratio}x slower than set lookup')
print(f'set is faster: {set_t < list_t}')
Output
list scan is ~32x slower than set lookup
set is faster: True

In production ETL deduplication — fix the algorithm, then measure improvement

PythonOrder deduplication: O(n^2) list vs O(n) set, result equivalence verifiedruns
import timeit


def dedup_list(rows):
    """O(n^2): linear scan for each row."""
    seen = []
    out = []
    for r in rows:
        if r not in seen:
            seen.append(r)
            out.append(r)
    return out


def dedup_set(rows):
    """O(n): hash lookup for each row."""
    seen: set = set()
    out = []
    for r in rows:
        if r not in seen:
            seen.add(r)
            out.append(r)
    return out


# Fixture: 2k rows, 50% duplicates, order preserved in output
rows = list(range(1_000)) * 2

setup_code = f'rows = {rows!r}'
n = 10
list_t = timeit.timeit('dedup_list(rows)', setup=setup_code,
                        globals={'dedup_list': dedup_list}, number=n)
set_t  = timeit.timeit('dedup_set(rows)',  setup=setup_code,
                        globals={'dedup_set':  dedup_set},  number=n)

ratio = round(list_t / set_t)
print(f'O(n^2) dedup is ~{ratio}x slower than O(n) set dedup')
print(f'results equal: {dedup_list(rows) == dedup_set(rows)}')
Output
O(n^2) dedup is ~69x slower than O(n) set dedup
results equal: True

See it happen Where algorithmic and micro-optimisation work

algorithmic fixesO(n²)→O(n)O(n log n)→O(n)stream vs loadmicro-optimisationslistcompstr.joinlocal vartypical gain10–10000x2–20x1.1–3x
  1. Algorithmic changes alter how many operations you do — the exponent changes.
  2. Micro-optimisations reduce the constant factor — the same number of operations, each slightly faster.
  3. Typical gains: algorithmic 10–10000x, micro 1.1–3x. No micro-optimisation overcomes a wrong algorithm.
Notice: micro-optimisations are multiplied by a constant; algorithmic improvements change the exponent.
Scenario A

In your world

The situation

Your nightly ETL checks 5 million transaction rows against a blocklist of 50 000 flagged account IDs. The blocklist is loaded once as a list. The job runs in 7 hours.

What actually breaks

The inner loop is if row_id not in blocklist_list — an O(n) scan repeated 5 000 000 times against a 50 000-element list. Total comparisons: 5 000 000 × 50 000 = 250 billion. Jenkins is green but finance waits until the next morning for results.

What you'd do on Monday

Convert the blocklist to a set once before the loop. The comparison becomes O(1) per row. Total: 5 000 000 lookups. A 200-element blocklist shows roughly 32× improvement; a 1 000-element list shows roughly 329× (both measured on CPython 3.12 on one machine; yours will vary). Either ratio scales to hours saved at 5 M rows.

PythonBlocklist from list to set: measure the fixruns
import timeit

# Simulate: 1k rows, blocklist of 200 IDs
setup = '''
rows = list(range(1_000))
blocklist_list = list(range(200))
blocklist_set  = set(blocklist_list)
'''

before = timeit.timeit('[r for r in rows if r not in blocklist_list]',
                        setup=setup, number=200)
after  = timeit.timeit('[r for r in rows if r not in blocklist_set]',
                        setup=setup, number=200)
ratio  = round(before / after)
print(f'before fix: ~{ratio}x slower than after fix')
print(f'fix is improvement: {after < before}')
Output
before fix: ~32x slower than after fix
fix is improvement: True
Scenario B

Imagine you work at Spotify

100 M tracks, real-time recommendations per session

The numbers

Spotify serves 100 M+ monthly active listeners. A recommendation engine filters candidate tracks against a per-user blocklist before ranking. Assume 10 000 requests/s at peak, each filtering 1 000 candidates against a 500-item blocklist. A worst-case miss on a 500-item list scan takes about 6.8 µs; a set lookup takes about 51 ns (measured on one machine, CPython 3.12). List path: 10 000 req/s × 1 000 tracks × 6.8 µs = 68 000 000 µs/s = 68 CPU-s/s ≈ 68 cores for the filter step alone. Set path: 10 000 × 1 000 × 51 ns = 510 000 000 ns/s = 0.51 CPU-s/s ≈ 1 core. The algorithmic fix eliminates 67 cores before any infrastructure conversation.

What the Scenario-A answer does here

Micro-optimising the filter loop body (list comprehension, local variable caching) saves at most 3× on the same O(n×m) computation: 68 cores ÷ 3 ≈ 23 cores still needed.

What flips at this scale

At 100 M sessions the design changes from "which Python idiom is faster" to "can you afford to reconstruct the blocklist set per session". The answer is yes (set construction from 500 items is O(500), negligible vs 1 000 lookups), and it moves the bottleneck to ranking, not filtering.

The trade-off you'd defend

I would defend building the set once per session request over any list-based filter, trading a trivial O(500) construction cost for O(1) lookup on 1 000 candidates.

Illustrative; session and track counts are from Spotify's publicly reported metrics; the per-session filter design and core arithmetic are constructed for this example.

Gotchas

Micro-optimising before profiling

Symptom

You rewrite three functions with list comprehensions; the pipeline is 3% faster, which disappears in the next deploy.

Cause

The functions you optimised were not the bottleneck. The hot path is somewhere else.

Fix

Run cProfile (Locate hot paths with cProfile) first. Accept no optimisation work until the hot function is identified.

Assuming "Pythonic" means fast

Symptom

Replacing a for-loop with a map() call makes no measurable difference to a pipeline that OOMs.

Cause

map() vs comprehension is a micro-optimisation. The OOM is caused by materialising the full dataset in memory — an algorithmic issue solvable by streaming (Stream a 20 GB file within a memory budget).

Fix

Separate the memory question (algorithmic: stream vs load) from the CPU question (data structure choice), then the constant-factor question (idiom choice). Tackle in that order.

Conflating "pure Python" with "slow"

Symptom

Team rewrites an ETL function in Cython after a week of work; performance improves 3× but the job is still too slow.

Cause

The hot loop was O(n²); Cython multiplied by a small constant. 3× × 7 hours = 2.3 hours, still too slow.

Fix

Profile to find the O(n²) loop, fix the algorithm, then consider compiled extensions only if the profiler still shows a single hot loop after the algorithmic fix.

Tricky questions

Is a list comprehension always faster than a for-loop with append?
Most people say

Yes — list comprehensions are always faster.

Actually

List comprehensions are typically 1.5–2× faster than for-loops with append for small to medium inputs because they run in an optimised bytecode path. But this is a constant factor. If the body of the loop has an O(n) lookup, the comprehension is still O(n²) — just 1.5× faster O(n²). The algorithmic complexity is unchanged.

Does using a set always improve performance over a list?
Most people say

Yes — sets are always faster.

Actually

For membership tests on large collections, yes. For operations that need ordered traversal, position-based indexing, or small collections (under ~20 elements) where construction overhead dominates, a list can be faster. Profile the specific use case.

Scenario question

You're handed a Python ETL job that filters 10 million rows against a 200 000-item blocklist. It runs in 14 hours. Your manager suggests switching to a compiled language. What do you try first?

Model answer

I'd profile first to confirm that the filter step is the actual bottleneck, not I/O or network. If the profiler confirms it, I'd check the blocklist data structure — if it is a list, switching to a set costs one line and should reduce the filter step by several orders of magnitude. I'd measure the result with timeit on a representative fixture. A 14-hour job that becomes 5 minutes after a one-line change is common, and it is the right first move before any language rewrite. The trade-off I'd name: a language rewrite is a months-long project with risk; a data-structure fix is a one-day experiment with a measured outcome.

Interview corner

CommonHow do you decide whether a performance problem is algorithmic or micro-level?
“I'd profile first to find the hot function, then look at the call count and the function's internal complexity. If it is called O(n) times and does O(n) work internally, I have an O(n²) problem — algorithmic. If it is called once with high tottime, I look at what it does: if it is a tight loop over a good algorithm, micro-optimisation is appropriate.”
TrickYou replace a for-loop with a list comprehension and see 2x speedup. Is the problem solved?
“I'd check whether the same loop has an O(n) membership test inside it. A 2× speedup on O(n²) is still O(n²) — it might go from 14 hours to 7 hours, which is still too slow. The question is: did the algorithmic complexity change? If not, the bottleneck is still there.”
Experience checkYou find a list membership test that runs 5 million times. What is the fix?
“I'd convert the list to a set before the loop — one line, and each membership test drops from O(n) to O(1). I'd verify the result is equivalent with an assertion and measure the speedup with timeit. On a 1 000-element list versus set, I've measured ~329× on CPython 3.12.”
DesignWhen is micro-optimisation appropriate?
“I'd apply micro-optimisations only after profiling confirms the hot function is algorithmically correct, its complexity is as low as the problem allows, and cProfile still shows it as the bottleneck. At that point I'd look at list comprehensions, str.join, and numpy for numeric loops — not before.”

Memory hook

Buy a dishwasher before you buy a gold-plated scrubbing brush: change how many operations you do per row before you change how fast each operation runs.

If you remember only one thing:

Profile first; then fix the algorithm; micro-optimise only after both steps confirm you are in the right function with the right complexity.

Self-check

  1. What is the difference between an algorithmic fix and a micro-optimisation?

    An algorithmic fix changes the complexity class — fewer total operations. A micro-optimisation reduces the constant factor — the same number of operations, each slightly faster.

  2. Why might a 3x speedup from a micro-optimisation fail to fix a pipeline?

    Because the underlying algorithm is O(n²). Multiplying by 3 does not change the exponent; the pipeline is still O(n²) and still too slow at production scale.

  3. When is it safe to apply micro-optimisations?

    After profiling confirms the hot function has the right algorithm (correct complexity class) and is still the bottleneck.

  4. A membership check in a hot loop runs 5 million times against a 10 000-item list. What is the fix?

    Replace the list with a set. Each lookup drops from O(10 000) to O(1), reducing total comparisons from 50 billion to 5 million.

Prove it

Write a function find_duplicates(rows, blocklist) that returns rows whose IDs appear in the blocklist. Implement it with a list-based scan and a set-based scan. Benchmark both with timeit on 20 000 rows and a 5 000-item blocklist. Print the ratio rounded to nearest integer. Assert both implementations return the same result. Paste the ratio.

py-09-06Card 6 of 6

Batch work against latency and memory budgets

MediumP2 · Very likelyKnow cold12 min
batchingmemory-budgetthroughputbounded-bufferetl

Why you care

An ETL script that loads 10 million rows into a list before inserting them will OOM on a 4 GB container and succeed on a 16 GB one — two different results for the same code depending on the machine. Batching makes the memory budget explicit and the result predictable. Interviewers ask about batching in ETL contexts because the answer reveals whether you design for bounded memory or rely on the next engineer to provision a bigger box.

Explain like I'm 5

You manage a warehouse that ships to a store. You could send a truck for every individual item — 10 000 trips. Or you could load 500 items per truck — 20 trips, and the truck is never more than 500 items full. Your warehouse (memory) never fills up regardless of how many items you are shipping. The batch size is the truck capacity — set it to what you can afford to hold, not to the total order size.

Core concept

A batch is a bounded window of rows processed together. Three parameters control a batched ETL loop:

  • batch_size: how many rows fit in memory at once. Derive it from the per-row allocation measured with tracemalloc (Trace allocation growth with tracemalloc) and your memory budget: batch_size = budget_bytes // bytes_per_row.
  • flush action: what happens at the end of each batch (INSERT, API call, file write). This is where latency accumulates; fewer batches means fewer round-trips.
  • remainder: the partial batch after the generator is exhausted. Always handle it — the last batch is almost never full.

Two failure modes to know:

  1. Batch too large: peak memory exceeds the container limit; the job is OOM-killed mid-run.
  2. Batch too small: thousands of round-trips to Postgres or S3; throughput collapses under per-call overhead.

Decision rule. Set batch_size high enough to amortise per-call overhead (typically 500–5 000 rows for Postgres executemany; 500–1 000 for S3 multipart). Measure peak memory with tracemalloc to confirm the chosen size fits the container. Use a generator (Iterator protocol, exhaustion and pull model) as the row source so the upstream read does not materialise the full dataset.

Minimal example Batch a stream and measure peak memory versus unbatched load

PythonBatched vs unbatched peak memory comparisonruns
import tracemalloc


def process_unbatched(n):
    tracemalloc.start()
    rows = list(range(n))           # full dataset in memory
    total = sum(rows)
    _, peak = tracemalloc.get_traced_memory()
    tracemalloc.stop()
    return total, peak


def process_batched(n, batch_size=1_000):
    def gen_rows(n):
        for i in range(n):
            yield i

    tracemalloc.start()
    total = 0
    batch = []
    for row in gen_rows(n):
        batch.append(row)
        if len(batch) >= batch_size:
            total += sum(batch)
            batch = []
    if batch:
        total += sum(batch)
    _, peak = tracemalloc.get_traced_memory()
    tracemalloc.stop()
    return total, peak


total_u, peak_u = process_unbatched(100_000)
total_b, peak_b = process_batched(100_000, batch_size=1_000)

print(f'totals match: {total_u == total_b}')
print(f'unbatched peak > batched peak: {peak_u > peak_b}')
ratio = peak_u // max(peak_b, 1)
print(f'peak ratio (unbatched / batched): {ratio}')
Output
totals match: True
unbatched peak > batched peak: True
peak ratio (unbatched / batched): 97

In production Batched ETL insert to sqlite3 with tracemalloc peak check

PythonBatched CSV-to-SQLite ETL with memory accounting (sqlite3 stands in for Postgres)runs
import csv, sqlite3, tracemalloc
from pathlib import Path


# --- fixture ---
Path('orders.csv').write_text(
    'id,amount,status\n'
    + '\n'.join(f'{i},{i * 5:.2f},SETTLED' for i in range(10_000))
)

BATCH_SIZE = 1_000


def row_stream(path):
    """Generator: one dict at a time — constant upstream memory."""
    with open(path, newline='') as f:
        yield from csv.DictReader(f)


def run_etl(path, batch_size):
    conn = sqlite3.connect(':memory:')
    conn.execute('CREATE TABLE orders (id INT, amount REAL, status TEXT)')

    tracemalloc.start(3)
    snap1 = tracemalloc.take_snapshot()

    batch = []
    rows_written = 0
    for row in row_stream(path):
        batch.append((int(row['id']), float(row['amount']), row['status']))
        if len(batch) >= batch_size:
            conn.executemany('INSERT INTO orders VALUES (?,?,?)', batch)
            rows_written += len(batch)
            batch = []
    if batch:
        conn.executemany('INSERT INTO orders VALUES (?,?,?)', batch)
        rows_written += len(batch)
    conn.commit()

    snap2 = tracemalloc.take_snapshot()
    _, peak = tracemalloc.get_traced_memory()
    tracemalloc.stop()

    stats = snap2.compare_to(snap1, 'lineno')
    net = sum(s.size_diff for s in stats if s.size_diff > 0)
    conn.close()
    return rows_written, peak, net


rows, peak, net = run_etl('orders.csv', BATCH_SIZE)
print(f'rows written: {rows}')
# Peak should be well under 5 MB for a 10k-row batched load
print(f'peak < 5_000_000: {peak < 5_000_000}')
print(f'net allocation > 0: {net > 0}')
Output
rows written: 10000
peak < 5_000_000: True
net allocation > 0: True

See it happen Batch window flowing through the ETL pipeline

row stream (generator)r0r1r2r3r4r5r6r7...batch buffer (max 3)database (flush)r0r1r2r3r4r5
  1. Rows sit in the stream; nothing is loaded yet.
  2. Row 0 enters the buffer.
  3. Rows 1 and 2 fill the buffer.
  4. Buffer full: flush batch to the database, then clear.
  5. Rows 3, 4, 5 fill the next batch.
  6. Flush again. Buffer memory stays bounded; only batch_size rows are ever live.
Notice: the buffer never holds more than batch_size rows; upstream memory is constant regardless of total file size.
Scenario A

In your world

The situation

Your nightly job loads settlement rows from S3 into a list, processes them all, then inserts in a single executemany. It worked fine on 500 000 rows three months ago.

What actually breaks

The settlement file grew to 8 million rows. The list holds 8 million dicts at ~247 bytes each (measured on CPython 3.12): 8 000 000 × 247 = 1 976 000 000 bytes ÷ 1 048 576 ≈ 1 884 MiB (≈ 1.84 GiB) before any processing. The container has 2 GiB RAM. The job is OOM-killed mid-run, leaving a partial write and a green Jenkins build (no exit code from OOM kill propagated to Jenkins).

What you'd do on Monday

Replace the full-load with a generator and a batch loop. Set batch_size to keep peak memory under 500 MB: 500_000_000 / 247 ≈ 2 024 000 rows per batch (or use 500 000 for safety margin). Each batch flushes to Postgres and is discarded before the next is loaded.

PythonShow peak memory grows linearly with rows if unbatchedruns
import tracemalloc

def peak_for_n(n):
    tracemalloc.start()
    rows = [{'id': i, 'amount': float(i)} for i in range(n)]
    _, peak = tracemalloc.get_traced_memory()
    tracemalloc.stop()
    del rows
    return peak

p1 = peak_for_n(10_000)
p2 = peak_for_n(20_000)
# Peak should roughly double when row count doubles (linear growth)
ratio = p2 / p1
print(f'peak doubles when rows double: {1.8 < ratio < 2.2}')
Output
peak doubles when rows double: True
Scenario B

Imagine you work at Swiggy

80 k order events/min at dinner peak

The numbers

Swiggy processes roughly 80 000 order events per minute during dinner peak — about 1 333 events/s. An ETL pipeline that aggregates these for reporting inserts rows into a data warehouse. At 247 bytes per event dict (measured on CPython 3.12 via tracemalloc), accumulating 1 minute of events (80 000 × 247 = 19 760 000 bytes ≈ 18.9 MB) is feasible. Accumulating 30 minutes (80 000 × 30 × 247 = 592 800 000 bytes ≈ 565 MB) exceeds a small container's safe working set. A batch size of 5 000 events (5 000 × 247 = 1 235 000 bytes ≈ 1.18 MB) keeps memory constant and requires 80 000 ÷ 5 000 = 16 flushes per minute — one flush every 3.75 seconds.

What the Scenario-A answer does here

Accumulating 30 minutes of events before flushing reduces database round-trips from 16/min to 1 per 30 min. At the cost of 565 MB peak — fine on a 4 GB instance, fatal on a 512 MB lambda.

What flips at this scale

At this scale the batch size becomes a deployment parameter tuned per environment. The design shift is: expose batch_size as a configurable constant (env var or config file) rather than a hard-coded number, and set the container memory limit based on the formula batch_size × bytes_per_row × safety_factor.

The trade-off you'd defend

I would defend a configurable batch size over a hard-coded one, trading one extra configuration parameter for the ability to right-size the deployment without a code change.

Illustrative; 247 bytes/row is measured on CPython 3.12 (tracemalloc on a 2-field dict); order volume is a widely cited public estimate for Swiggy peak, not a published engineering figure.

Gotchas

Forgetting to flush the remainder batch

Symptom

The job reports n rows written, but the row count in the database is n minus the last batch (up to batch_size-1 rows).

Cause

The flush only happens inside the if len(batch) >= batch_size branch. The final partial batch is never written.

Fix

Add a flush after the loop: if batch: conn.executemany(...).

Setting batch_size by instinct, not measurement

Symptom

Batch size of 10 000 causes OOM on the production container but works on the developer's laptop.

Cause

The developer's laptop has 16 GB RAM; the container has 2 GB. 10 000 × 247 bytes per dict = 2.47 MB — fine in isolation, but the dict plus the tuple conversion plus the executemany buffer puts peak closer to 10 MB per batch, which is still safe. The problem is usually a different path: the entire source file is loaded into a list before the batch loop starts.

Fix

Use a generator as the row source. Measure batch peak with tracemalloc on the production container size before deploying.

Counting bytes with sys.getsizeof and missing referenced objects

Symptom

sys.getsizeof reports 88 bytes for a list of 1 000 dicts; you set batch_size to 100 000 and OOM.

Cause

sys.getsizeof returns the shallow container size. It does not include the dicts the list points to.

Fix

Use tracemalloc on a representative sample to measure peak per batch.

Tricky questions

Does a smaller batch size always use less memory?
Most people say

Yes — smaller batches always mean less memory.

Actually

A smaller batch does hold fewer rows in memory at once, but if the row source is a list (not a generator), the full dataset is already in memory before the batch loop starts. The batch controls the write buffer, not the read buffer. Use a generator for the source to make the upstream memory constant.

Does a larger batch size always give better throughput?
Most people say

Yes — fewer flushes means higher throughput.

Actually

Up to a point. Each flush to Postgres or S3 has a fixed overhead (TCP round-trip, lock acquisition). Reducing flushes from 10 000 to 1 000 saves real time. Reducing from 1 000 to 10 saves less, and holding 100× more rows in the buffer may trigger a GC pause that costs more than the round-trips saved. Measure throughput at 500, 1 000 and 5 000 row batches before deciding.

Scenario question

You're handed an ETL job that OOMs on an 8 GB container when the input grows beyond 5 million rows. How do you fix it?

Model answer

I'd start by confirming with tracemalloc.get_traced_memory() peak whether the OOM is from the row buffer or from some other allocation. If the peak matches n_rows × bytes_per_row, the problem is that the source is materialised as a list. I'd replace the list with a generator that reads one row at a time and feeds a batch loop with a bounded buffer. I'd set batch_size = target_peak_bytes / bytes_per_row and validate with tracemalloc that the peak stays flat as row count grows. The trade-off: more database round-trips versus predictable memory. I'd present the measured batch size to the team before deploying, since it affects both memory and throughput SLOs.

Interview corner

CommonWhat is the purpose of batching in an ETL pipeline?
“I'd explain that batching bounds the working set: peak memory becomes a function of batch_size, not total row count. Without it, loading 10 million rows materialises everything at once, and memory scales linearly with the file. With batching it stays constant regardless of file size.”
TrickYou batch the inserts but the job still OOMs. What do you check first?
“I'd check whether the row source is a generator or a list. If it is a list, the full dataset is already in memory before the batch loop starts — the batch only controls the write buffer, not the read buffer. I'd replace the list with a generator, then verify peak with tracemalloc.”
Experience checkHow do you choose a batch size for an ETL insert to Postgres?
“I'd measure the per-row peak allocation with tracemalloc on a representative sample, divide the container's safe working set by that number to get a ceiling, then tune downward until the per-batch round-trip overhead is acceptable. I'd typically start at 1 000 rows and benchmark at 500 and 5 000 to find the throughput knee.”
DesignHow would you expose batch size to operations without a code change?
“I'd read it from an environment variable with a sensible default, so operations can tune it per deployment without touching the code. The formula batch_size × bytes_per_row × safety_factor gives the expected peak memory; I'd document that in the deploy config so the container memory limit can be set correctly.”

Memory hook

The warehouse truck never fills beyond capacity: set the batch size to what the truck can carry, not to the size of the entire order — memory stays bounded regardless of how large the shipment grows.

If you remember only one thing:

Use a generator as the row source and flush a bounded buffer; peak memory becomes batch_size × bytes_per_row, not total_rows × bytes_per_row.

Self-check

  1. What determines peak memory in a batched ETL loop?

    The batch_size multiplied by the bytes per row (measured with tracemalloc), provided the row source is a generator — not the total number of rows.

  2. A batched loop uses a list as its row source. Will batching control peak memory?

    No. The list materialises the full dataset in memory before the loop starts. Peak memory is total_rows × bytes_per_row. The batch only controls the write buffer. Replace the list source with a generator.

  3. What is the failure mode of a batch that is too small?

    Too many round-trips to the database or API: thousands of small inserts instead of hundreds of larger ones, with fixed per-call overhead on each. Throughput collapses.

  4. Why does the remainder batch need explicit handling?

    The flush inside the loop only fires when len(batch) >= batch_size. After the loop ends, the last partial batch is still in the buffer. Without a post-loop flush, up to batch_size - 1 rows are never written.

Prove it

Write a batched ETL function that reads rows from a generator (integers 0 to 499 999), processes them in batches of 5 000, and inserts them into a sqlite3 in-memory table. Before the loop, call tracemalloc.start(). After the loop, call tracemalloc.get_traced_memory(). Print peak < 10_000_000 (should be True for a 5 000-row batch of ints). Then repeat with batch_size=500_000 and print the new peak. Paste both peaks.