Shelf · Python · Chapter 4

Chapter 4: Iterators, generators & lazy pipelines

Every ETL script you have written already runs on the iterator protocol; this chapter makes it visible. By the end you can stream a file of any size inside a fixed memory budget and explain exactly why a second loop over the same reader silently returns nothing.

Difficulty
Priority
py-04-01Card 1 of 6

Iterator protocol, exhaustion and pull model

MediumP2 · Very likelyKnow cold10 min
iteratorsprotocolfor-loopexhaustion

Why you care

Half the "the job succeeded but loaded 0 rows" incidents in Python ETL come from one fact: some objects can be looped over once and only once. csv.reader, a file handle, map(), zip() and a generator are all one-shot. A list, a dict and a range are not. If you cannot tell the two apart at a glance, you will write a second pass that silently does nothing, and no exception will tell you. Interviewers ask "what is the difference between an iterable and an iterator" precisely because the answer predicts whether you have been bitten by this.

Explain like I'm 5

A book is an iterable: anyone can start reading it from page one. A bookmark is an iterator: it remembers where one particular reader is. iter(book) hands you a fresh bookmark. next(bookmark) reads the page and moves the bookmark forward. When the bookmark falls off the last page it shouts "StopIteration" and the reader stops. If you give someone your used bookmark instead of the book, they start at the end and read nothing.

Core concept

Two methods make up the whole protocol:

  • An iterable has __iter__(), which returns an iterator. Lists, tuples, dicts, sets, strings, range, files and your own classes can be iterables.
  • An iterator has __next__(), which returns the next value or raises StopIteration, and __iter__(), which returns itself. That last rule is why an iterator is also an iterable, and why looping over it twice gives you nothing the second time: the second iter() call hands back the same, already-advanced object.

A for loop is sugar for: call iter(x) once, then call next() until StopIteration, which the loop swallows. list(), sum(), max(), in, unpacking, zip() and dict() all do the same pull. Pulling is the key word: the consumer asks for one value at a time and the producer does just enough work to hand it over. Nothing is computed ahead. That pull model is what later lets a 20 GB file flow through a script that holds one line at a time (Stream a 20 GB file within a memory budget).

The practical test: iter(x) is x is True for an iterator (one-shot) and False for a container (re-iterable). When your function accepts "rows", decide which one it needs. If it needs two passes, either demand a container, call list() deliberately, or accept a re-iterable object whose __iter__ starts over each time.

Under the hood

CPython compiles for x in xs: to GET_ITER followed by a FOR_ITER loop; FOR_ITER calls the iterator's tp_iternext slot and jumps out when it signals exhaustion, so there is no Python-level try/except cost per item. iter() has two fallbacks worth knowing. If a class has no __iter__ but has __getitem__, Python calls it with 0, 1, 2 … until IndexError: the old sequence protocol. And iter(callable, sentinel) calls the function until it returns the sentinel, which is how you turn f.read(65536) into a chunk stream: iter(lambda: f.read(65536), b"").

Minimal example One list, one iterator, exhaustion

PythonIterator state lives outside the listruns
nums = [10, 20, 30]
it = iter(nums)
print(next(it), next(it))
print(list(it))
print(list(it))
print(iter(it) is it, iter(nums) is nums)
Output
10 20
[30]
[]
True False

In production A re-iterable reader for a two-pass job

A class whose __iter__ opens the file again gives every loop its own fresh iterator. The yield from inside is covered in Suspend computation with yield; here it simply means "produce every row of the reader".

PythonTwo passes that both see every rowruns
import csv
from pathlib import Path


class CsvRows:
    """Re-iterable: each for-loop reopens the file and gets a new iterator."""

    def __init__(self, path):
        self.path = Path(path)

    def __iter__(self):
        with self.path.open(newline="") as f:
            yield from csv.DictReader(f)


Path("orders.csv").write_text("id,amount\n1,250\n2,90\n3,410\n")

rows = CsvRows("orders.csv")
total = sum(int(r["amount"]) for r in rows)
big = [r["id"] for r in rows if int(r["amount"]) > 200]
print(total, big)

with open("orders.csv", newline="") as f:
    one_shot = csv.DictReader(f)
    print(sum(1 for _ in one_shot), sum(1 for _ in one_shot))
Output
750 ['1', '3']
3 0

See it happen What a for loop actually does

for looporders (list)iteratoriter(orders)new list_iteratornext()10next() x2 -> 20, 30StopIterationsecond loop: iter(it)StopIteration
  1. The loop calls iter() exactly once, before the first value.
  2. The list hands back a separate object that remembers the position (index 0).
  3. Pull one value. Nothing else is computed.
  4. The iterator returns 10 and moves its index to 1.
  5. Two more pulls, two more values.
  6. Nothing left: the loop catches StopIteration and ends quietly.
  7. Loop again over the iterator, not the list: iter(it) returns the same exhausted object.
  8. Zero iterations, no error. This is the silent 0-row load.
Notice: the list never moves. Only the iterator has a position, so a second loop over the list starts over, while a second loop over the iterator starts at the end.
Scenario A

In your world

The situation

Your nightly job pulls settlements.csv from S3, wraps it in csv.reader, logs row_count = sum(1 for _ in reader) for the audit table, then loops over reader again to insert into Postgres.

What actually breaks

The audit table says 48,213 rows. The target table gets 0. Jenkins shows green, because an empty loop is not an error. Nobody notices until finance asks where yesterday went.

What you'd do on Monday

Make one pass do both jobs, or make the source re-iterable. The Monday fix is to count while you load and assert the counts match:

PythonCount inside the load loop (needs psycopg + a real connection)
loaded = 0
for row in reader:
    cur.execute(INSERT_SQL, row)
    loaded += 1
log.info("loaded %d rows", loaded)
if loaded == 0:
    raise RuntimeError("0 rows loaded from settlements.csv, refusing to report success")
Scenario B

Imagine you work at Razorpay

60M settlement rows per night

The numbers

Illustrative: 60M rows × ~180 bytes ≈ 10.8 GB of CSV a night, split into 600 files of 100k rows. A two-pass design reads 21.6 GB from S3 instead of 10.8 GB, and at ~100 MB/s per worker that is 216 s versus 108 s per worker-GB-set: a second pass doubles I/O cost.

What the Scenario-A answer does here

The Scenario A fix (count while loading) still works, but "just call list(reader) so you can loop twice" does not: 100k rows × ~600 bytes of Python objects per row ≈ 60 MB per file per worker, times 32 workers ≈ 1.9 GB of RAM spent only on holding data you already streamed once.

What flips at this scale

At this scale you design for exactly one pass: every metric (count, checksum, min/max date) is accumulated in the same loop that writes, and the reconciliation compares those in-flight numbers with the source manifest. Re-iterable sources become a cost you pay deliberately, not a convenience.

The trade-off you'd defend

I would defend single-pass accumulation plus a manifest check over re-reading, trading slightly more complex loop code for half the S3 egress and constant memory. The one exception is a small lookup file, which I would load into a dict once.

Illustrative numbers chosen to make the arithmetic visible; not Razorpay-published figures.

Gotchas

A second loop over a reader returns nothing

Symptom

The second for over csv.reader, a file, map() or a generator runs zero times, with no error.

Cause

Those objects are iterators; iter() returns the same exhausted object.

Fix

Do everything in one pass, or keep a re-iterable source (a list you chose to build, or a class whose __iter__ reopens the file).

x in iterator consumes it

Symptom

After if "ERROR" in lines: the next loop over lines starts in the middle.

Cause

Membership on an iterator pulls values until it finds a match, and those values are gone.

Fix

Test membership on a container (a set is O(1)), or accumulate the flag inside the main loop.

Sharing one generator between threads

Symptom

ValueError: generator already executing, or duplicated and missing rows when several threads call next() on one iterator.

Cause

Generators are not thread-safe; two threads can re-enter the same frame.

Fix

Give each worker its own iterator, or hand rows out through a queue.Queue fed by a single producer (Runtime concurrency: threads, processes, async and GIL).

Tricky questions

Is a list an iterator?
Most people say

Yes, you can loop over it.

Actually

A list is an iterable. next([1, 2]) raises TypeError: 'list' object is not an iterator. iter([1, 2]) returns a separate list_iterator, which is why the list can be looped over any number of times.

Why does an iterator need an __iter__ method at all?
Most people say

It doesn't; only __next__ matters.

Actually

for, list() and zip() always call iter() first. Returning self lets you pass an iterator anywhere an iterable is expected, and it is exactly that return self that makes a second loop start at the end.

Scenario question

You're handed a function def dedupe_and_count(rows) that returns the correct count when called with a list but returns 0 duplicates, not an error, when called with the output of csv.reader. The code does unique = set(rows) and then return len(list(rows)) - len(unique). What is going on and how do you fix it without changing callers?

Model answer

set(rows) consumes the reader, so list(rows) sees an exhausted iterator and returns an empty list, and the result is 0 - n, or 0 if the caller clamps it. I would make it a single pass: loop once, keep a seen set and a dupes counter, and return dupes. That works for both lists and iterators and it holds only the unique keys, not every row. The trade-off is that I store keys, not full rows; if callers also need the rows, I would return them explicitly rather than rely on a second pass.

Interview corner

CommonWhat is the difference between an iterable and an iterator?
“I'd say an iterable is anything you can ask for an iterator, via __iter__, and an iterator is the thing with a position, via __next__. The iterator's own __iter__ returns itself, which is why lists can be looped repeatedly but a csv.reader or a generator can only be looped once.”
TrickYou loop over the same object twice and the second loop does nothing. Why?
“I'd suspect it's an iterator, not a container, so the second iter() returned the same exhausted object. I check with iter(x) is x. The fix is either a single pass or a source whose __iter__ starts over.”
Experience checkWhere has iterator exhaustion hurt you in real code?
“In my ETL work the classic one was counting rows from a csv.reader for an audit log and then loading from the same reader, which loaded zero rows while the job reported success. I now accumulate counts in the load loop and fail loudly on zero.”
CommonWhat does a for loop do under the hood?

Calls iter() once, then next() until StopIteration, which it swallows.

“I'd describe it as it = iter(xs) followed by repeated next(it) inside a try that exits on StopIteration. Everything that consumes, like sum, list, unpacking and in, uses the same pull.”

Memory hook

A list is a book; an iterator is a bookmark. Hand someone your bookmark and they start at the end.

If you remember only one thing:

If iter(x) is x, you get one pass. Plan for it.

Self-check

  1. What does iter(it) is it return for a file object, and why?

    True. A file object is its own iterator, so looping it twice reads nothing the second time unless you seek(0).

  2. Name three built-ins that return one-shot iterators.

    map(), filter(), zip(), enumerate(), reversed() and open(); also csv.reader and every generator.

  3. How would you turn f.read(65536) into something a for loop can consume?

    for chunk in iter(lambda: f.read(65536), b""):, the two-argument form of iter() with a sentinel.

Prove it

Write class Countdown with __iter__ and __next__ that counts from n to 1. Show that list(c) twice returns [3, 2, 1] then []. Then change it into a re-iterable, where __iter__ returns a new iterator, and show [3, 2, 1] twice. Paste both outputs.

py-04-02Card 2 of 6

Suspend computation with yield

MediumP2 · Very likelyKnow cold9 min
generatorsyieldlazy-evaluationiterators

Why you care

Writing an iterator class by hand means __iter__, __next__, instance state and a StopIteration. A generator function gives you all of that from one keyword. More importantly, it changes when your code runs: the body does not start until someone pulls. That is the source of both the memory win and the most confusing bugs, like validation that "doesn't run" and files that stay open. Every Python backend uses it daily, from FastAPI yield dependencies to pytest fixtures to paginated API clients.

Explain like I'm 5

A normal function is a vending machine: press the button and it drops everything at once. A generator is a waiter who brings one plate, then stands still exactly where they are, remembering the order, until you wave again. Waving is next(). When the kitchen is empty the waiter says "that's all" (StopIteration) and leaves.

Core concept

Any function containing yield is a generator function. Calling it runs none of the body; it returns a generator object, an iterator whose frame (local variables plus the current line) is frozen.

  • next(gen) runs the body until the next yield, hands that value out, and freezes again with every local intact.
  • Falling off the end, or return value, raises StopIteration. A for loop swallows it; the return value rides along as StopIteration.value, which for ignores and yield from captures (Delegate generators and exchange values).
  • A generator is single-pass like every iterator. Call the function again for a fresh run.
  • gen.close() raises GeneratorExit at the paused yield, so finally blocks and with blocks inside the generator run. If nobody closes it, that cleanup happens only when the object is garbage-collected.

Use a generator when producing values is expensive or unbounded, like pages from an API, rows from a cursor or lines from a file, and the consumer may stop early. Use a list when you need len(), indexing, or several passes.

Minimal example The body waits for the first pull

PythonNothing runs until next()runs
def countdown(n):
    print("start")
    while n > 0:
        yield n
        n -= 1
    print("done")


g = countdown(3)
print("created")
print(next(g))
print(next(g))
print(list(g))
Output
created
start
3
2
done
[1]

In production Keyset pagination as a generator

The consumer sees a flat stream of rows; the generator fetches the next page only when the previous one is used up, and reports how many pages it fetched through its return value.

PythonPages fetched lazily, count returned via StopIteration.valueruns
def fetch_page(after_id, limit):
    # stand-in for: SELECT ... WHERE id > %s ORDER BY id LIMIT %s
    table = list(range(1, 8))
    return [i for i in table if i > after_id][:limit]


def all_rows(page_size=3):
    last_id, pages = 0, 0
    while True:
        page = fetch_page(last_id, page_size)
        if not page:
            return pages
        pages += 1
        yield from page
        last_id = page[-1]


print(list(all_rows()))

gen = all_rows()
try:
    while True:
        next(gen)
except StopIteration as stop:
    print("pages fetched:", stop.value)
Output
[1, 2, 3, 4, 5, 6, 7]
pages fetched: 3

See it happen The generator's frame, frozen and resumed

next()StopIterationcallerg = countdown(3)countdown frameFINISHED
  1. Calling countdown(3) builds the frame but runs no code. Not even print('start').
  2. First next(): the body starts, prints 'start', runs to the first yield.
  3. yield hands 3 out and freezes the frame with n = 3.
  4. Next pull resumes right after the yield: n -= 1, loop, yield 2.
  5. list(g) keeps pulling: yield 1, then the loop ends and 'done' prints.
  6. Falling off the end raises StopIteration; list() stops. The frame is gone: calling next(g) again raises StopIteration immediately.
Notice: the frame keeps n between pulls. That is the whole trick; there is no hidden list anywhere.
Scenario A

In your world

The situation

Your extract_rows(path) generator validates that the file exists and raises ValueError for a bad path, then yields parsed rows. The job calls rows = extract_rows(cfg.path) at startup, connects to Postgres, truncates the staging table, and only then loops.

What actually breaks

With a wrong path, the error fires at the first loop iteration, after the truncate. Staging is now empty, the load failed, and the previous good data is gone.

What you'd do on Monday

Split validation from generation. The outer function checks eagerly and returns the inner generator:

PythonEager validation, lazy rowsruns
from pathlib import Path


def extract_rows(path):
    p = Path(path)
    if not p.exists():
        raise ValueError(f"missing input {p}")   # runs at call time

    def rows():
        with p.open() as f:
            for line in f:
                yield line.rstrip("\n").split(",")
    return rows()


try:
    extract_rows("nope.csv")
    print("truncate staging")                    # never reached
except ValueError as e:
    print("failed before touching the DB:", e)
Output
failed before touching the DB: missing input nope.csv
Scenario B

Imagine you work at Swiggy

80k order events per minute

The numbers

Illustrative: 80k events/min ≈ 1,333 events/s. A consumer that builds a list of one minute's events holds 80k × ~1 KB ≈ 80 MB per worker before processing anything; a generator pipeline holds one event plus the current batch, say 500 × 1 KB ≈ 0.5 MB.

What the Scenario-A answer does here

The Scenario A fix (eager validation) still applies, but "collect the minute into a list, then process" adds up to 60 s of latency to every event and needs 80 MB × 20 workers ≈ 1.6 GB just for buffers.

What flips at this scale

At stream scale a generator is not a style choice; it is how you get latency bounded by batch size instead of window size. The flip is that you now also need an explicit flush rule (500 events or 2 seconds, whichever first), because a lazy stream never "ends" to trigger the last write.

The trade-off you'd defend

I would take a generator pipeline with size-or-time batching, accepting more moving parts (a timer, a flush path, shutdown handling) for bounded memory and sub-second latency.

Illustrative figures for the arithmetic; not Swiggy-published numbers.

Gotchas

Argument checks run late

Symptom

A generator called with bad input raises only when first iterated, often far from the call site and after side effects.

Cause

Calling a generator function executes none of its body, including the checks at the top.

Fix

Validate in an ordinary function that then returns the inner generator (see Scenario A).

Abandoned generators hold resources

Symptom

Too many open files, or a DB cursor held open, after breaking out of loops over generators that open() files.

Cause

The generator paused inside its with block; cleanup runs only on close() or garbage collection.

Fix

Close explicitly: with contextlib.closing(gen()) as g: or call g.close() in finally (Context managers and deterministic cleanup).

The return value vanishes in a for loop

Symptom

return pages in a generator never reaches the caller.

Cause

for discards StopIteration.value.

Fix

Capture it with yield from in a delegating generator, or catch StopIteration manually as in the production example.

Tricky questions

Does calling a generator function execute its code?
Most people say

Yes, up to the first yield.

Actually

No. It only creates the generator object. The first next() runs the body up to the first yield.

Can a generator be restarted?
Most people say

Yes, loop over it again.

Actually

No. It is a single-pass iterator. Call the generator function again to get a new one.

Scenario question

You're handed a paginated REST client that returns list[dict] for every page and a job that needs only the first 50 records matching a filter. The API has 2,000 pages. How would you restructure it?

Model answer

I would make the client a generator that yields records page by page, and the job would use itertools.islice(filter(pred, client.records()), 50). Because generators are pulled, the client stops requesting pages as soon as 50 matches arrive: if matches are dense that might be 2 pages instead of 2,000. The trade-off is that failures now surface mid-stream, so I would put retry logic around the page fetch inside the generator, and make sure the HTTP session is closed with closing() when the consumer stops early.

Interview corner

CommonWhat is a generator and when would you use one?
“I'd call it a function with yield that returns an iterator whose frame pauses between values. I use it when producing values is expensive or unbounded, like pages, rows or lines, and the consumer may stop early, because it gives me constant memory and lazy work.”
TrickYou call a generator function with a bad argument and nothing fails. Why?
“I'd explain the body hasn't run yet; calling a generator function only builds the object. My fix is an eager wrapper that validates and then returns the inner generator.”
CommonGenerator versus list comprehension?
“I'd pick a list when I need len, indexing or multiple passes, and a generator when I only stream through once or the data is large. The generator trades random access for constant memory.”
Experience checkWhere does yield show up in frameworks you've used?
“I've seen it in pytest fixtures and FastAPI dependencies: code before yield is setup, code after is teardown. It's the same pause-and-resume mechanism, so teardown runs when the framework closes the generator.”

Memory hook

A generator is a paused function that remembers where it was standing.

If you remember only one thing:

Nothing inside a generator runs until the first next(), so validate outside it.

Self-check

  1. What is printed by g = countdown(3) in the minimal example?

    Nothing. The body, including print("start"), runs on the first next(g).

  2. How do you get a generator's return value?

    From StopIteration.value, either by catching it or with result = yield from gen inside another generator.

  3. Why can a paused generator keep a file open?

    It is frozen inside the with block. Cleanup runs only on close(), exhaustion, or garbage collection.

Prove it

Write read_lines(path) as a generator that prints "open" and "close" around a with open(...) block. Loop over it and break after two lines. Show that "close" prints only after you call .close() (or del the generator), then fix it with contextlib.closing. Paste the output.

py-04-03Card 3 of 6

Choose lazy generator expressions

EasyP2 · Very likelyKnow cold8 min
generator-expressionscomprehensionslazy-evaluationmemory

Why you care

Switching [ ... ] for ( ... ) is the cheapest memory fix in Python, and one of the most misunderstood. It helps only when the source is itself lazy and the consumer takes one pass. It also delays when conditions are evaluated, which produces a genuinely surprising bug. For an interviewer, "list comprehension or generator expression, and why?" separates people who memorised the syntax from people who understand evaluation order.

Explain like I'm 5

A list comprehension is cooking every dish on the menu before anyone orders. A generator expression is a cook who makes one dish each time an order slip arrives. If only three people order, you cooked three dishes. But the cook reads the recipe card at order time, so if someone swaps the card midway, later dishes follow the new card.

Core concept

(expr for x in source if cond) builds a generator, the same object a generator function returns, without writing a function.

  • Eager vs lazy. A list comprehension runs fully and stores every result. A generator expression stores nothing but its state; each next() computes one result.
  • What is evaluated when. Only the first for source is evaluated immediately. Every condition, later for clause and expression is evaluated during iteration, using whatever the names refer to at that moment.
  • Consumers. sum(), max(), any(), all(), "".join(), dict() and set() accept a generator directly: sum(x.amount for x in rows) needs no extra brackets. any() and all() short-circuit, so they stop pulling on the first decisive value.
  • Where it does not help. If source is already a list you built, the list is the memory; wrapping it in a generator saves only the second copy. The generator also keeps a reference to the source, so the source cannot be freed while the generator lives.

Rule of thumb: a generator expression for a single pass into a reducer or a writer; a list when you need len(), indexing, sorting, or a second pass.

Minimal example Same values, very different memory

PythonSize of a list vs a generator of one million squaresruns
import sys

squares_list = [n * n for n in range(1_000_000)]
squares_gen = (n * n for n in range(1_000_000))

print(sys.getsizeof(squares_list) > 8_000_000)
print(sys.getsizeof(squares_gen) < 500)
print(sum(squares_gen))
print(sum(squares_gen))
Output
True
True
333332833333500000
0

In production A lazy log scan that stops at the first hit

Each stage pulls from the previous one, and next(..., None) stops the whole chain as soon as one match is found. All consumption happens inside the with, while the file is still open.

PythonChained generator expressions with early exitruns
from pathlib import Path

Path("events.log").write_text(
    "INFO job=load rows=120\n"
    "ERROR job=load code=DUP\n"
    "INFO job=push rows=80\n"
    "ERROR job=push code=TIMEOUT\n"
    "ERROR job=push code=TIMEOUT\n"
)

with open("events.log") as f:
    errors = (line.rstrip("\n") for line in f if line.startswith("ERROR"))
    codes = (line.split("code=")[1] for line in errors)
    first_timeout = next((c for c in codes if c == "TIMEOUT"), None)
    print("lines left unread:", sum(1 for _ in f))

print(first_timeout)
Output
lines left unread: 1
TIMEOUT
Scenario A

In your world

The situation

Your job computes total = sum([row["amount"] for row in rows]) over 3M rows fetched with cur.fetchall(), and the container gets OOM-killed once a quarter when volume spikes.

What actually breaks

fetchall() already materialised 3M tuples, and the list comprehension builds a second 3M-element list. Swapping in a generator expression removes only the second list; the first is still there.

What you'd do on Monday

Make the source lazy too: iterate the cursor in chunks (cur.fetchmany(10_000) or a server-side cursor), and feed a generator expression into sum(). Better still, push the sum into SQL (SELECT SUM(amount)), which moves zero rows (Choose restartable chunks rather than arbitrary batches).

Scenario B

Imagine you work at Flipkart

300M catalogue rows in a nightly price job

The numbers

Illustrative: 300M rows × ~120 bytes of Python object per price float and key ≈ 36 GB if listed; a streamed genexp into a batched writer holds ~10k rows ≈ 1.2 MB at a time. Spread across 60 workers that is 5M rows per worker, 600 MB per worker if listed.

What the Scenario-A answer does here

The Scenario A fix of a genexp over fetchall() is still ~600 MB per worker, because the source is eager. The late-binding trap also becomes real: a genexp filtering on price > threshold, created before a config reload that changes threshold, silently applies the new value halfway through a run.

What flips at this scale

At this scale the whole chain must be lazy end to end, from a server-side cursor through genexps to a batched writer, and any value a condition depends on must be bound at creation time (a default argument, a local, or functools.partial), not read from mutable module state.

The trade-off you'd defend

I would defend fully lazy chains with frozen parameters, accepting that debugging is harder, since you cannot print an intermediate list, in exchange for flat memory. For debugging I would add counters, not lists.

Illustrative sizing, not Flipkart-published figures.

Gotchas

Conditions read variables at pull time

Symptom

A generator filtered with x > threshold uses a value set after the generator was created.

Cause

Only the first for source is evaluated eagerly; the condition runs lazily on every pull.

Fix

Bind the value when you create it: limit = threshold just before, or build the generator inside a function that takes threshold as a parameter.

Consuming a file-backed genexp after the with closed it

Symptom

ValueError: I/O operation on closed file when the generator is used after returning from a function.

Cause

The genexp pulls from f lazily, and f was closed when the with block exited.

Fix

Consume inside the with, or make the whole thing a generator function that owns the with block (Suspend computation with yield).

A genexp over a list saves almost nothing

Symptom

Memory is unchanged after replacing [...] with (...).

Cause

The source list already holds everything; the genexp removes only the second copy.

Fix

Make the source lazy: file iteration, fetchmany, a paginated generator.

Tricky questions

What does this print? t = 10; g = (x for x in [5, 50, 500] if x > t); t = 100; print(list(g))
Most people say

[50, 500], because t was 10 when g was created.

Actually

[500]. The condition runs at pull time and sees t = 100. Only the source list [5, 50, 500] was captured eagerly.

Is sum(x for x in xs) a generator expression inside a tuple?
Most people say

It needs double parentheses: sum((x for x in xs)).

Actually

When a genexp is the only argument, the call's parentheses are enough. With two arguments you need its own: sum((x for x in xs), start).

Scenario question

You're handed a reporting script that does rows = cur.fetchall(), then bad = [r for r in rows if not valid(r)], then if any([r["amount"] < 0 for r in rows]): alert(). It runs out of memory on month-end. What do you change, in order?

Model answer

I would change three things, cheapest first. The any([...]) should be any(r["amount"] < 0 for r in rows): it short-circuits and builds no list. bad only needs a count, so it becomes sum(1 for r in rows if not valid(r)). The real fix is the source: I would replace fetchall() with chunked fetchmany, or move the checks into SQL (COUNT(*) FILTER (WHERE amount < 0)), because a lazy expression over an eager list still holds the list. The trade-off is that with a streamed source I can only make one pass, so I would compute both checks in a single loop.

Interview corner

CommonList comprehension or generator expression: how do you choose?
“I'd use a generator expression when the result is consumed once, especially by sum, any, join or a writer, and a list when I need len, indexing, sorting or a second pass. The generator trades random access and reuse for constant memory.”
TrickWhen is a generator expression not a memory win?
“I'd point out that if the source is already a materialised list, the genexp saves only the output copy. I fix memory at the source, with lazy file iteration or chunked DB fetches.”
TrickWhich parts of a generator expression are evaluated immediately?
“Only the first for's iterable. I remember it because conditions read variables at pull time, which is a real bug if a config value changes mid-run.”

Memory hook

Brackets cook everything now; parentheses cook to order, from whatever recipe is on the card at that moment.

If you remember only one thing:

Lazy output over an eager source is still eager. Fix the source first.

Self-check

  1. Why does the second sum(squares_gen) return 0?

    The generator was exhausted by the first sum; generator expressions are single-pass iterators.

  2. Why does any(big_check(x) for x in xs) beat any([big_check(x) for x in xs])?

    The generator short-circuits and stops calling big_check at the first True; the list version calls it for every element first.

  3. In the production example, why is one line left unread?

    next() stopped the chain at the first TIMEOUT, so the last line of the file was never pulled.

Prove it

Using tracemalloc, measure peak memory for sum([i for i in range(5_000_000)]) versus sum(i for i in range(5_000_000)), and print both peaks in MB. Then show the late-binding bug with a threshold variable and fix it by binding the value inside a function.

py-04-04Card 4 of 6

Compose streams with itertools

MediumP3 · Role-dependentTalk about it9 min
itertoolspipelinesbatchingiterators

Why you care

itertools turns hand-written loops with counters and flags into composable, lazy building blocks: take the first n, glue many sources, batch, group, pair adjacent rows. In ETL code that means fewer off-by-one bugs and no accidental lists. It is also a quiet interview signal: reaching for islice or batched instead of writing a counter says you think in streams. Know five tools cold and know that the rest exist.

Explain like I'm 5

Think of pipes and fittings in a water system. chain joins several pipes into one. islice is a valve that lets exactly n litres through. batched fills buckets of a fixed size. groupby puts a divider every time the colour of the water changes. None of them store water; they only shape how it flows.

Core concept

The ones worth knowing cold, all lazy and all returning iterators:

  • islice(it, stop) / islice(it, start, stop, step): slicing for any iterator. It consumes what it skips.
  • chain(a, b, …) and chain.from_iterable(list_of_iterables): one stream from many sources, like 40 daily files read as one.
  • batched(it, n) (Python 3.12+): tuples of up to n items; the last may be shorter. 3.13 added strict=True to fail on a short final batch. This is the standard shape for executemany or bulk API calls.
  • groupby(it, key): groups consecutive items with the same key. Unsorted input gives repeated groups. Each group shares the underlying iterator, so it becomes invalid once you advance to the next group.
  • pairwise(it) (3.10+): (a, b), (b, c)…, handy for gaps between timestamps.

Also know that these exist: accumulate (running totals), count, repeat, cycle, zip_longest, takewhile, dropwhile, starmap, product, permutations, combinations, tee. And from outside itertools, heapq.merge, which lazily merges already-sorted streams.

tee(it, n) gives n independent iterators but buffers everything one consumer has seen and another has not, so if one runs far ahead, tee quietly becomes a list. Freshness note: since Python 3.14, itertools objects no longer support copy, deepcopy or pickle (deprecated in 3.12), so do not try to checkpoint a pipeline by pickling it. Persist the last processed key instead.

Minimal example Five tools, one screen

Pythonislice, chain, batched, groupby, pairwiseruns
from itertools import batched, chain, groupby, islice, pairwise

files = [["a1", "a2"], ["b1"], ["c1", "c2", "c3"]]
rows = chain.from_iterable(files)
print(list(islice(rows, 4)))
print(list(rows))
print([list(b) for b in batched(range(7), 3)])
print([(k, len(list(g))) for k, g in groupby("xxyx")])
print([b - a for a, b in pairwise([3, 5, 9, 10])])
Output
['a1', 'a2', 'b1', 'c1']
['c2', 'c3']
[[0, 1, 2], [3, 4, 5], [6]]
[('x', 2), ('y', 1), ('x', 1)]
[2, 4, 1]

In production Merge sorted exchange files, total per client, write in batches

Both inputs are sorted by client id, so heapq.merge plus groupby produces per-client totals with only one group in memory, and batched shapes the writes.

PythonStreaming merge-group-batch pipelineruns
import heapq
from itertools import batched, groupby
from operator import itemgetter

nse = [("C1", 100), ("C3", 40)]
bse = [("C1", 25), ("C2", 70), ("C3", 5)]

merged = heapq.merge(nse, bse, key=itemgetter(0))
totals = (
    (client, sum(qty for _, qty in trades))
    for client, trades in groupby(merged, key=itemgetter(0))
)
for batch in batched(totals, 2):
    print("INSERT batch:", batch)
Output
INSERT batch: (('C1', 125), ('C2', 70))
INSERT batch: (('C3', 45),)

See it happen Pull flows backwards through the pipeline

nse.csvbse.csvheapq.mergeC3: 451 of 23 rows
  1. Building the pipeline reads nothing. Every stage is an idle iterator.
  2. The for loop asks batched for a batch. batched asks groupby for group 1.
  3. groupby pulls from merge until the key changes: C1 from nse and C1 from bse.
  4. It pulled C2 to notice the key changed. Group 2 is C2: 70, and the batch is full.
  5. The batch goes to INSERT. At most 2 totals and 1 group are ever held.
  6. The loop pulls again: C3 = 40 + 5, the sources end, and the short final batch flushes.
Notice: data moves left to right, but requests move right to left. The writer sets the pace, and nothing upstream runs ahead of it.
Scenario A

In your world

The situation

Your job concatenates 30 daily CSVs with all_rows = [] and all_rows.extend(...) per file, then loops for i in range(0, len(all_rows), 1000) to insert in slices.

What actually breaks

The list costs memory for every file at once, and the slicing loop has an off-by-one history (the last partial batch was skipped once, which is why there is a # fix comment on it).

What you'd do on Monday

Stream it: rows = chain.from_iterable(read_csv(p) for p in paths), then for batch in batched(rows, 1000): cur.executemany(SQL, batch). There is no index arithmetic, the last short batch is automatic, and one file is open at a time.

Scenario B

Imagine you work at Zerodha

40 exchange files, 25M trades a day

The numbers

Illustrative: 25M trades × ~150 bytes ≈ 3.75 GB across 40 files. A list-based per-client total with a dict of 12M clients × ~200 bytes ≈ 2.4 GB of RAM. A merge + groupby over files pre-sorted by client holds one group at a time: kilobytes.

What the Scenario-A answer does here

The Scenario A fix (chain + batched) streams rows, but it still needs the dict for per-client totals, because the files are not grouped. groupby on unsorted input silently emits the same client many times, and the upsert then overwrites totals with partial sums.

What flips at this scale

At this scale the sort order becomes a contract. The upstream export sorts by client id, you use heapq.merge + groupby for O(1) memory, and you assert monotonic keys as you stream (compare with the previous key and fail on a step backwards) so a badly sorted file fails loudly instead of corrupting totals.

The trade-off you'd defend

I would demand sorted inputs and assert them in-stream, trading a dependency on upstream ordering, and an extra sort step if they cannot guarantee it, for constant-memory aggregation. If sorting is impossible, the dict is honest and I would size the worker for it.

Illustrative sizes for the arithmetic; not Zerodha-published figures.

Gotchas

groupby on unsorted data

Symptom

The same key appears in several groups; totals are split or overwritten.

Cause

groupby groups only consecutive equal keys.

Fix

Sort by the same key first, or guarantee sorted input and assert that keys only move forward.

Using a group after advancing

Symptom

list(group) is empty when you saved groups for later.

Cause

All groups share one underlying iterator; moving to the next group drains the previous one.

Fix

Materialise each group inside the loop (list(g)) if you really need it later.

islice does not rewind

Symptom

After islice(rows, 100) for a preview, the main loop misses the first 100 rows.

Cause

islice consumes the underlying iterator.

Fix

Preview from a separate iterator (reopen the file), or preview the batch you are already processing.

tee turning into a list

Symptom

Memory grows steadily when two consumers of tee run at different speeds.

Cause

tee buffers every item the slow consumer has not seen yet.

Fix

Compute both results in one loop instead of teeing.

Tricky questions

Is batched just zip(*[iter(xs)] * n)?
Most people say

Yes, it's the same idiom.

Actually

The zip idiom silently drops the incomplete final batch; batched keeps it as a shorter tuple. That dropped tail is exactly the "last 734 rows never load" bug.

Does chain(a, b) copy a and b?
Most people say

Yes, it builds a combined list.

Actually

No. It holds references and pulls from a until it is exhausted, then from b. It is O(1) memory.

Scenario question

You're handed a reconciliation script that loads two 5 GB sorted CSV exports into dicts to find ids present in one but not the other. It needs 22 GB of RAM. Redesign it.

Model answer

Because both files are sorted by id, I would walk them together like a merge step: read one row from each and advance whichever has the smaller id, emitting "only in A" or "only in B" when they differ. That is O(1) memory and a single pass per file. In Python I would write it as a generator over two iterators, and use groupby if ids can repeat. The trade-off is that I depend on sort order, so I would assert it while streaming and fail fast on a regression. If the exports cannot be sorted, I would load them into Postgres and use an anti-join instead of RAM.

Interview corner

CommonWhich itertools do you actually use and for what?
“I'd name islice for previews and limits, chain.from_iterable to read many files as one stream, batched for bulk inserts, groupby on sorted data for per-key aggregates, and pairwise for deltas between rows. All of them are lazy, so they keep memory flat.”
TrickWhat goes wrong with groupby on a list of dicts from a query?
“Unless the query has ORDER BY on the grouping key, I'd get the same key in several groups. I sort, or I push the grouping into SQL, which is usually better anyway.”
DesignHow would you batch 10M rows into Postgres from Python?
“I'd stream rows through batched(rows, 5_000) into executemany or, better, COPY, commit per batch, and record the last key per batch so a restart resumes instead of duplicating. The batch size is a trade-off between round trips and transaction size.”

Memory hook

itertools are plumbing fittings: they shape the flow and never hold the water.

If you remember only one thing:

groupby groups neighbours, not values. Sort first.

Self-check

  1. What does list(batched("abcde", 2)) return?

    [('a', 'b'), ('c', 'd'), ('e',)].

  2. Why can the pipeline in the production example aggregate millions of clients in constant memory?

    Inputs are sorted by client, so groupby sees one client's trades at a time and batched holds at most one batch.

  3. What changed for itertools objects in Python 3.14?

    copy, deepcopy and pickle support was removed, so persist a resume key instead of pickling pipeline state.

Prove it

Generate two sorted files of 100,000 ids each, with about 1% of ids unique to each side, and write the merge-walk from the scenario question as a generator. Print the count of ids only in A and only in B, and show with tracemalloc that peak memory stays under 1 MB.

py-04-05Card 5 of 6

Stream a 20 GB file within a memory budget

MediumP2 · Very likelyKnow cold10 min
streaminglarge-filesmemorybatchingetl

Why you care

"How would you process a 20 GB file on a machine with 4 GB of RAM?" is the most common practical Python question for data-heavy backend roles, and it is literally your job. A strong answer is not "use a generator". It is a small, correct design: stream records, bound the batch, commit per batch with a resume point, account for the one record that is bigger than your budget, and clean up deterministically. This card is that design, built from Iterator protocol, exhaustion and pull model to Compose streams with itertools.

Explain like I'm 5

You cannot drink a swimming pool in one gulp, but you can drink it with a cup. The cup is your batch. What matters is that the cup never gets bigger: however large the pool, you only ever hold one cupful. The catch is that if something in the pool is bigger than the cup, like a beach ball, you need a plan before you meet it.

Core concept

The design, in order:

  1. Stream the source. for line in f reads through an ~8 KB buffer. gzip.open(path, "rt") streams compressed files. For S3, iterate the body in chunks or lines instead of .read(). Never read(), readlines() or list(reader).
  2. Parse lazily. A generator turns raw lines into records and sends bad lines to a reject list with the line number instead of crashing the job.
  3. Bound the batch. batched(records, 5_000) fixes working memory at about batch size × record size.
  4. Commit per batch with a checkpoint. One transaction per batch, and store the last id or byte offset, so a crash at batch 9,000 resumes at 9,000 instead of starting over or duplicating (Choose restartable chunks rather than arbitrary batches).
  5. Clean up deterministically. with on the file and the connection, so an exception mid-stream closes both (Context managers and deterministic cleanup).

The memory bound is batch size × average record size + the largest single record. Line iteration is only as bounded as your longest line. A file that is one giant JSON array, or a CSV with a multi-megabyte quoted field, breaks the assumption. For JSON, insist on NDJSON (one object per line) or use a streaming parser. For CSV, set csv.field_size_limit deliberately and reject oversized records.

Measure, don't guess: tracemalloc peak for Python objects, or RSS from the container metrics.

Minimal example Streaming vs readlines, measured

PythonSame answer, very different peak memoryruns
import tracemalloc

with open("big.csv", "w") as f:
    f.write("id,amount\n")
    for i in range(200_000):
        f.write(f"{i},{i % 997}\n")


def stream_total(path):
    total = 0
    with open(path) as f:
        next(f)                                    # skip header
        for line in f:
            total += int(line.split(",")[1])
    return total


def eager_total(path):
    with open(path) as f:
        lines = f.readlines()[1:]
    return sum(int(line.split(",")[1]) for line in lines)


for fn in (stream_total, eager_total):
    tracemalloc.start()
    result = fn("big.csv")
    peak = tracemalloc.get_traced_memory()[1]
    tracemalloc.stop()
    print(fn.__name__, result, "peak under 1 MB:", peak < 1_000_000)
Output
stream_total 99480900 peak under 1 MB: True
eager_total 99480900 peak under 1 MB: False

In production Batched, transactional load with rejects

sqlite3 stands in for Postgres so this runs anywhere; with psycopg the shape is identical, with COPY or executemany per batch and with conn.transaction():.

PythonStream, parse, batch, commit per batchruns
import csv
import sqlite3
from itertools import batched
from pathlib import Path

Path("orders.csv").write_text(
    "id,amount\n" + "".join(f"{i},{i * 3}\n" for i in range(1, 12)) + "bad,row\n"
)


def parse(rows, rejects):
    for lineno, row in enumerate(rows, start=2):
        try:
            yield int(row["id"]), int(row["amount"])
        except (KeyError, ValueError):
            rejects.append(lineno)


def load(path, db, batch_size=4):
    rejects = []
    with open(path, newline="") as f:
        records = parse(csv.DictReader(f), rejects)
        for n, batch in enumerate(batched(records, batch_size), start=1):
            with db:                               # one transaction per batch
                db.executemany("INSERT INTO orders VALUES (?, ?)", batch)
            print(f"batch {n}: {len(batch)} rows committed")
    return rejects


db = sqlite3.connect(":memory:")
db.execute("CREATE TABLE orders (id INTEGER PRIMARY KEY, amount INTEGER)")
print("rejected lines:", load("orders.csv", db))
print("rows in table:", db.execute("SELECT COUNT(*) FROM orders").fetchone()[0])
Output
batch 1: 4 rows committed
batch 2: 4 rows committed
batch 3: 3 rows committed
rejected lines: [13]
rows in table: 11

See it happen Memory stays one batch wide

file (20 GB)r1r2r3r4r5r6...batch (max 4)r1r2r3r4r5r6...committed0
  1. Nothing is read yet. The file stays on disk; memory holds an empty batch.
  2. The reader pulls r1 and r2. Each row is parsed and appended to the batch.
  3. After r3 and r4 the batch is full: 4 rows is the entire working set.
  4. Commit the batch in one transaction and record the checkpoint (last id = 4).
  5. The same cup refills with r5 and r6. Memory never grows with file size.
  6. The naive readlines() version: every row is held at once, and memory grows with the file.
Notice: the batch row has a fixed width in every frame except the last. That fixed width is what "bounded memory" means.
Scenario A

In your world

The situation

A vendor starts sending a 20 GB monthly dump to S3. Your current script does body = obj["Body"].read().decode() and for line in body.splitlines(), on a 4 GB container.

What actually breaks

read() tries to hold 20 GB of bytes, then decode() makes a second copy and splitlines() a third. The container is OOM-killed in the first minute and the retry does the same.

What you'd do on Monday

Stream the body and keep everything else lazy:

PythonStreaming an S3 object (needs boto3 + AWS credentials)
import io
from itertools import batched

import boto3

obj = boto3.client("s3").get_object(Bucket=BUCKET, Key=KEY)
with io.TextIOWrapper(obj["Body"], encoding="utf-8", newline="") as text:
    for batch in batched(parse(csv.DictReader(text), rejects), 5_000):
        copy_batch(conn, batch)          # COPY + commit + checkpoint
Scenario B

Imagine you work at Meesho

20 GB supplier catalogue dump, 4 GB workers

The numbers

Illustrative: 20 GB ÷ ~200 bytes/row ≈ 100M rows. With batches of 5,000 that is 20,000 commits. At ~1 KB per parsed row in Python, a batch is ~5 MB, so the working set is far under 4 GB. At 50k rows/s the load takes ~33 minutes on one worker; split by byte range across 8 workers, about 4 minutes.

What the Scenario-A answer does here

The Scenario A design (one streaming worker) holds memory, but a crash at minute 30 restarts from zero, and one supplier sends a single 2.3 GB line (a product description field full of embedded base64 images) that blows the "one line at a time" bound.

What flips at this scale

At this size you add three things. Checkpoints: store the last committed id or byte offset, so a restart resumes. Parallelism by byte range: seek to an offset, skip to the next newline, and stop at the range end. A per-record size guard: reject any line over, say, 1 MB into a quarantine file instead of parsing it.

The trade-off you'd defend

I would defend checkpoints plus a record-size guard over simply buying a bigger box, accepting more code paths in exchange for a job whose memory and restart cost stay fixed as the vendor grows. Parallel byte ranges come only after a single worker proves too slow.

Illustrative figures to make the arithmetic concrete; not Meesho-published numbers.

Gotchas

A list somewhere in the pipeline

Symptom

Memory grows with file size even though "it uses generators".

Cause

One list(), readlines(), sorted() or fetchall() in the chain materialises everything before it.

Fix

Search the pipeline for eager calls; replace them with lazy ones, or push sorting and grouping into the database.

The largest record, not the average, sets the bound

Symptom

A job that ran for months OOMs on one file.

Cause

A single enormous line or CSV field; line iteration must hold the whole line.

Fix

Guard record size (length check, csv.field_size_limit), quarantine oversized records, and prefer NDJSON over one big JSON document.

Restarting duplicates rows

Symptom

After a crash and rerun, row counts are higher than the source.

Cause

Committed batches are reinserted from the beginning.

Fix

Checkpoint the last committed key per batch in the same transaction, or upsert on a natural key.

Tricky questions

Does for line in f read the whole file into memory?
Most people say

Yes, Python loads the file and then splits it.

Actually

No. File objects are iterators over a buffered stream, which reads a chunk (around 8 KB) and yields line by line. Memory is one buffer plus the current line.

Is a generator enough to guarantee bounded memory?
Most people say

Yes, generators are lazy.

Actually

Only if every stage is lazy and records are bounded. A single sorted(), a dict of all keys or one giant line breaks it. The bound is batch × record size + the largest record.

Scenario question

You're handed "process a 20 GB CSV on a 4 GB machine and load it into Postgres; the job must be restartable." Walk me through your design.

Model answer

I would stream the file with csv.reader over the file object, never read(), and parse each row in a generator that sends bad rows to a reject file with line numbers. I would batch 5,000 rows with itertools.batched and load each batch with COPY inside one transaction, which also writes the batch's last line number to a checkpoint table. On restart I skip to the checkpoint. I would cap record size so one huge row cannot blow the memory bound, and measure the peak with tracemalloc on a sample. The trade-off is batch size: bigger batches mean fewer round trips but longer transactions and more work to redo; I would start at about 5,000 and tune using throughput and lock time.

Interview corner

DesignProcess a 20 GB file with 4 GB of RAM: how?
“I'd stream it: iterate the file, parse lazily in a generator, batch with itertools.batched, and commit per batch with a checkpoint for restarts. Memory is bounded by batch size times record size plus the largest record, so I'd also guard against oversized rows.”
TrickYour streaming job still runs out of memory. Where do you look?
“I'd look for the one eager call in the chain, a list, sorted, readlines, fetchall or a dict of all keys, and then at record size, because one giant line breaks line-by-line streaming. I confirm with tracemalloc snapshots rather than guessing.”
Experience checkHow do you make a big load restartable?
“I commit per batch and write a checkpoint, the last id or line number, in the same transaction, so a restart resumes exactly after the last committed batch. Where the source has a natural key I also upsert, so a replay can't duplicate.”
DesignWould you parallelise it?
“Only after measuring one worker. If needed, I'd split by byte ranges and snap each range to the next newline, but I'd keep checkpoints per range. Often the database, not Python, is the bottleneck, so COPY beats more workers.”

Memory hook

Drink the pool with a cup, and check for beach balls.

If you remember only one thing:

Memory is batch × record size + the largest record. Every eager call breaks it.

Self-check

  1. Why is eager_total's peak much higher than stream_total's?

    readlines() holds all 200k line strings, plus the slice copy, at once; streaming holds one buffered line.

  2. What must be in the same transaction as each batch insert to make restarts safe?

    The checkpoint update, the last committed key or line number, so both commit or roll back together.

  3. Why prefer NDJSON over a single JSON array for huge exports?

    Each line is an independent record you can stream; a single array must be parsed whole, unless you use a streaming parser.

Prove it

Generate a 1 GB CSV locally (a loop writing about 10M rows). Load it into SQLite with the production pattern, using batch size 10,000 and a checkpoint table. Kill the process midway with Ctrl+C, rerun, and show that it resumes from the checkpoint with no duplicate ids. Report peak memory from tracemalloc.

py-04-06Card 6 of 6

Delegate generators and exchange values

HardP4 · DifferentiatorTalk about it9 min
generatorssendyield-fromcoroutines

Why you care

You will rarely write gen.send() at work. You will use its descendants every day: yield from is how generators delegate and return values, @contextmanager, pytest fixtures and FastAPI yield dependencies are built on generator control (next, throw, close), and async/await grew out of generator coroutines. This card exists for depth questions ("what does yield from do beyond a loop?") and so you can say clearly why this is not asyncio.

Explain like I'm 5

So far the waiter only brought plates to you. send() lets you hand the waiter a note while they are standing there, and they read it before fetching the next plate. yield from is the head waiter saying "my colleague will serve you for a while": everything you say goes to the colleague, and when the colleague is done they tell the head waiter how it went.

Core concept

send(value) resumes a paused generator and makes the paused yield expression evaluate to value, so x = yield out both emits out and receives x. A just-created generator must be primed with next(g) or g.send(None) first, because there is no paused yield to receive anything.

yield from sub delegates to a sub-iterator:

  • every value sub yields goes straight to the caller;
  • send(), throw() and close() from the caller are forwarded to sub;
  • when sub returns, yield from evaluates to its return value.

A plain for x in sub: yield x does the first part but not forwarding or return values.

Not asyncio. Native coroutines are async def and use await; they are driven by an event loop, not by your next() calls. The old generator-based @asyncio.coroutine style was removed in Python 3.11. Generators with send are a synchronous control-flow tool: accumulators, parsers and state machines (Follow the event loop through await).

Under the hood

PEP 380 defines yield from precisely. While delegating, the outer generator's gi_yieldfrom points at the sub-generator, and the interpreter forwards send, throw and close without running outer code. close() raises GeneratorExit at the paused yield; if the generator catches it and yields again, Python raises RuntimeError("generator ignored GeneratorExit"). @contextlib.contextmanager is built on exactly this machinery: __enter__ calls next(), __exit__ calls next() again on success or throw(exc) on failure, which is why an exception inside a with appears at the yield line of your context manager.

Minimal example A running average you feed with send()

Pythonsend() both delivers a value and resumesruns
def running_avg():
    total = count = 0
    avg = None
    while True:
        value = yield avg
        total += value
        count += 1
        avg = total / count


acc = running_avg()
next(acc)                        # prime: run to the first yield
print(acc.send(10))
print(acc.send(20))
print(acc.send(60))

try:
    running_avg().send(5)
except TypeError as e:
    print("TypeError:", e)
Output
10.0
15.0
30.0
TypeError: can't send non-None value to a just-started generator

In production Delegation that collects per-file counts

yield from passes every row up to the caller and hands each sub-generator's return value back to the delegator: exactly what a plain for loop cannot do.

Pythonyield from returns the sub-generator's valueruns
def read_file(name, rows):
    count = 0
    for r in rows:
        yield f"{name}:{r}"
        count += 1
    return count


def all_files(files):
    counts = {}
    for name, rows in files.items():
        counts[name] = yield from read_file(name, rows)
    return counts


gen = all_files({"a.csv": [1, 2], "b.csv": [3]})
out = []
while True:
    try:
        out.append(next(gen))
    except StopIteration as stop:
        print(out)
        print("per-file counts:", stop.value)
        break
Output
['a.csv:1', 'a.csv:2', 'b.csv:3']
per-file counts: {'a.csv': 2, 'b.csv': 1}

See it happen yield from is a transparent pipe

callerall_filesread_file(a)next()yield from: forward next()'a.csv:1'next() -> 'a.csv:2'return 2next file: b.csvStopIteration(counts)
  1. The caller only ever talks to the outer generator.
  2. all_files is paused at yield from and forwards the request.
  3. The value goes straight to the caller; no outer code runs.
  4. The next pull is forwarded too.
  5. The sub-generator finishes; yield from evaluates to 2 and counts['a.csv'] = 2.
  6. Only now does outer code run again, starting the next delegation.
  7. All files done: the outer return value rides on StopIteration.
Notice: between delegation start and the sub-generator's return, the outer generator executes no code at all. It is a pipe, not a loop.
Scenario A

In your world

The situation

Your team writes a pytest fixture and a FastAPI dependency the same way: open a DB connection, yield conn, then close it. Someone adds try/except Exception: pass around the yield "to be safe".

What actually breaks

When a test or request fails, the exception is thrown into the generator at the yield, and the bare except swallows it. Teardown runs, but the failure disappears: the request returns 200 and the test goes green.

What you'd do on Monday

Use try/finally around the yield for cleanup, and never swallow exceptions there. If you must handle one, re-raise:

PythonCleanup with finally, exceptions still propagateruns
from contextlib import contextmanager


@contextmanager
def connection():
    print("open")
    try:
        yield "conn"
    finally:
        print("close")


try:
    with connection() as c:
        raise ValueError("insert failed")
except ValueError as e:
    print("caller still sees:", e)
Output
open
close
caller still sees: insert failed
Scenario B

Imagine you work at Dream11

150k score updates per minute into a state machine

The numbers

Illustrative: 150k updates/min ≈ 2,500/s across ~40k live contests, about 3.75 updates per contest per minute. A per-contest state machine written as a send()-driven generator holds a few hundred bytes of state; 40k of them ≈ 40k × ~1 KB ≈ 40 MB per process.

What the Scenario-A answer does here

The Scenario A pattern (generators as context managers) is fine for resources, but one-generator-per-contest driven by send() in a single process has no persistence: a restart loses every contest's in-memory state, and a single bad update raising inside one generator kills that state machine permanently (a generator that raised is finished).

What flips at this scale

At this scale, send-driven generators are a neat way to write the logic, but state must live outside the process: in Redis or Postgres with an explicit state column. The consumer becomes async I/O around a pure transition function state, event -> new_state, and generators are replaced by plain functions that are easy to persist, replay and test.

The trade-off you'd defend

I would keep generator coroutines for local parsing and protocol code, and use explicit, persisted state machines for business state, trading elegance for restartability and observability.

Illustrative figures chosen for the arithmetic; not Dream11-published numbers.

Gotchas

Sending to an unprimed generator

Symptom

TypeError: can't send non-None value to a just-started generator.

Cause

There is no paused yield to receive the value yet.

Fix

Call next(g) (or g.send(None)) once first, or wrap creation in a small helper that primes it.

Swallowing exceptions around yield in context managers

Symptom

Errors inside with blocks, fixtures or FastAPI dependencies disappear.

Cause

The failure is thrown into the generator at yield; a broad except eats it.

Fix

try/finally for cleanup; re-raise if you catch anything.

A generator that raised is finished

Symptom

After one bad send(), every later send() raises StopIteration.

Cause

An unhandled exception inside a generator terminates its frame.

Fix

Handle expected errors inside the loop, or build a new generator on failure.

Tricky questions

Is yield from sub the same as for x in sub: yield x?
Most people say

Yes, it's shorthand.

Actually

Only for plain iteration. yield from also forwards send(), throw() and close() to sub, and evaluates to sub's return value. The loop version drops all three.

Are generator coroutines the same thing as async def coroutines?
Most people say

Yes, await is just yield from.

Actually

They share history, but native coroutines are a separate type, driven by an event loop, and cannot be iterated with next(). Generator-based asyncio coroutines were removed in 3.11.

Scenario question

You're handed a parser for a vendor's line protocol, written as a 300-line class with self.state flags ("in_header", "in_body", "expect_trailer"). Bugs cluster around state transitions. Would you rewrite it with a generator driven by send()?

Model answer

I would consider it, because a generator keeps the state implicit in where the code is paused: the header loop, then the body loop, then the trailer. The transitions are just the order of the code, which removes the flag combinations that cause bugs. I would prime it once, send() each line, and return the parsed record via StopIteration.value or yield records as they complete. The trade-off is that fewer engineers read send() fluently and the state cannot be persisted or inspected, so if the parser must survive restarts or be debugged in production, I would instead write an explicit state enum with a transition table and tests for every edge.

Interview corner

TrickWhat does yield from do that a for-loop doesn't?
“I'd say it forwards send, throw and close to the sub-generator and returns the sub-generator's return value; the for-loop only re-yields values. That's why yield from is used for delegation, not just flattening.”
CommonWhat is send() for?
“I'd describe it as resuming a generator while passing a value into the paused yield. I've used the idea for accumulators and parsers; you have to prime the generator with next() first.”
Experience checkWhere do you rely on generator control in everyday code?
“I rely on it in @contextmanager, pytest fixtures and FastAPI yield dependencies. The framework calls next() for setup and throw() or next() for teardown, so I always use try/finally around the yield.”
DesignGenerator coroutine or asyncio?
“I'd use asyncio for concurrent I/O, because it has an event loop and cancellation. Generator coroutines are synchronous control flow. I'd use them for local parsing, never to fake concurrency.”

Memory hook

send hands the waiter a note; yield from hands you to a colleague, who reports back when done.

If you remember only one thing:

yield from = values out + send/throw/close in + return value back.

Self-check

  1. Why must a generator be primed before send(value)?

    Execution has not reached any yield yet, so there is nothing to receive the value.

  2. What does x = yield from sub() assign to x?

    The value sub() returns, taken from its StopIteration.value.

  3. Why can an exception inside a with block appear at the yield line of a @contextmanager?

    __exit__ throws the exception into the generator at the paused yield.

Prove it

Rewrite the running average as a class with add(value) returning the average, and as a generator with send. Write three asserts that both give identical results for the same inputs. Then write a @contextmanager that prints "rollback" on exception and "commit" otherwise, and show both paths with output.