Iterator protocol, exhaustion and pull model
Why you care
Half the "the job succeeded but loaded 0 rows" incidents in Python ETL come from one fact: some objects can be looped over once and only once. csv.reader, a file handle, map(), zip() and a generator are all one-shot. A list, a dict and a range are not. If you cannot tell the two apart at a glance, you will write a second pass that silently does nothing, and no exception will tell you. Interviewers ask "what is the difference between an iterable and an iterator" precisely because the answer predicts whether you have been bitten by this.
Explain like I'm 5
A book is an iterable: anyone can start reading it from page one. A bookmark is an iterator: it remembers where one particular reader is. iter(book) hands you a fresh bookmark. next(bookmark) reads the page and moves the bookmark forward. When the bookmark falls off the last page it shouts "StopIteration" and the reader stops. If you give someone your used bookmark instead of the book, they start at the end and read nothing.
Core concept
Two methods make up the whole protocol:
- An iterable has
__iter__(), which returns an iterator. Lists, tuples, dicts, sets, strings,range, files and your own classes can be iterables. - An iterator has
__next__(), which returns the next value or raisesStopIteration, and__iter__(), which returns itself. That last rule is why an iterator is also an iterable, and why looping over it twice gives you nothing the second time: the seconditer()call hands back the same, already-advanced object.
A for loop is sugar for: call iter(x) once, then call next() until StopIteration, which the loop swallows. list(), sum(), max(), in, unpacking, zip() and dict() all do the same pull. Pulling is the key word: the consumer asks for one value at a time and the producer does just enough work to hand it over. Nothing is computed ahead. That pull model is what later lets a 20 GB file flow through a script that holds one line at a time (Stream a 20 GB file within a memory budget).
The practical test: iter(x) is x is True for an iterator (one-shot) and False for a container (re-iterable). When your function accepts "rows", decide which one it needs. If it needs two passes, either demand a container, call list() deliberately, or accept a re-iterable object whose __iter__ starts over each time.
Under the hood
CPython compiles for x in xs: to GET_ITER followed by a FOR_ITER loop; FOR_ITER calls the iterator's tp_iternext slot and jumps out when it signals exhaustion, so there is no Python-level try/except cost per item. iter() has two fallbacks worth knowing. If a class has no __iter__ but has __getitem__, Python calls it with 0, 1, 2 … until IndexError: the old sequence protocol. And iter(callable, sentinel) calls the function until it returns the sentinel, which is how you turn f.read(65536) into a chunk stream: iter(lambda: f.read(65536), b"").
Minimal example One list, one iterator, exhaustion
nums = [10, 20, 30]
it = iter(nums)
print(next(it), next(it))
print(list(it))
print(list(it))
print(iter(it) is it, iter(nums) is nums)10 20
[30]
[]
True FalseIn production A re-iterable reader for a two-pass job
A class whose __iter__ opens the file again gives every loop its own fresh iterator. The yield from inside is covered in Suspend computation with yield; here it simply means "produce every row of the reader".
import csv
from pathlib import Path
class CsvRows:
"""Re-iterable: each for-loop reopens the file and gets a new iterator."""
def __init__(self, path):
self.path = Path(path)
def __iter__(self):
with self.path.open(newline="") as f:
yield from csv.DictReader(f)
Path("orders.csv").write_text("id,amount\n1,250\n2,90\n3,410\n")
rows = CsvRows("orders.csv")
total = sum(int(r["amount"]) for r in rows)
big = [r["id"] for r in rows if int(r["amount"]) > 200]
print(total, big)
with open("orders.csv", newline="") as f:
one_shot = csv.DictReader(f)
print(sum(1 for _ in one_shot), sum(1 for _ in one_shot))750 ['1', '3']
3 0See it happen What a for loop actually does
- The loop calls iter() exactly once, before the first value.
- The list hands back a separate object that remembers the position (index 0).
- Pull one value. Nothing else is computed.
- The iterator returns 10 and moves its index to 1.
- Two more pulls, two more values.
- Nothing left: the loop catches StopIteration and ends quietly.
- Loop again over the iterator, not the list: iter(it) returns the same exhausted object.
- Zero iterations, no error. This is the silent 0-row load.
In your world
The situation
Your nightly job pulls settlements.csv from S3, wraps it in csv.reader, logs row_count = sum(1 for _ in reader) for the audit table, then loops over reader again to insert into Postgres.
What actually breaks
The audit table says 48,213 rows. The target table gets 0. Jenkins shows green, because an empty loop is not an error. Nobody notices until finance asks where yesterday went.
What you'd do on Monday
Make one pass do both jobs, or make the source re-iterable. The Monday fix is to count while you load and assert the counts match:
loaded = 0
for row in reader:
cur.execute(INSERT_SQL, row)
loaded += 1
log.info("loaded %d rows", loaded)
if loaded == 0:
raise RuntimeError("0 rows loaded from settlements.csv, refusing to report success")Imagine you work at Razorpay
The numbers
Illustrative: 60M rows × ~180 bytes ≈ 10.8 GB of CSV a night, split into 600 files of 100k rows. A two-pass design reads 21.6 GB from S3 instead of 10.8 GB, and at ~100 MB/s per worker that is 216 s versus 108 s per worker-GB-set: a second pass doubles I/O cost.
What the Scenario-A answer does here
The Scenario A fix (count while loading) still works, but "just call list(reader) so you can loop twice" does not: 100k rows × ~600 bytes of Python objects per row ≈ 60 MB per file per worker, times 32 workers ≈ 1.9 GB of RAM spent only on holding data you already streamed once.
What flips at this scale
At this scale you design for exactly one pass: every metric (count, checksum, min/max date) is accumulated in the same loop that writes, and the reconciliation compares those in-flight numbers with the source manifest. Re-iterable sources become a cost you pay deliberately, not a convenience.
The trade-off you'd defend
I would defend single-pass accumulation plus a manifest check over re-reading, trading slightly more complex loop code for half the S3 egress and constant memory. The one exception is a small lookup file, which I would load into a dict once.
Illustrative numbers chosen to make the arithmetic visible; not Razorpay-published figures.
Gotchas
A second loop over a reader returns nothing
The second for over csv.reader, a file, map() or a generator runs zero times, with no error.
Those objects are iterators; iter() returns the same exhausted object.
Do everything in one pass, or keep a re-iterable source (a list you chose to build, or a class whose __iter__ reopens the file).
x in iterator consumes it
After if "ERROR" in lines: the next loop over lines starts in the middle.
Membership on an iterator pulls values until it finds a match, and those values are gone.
Test membership on a container (a set is O(1)), or accumulate the flag inside the main loop.
Sharing one generator between threads
ValueError: generator already executing, or duplicated and missing rows when several threads call next() on one iterator.
Generators are not thread-safe; two threads can re-enter the same frame.
Give each worker its own iterator, or hand rows out through a queue.Queue fed by a single producer (Runtime concurrency: threads, processes, async and GIL).
Tricky questions
Yes, you can loop over it.
A list is an iterable. next([1, 2]) raises TypeError: 'list' object is not an iterator. iter([1, 2]) returns a separate list_iterator, which is why the list can be looped over any number of times.
__iter__ method at all?It doesn't; only __next__ matters.
for, list() and zip() always call iter() first. Returning self lets you pass an iterator anywhere an iterable is expected, and it is exactly that return self that makes a second loop start at the end.
Scenario question
You're handed a function def dedupe_and_count(rows) that returns the correct count when called with a list but returns 0 duplicates, not an error, when called with the output of csv.reader. The code does unique = set(rows) and then return len(list(rows)) - len(unique). What is going on and how do you fix it without changing callers?
Model answer
set(rows) consumes the reader, so list(rows) sees an exhausted iterator and returns an empty list, and the result is 0 - n, or 0 if the caller clamps it. I would make it a single pass: loop once, keep a seen set and a dupes counter, and return dupes. That works for both lists and iterators and it holds only the unique keys, not every row. The trade-off is that I store keys, not full rows; if callers also need the rows, I would return them explicitly rather than rely on a second pass.
Interview corner
CommonWhat is the difference between an iterable and an iterator?
__iter__, and an iterator is the thing with a position, via __next__. The iterator's own __iter__ returns itself, which is why lists can be looped repeatedly but a csv.reader or a generator can only be looped once.”TrickYou loop over the same object twice and the second loop does nothing. Why?
iter() returned the same exhausted object. I check with iter(x) is x. The fix is either a single pass or a source whose __iter__ starts over.”Experience checkWhere has iterator exhaustion hurt you in real code?
csv.reader for an audit log and then loading from the same reader, which loaded zero rows while the job reported success. I now accumulate counts in the load loop and fail loudly on zero.”CommonWhat does a for loop do under the hood?
Calls iter() once, then next() until StopIteration, which it swallows.
it = iter(xs) followed by repeated next(it) inside a try that exits on StopIteration. Everything that consumes, like sum, list, unpacking and in, uses the same pull.”Memory hook
A list is a book; an iterator is a bookmark. Hand someone your bookmark and they start at the end.
If iter(x) is x, you get one pass. Plan for it.
Self-check
What does
iter(it) is itreturn for a file object, and why?True. A file object is its own iterator, so looping it twice reads nothing the second time unless youseek(0).Name three built-ins that return one-shot iterators.
map(),filter(),zip(),enumerate(),reversed()andopen(); alsocsv.readerand every generator.How would you turn
f.read(65536)into something a for loop can consume?for chunk in iter(lambda: f.read(65536), b""):, the two-argument form ofiter()with a sentinel.
Write class Countdown with __iter__ and __next__ that counts from n to 1. Show that list(c) twice returns [3, 2, 1] then []. Then change it into a re-iterable, where __iter__ returns a new iterator, and show [3, 2, 1] twice. Paste both outputs.