Choose collections and dict/set idioms
Why you care
A bare dict with manual increment logic for counting is three lines where Counter is one — and the bare dict raises KeyError until you initialise every key, which means the first missing key corrupts your audit numbers silently. On the grouping side, a defaultdict(list) removes the if-key-exists dance that senior reviewers flag in every code review. Interviewers ask about collections because the answer exposes whether you know the standard toolkit or reinvent it from scratch every time, and reinventing it introduces bugs the stdlib version has already fixed.
Explain like I'm 5
Imagine a ticket counter at a railway station. You are the ticket clerk and you have a tally sheet. With a plain dict, before you mark a tally you must first check whether the row even exists, draw it in, then add one. With a Counter, you just mark the tally — if the row was blank it starts at zero automatically. defaultdict is the same idea for lists: the slot appears the moment you first put something in, so you never have to check first.
Core concept
collections gives you five types that solve the counting, grouping and queue problems without reinventing them:
Counter— adictsubclass for tallying.c[missing_key]returns0, neverKeyError..most_common(n)returns thenhighest-count pairs sorted descending. Arithmetic operators (c1 - c2,c1 + c2) produce a newCounterwith zero and negative counts stripped.defaultdict(factory)— adictwhose factory function (e.g.list,int,set) is called for any missing key, sod[key].append(val)is always safe.namedtuple— an immutable row type with named fields. Lighter than a dataclass, hashable, and serialises cleanly to a dict with._asdict().deque— a double-ended queue withO(1)appends and pops at both ends. Use it for sliding-window accumulation and bounded buffers (maxlen=Nauto-discards the oldest entry when you append).OrderedDict— preserves insertion order (plaindictalso preserves order since 3.7) and adds.move_to_end(key)needed for LRU logic.
Decision rule: use Counter for any counting problem; defaultdict(list) for grouping; deque for bounded sliding windows; namedtuple for lightweight rows that must be hashable. Use a plain dict everywhere else — it is faster and more readable for simple key-value stores. Converting a list to a set for deduplication discards insertion order silently; use dict.fromkeys(items) instead when order matters.
Minimal example Counter and defaultdict basics
from collections import Counter, defaultdict
events = ['settled', 'failed', 'settled', 'pending', 'settled', 'failed']
c = Counter(events)
print(c.most_common(2))
print(c['unknown'])
rows = [('INR', 1200), ('USD', 50), ('INR', 800), ('USD', 30)]
by_ccy = defaultdict(list)
for ccy, amt in rows:
by_ccy[ccy].append(amt)
print(sorted(by_ccy.keys()))
print(sorted(by_ccy['INR']))[('settled', 3), ('failed', 2)]
0
['INR', 'USD']
[800, 1200]In production Settlement aggregation by currency and status
from collections import Counter, defaultdict, namedtuple
Settlement = namedtuple('Settlement', ['id', 'currency', 'amount', 'status'])
rows = [
Settlement('S01', 'INR', 1200, 'settled'),
Settlement('S02', 'USD', 50, 'settled'),
Settlement('S03', 'INR', 800, 'failed'),
Settlement('S04', 'USD', 30, 'settled'),
Settlement('S05', 'INR', 400, 'pending'),
Settlement('S06', 'INR', 600, 'settled'),
]
totals = defaultdict(int)
status_by_ccy = defaultdict(Counter)
for s in rows:
totals[s.currency] += s.amount
status_by_ccy[s.currency][s.status] += 1
for ccy in sorted(totals):
print(ccy, totals[ccy], dict(status_by_ccy[ccy]))INR 3000 {'settled': 2, 'failed': 1, 'pending': 1}
USD 80 {'settled': 2}In your world
The situation
Your nightly ETL reads 50K booking rows and counts bookings per route. You build a plain dict and increment counts[route] += 1.
What actually breaks
The first unseen route raises KeyError: 'DEL-BOM' and kills the job. Jenkins marks it failed. You add counts[route] = counts.get(route, 0) + 1 — job passes, but now you have four more places with the same get-default dance and each is a maintenance risk.
What you'd do on Monday
Replace every counting dict with Counter() and every grouping dict with defaultdict(int) or defaultdict(list). Counter(row['route'] for row in rows) counts the entire dataset in one expression. The KeyError is gone and the code is three lines instead of twelve.
from collections import Counter, defaultdict
import csv
with open('bookings.csv') as f:
rows = list(csv.DictReader(f))
route_counts = Counter(r['route'] for r in rows)
by_class = defaultdict(list)
for r in rows:
by_class[r['class']].append(r['route'])Imagine you work at IRCTC
The numbers
1,400,000 bookings/day ÷ 86,400 s/day ≈ 16 booking events per second. A daily aggregation job counts bookings per (source_station, destination_station, class) triple. Assuming 10,000 distinct triples and 24 bytes per Counter entry plus Python dict overhead of roughly 100 bytes per slot, the Counter peaks at ≈10,000 × 124 bytes ≈ 1.2 MB. Reading 1.4 M rows from a CSV at 200 bytes per row is 280 MB of I/O; the Counter accumulation is negligible in both time and memory.
What the Scenario-A answer does here
The Scenario A fix — Counter(r['triple'] for r in rows) — still works at 1.4 M rows; the loop completes in under two seconds on typical hardware. The bottleneck is I/O, not the Counter.
What flips at this scale
What flips is when you need a live leaderboard updated per event rather than once per day. At 16 events/second, calling .most_common(20) after every event is fine — it is O(n log k) with n=10,000 keys and k=20 (CPython uses heapq.nlargest under the hood). At 100,000 events/second (a hypothetical online scenario), you need an incremental data structure: a Redis ZSET or a SortedList from sortedcontainers that keeps the top-N sorted without a full scan.
The trade-off you'd defend
I would defend using Counter for the batch aggregation and offloading any real-time leaderboard to Redis, trading in-process simplicity for an external dependency. A batch job does not need sub-second freshness.
Illustrative. 1.4 M bookings/day sourced from IRCTC public statements circa 2023. Counter memory estimate computed above.
Gotchas
Converting a list to a set discards insertion order and duplicates silently
list(set(ids)) returns fewer items than the input, in a different order each run.
A set is unordered and stores only unique values. Duplicate items and original order vanish without an error.
To deduplicate while preserving order, use list(dict.fromkeys(ids)). To just deduplicate with no order concern, set is fine.
Counter[missing_key] returns 0; a plain dict[missing_key] raises KeyError
Switching from Counter to a plain dict for a lookup breaks with KeyError on the first key that was never incremented.
Counter silently returns 0 for missing keys. A plain dict raises KeyError.
When switching types, audit every read site. Use d.get(key, 0) for a plain dict, or keep a Counter if zero-default semantics are what you want.
defaultdict creates the key on read, not just on write
len(d) grows after a read loop that checks if d[key] for keys not yet added.
d[key] invokes the factory and inserts the key with its default; d.get(key) does not.
Use d.get(key) for read-only lookups. Use d[key] only when you intend to create-and-fill the entry.
Tricky questions
Counter({'a': 0}) contain the key 'a'?No — a zero count means the item is absent.
Yes. 'a' in Counter({'a': 0}) is True. The arithmetic operators strip zero counts, but plain __contains__ does not. Code that checks if c['a']: treats zero as falsy and misses the key — that is the trap.
defaultdict drop in for dict everywhere?Yes — it's a dict subclass, so it is compatible everywhere.
Almost. defaultdict serialises fine with json.dumps, but the receiving end gets a plain dict (no factory). Also, any code that relies on KeyError as a sentinel for a missing key will never get that error from a defaultdict. The factory fires instead, inserting a default and hiding the missing-key condition.
Scenario question
You're handed a 2 M-row CSV of train bookings with columns train_no, from_stn, to_stn, class, status. Produce a report: for each (from_stn, class) pair, how many bookings have each status? Memory is limited to 500 MB.
Model answer
I'd use a defaultdict(Counter) keyed by (from_stn, class). Each row is one O(1) Counter update, so the total is O(n). Memory is proportional to the number of distinct (from_stn, class) pairs — perhaps 5,000 combos at 200 bytes each is 1 MB, well within the limit. I'd stream the CSV with csv.DictReader so the row data never all sits in memory at once. The trade-off: if I need cross-group totals (total by class across all stations), I do a second pass over the dict, but for 5,000 keys that is trivial.
Interview corner
CommonWhen would you use Counter over a plain dict with manual increment?
Counter whenever I'm building frequency tallies. It removes the get-or-default boilerplate, gives me .most_common() for free, and supports subtraction between two counters. The overhead over a plain int dict is tiny and worth the readability.”TrickCounter({'a': -1}) — is 'a' in the Counter?
if c['a']: instead of 'a' in c — a zero or negative count is falsy.”Experience checkYour grouping dict crashed with KeyError after months of running fine. What happened?
d[key].append(val) fails the first time key is missing. My fix is defaultdict(list) and a regression test that sends in a novel key the test data never used before.”DesignYou need a bounded event buffer — keep only the last 1,000 events. What do you use?
deque(maxlen=1000). It evicts the oldest entry automatically at O(1) cost. A list with lst[-1000:] copies the full slice every time, which is O(n) and wasteful at high throughput.”Memory hook
Like a ticket clerk whose tally sheet starts every new row at zero, Counter and defaultdict remove the "first check if the key exists" boilerplate that litters every manual counting loop.
Use Counter for counting, defaultdict(list) for grouping — never write d.get(k, 0) + 1 in a loop again.
Self-check
What does
Counter('aabbc')['z']return?0— missing keys return zero, neverKeyError.How do you deduplicate a list while preserving insertion order?
list(dict.fromkeys(items)).What is the risk of accessing a
defaultdictkey you only want to read?The access creates the key with the default value, which inflates the dict and can hide missing-key bugs.
When does Counter arithmetic strip keys?
The
+,-,&and|operators remove keys with zero or negative counts. Direct construction and increment do not.What does
deque([1,2,3], maxlen=3)produce afterappendleft(0)?deque([0, 1, 2])— the rightmost item (3) is evicted.
Write a script that reads a CSV with columns route,status,amount (at least 6 rows, two routes, three statuses). Use defaultdict(Counter) to count statuses per route and Counter to find the top route by total booking count. Print both results. Create the CSV with Path.write_text. Paste the output.