Shelf · Python · Chapter 13

Chapter 13: Stdlib survival kit & idiom review

The stdlib has a handful of tools that show up in almost every ETL job — Counter, pathlib, datetime, subprocess, logging — and each one has one trap that silently wrecks your output. This chapter covers the mechanics, the traps and the decision rules so you can use them correctly under interview pressure.

Difficulty
Priority
py-13-01Card 1 of 6

Choose collections and dict/set idioms

EasyP1 · Must knowKnow cold10 min
collectionscounterdefaultdictdequenamedtupledict-idiomsset-idioms

Why you care

A bare dict with manual increment logic for counting is three lines where Counter is one — and the bare dict raises KeyError until you initialise every key, which means the first missing key corrupts your audit numbers silently. On the grouping side, a defaultdict(list) removes the if-key-exists dance that senior reviewers flag in every code review. Interviewers ask about collections because the answer exposes whether you know the standard toolkit or reinvent it from scratch every time, and reinventing it introduces bugs the stdlib version has already fixed.

Explain like I'm 5

Imagine a ticket counter at a railway station. You are the ticket clerk and you have a tally sheet. With a plain dict, before you mark a tally you must first check whether the row even exists, draw it in, then add one. With a Counter, you just mark the tally — if the row was blank it starts at zero automatically. defaultdict is the same idea for lists: the slot appears the moment you first put something in, so you never have to check first.

Core concept

collections gives you five types that solve the counting, grouping and queue problems without reinventing them:

  • Counter — a dict subclass for tallying. c[missing_key] returns 0, never KeyError. .most_common(n) returns the n highest-count pairs sorted descending. Arithmetic operators (c1 - c2, c1 + c2) produce a new Counter with zero and negative counts stripped.
  • defaultdict(factory) — a dict whose factory function (e.g. list, int, set) is called for any missing key, so d[key].append(val) is always safe.
  • namedtuple — an immutable row type with named fields. Lighter than a dataclass, hashable, and serialises cleanly to a dict with ._asdict().
  • deque — a double-ended queue with O(1) appends and pops at both ends. Use it for sliding-window accumulation and bounded buffers (maxlen=N auto-discards the oldest entry when you append).
  • OrderedDict — preserves insertion order (plain dict also preserves order since 3.7) and adds .move_to_end(key) needed for LRU logic.

Decision rule: use Counter for any counting problem; defaultdict(list) for grouping; deque for bounded sliding windows; namedtuple for lightweight rows that must be hashable. Use a plain dict everywhere else — it is faster and more readable for simple key-value stores. Converting a list to a set for deduplication discards insertion order silently; use dict.fromkeys(items) instead when order matters.

Minimal example Counter and defaultdict basics

PythonCount events and group by currencyruns
from collections import Counter, defaultdict

events = ['settled', 'failed', 'settled', 'pending', 'settled', 'failed']
c = Counter(events)
print(c.most_common(2))
print(c['unknown'])

rows = [('INR', 1200), ('USD', 50), ('INR', 800), ('USD', 30)]
by_ccy = defaultdict(list)
for ccy, amt in rows:
    by_ccy[ccy].append(amt)
print(sorted(by_ccy.keys()))
print(sorted(by_ccy['INR']))
Output
[('settled', 3), ('failed', 2)]
0
['INR', 'USD']
[800, 1200]

In production Settlement aggregation by currency and status

PythonGroup settlements by currency, count by statusruns
from collections import Counter, defaultdict, namedtuple

Settlement = namedtuple('Settlement', ['id', 'currency', 'amount', 'status'])

rows = [
    Settlement('S01', 'INR', 1200, 'settled'),
    Settlement('S02', 'USD',   50, 'settled'),
    Settlement('S03', 'INR',  800, 'failed'),
    Settlement('S04', 'USD',   30, 'settled'),
    Settlement('S05', 'INR',  400, 'pending'),
    Settlement('S06', 'INR',  600, 'settled'),
]

totals = defaultdict(int)
status_by_ccy = defaultdict(Counter)

for s in rows:
    totals[s.currency] += s.amount
    status_by_ccy[s.currency][s.status] += 1

for ccy in sorted(totals):
    print(ccy, totals[ccy], dict(status_by_ccy[ccy]))
Output
INR 3000 {'settled': 2, 'failed': 1, 'pending': 1}
USD 80 {'settled': 2}
Scenario A

In your world

The situation

Your nightly ETL reads 50K booking rows and counts bookings per route. You build a plain dict and increment counts[route] += 1.

What actually breaks

The first unseen route raises KeyError: 'DEL-BOM' and kills the job. Jenkins marks it failed. You add counts[route] = counts.get(route, 0) + 1 — job passes, but now you have four more places with the same get-default dance and each is a maintenance risk.

What you'd do on Monday

Replace every counting dict with Counter() and every grouping dict with defaultdict(int) or defaultdict(list). Counter(row['route'] for row in rows) counts the entire dataset in one expression. The KeyError is gone and the code is three lines instead of twelve.

PythonReplace manual counting in a Jenkins ETL job
from collections import Counter, defaultdict
import csv

with open('bookings.csv') as f:
    rows = list(csv.DictReader(f))

route_counts = Counter(r['route'] for r in rows)
by_class = defaultdict(list)
for r in rows:
    by_class[r['class']].append(r['route'])
Scenario B

Imagine you work at IRCTC

1.4 M bookings per day

The numbers

1,400,000 bookings/day ÷ 86,400 s/day ≈ 16 booking events per second. A daily aggregation job counts bookings per (source_station, destination_station, class) triple. Assuming 10,000 distinct triples and 24 bytes per Counter entry plus Python dict overhead of roughly 100 bytes per slot, the Counter peaks at ≈10,000 × 124 bytes ≈ 1.2 MB. Reading 1.4 M rows from a CSV at 200 bytes per row is 280 MB of I/O; the Counter accumulation is negligible in both time and memory.

What the Scenario-A answer does here

The Scenario A fix — Counter(r['triple'] for r in rows) — still works at 1.4 M rows; the loop completes in under two seconds on typical hardware. The bottleneck is I/O, not the Counter.

What flips at this scale

What flips is when you need a live leaderboard updated per event rather than once per day. At 16 events/second, calling .most_common(20) after every event is fine — it is O(n log k) with n=10,000 keys and k=20 (CPython uses heapq.nlargest under the hood). At 100,000 events/second (a hypothetical online scenario), you need an incremental data structure: a Redis ZSET or a SortedList from sortedcontainers that keeps the top-N sorted without a full scan.

The trade-off you'd defend

I would defend using Counter for the batch aggregation and offloading any real-time leaderboard to Redis, trading in-process simplicity for an external dependency. A batch job does not need sub-second freshness.

Illustrative. 1.4 M bookings/day sourced from IRCTC public statements circa 2023. Counter memory estimate computed above.

Gotchas

Converting a list to a set discards insertion order and duplicates silently

Symptom

list(set(ids)) returns fewer items than the input, in a different order each run.

Cause

A set is unordered and stores only unique values. Duplicate items and original order vanish without an error.

Fix

To deduplicate while preserving order, use list(dict.fromkeys(ids)). To just deduplicate with no order concern, set is fine.

Counter[missing_key] returns 0; a plain dict[missing_key] raises KeyError

Symptom

Switching from Counter to a plain dict for a lookup breaks with KeyError on the first key that was never incremented.

Cause

Counter silently returns 0 for missing keys. A plain dict raises KeyError.

Fix

When switching types, audit every read site. Use d.get(key, 0) for a plain dict, or keep a Counter if zero-default semantics are what you want.

defaultdict creates the key on read, not just on write

Symptom

len(d) grows after a read loop that checks if d[key] for keys not yet added.

Cause

d[key] invokes the factory and inserts the key with its default; d.get(key) does not.

Fix

Use d.get(key) for read-only lookups. Use d[key] only when you intend to create-and-fill the entry.

Tricky questions

Does Counter({'a': 0}) contain the key 'a'?
Most people say

No — a zero count means the item is absent.

Actually

Yes. 'a' in Counter({'a': 0}) is True. The arithmetic operators strip zero counts, but plain __contains__ does not. Code that checks if c['a']: treats zero as falsy and misses the key — that is the trap.

Can a defaultdict drop in for dict everywhere?
Most people say

Yes — it's a dict subclass, so it is compatible everywhere.

Actually

Almost. defaultdict serialises fine with json.dumps, but the receiving end gets a plain dict (no factory). Also, any code that relies on KeyError as a sentinel for a missing key will never get that error from a defaultdict. The factory fires instead, inserting a default and hiding the missing-key condition.

Scenario question

You're handed a 2 M-row CSV of train bookings with columns train_no, from_stn, to_stn, class, status. Produce a report: for each (from_stn, class) pair, how many bookings have each status? Memory is limited to 500 MB.

Model answer

I'd use a defaultdict(Counter) keyed by (from_stn, class). Each row is one O(1) Counter update, so the total is O(n). Memory is proportional to the number of distinct (from_stn, class) pairs — perhaps 5,000 combos at 200 bytes each is 1 MB, well within the limit. I'd stream the CSV with csv.DictReader so the row data never all sits in memory at once. The trade-off: if I need cross-group totals (total by class across all stations), I do a second pass over the dict, but for 5,000 keys that is trivial.

Interview corner

CommonWhen would you use Counter over a plain dict with manual increment?
“I'd use Counter whenever I'm building frequency tallies. It removes the get-or-default boilerplate, gives me .most_common() for free, and supports subtraction between two counters. The overhead over a plain int dict is tiny and worth the readability.”
TrickCounter({'a': -1}) — is 'a' in the Counter?
“I'd say yes — negative counts are valid and the key is present. The arithmetic operators strip non-positives, but a Counter built directly can hold negatives. The trap is testing if c['a']: instead of 'a' in c — a zero or negative count is falsy.”
Experience checkYour grouping dict crashed with KeyError after months of running fine. What happened?
“I'd guess a new key appeared in the data that never showed up in testing. A bare d[key].append(val) fails the first time key is missing. My fix is defaultdict(list) and a regression test that sends in a novel key the test data never used before.”
DesignYou need a bounded event buffer — keep only the last 1,000 events. What do you use?
“I'd use deque(maxlen=1000). It evicts the oldest entry automatically at O(1) cost. A list with lst[-1000:] copies the full slice every time, which is O(n) and wasteful at high throughput.”

Memory hook

Like a ticket clerk whose tally sheet starts every new row at zero, Counter and defaultdict remove the "first check if the key exists" boilerplate that litters every manual counting loop.

If you remember only one thing:

Use Counter for counting, defaultdict(list) for grouping — never write d.get(k, 0) + 1 in a loop again.

Self-check

  1. What does Counter('aabbc')['z'] return?

    0 — missing keys return zero, never KeyError.

  2. How do you deduplicate a list while preserving insertion order?

    list(dict.fromkeys(items)).

  3. What is the risk of accessing a defaultdict key you only want to read?

    The access creates the key with the default value, which inflates the dict and can hide missing-key bugs.

  4. When does Counter arithmetic strip keys?

    The +, -, & and | operators remove keys with zero or negative counts. Direct construction and increment do not.

  5. What does deque([1,2,3], maxlen=3) produce after appendleft(0)?

    deque([0, 1, 2]) — the rightmost item (3) is evicted.

Prove it

Write a script that reads a CSV with columns route,status,amount (at least 6 rows, two routes, three statuses). Use defaultdict(Counter) to count statuses per route and Counter to find the top route by total booking count. Print both results. Create the CSV with Path.write_text. Paste the output.

py-13-02Card 2 of 6

Work with paths and explicit JSON data

EasyP2 · Very likelyKnow cold10 min
pathlibjsonpath-operationsserializationeval-dangertomllib

Why you care

Every ETL script that builds file paths with os.path.join or string concatenation will eventually produce a path with a doubled separator, a missing subdirectory, or a wrong platform separator on the next developer's machine. Meanwhile, eval(config_line) to avoid writing a JSON parser is a remote code execution hole when config values come from outside the process. Interviewers ask about pathlib and JSON because both topics expose whether you know the safe boundary between data and code.

Explain like I'm 5

A Path object is like a postal address object: you can ask it for the street name, the postcode, or the city without slicing a string, and you can extend it by joining parts with /. JSON is the agreed envelope format for moving structured data between systems. eval() is asking the postman to execute the letter — fine when you wrote it, catastrophic when the sender is unknown.

Core concept

pathlib.Path replaces os.path for nearly everything:

  • Join with /: base / 'data' / fname instead of os.path.join(base, 'data', fname).
  • Inspect: .name, .stem, .suffix, .parent, .parts.
  • Read/write text: .read_text(encoding='utf-8'), .write_text(content, encoding='utf-8').
  • Create directories: .mkdir(parents=True, exist_ok=True).
  • Glob files: sorted(dir.glob('*.csv')) — sort explicitly, glob order is OS-dependent.
  • Test existence: .exists(), .is_file(), .is_dir().
  • Convert to string for old APIs: str(p).

json is the stdlib's safe data serialiser:

  • json.loads(s) parses a string to a Python object. Raises json.JSONDecodeError on malformed input.
  • json.dumps(obj) serialises to a string. Pass indent=2 for human-readable output.
  • json.load(fp) / json.dump(obj, fp) operate on file objects.
  • Only these types round-trip: dict, list, str, int, float, bool, None. datetime, set, bytes and custom classes do not — convert them first.

tomllib (stdlib since 3.11) parses TOML, which is a better format for configuration files than hand-rolled Python dict literals. Use it instead of eval() for config.

Decision rule: never use string concatenation or os.path for new code — pathlib is cleaner in every case. Never use eval() or ast.literal_eval() to parse external data — json.loads() is safe and explicit. Use ast.literal_eval() only when you control the string (e.g., a Python literal you wrote yourself in a test). When not to use json: for config files with comments, use tomllib; for large streaming data, use JSONL with a line-by-line loop.

Under the hood

Path is an abstract base with two concrete subclasses: PosixPath on Linux/macOS and WindowsPath on Windows. The / operator calls __truediv__, which normalises separators for the current OS. When you pass a Path to open(), os.stat(), or most stdlib functions, Python calls os.fspath(p) which returns str(p) — so Path slots in wherever a string path is accepted. One non-obvious rule: a right-hand operand that is itself absolute replaces the entire left side: Path('a/b') / '/c' gives Path('/c'), discarding 'a/b'.

Minimal example Path operations and JSON roundtrip

PythonPath name/suffix/stem and JSON loads/dumpsruns
from pathlib import Path
import json

p = Path('reports') / '2024' / 'settlements.json'
print(p.name, p.suffix, p.stem)

data = {'run': 'R-001', 'rows': 48213, 'errors': 7}
encoded = json.dumps(data)
decoded = json.loads(encoded)
print(decoded['rows'], decoded['errors'])
Output
settlements.json .json settlements
48213 7

In production Catalogue loader with pathlib and JSON

PythonWrite JSON fixture, load it, validate types, summariseruns
from pathlib import Path
import json

base = Path('catalogue')
base.mkdir(exist_ok=True)

items = [
    {'sku': 'BB-0001', 'name': 'Milk 1L',    'price': 62,  'active': True},
    {'sku': 'BB-0002', 'name': 'Bread',       'price': 45,  'active': True},
    {'sku': 'BB-0003', 'name': 'Eggs 12pk',   'price': 90,  'active': False},
    {'sku': 'BB-0004', 'name': 'Butter 200g', 'price': 55,  'active': True},
]
cat_file = base / 'items.json'
cat_file.write_text(json.dumps(items, indent=2), encoding='utf-8')

raw = cat_file.read_text(encoding='utf-8')
try:
    catalogue = json.loads(raw)
except json.JSONDecodeError as e:
    raise SystemExit(f'invalid catalogue JSON: {e}') from e

active = [i for i in catalogue if i['active']]
skus = sorted(i['sku'] for i in active)
total = sum(i['price'] for i in active)
print('active:', len(active))
print('first sku:', skus[0])
print('price total:', total)
Output
active: 3
first sku: BB-0001
price total: 162
Scenario A

In your world

The situation

Your ETL reads a JSON config file that holds S3 prefix and batch-size values. A colleague originally parsed it with eval() because the config was written as a Python dict literal.

What actually breaks

A change request adds a cleanup_cmd field. Someone testing locally sets cleanup_cmd: __import__('os').system('rm -rf /'). Your eval(f.read()) call executes it. The job is green; the filesystem is not.

What you'd do on Monday

Convert the config to valid JSON (double quotes, no trailing commas, no Python expressions). Replace eval(f.read()) with json.loads(f.read()). If the config genuinely needs comments, migrate to TOML and use tomllib.loads() (stdlib since 3.11). Never use eval() on external input.

Scenario B

Imagine you work at BigBasket

500 dark stores × 5 K SKUs each

The numbers

500 stores × 5,000 SKUs/store × ~400 bytes/SKU (name, price, stock, metadata as JSON) ≈ 1 GB of raw JSON if loaded as one object. json.loads on 1 GB takes roughly 5–10 s and peaks at 2–3× the raw size in resident memory, roughly 2–3 GB. A typical container has 4 GB RAM; loading all stores at once pushes close to the limit.

What the Scenario-A answer does here

The Scenario A pattern — json.loads(cat_file.read_text()) — works for a 1,000-item per-store file. It fails when you try to load all 500 stores into one dict.

What flips at this scale

The design change is to process one store file at a time in a loop: for f in sorted(store_dir.glob('*.json')): process(json.loads(f.read_text())). For even larger datasets, switch to JSONL (newline-delimited JSON, one object per line) and stream line by line: for line in fp: process(json.loads(line)). Peak memory drops from O(total) to O(one object).

The trade-off you'd defend

I would defend JSONL over a single large JSON array, trading human readability (a pretty-printed array is easier to inspect manually) for streaming safety. A large JSON file cannot be parsed incrementally without a third-party library like ijson; JSONL can with a plain for-loop.

Illustrative. BigBasket dark-store count sourced from published category strategy materials, 2023. Memory multiplier and timing are estimates derived from benchmarks on commodity hardware.

Gotchas

json.dumps raises TypeError for non-serialisable types

Symptom

TypeError: Object of type datetime is not JSON serializable.

Cause

datetime, set, bytes, and custom objects are not in JSON's type system.

Fix

Convert before encoding: dt.isoformat() for datetimes, list(s) for sets, s.decode() for bytes. Pass default=str as a quick fallback for debugging, not production.

Path.glob() order is not guaranteed

Symptom

A job that relies on files being processed alphabetically produces different results on re-runs or a different OS.

Cause

glob returns entries in OS-dependent order, often inode order on Linux ext4.

Fix

Always wrap in sorted(): sorted(data_dir.glob('*.csv')).

Path('a') / '/b' silently discards 'a'

Symptom

A dynamically built path points to an absolute location the developer did not intend.

Cause

A right-hand Path operand that is absolute replaces the left side entirely.

Fix

Strip leading slashes from user-supplied path components before joining, or use p.joinpath(part.lstrip('/')).

Tricky questions

Does json.loads execute Python code?
Most people say

It can if the JSON string contains something like __import__.

Actually

No. json.loads is a data parser and raises json.JSONDecodeError on any input that is not valid JSON. A JSON string "__import__('os')" is decoded as a Python str, not executed. Only eval() and exec() execute code.

What does Path.read_text() use for encoding by default?
Most people say

UTF-8 — that's the standard.

Actually

locale.getpreferredencoding(False), which differs across platforms. A file written on Windows with cp1252 encoding fails with UnicodeDecodeError when read_text() is called on Linux. Always pass encoding='utf-8' explicitly.

Scenario question

You're handed a script that builds output file paths with string concatenation: path = base + '/' + year + '/' + month + '/' + fname. It fails intermittently with FileNotFoundError. What do you do?

Model answer

I'd rewrite the construction as Path(base) / year / month / fname, which normalises separators and never doubles a slash. The FileNotFoundError almost certainly means the intermediate directory does not exist — I'd add path.parent.mkdir(parents=True, exist_ok=True) before the write. The trade-off: mkdir(exist_ok=True) issues a syscall each time; for tight loops I'd call it once per date prefix rather than once per file.

Interview corner

CommonWhy use pathlib.Path instead of os.path?
“I'd use Path because the / operator reads like an actual path, .name and .stem avoid string slicing, and I can call .read_text() directly instead of nesting open(). The old os.path style scatters the path string through nested calls, which is harder to read and refactor.”
TrickPath('a') / '/b' — what do you get?
“I'd get Path('/b'). An absolute right-hand operand replaces the left side entirely. It catches people who join a base path with a user-supplied filename that accidentally starts with a slash.”
Experience checkYour JSON config file sometimes fails to parse. How do you diagnose it?
“I'd catch json.JSONDecodeError — it includes the line number and column in its message — log the full text, and check whether the file was written atomically. A crash during a write produces truncated JSON that the next run fails to parse. I'd also add a schema validation step if the config has required fields.”

Memory hook

A Path is an address object, not a string — join it with /, read it with .read_text(encoding='utf-8'), never slice it. JSON is the envelope; eval() is a trapdoor.

If you remember only one thing:

Use Path(base) / sub / fname for all path construction and json.loads() for all external data — never str + or eval().

Self-check

  1. What method creates intermediate directories with exist_ok=True?

    path.mkdir(parents=True, exist_ok=True).

  2. json.dumps raises TypeError for a datetime. What is the safest fix?

    Convert the datetime to .isoformat() before serialising. Do not rely on str(datetime) — str(dt) uses a space separator (2024-03-15 09:15:30) while .isoformat() uses 'T' (2024-03-15T09:15:30), so they are not the same string. Being explicit prevents consumers from receiving a non-standard format.

  3. What does Path('data') / '/archive/2024' produce?

    Path('/archive/2024') — the absolute right-hand side replaces the left entirely.

  4. What exception does json.loads raise on invalid input?

    json.JSONDecodeError.

  5. Name the stdlib module added in Python 3.11 for parsing TOML config files.

    tomllib.

Prove it

Write a script that creates a directory output/2024/Q1/ using pathlib, writes a list of three dicts as a JSON file in it (json.dumps), reads the file back with json.loads, filters records where amount > 100, and prints the filtered count. Create the file in the current working directory. Paste the output.

py-13-03Card 3 of 6

Make timezones part of the data contract

MediumP2 · Very likelyKnow cold11 min
datetimetimezonezoneinfoaware-naiveutcistfromisoformat

Why you care

A naive datetime has no timezone label. When you subtract two naive datetimes that were captured in different timezones — one from a database in UTC, one from an API in IST — you get a duration that is off by the UTC offset, silently. In financial ETL, a trade stamped "one minute after market close" can become "one minute before market close" after a timezone mix-up, and that flips a reconciliation status from passed to failed. Interviewers ask about timezone-aware datetimes because almost every backend stores in UTC, displays in local time, and the path between the two is where off-by-an-hour bugs live.

Explain like I'm 5

Imagine two clocks. A naive datetime is a clock face with no city label — it shows a time but not which zone. An aware datetime is the same clock face with a sticker reading "IST" or "UTC". When you subtract two labelled clocks, Python converts both to a shared reference (UTC) before computing the gap. When you subtract two unlabelled clocks, Python subtracts the face values directly and gives you a wrong answer with no complaint.

Core concept

Every datetime has a .tzinfo attribute. When it is None, the object is naive. When it holds a tzinfo instance, the object is aware.

Fixed UTC offsets — always available in the stdlib:

Pythonillustrative; fixed offsets are stdlib, no extra package
import datetime
IST = datetime.timezone(datetime.timedelta(hours=5, minutes=30))
UTC = datetime.timezone.utc

Named IANA timezones — stdlib zoneinfo module (3.9+), but requires the tzdata PyPI package on Windows where no system IANA database exists:

Pythonneeds tzdata package on Windows: pip install tzdata
from zoneinfo import ZoneInfo
IST = ZoneInfo('Asia/Kolkata')

Rules:

  • Never mix aware and naive. aware - naive raises TypeError: can't subtract offset-naive and offset-aware datetimes.
  • Store and compare in UTC. Convert to local time only at display time.
  • Parse ISO 8601 strings with datetime.fromisoformat() (3.7+). Strings with an offset suffix like +05:30 produce aware datetimes.
  • Avoid datetime.utcnow() — it returns a naive UTC datetime, which is contradictory. Use datetime.now(tz=datetime.timezone.utc) instead. utcnow() is deprecated in 3.12 and will be removed in a future version.
  • Avoid datetime.fromtimestamp(ts) without tz= — it converts using the local system clock and produces machine-dependent results.

Decision rule: if your system uses one fixed-offset timezone and DST is not a concern (e.g., IST is always UTC+5:30, no DST), datetime.timezone(timedelta(...)) is sufficient and has no external dependencies. If you need named zones with DST transitions, use zoneinfo.ZoneInfo with tzdata installed. Always target UTC for storage.

Under the hood

A datetime stores six integers (year, month, day, hour, minute, second) plus microseconds. The .tzinfo attribute is a pointer to a tzinfo subclass whose .utcoffset() method returns a timedelta. When Python computes a - b, it calls both .utcoffset() methods and adjusts each value to a common UTC epoch before subtracting. If either call returns None (naive), Python raises TypeError immediately. The dangerous case is two naive datetimes that are actually in different real timezones: both .utcoffset() calls return None, so Python subtracts face values and produces a wrong result with no error. This is the silent bug datetime.utcnow() causes: it looks like UTC but carries no label, so mixing it with a naive local-time datetime produces garbage arithmetic.

Minimal example Aware datetime, UTC conversion, and the TypeError trap

PythonFixed-offset IST datetime, convert to UTC, hit the naive/aware TypeErrorruns
import datetime

IST = datetime.timezone(datetime.timedelta(hours=5, minutes=30))
UTC = datetime.timezone.utc

trade = datetime.datetime(2024, 3, 15, 9, 15, 30, tzinfo=IST)
print(trade.isoformat())

utc_trade = trade.astimezone(UTC)
print(utc_trade.isoformat())

naive = datetime.datetime(2024, 3, 15, 9, 15, 30)
try:
    diff = trade - naive
except TypeError as e:
    print('TypeError:', e)
Output
2024-03-15T09:15:30+05:30
2024-03-15T03:45:30+00:00
TypeError: can't subtract offset-naive and offset-aware datetimes

In production Trade timestamp normalisation pipeline

PythonParse IST trade events, store in UTC, check session boundsruns
import datetime

IST = datetime.timezone(datetime.timedelta(hours=5, minutes=30))
UTC = datetime.timezone.utc

MARKET_OPEN  = datetime.datetime(2024, 3, 15,  9, 15, 0, tzinfo=IST)
MARKET_CLOSE = datetime.datetime(2024, 3, 15, 15, 30, 0, tzinfo=IST)

raw_trades = [
    {'id': 'T01', 'ts_ist': '2024-03-15T09:15:45+05:30', 'symbol': 'RELIANCE'},
    {'id': 'T02', 'ts_ist': '2024-03-15T12:00:00+05:30', 'symbol': 'INFY'},
    {'id': 'T03', 'ts_ist': '2024-03-15T15:29:55+05:30', 'symbol': 'TCS'},
]

def normalise(trade):
    ts = datetime.datetime.fromisoformat(trade['ts_ist'])   # aware +05:30
    ts_utc = ts.astimezone(UTC)
    in_session = MARKET_OPEN <= ts <= MARKET_CLOSE
    return {**trade, 'ts_utc': ts_utc.isoformat(), 'in_session': in_session}

for t in [normalise(t) for t in raw_trades]:
    print(t['id'], t['ts_utc'], t['in_session'])
Output
T01 2024-03-15T03:45:45+00:00 True
T02 2024-03-15T06:30:00+00:00 True
T03 2024-03-15T09:59:55+00:00 True

See it happen Aware minus naive raises; aware minus aware normalises first

your codedatetimePython runtimetrade - utc_trade (both aware).utcoffset() → +05:30, .utcoffset() → +00:00correct timedelta (5h 30m)trade - naive_dt (aware minus naive).utcoffset() → +05:30, .utcoffset() → NoneTypeError: can't subtract offset-naive and offset-aware datetimes
  1. Both datetimes have .tzinfo set; subtraction is allowed.
  2. Python fetches both offsets and normalises each to a UTC epoch value.
  3. Subtraction succeeds on the normalised UTC values.
  4. naive_dt.tzinfo is None; Python cannot normalise it.
  5. One side has no offset; Python cannot find a common epoch.
  6. The TypeError is the good news. The silent bug is two naive datetimes in different real zones: same face values, wrong answer, no error.
Notice: the TypeError only fires when one side is aware and the other is naive; two naive datetimes in different real timezones subtract silently and produce a wrong result.
Scenario A

In your world

The situation

Your ETL fetches trade timestamps from two APIs: one returns naive UTC strings, the other returns naive IST strings. You store both as naive datetimes in the same Postgres column.

What actually breaks

A query for "trades in the last hour" compares a naive UTC timestamp from the DB against datetime.now() (local time). The 5:30 offset causes trades near session boundaries to appear in the wrong window. Reconciliation rejects 47 valid trades.

What you'd do on Monday

Establish a timezone contract at the ingestion boundary. Parse every incoming timestamp with datetime.fromisoformat() and attach the correct tzinfo from the source's documentation. Convert to UTC with .astimezone(UTC) before storing. Store as TIMESTAMPTZ in Postgres. All internal comparisons then use UTC-aware objects and the offset arithmetic is automatic.

Scenario B

Imagine you work at Upstox

10 M trades per day

The numbers

NSE equity session runs 09:15–15:30 = 375 minutes = 22,500 seconds. 10,000,000 trades/day ÷ 22,500 s/session ≈ 444 trades/s at peak. Each trade carries a timestamp string (≈30 bytes), one fromisoformat() call, and one astimezone() call. fromisoformat() is a fast C implementation; benchmarks show roughly 200,000 calls/s per core on 3.12. At 444 trades/s, datetime normalisation uses well under 1% of one core. The bottleneck is the downstream DB write, not the datetime arithmetic.

What the Scenario-A answer does here

The Scenario A fix — fromisoformat then astimezone(UTC) — still works at 444 trades/s. The transformation is stateless and can be vectorised or parallelised easily.

What flips at this scale

What flips is DST handling. Upstox trades Indian equities, and India has no DST, so a fixed offset (UTC+05:30) is correct for all time. If the system were expanded to cover US equities (EDT/EST transition in March and November), a fixed offset would produce 1-hour errors on DST transition dates. You would need zoneinfo.ZoneInfo('America/New_York') with the tzdata package to handle those correctly.

The trade-off you'd defend

I would defend using fixed offsets for IST-only systems (no external dependency, deterministic, correct) and switching to named IANA zones only when the system must handle multiple markets with DST. Adding tzdata is a small dependency, but it is another package to pin and update.

Illustrative. 10 M trades/day is a published order-of-magnitude figure for mid-tier Indian brokers. fromisoformat throughput estimated from Python 3.12 benchmark data; not Upstox-published.

Gotchas

datetime.utcnow() returns a naive UTC datetime

Symptom

Code that mixes utcnow() values with aware datetimes raises TypeError, or the comparison silently uses local time when the developer intended UTC.

Cause

utcnow() returns a naive object despite the utc in its name. It is deprecated in 3.12.

Fix

Use datetime.now(tz=datetime.timezone.utc) everywhere. It returns an aware datetime anchored to UTC.

datetime.fromtimestamp(ts) uses the local system clock

Symptom

The same Unix timestamp produces a different datetime on two machines in different timezones.

Cause

fromtimestamp without tz= converts using time.localtime(), which is machine-dependent.

Fix

datetime.fromtimestamp(ts, tz=datetime.timezone.utc) converts to UTC explicitly and is machine-independent.

Comparing two naive datetimes from different timezones silently produces a wrong result

Symptom

Arithmetic on two naive datetimes returns a timedelta that is off by exactly the UTC offset difference, with no exception.

Cause

Python subtracts naive datetime face values directly, assuming they share a timezone.

Fix

Attach the correct tzinfo at the point where you parse or receive the timestamp, before storing or comparing.

Tricky questions

Does datetime.fromisoformat('2024-03-15T09:15:30+05:30') return a naive or aware datetime?
Most people say

Naive — fromisoformat just parses the string; the +05:30 is ignored.

Actually

Aware. Since Python 3.7, fromisoformat parses offset suffixes and attaches a datetime.timezone object. The result has .tzinfo set to UTC+05:30.

Is datetime.now(tz=UTC) the same as datetime.utcnow()?
Most people say

Yes — both give the current UTC time.

Actually

No. datetime.utcnow() returns a naive datetime (.tzinfo is None). datetime.now(tz=UTC) returns an aware datetime with UTC+00:00. Mixing the naive result of utcnow() with aware datetimes raises TypeError.

Scenario question

You're handed a CSV of insurance policy timestamps. Some rows have +05:30, some have +00:00, some have no offset at all. The job must calculate how many days until each policy expires from a fixed reference date. What do you do?

Model answer

I'd start by categorising the rows: those with an offset suffix can be parsed with fromisoformat and are already aware. Those without an offset need a source-system timezone documented somewhere — I'd check with the team whether naked timestamps from the old system are IST or UTC. Once I know, I'd attach the correct tzinfo on parse. I'd set a fixed reference datetime with UTC timezone (e.g., datetime(2024, 3, 15, 0, 0, tzinfo=UTC)) and subtract each aware expiry from it. The trade-off: if I cannot confirm the source timezone for naked timestamps, I'd reject those rows and log a warning rather than guess.

Interview corner

CommonWhat is the difference between a naive and an aware datetime?
“I'd say a naive datetime has no timezone — it's just a clock face with no label. An aware datetime has a tzinfo attached, so Python can convert both sides of a subtraction to a common UTC epoch before computing the gap. Without that label, Python subtracts face values and can give you a wrong answer silently.”
TrickWhy is datetime.utcnow() dangerous?
“I'd say it returns a naive datetime despite the name. If you store it in a list alongside aware datetimes and later subtract them, you get a TypeError. Worse, if both sides are naive but in different real timezones, you get a wrong timedelta with no error. The fix is datetime.now(tz=UTC), which always returns an aware datetime.”
Experience checkYour reconciliation job shows a 5h 30m discrepancy on trade timestamps. What is your first hypothesis?
“I'd suspect a naive IST datetime is being compared against a UTC reference without a timezone conversion — a classic IST-to-UTC mix-up. My first check is whether the timestamps from each source are labelled with tzinfo and whether the storage layer uses TIMESTAMPTZ or bare TIMESTAMP in Postgres.”
DesignThe system needs to support both IST and New York exchange timestamps. What timezone approach do you use?
“I'd use zoneinfo.ZoneInfo with the tzdata package, which handles DST transitions for named IANA zones. IST has no DST so a fixed offset would also work for IST, but using named zones everywhere is consistent and avoids a two-codepath system. The trade-off is adding tzdata as a dependency, which must be pinned in requirements.”

Memory hook

Like a clock face without a city label, a naive datetime looks correct but is meaningless when moved between systems — attach the timezone at the boundary, convert to UTC, and compare aware objects only.

If you remember only one thing:

Parse every external timestamp into an aware datetime immediately, store in UTC, and never let a naive datetime cross a function boundary.

Self-check

  1. What does datetime.fromisoformat('2024-03-15T09:15:30+05:30').tzinfo return?

    UTC+05:30 — an instance of datetime.timezone with a 5h30m offset.

  2. What is wrong with datetime.utcnow()?

    It returns a naive datetime despite the name, deprecated in 3.12. Use datetime.now(tz=datetime.timezone.utc) instead.

  3. What exception does aware_dt - naive_dt raise?

    TypeError: can't subtract offset-naive and offset-aware datetimes.

  4. How do you convert an aware IST datetime to UTC?

    dt_ist.astimezone(datetime.timezone.utc).

  5. Why does datetime.fromtimestamp(ts) produce machine-dependent results?

    It uses the local system timezone unless tz= is passed.

Prove it

Write a script that defines a fixed IST aware datetime for market open (2024-03-15 09:15:00+05:30) and a fixed UTC close time (2024-03-15 10:00:00+00:00). Compute the session duration as a timedelta, then try to subtract a naive datetime from the IST one and print the caught TypeError message. Paste the output.

py-13-04Card 4 of 6

Review subprocess and regex boundaries

MediumP3 · Role-dependentTalk about it12 min
subprocessreregexshell-injectiontimeoutfullmatchnamed-groups

Why you care

Two things in this area cause real incidents: subprocess called with shell=True and a user-controlled string, which is a shell injection hole; and re.match used when re.fullmatch is needed, which silently validates strings that have trailing garbage. An ETL job that calls an external tool or validates booking references is doing both, and the failures are quiet — the job succeeds with wrong data or, in the injection case, with a shell command running under your service account. Interviewers ask about these because they expose whether you think about the input boundary.

Explain like I'm 5

subprocess with shell=True is asking the OS to hand your string to a shell interpreter. If the string came from the outside, the interpreter will run whatever the attacker appended after a ; or &&. A list of arguments instead of a string bypasses the shell entirely — the OS invokes the program directly. re.match is a clerk who checks only the front of your ID; re.fullmatch checks the whole card.

Core concept

subprocess.run — the right choice for almost every case:

  • Pass arguments as a list, never as a string with shell=True on user-controlled data.
  • capture_output=True captures both stdout and stderr; text=True decodes as UTF-8.
  • timeout=N raises subprocess.TimeoutExpired if the process takes longer than N seconds.
  • check=True raises subprocess.CalledProcessError on non-zero exit code.
  • result.returncode, result.stdout, result.stderr give you the outcome.

re — compile patterns that are reused:

  • re.compile(r'...') compiles once; .fullmatch(), .match(), .search(), .findall() on the compiled object are faster in loops.
  • re.fullmatch requires the entire string to match the pattern. Use it for validation.
  • re.match anchors at the start but allows trailing characters. Use it only when you genuinely want a prefix match.
  • re.search finds the first match anywhere in the string. Use it for extraction.
  • Named groups (?P<name>...) make extraction readable: m.group('date') instead of m.group(1).
  • Raw strings r'...' prevent Python from interpreting \d, \w etc. as escape sequences before the regex engine sees them.

Decision rule: use subprocess.run with a list; add timeout on any process you do not fully control; add check=True when a non-zero exit should abort the job; use re.fullmatch for validation, re.search for extraction. When not to use subprocess.run: do not spawn a process per item in a tight loop — each spawn carries significant OS overhead (illustrative: tens to hundreds of milliseconds on most platforms); prefer a Python function call, a C extension, or batching many items into a single process call instead.

Under the hood

When you pass a list to subprocess.run (not shell=True), Python calls os.execvp (or CreateProcess on Windows) directly with the argument vector. The shell is never involved, so shell metacharacters (; & | > $) in the arguments are passed literally to the target program. With shell=True and a string argument, Python invokes /bin/sh -c "your string", which interprets every metacharacter. On Windows the default shell is cmd.exe. The regex engine in CPython is a backtracking NFA. Pathological patterns like (a+)+b on a string of as have exponential worst-case time — catastrophic backtracking. The re module has no built-in timeout parameter; the match runs at the C level and cannot be interrupted by Python threads or timers. Real mitigations: (1) avoid nested quantifiers over the same character class ((a+)+ → a+); (2) use atomic groups (?>...) or possessive quantifiers *+ ++ ?+, both added to re in Python 3.11+, which prevent the engine from backtracking into already-matched subgroups; (3) cap input length before calling the regex; (4) if you must accept arbitrary patterns from users, use the third-party regex module which accepts a timeout= argument.

Minimal example subprocess.run and re.fullmatch basics

PythonRun a subprocess safely, validate a booking ref with fullmatchruns
import subprocess
import re

result = subprocess.run(
    ['python', '-c', 'print(2 + 2)'],
    capture_output=True, text=True, timeout=5,
)
print('exit:', result.returncode)
print('out:', result.stdout.strip())

BOOKING_PAT = re.compile(r'^MMT-(?P<date>\d{8})-(?P<seq>\d{6})$')
refs = ['MMT-20240315-000001', 'MMT-2024-001', 'MMT-20240315-0000011']
for ref in refs:
    print(ref, bool(BOOKING_PAT.fullmatch(ref)))
Output
exit: 0
out: 4
MMT-20240315-000001 True
MMT-2024-001 False
MMT-20240315-0000011 False

In production Booking reference validator and external checksum runner

PythonValidate MakeMyTrip booking refs, extract named groups, run a checksum scriptruns
import subprocess
import re
from pathlib import Path

BOOKING_PAT = re.compile(r'^MMT-(?P<date>\d{8})-(?P<seq>\d{6})$')

# Write a fixture checksum script
Path('checksum.py').write_text(
    'import hashlib, sys\n'
    'print(hashlib.md5(sys.argv[1].encode()).hexdigest()[:8])\n'
)

bookings = [
    'MMT-20240315-000001',
    'MMT-2024-99',          # invalid: date too short
    'MMT-20240316-000099',
]

for ref in bookings:
    m = BOOKING_PAT.fullmatch(ref)
    if not m:
        print(f'SKIP invalid ref: {ref}')
        continue
    result = subprocess.run(
        ['python', 'checksum.py', ref],
        capture_output=True, text=True, timeout=5, check=True,
    )
    print(f'ref={ref} date={m.group("date")} seq={m.group("seq")} cs={result.stdout.strip()}')
Output
ref=MMT-20240315-000001 date=20240315 seq=000001 cs=18449141
SKIP invalid ref: MMT-2024-99
ref=MMT-20240316-000099 date=20240316 seq=000099 cs=3566ba5a
Scenario A

In your world

The situation

Your Jenkins job validates S3 file names against a pattern, then calls an external checksum tool to verify each file. You pass the file name into os.system(f'checksum {fname}').

What actually breaks

A filename arrives as valid_name.csv; rm -rf /tmp/workspace. The shell interprets the semicolon and executes rm -rf /tmp/workspace under your service account. Jenkins workspace is gone; the next build fails with missing files.

What you'd do on Monday

Replace os.system(f'...') with subprocess.run(['checksum', fname], capture_output=True, text=True, timeout=30, check=True). Pass the filename as a separate list element — the shell is never invoked, so no metacharacter in fname is interpreted.

Scenario B

Imagine you work at MakeMyTrip

2 M booking refs per day

The numbers

2,000,000 booking refs/day ÷ 86,400 s/day ≈ 23 refs/s on average; peak flight search periods reach 10× that, so plan for 230 refs/s. A compiled re.compile pattern costs one compile (~50 µs); .fullmatch() on a 20-character string takes roughly 1–5 µs. At 230 calls/s, regex validation uses well under 0.1% of a single CPU core. The external checksum subprocess is the bottleneck: each subprocess.run call has ~5–10 ms of overhead on Linux.

What the Scenario-A answer does here

Calling subprocess.run per-booking at 230 refs/s means 230 × 5 ms = 1.15 s per second of real time — the validation cannot keep up with the input rate. This is fine for a batch job but not for an inline validation in a request path.

What flips at this scale

The design change is to move validation away from spawning a process per item. Either port the checksum to a Python function (so it is a function call, not a process spawn), or batch 1,000 refs per subprocess call and parse the multi-line output. At 1,000 refs per call the amortised overhead drops from 5 ms/ref to 5 µs/ref.

The trade-off you'd defend

I would defend porting the checksum to Python if the checksum logic is simple (e.g., a hash), trading the subprocess boundary for speed. If the external tool does something Python cannot reproduce (e.g., a vendor-specific signing step), I would batch the calls and accept the subprocess overhead.

Illustrative. Subprocess overhead of 5–10 ms is a measured order-of-magnitude on Linux with CPython 3.12; exact values depend on hardware and OS scheduling. MakeMyTrip booking volume is illustrative.

subprocess.run vs os.system vs os.popen

subprocess.runos.systemos.popen
Capture outputyes, via capture_outputno, goes to terminalstdout only, via file object
Timeoutyesnono
Shell injection risklow if args is a listhigh (always uses shell)high (always uses shell)
Exit coderesult.returncodereturn valuenone directly
Recommended for new codeyesnono
Notesprefer list args + check=Truelegacy tool, avoiddeprecated pattern, avoid

Gotchas

re.match passes strings with trailing garbage

Symptom

re.match(r'\d{8}', '20240315_extra') returns a match object, so the validation passes even though the string has a suffix.

Cause

re.match anchors at the start but not the end. It matches the prefix and ignores what follows.

Fix

Use re.fullmatch for validation. It requires the pattern to cover the entire string.

subprocess.run with shell=True and a user-supplied string is a shell injection

Symptom

A crafted filename like file.csv; curl evil.com causes the subprocess to issue an unexpected network request.

Cause

With shell=True, the entire string argument is passed to /bin/sh -c, which interprets metacharacters.

Fix

Pass arguments as a list without shell=True. Each list element is passed literally to the program with no shell interpretation.

Missing timeout causes a hanging job

Symptom

A Jenkins job runs for hours and never finishes; the external process it called is blocked on I/O or a deadlock.

Cause

subprocess.run with no timeout waits forever by default.

Fix

Always set a timeout appropriate to the expected runtime of the called process. Catch subprocess.TimeoutExpired and log it as an error so the job fails fast.

Tricky questions

Does re.match(r'\d+', '123abc') return a match?
Most people say

No — the string contains non-digit characters, so it should not match.

Actually

Yes. re.match anchors at the start and matches 123; the abc is ignored. re.fullmatch(r'\d+', '123abc') returns None.

Does shell=True with a hardcoded string cause a security risk?
Most people say

Only when user data is involved.

Actually

A hardcoded string is safe from injection, but shell=True still has costs: it forks a shell process (slower), breaks on platforms where /bin/sh is not available, and encourages copy-paste reuse with a dynamic string later. Prefer a list even for hardcoded commands.

Scenario question

You're handed an ETL script that validates hundreds of S3 filenames against a regex and then calls an external binary to compute a checksum for each valid file. It's running slowly. What do you change?

Model answer

I'd profile two bottlenecks: the regex and the subprocess. For the regex, I'd confirm the pattern is compiled once with re.compile() outside the loop — calling re.fullmatch(pattern_str, ...) recompiles the pattern on every call, which is unnecessary overhead. For the subprocess, I'd measure whether calling one process per file is the bottleneck. If it is, I'd batch multiple filenames as arguments to a single process call and split the output. The trade-off: batching complicates error attribution — if one filename fails, the error message needs enough context to identify which one.

Interview corner

CommonWhen should you use shell=True with subprocess?
“I'd say almost never for new code. The only time I'd use it is when I genuinely need a shell feature like glob expansion or piping, and only when the command string is completely hardcoded. Any user-supplied or external data must go in as a list element, not interpolated into a shell string.”
TrickWhat is the difference between re.match and re.fullmatch?
“I'd say re.match anchors at the start but not the end, so it passes strings with trailing characters the pattern does not cover. re.fullmatch requires the pattern to consume the entire string. For validation, re.fullmatch is almost always what you want; re.match is a source of false positives that cost two months of debugging to find in a booking reference pipeline.”
Experience checkYour subprocess call blocks the entire ETL job for twenty minutes. What happened and how do you fix it?
“I'd bet the external process is waiting for stdin or stuck in a network call with no timeout. My fix is to add timeout=30 (or whatever bound the process should respect) and catch subprocess.TimeoutExpired so the job logs the failure and moves on rather than hanging indefinitely.”

Memory hook

A list of arguments goes straight to the OS and bypasses the shell entirely — no interpreter reads your filename and no semicolon becomes a second command. A re.fullmatch check reads the whole ID card, not just the front — trailing garbage cannot sneak past it.

If you remember only one thing:

Pass subprocess arguments as a list (never shell=True on external data) and use re.fullmatch for validation.

Self-check

  1. What argument prevents subprocess.run from interpreting shell metacharacters?

    Passing arguments as a list without shell=True.

  2. Which re function anchors at the start but allows trailing characters?

    re.match.

  3. What exception does subprocess.run raise when the process exceeds the timeout?

    subprocess.TimeoutExpired.

  4. What exception does subprocess.run(check=True) raise on a non-zero exit code?

    subprocess.CalledProcessError.

  5. How do you extract a named group date from a compiled regex match?

    m.group('date').

Prove it

Write a script that compiles a regex pattern for a settlement file name of the form SETTLE-YYYYMMDD-NNN.csv (8-digit date, 3-digit sequence), validates five sample filenames (mix of valid and invalid), and prints VALID or INVALID for each. Use re.fullmatch. Paste the output.

py-13-05Card 5 of 6

Configure logging for useful diagnostics

MediumP2 · Very likelyKnow cold12 min
loggingbasicconfigdictconfighandlerslevelspropagationstructured-logs

Why you care

print() debugging dies the moment the job runs on a Jenkins agent you cannot attach to. A proper logging setup gives you level-filtered output, a persistent file trail, and enough context (module name, level, timestamp) to reconstruct what happened. The trap is setting up logging incorrectly — using the root logger directly, configuring handlers twice, or logging sensitive data like passwords and credit card numbers that end up in your log aggregator. Interviewers ask about logging configuration because it separates developers who can debug production from those who are flying blind.

Explain like I'm 5

Imagine a reporting chain in a large company. Every department has its own manager (named logger). When a report (log record) is filed, it travels up the management chain to the CEO (root logger), who distributes it to all notice boards (handlers). A handler is a notice board: one goes to the console, one to a file. A formatter is the template each notice board uses to print reports. If you accidentally put a notice board at the department level and the CEO level, every report gets printed twice.

Core concept

Logger hierarchy: loggers are named with dot-separated names. logging.getLogger('etl.orders') is a child of logging.getLogger('etl'), which is a child of the root logger. A record travels up the chain until it hits a logger with propagate=False or the root.

Levels (low to high): DEBUG, INFO, WARNING, ERROR, CRITICAL. A logger only processes records at or above its set level. The root logger defaults to WARNING.

Key calls:

  • logging.basicConfig(level=..., format=..., stream=...) — one-shot setup for scripts. Must be called before the first getLogger or it has no effect.
  • logging.config.dictConfig(config_dict) — the production choice; separates config from code.
  • logger.info('loaded %d rows', count) — use % formatting, not f-strings, so the string is only built if the level passes.
  • logger.exception('msg') — logs at ERROR and appends the current traceback. Call inside an except block.

Decision rule: use basicConfig in one-off scripts. Use dictConfig in any package or service. Never call logging.getLogger() with no argument (the root logger) in library code — always name your logger logging.getLogger(__name__). Never log raw exception objects with sensitive fields; extract and redact the fields you need before logging.

Under the hood

A log record's journey: logger.info('msg') creates a LogRecord, checks the logger's effective level (walking up the hierarchy if the logger has NOTSET), then passes the record to the logger's handlers and — if propagate is True — to the parent's handlers. Each handler checks its own level before calling its formatter and writing the output. Two handlers on two levels of the hierarchy that both pass the same record is how duplicate log lines happen: the record fires on the child handler, then propagates to the parent handler. Fix it with propagate=False on the child or by attaching handlers only to the root.

Minimal example basicConfig and named logger

PythonSet up logging with basicConfig, use a named loggerruns
import logging
import sys

logging.basicConfig(
    stream=sys.stdout,
    level=logging.DEBUG,
    format='%(levelname)s %(name)s %(message)s',
)
log = logging.getLogger('etl')
log.info('loaded %d rows', 42)
log.warning('skipped %d duplicates', 3)
Output
INFO etl loaded 42 rows
WARNING etl skipped 3 duplicates

In production Logging for a policy ingest job

PythonCapture log output and verify level filteringruns
import logging
import sys
import io

buf = io.StringIO()
handler = logging.StreamHandler(buf)
handler.setFormatter(logging.Formatter('%(levelname)s %(name)s %(message)s'))

logger = logging.getLogger('policy.ingest')
logger.handlers.clear()
logger.addHandler(handler)
logger.setLevel(logging.DEBUG)
logger.propagate = False

logger.info('batch start run=%s size=%d', 'run-2024-03-15', 500)
logger.debug('fetched %d policies from S3', 500)
logger.warning('policy %s missing premium_amount, skipping', 'POL-0042')
logger.info('inserted %d, skipped %d', 499, 1)

print(buf.getvalue(), end='')
Output
INFO policy.ingest batch start run=run-2024-03-15 size=500
DEBUG policy.ingest fetched 500 policies from S3
WARNING policy.ingest policy POL-0042 missing premium_amount, skipping
INFO policy.ingest inserted 499, skipped 1

See it happen Logger hierarchy and handler propagation

propagatespropagatespropagateshandleshandlesroot (WARNING)etl (propagate=F)etl.ordersetl.loaderStreamHandlerFileHandler
  1. Logger hierarchy: every named logger propagates records up to root unless propagate=False stops them.
  2. A DEBUG record from etl.orders travels up through etl (DEBUG level passes) then to root, where both handlers fire.
  3. Set propagate=False on etl to contain all etl output at the etl logger. Root handlers go silent for etl traffic; add handlers to etl directly instead.
Notice: records travel from child to parent; handlers fire at every level the record reaches, so a handler on both etl and root prints each record twice.
Scenario A

In your world

The situation

Your nightly ETL job processes 500K policy rows. You added print() statements for debugging, but Jenkins captures stdout inconsistently — some print calls appear in the log, some do not, and there is no severity information.

What actually breaks

An error in row 412,089 causes a silent wrong calculation. The print output is mixed with stdout from other jobs in Jenkins. You cannot tell when the error first appeared, what the row contained, or which module raised it.

What you'd do on Monday

Replace every print() with logger = logging.getLogger(__name__) and the appropriate level call: logger.debug for trace data, logger.info for milestones, logger.warning for skipped rows, logger.error for failures. Configure one StreamHandler to stderr in a top-level main() or a dictConfig call so the format includes the module name, level, and timestamp. Use logger.exception('failed on row %d', row_num) inside the except block to capture the traceback automatically.

Scenario B

Imagine you work at PolicyBazaar

500 K policies per batch

The numbers

500,000 policies per nightly batch. With one log call per policy (at DEBUG level) that is 500,000 LogRecord creations. Each LogRecord creation is roughly 2–5 µs; 500,000 × 5 µs = 2.5 s of logging overhead per batch. If DEBUG is enabled in production (it should not be) the overhead is 0.5% of an assumed 8-minute batch window.

What the Scenario-A answer does here

Logging one record per policy at INFO level is the Scenario A fix. At 500,000 records/batch with a FileHandler and UTF-8 formatting, each write is roughly 200 bytes → 100 MB of log data per batch, written sequentially. That is fine for a nightly batch.

What flips at this scale

What flips is when the batch runs on a shared host and logs go to a centralised aggregator like ELK or CloudWatch. Each log line becomes a network write. At 500,000 lines/batch × 200 bytes = 100 MB of log traffic per batch per host, across 10 concurrent jobs that is 1 GB of log data per batch window. The flip is to log only the summary at INFO level (logger.info('batch done: inserted=%d skipped=%d', ok, skip)) and emit per-row diagnostics only at DEBUG (disabled in production). Structured JSON logging via python-json-logger or a custom formatter allows the aggregator to index fields like policy_id efficiently.

The trade-off you'd defend

I would defend INFO-level summaries with DEBUG per-row detail, trading verbosity for log storage cost. The policy ID and error context in a WARNING line are enough to debug most failures without logging every row.

Illustrative. LogRecord creation overhead of 2–5 µs measured on CPython 3.12 with a StreamHandler; not PolicyBazaar-published.

Gotchas

logging.basicConfig has no effect when the root logger already has handlers

Symptom

basicConfig is called at the top of the script but produces no output, or uses the wrong format.

Cause

basicConfig only configures the root logger if it has no handlers. It does NOT fire when root already has handlers — which happens if an earlier basicConfig ran, or if a library or framework called logging.getLogger() and then attached a handler to root. (getLogger() alone does not attach handlers; the handlers come from a later addHandler or basicConfig call.)

Fix

Call basicConfig before any library import that configures root logging. Use force=True (added in Python 3.8) to replace an existing root configuration unconditionally: logging.basicConfig(level=logging.DEBUG, format='...', force=True). For production packages, use logging.config.dictConfig which always replaces the configuration.

Duplicate log lines appear in the output

Symptom

Every log line prints twice — once for the named logger and once for the root logger.

Cause

You added a handler to a named logger (e.g. etl) and a handler to the root logger, and etl.propagate is True (the default). Each record fires both handlers.

Fix

Either set propagate=False on the named logger, or attach handlers only to the root logger and let propagation do the work.

Logging sensitive data is a compliance violation

Symptom

User passwords, credit card numbers or PII appear in the log file and eventually in a log aggregator visible to the operations team.

Cause

logger.debug('processing: %s', row) logs the full row dict, which may contain password or card_number fields.

Fix

Extract only the fields you need before logging: logger.debug('processing id=%s amount=%d', row['id'], row['amount']). Never log raw dicts or exception objects that contain user data.

Tricky questions

Does logger.debug('msg %s', value) build the formatted string immediately?
Most people say

Yes — %s formatting happens when the call is made.

Actually

No. With %-style arguments, the string is only formatted if the logger's effective level passes the DEBUG check. If the logger is at INFO, 'msg %s' % value is never computed. Using f-strings (logger.debug(f'msg {value}')) always builds the string, even when debug is disabled — wasted work.

Does propagate=False prevent a record from being processed by the named logger's own handlers?
Most people say

Yes — setting propagate=False stops all processing.

Actually

No. propagate=False only stops the record from travelling to the parent logger after the named logger's own handlers have already fired. The named logger's handlers still run.

Scenario question

You're handed a Python ETL package with a utils/logger.py module that calls logging.basicConfig(level=logging.DEBUG) at import time. In production, every INFO message from every library the package uses floods the log. What do you do?

Model answer

I'd remove the basicConfig call from utils/logger.py entirely. Library code should never configure the root logger — that is the application's responsibility. I'd replace it with logging.getLogger(__name__) in each module and leave configuration to the calling application. In the application entrypoint, I'd use dictConfig to set the root logger to WARNING and the package's own logger (e.g., etl) to INFO or DEBUG as needed. The trade-off: library developers lose the "works out of the box" convenience, but the application operator gains full control over what appears in the log.

Interview corner

CommonWhy should library code never call logging.basicConfig?
“I'd say because basicConfig configures the root logger, which affects every logger in the entire process. If my library configures the root at DEBUG, every third-party library in the application suddenly emits debug output into the operator's log. The correct pattern is to call getLogger(__name__) and let the application configure root logging however it wants.”
Tricklogger.debug(f'value={some_object}') vs logger.debug('value=%s', some_object) — which is faster when DEBUG is disabled?
“I'd say the percent-style is faster. The f-string builds the formatted string unconditionally before the function is even called. With percent-style arguments, CPython skips the formatting step entirely if the effective level is above DEBUG. For objects with an expensive __str__, this matters.”
Experience checkYour ETL job is producing duplicate log lines in CloudWatch. What is the likely cause?
“I'd check whether the package logger has its own handler and also propagates to root, which also has a handler. Each record fires both. The fix is either propagate=False on the package logger or removing the handler from the package logger and relying on propagation to root.”
DesignHow would you add a request-ID field to every log line in a long-running ETL job?
“I'd use a logging.Filter that adds the request ID to the record, or a LoggerAdapter that wraps the logger and pre-populates the extra dict. Both avoid repeating the ID on every individual log call. The trade-off: a Filter mutates the record in place, which can be surprising; an Adapter is more explicit but requires passing it around.”

Memory hook

Like a company reporting chain, log records travel up from named loggers to the root; handlers are notice boards at each level — put a notice board at two levels and every report gets posted twice.

If you remember only one thing:

Name every logger with getLogger(__name__), configure handlers only once at the root or with dictConfig, and use %-style arguments to avoid formatting strings the logger will never emit.

Self-check

  1. What does logger.exception('msg') do that logger.error('msg') does not?

    It appends the current exception traceback to the log record. Call it inside an except block.

  2. Why use logger.info('value=%s', val) instead of logger.info(f'value={val}')?

    The percent-style arguments are only formatted if the logger's effective level passes, so no string is built when INFO is disabled. The f-string is built unconditionally.

  3. What is the default level of the root logger?

    WARNING.

  4. How do you prevent a child logger from sending its records to the root logger?

    Set logger.propagate = False.

  5. What is the production alternative to logging.basicConfig for configuring logging?

    logging.config.dictConfig(config_dict).

Prove it

Write a script that configures two loggers: etl.reader at DEBUG and etl.writer at INFO. Capture output to a StringIO. Emit DEBUG, INFO and WARNING from each logger. Print only the lines that appear in the output. Confirm that etl.reader emits all three levels and etl.writer omits DEBUG. Paste the output.

py-13-06Card 6 of 6

Run the is-this-Pythonic review checklist

MediumP5 · SkippableKnow it exists12 min
pythonicanti-patternsmutable-defaultscomprehensionseafpnone-comparisoncode-review

Why you care

A senior engineer reviewing your pull request will flag five things before looking at business logic: mutable default arguments, comprehensions used for their side effects, over-broad except Exception blocks that swallow bugs, comparisons against None with == instead of is, and names so short or generic that a reviewer cannot tell what they hold. Each one is either a latent bug or a readability problem that slows every future edit. Interviewers give code review exercises precisely to test this checklist. Passing it signals you can be trusted to review others' code.

Explain like I'm 5

Imagine a form that has a pre-filled box. Every person who fills in the form writes in the same box — they do not get a fresh one. That is the mutable default argument: the list or dict in the function signature is shared across all calls, not created fresh each time. A comprehension used for side effects is a photocopier that also keeps notes — you wanted the notes but got a pile of photocopies instead. Neither is what you meant.

Core concept

The Pythonic review checklist — five items that appear in almost every code review:

  1. Mutable default arguments. def f(items=[]): shares one list across all calls. Use def f(items=None): and if items is None: items = [] inside. This applies to dict, set and any other mutable type. ([[py-02-02]] covers this in depth.)
  1. Comprehensions with side effects. [results.append(x) for x in data] builds a list of None values that you immediately discard while mutating results as a side effect. The intent is a loop; write a loop. List comprehensions are for building lists you keep, not for iterating with effects.
  1. Over-broad except Exception. except Exception: catches TypeError, AttributeError and every programming mistake in the block. When those exceptions appear during testing, you want them to propagate so you can fix them. Catch only the exception you can handle: except ValueError: for parsing, except FileNotFoundError: for missing files.
  1. == None instead of is None. x == None calls x.__eq__(None), which a custom __eq__ can override. x is None tests object identity and is always correct for None. PEP 8 mandates is None; linters flag == None.
  1. Names that hide intent. data, result, temp, x in module-level or function-level code obscure what the variable holds. Name it for what it contains: pending_orders, row_count, error_messages. One-letter names are fine in list comprehensions and short lambdas where the scope is one line.

Decision rule: treat each item on the checklist as a mandatory fix. Items 1 and 3 are bugs; items 2, 4 and 5 are readability problems that become bugs under maintenance. When you spot any of them in a review, block the PR with a CHANGES REQUESTED comment, not a nit:.

Under the hood

The mutable default trap is a CPython detail: function default values are evaluated once at function definition time and stored in func.__defaults__. They are not re-evaluated on each call. You can inspect them: f.__defaults__ returns the tuple of defaults. id(f.__defaults__[0]) is the same on every call unless you explicitly create a new object inside the function body. This is intentional — it allows decorators to use a mutable default as a cache — but it surprises everyone who expects "default value" to mean "value constructed at call time".

Minimal example Mutable default and comprehension side-effect traps

PythonShow mutable default persists and comprehension side effect returns Noneruns
def bad_collect(item, acc=[]):
    acc.append(item)
    return acc

print(bad_collect('a'))
print(bad_collect('b'))   # ['a', 'b'] — not ['b']

results = []
returned = [results.append(x * 2) for x in range(4)]
print('side effect:', results)
print('returned:', returned)
Output
['a']
['a', 'b']
side effect: [0, 2, 4, 6]
returned: [None, None, None, None]

In production Before-and-after code review for an Urban Company service booking loader

PythonFix mutable default, broad except, and comprehension side effectruns
from pathlib import Path
import json

Path('bookings.json').write_text(json.dumps([
    {'id': 'UC-001', 'service': 'cleaning', 'amount': 499},
    {'id': 'UC-002', 'service': 'plumbing', 'amount': None},
    {'id': 'UC-003', 'service': 'cleaning', 'amount': 750},
]))

# --- BEFORE (anti-patterns) ---
def load_valid_bad(path, errors=[]):          # mutable default
    data = json.loads(Path(path).read_text())
    [errors.append(r['id']) for r in data if r['amount'] is None]  # side-effect comprehension
    try:
        return [r for r in data if r['amount'] is not None]
    except Exception:                          # too broad
        return []

# --- AFTER (fixed) ---
def load_valid(path, errors=None):
    if errors is None:
        errors = []
    data = json.loads(Path(path).read_text())
    for r in data:
        if r['amount'] is None:
            errors.append(r['id'])
    try:
        return [r for r in data if r['amount'] is not None]
    except json.JSONDecodeError as e:
        raise ValueError(f'bad JSON in {path}: {e}') from e

errs = []
valid = load_valid('bookings.json', errs)
print('valid:', len(valid))
print('errors:', errs)
print('total amount:', sum(r['amount'] for r in valid))
Output
valid: 2
errors: ['UC-002']
total amount: 1249
Scenario A

In your world

The situation

Your ETL accumulates validation errors with def collect(row, errs=[]). The job runs nightly via Jenkins, so the Python process restarts and the bug stays hidden — the default list is empty at process start. A load test runs 50 small jobs in-process within a test suite.

What actually breaks

The third test run finds errors from all previous runs in errs because the list is shared across calls. The test asserts len(errs) == 1 and gets len(errs) == 3. The test failure is confusing because the function looks correct in isolation.

What you'd do on Monday

Use def collect(row, errs=None): with if errs is None: errs = [] inside. Each call gets a fresh list unless the caller explicitly passes one. The test passes, and the nightly job is safe because the Jenkins restart would have hidden the bug anyway — the test caught it first.

Scenario B

Imagine you work at Urban Company

500 K service bookings per month

The numbers

500,000 bookings/month ÷ 30 days ÷ 86,400 s/day ≈ 0.2 bookings/s on average; peak weekend mornings see roughly 10× that, about 2 bookings/s. At this scale, code quality and latency of the booking pipeline matter for on-call maintainability, not raw throughput. A broad except Exception in the booking validator that silently swallows a KeyError on a missing service_type field means 500,000 bookings per month accumulate with missing metadata, and the bug is only noticed at month-end reconciliation.

What the Scenario-A answer does here

The Scenario A fix — replacing broad except with except KeyError for field access — catches missing fields immediately. At 0.2 bookings/s the latency overhead of a stricter except is zero.

What flips at this scale

What flips at scale is not performance but failure detectability. At 100,000 bookings/day (a hypothetical ten-times scenario), a broad except that silently swallows 0.1% of bookings means 100 silent failures per day, each accumulating in the database with missing metadata. Detecting this requires a separate audit query rather than an alert on the exception rate.

The trade-off you'd defend

I would defend narrow exceptions (except KeyError, except ValueError) over broad ones at any scale, trading the "never crashes in production" comfort for immediate visibility of programming errors. A crash that pages the on-call is better than a silent data quality hole.

Illustrative. Urban Company booking volume is a public order-of-magnitude estimate; exact figures are not published.

Pythonic vs non-Pythonic patterns

PatternNon-PythonicPythonic
Countingif k in d: d[k] += 1 else: d[k] = 1Counter(items) or d[k] = d.get(k,0) + 1
Groupingif k not in d: d[k] = []; d[k].append(v)defaultdict(list); d[k].append(v)
Mutable defaultdef f(lst=[]):def f(lst=None): if lst is None: lst = []
None checkif x == None:if x is None:
Side-effect loop[log(x) for x in items]for x in items: log(x)
Broad exceptexcept Exception: passexcept ValueError: handle()

Gotchas

Mutable default shared across calls is invisible in single-call tests

Symptom

assert collect(row) == [row_id] passes in isolation but assert collect(row2) == [row_id2] fails with [row_id, row_id2] in a test suite.

Cause

The default list is shared across all calls in the same Python process; the test suite accumulates state.

Fix

Use None as the default and create a new list inside the function.

except Exception: pass hides bugs for months

Symptom

A field rename in the data source causes a KeyError, but the function returns an empty result silently. The downstream report shows zero rows for weeks.

Cause

except Exception: pass catches and discards KeyError, AttributeError, and every other programming error alongside the input errors you intended to swallow.

Fix

Catch only the specific exception you expect from user data (e.g., ValueError, json.JSONDecodeError). Let programming errors propagate.

Comprehension side effects are invisible at code review

Symptom

[cache.update({k: v}) for k, v in pairs] looks like it builds a list, but it silently returns a list of None and is only used for the side effect.

Cause

A list comprehension is syntactically identical whether it builds a useful list or discards one. The intent is hidden.

Fix

Use a for loop for side-effect iteration. Reserve comprehensions for building a list, dict or set you keep.

Tricky questions

Is x == None ever wrong when x is a standard Python object?
Most people say

No — == and is both test equality for None because None is a singleton.

Actually

A custom class can override __eq__ to return True for == comparison with None. x is None always tests object identity and is always correct. PEP 8 mandates is None; x == None generates a SyntaxWarning in some linter configurations.

Does a for loop creating a comprehension behave differently from a for loop?
Most people say

No — [f(x) for x in items] and for x in items: f(x) do the same thing.

Actually

The comprehension builds and returns a list of return values; the for-loop discards them. If f(x) returns None, the comprehension produces [None, None, …] which is immediately garbage-collected if you do not assign it. The for-loop is clearer for side-effect-only iteration and does not waste memory constructing the throwaway list.

Scenario question

You're reviewing a PR. The diff includes def process(items, cache={}): — a mutable dict default — and [db.insert(r) for r in items] — a comprehension for a side effect. The author says "it works in our tests." What do you say?

Model answer

I'd say the tests pass because each test probably calls process in a fresh subprocess, or the tests are small enough that the cache accumulation is not visible. The mutable default is a latent bug: on a long-running worker that calls process many times, the cache grows forever and is never cleared between logical batches unless the caller explicitly passes a fresh dict. The comprehension is a readability issue: a reader must read the body to discover it is not building a useful list. I'd request changes on both: def process(items, cache=None) with the None-guard inside, and a for r in items: db.insert(r) loop.

Interview corner

CommonWhat is wrong with def f(items=[])?
“I'd say the list is created once at function definition time and shared across all calls. Appending to items in the function body mutates the same list every time. The fix is items=None with if items is None: items = [] inside — that creates a fresh list for each call.”
TrickWhy use is None instead of == None?
“I'd say is tests object identity, which is always correct for the singleton None. == calls __eq__, which a custom class can override to return True for comparison with None. PEP 8 mandates is None and linters flag == None, so it also avoids style warnings.”
Experience checkYou see except Exception: return {} in a code review. What is the risk?
“I'd say it swallows every exception in the block: KeyError from a missing field, TypeError from a wrong type, AttributeError from a renamed method — all silently return {} instead of crashing. The caller sees an empty dict and may not notice the bug for weeks. I'd ask for the specific exception class the author expects and let everything else propagate.”
DesignWhen is a comprehension with a side effect acceptable?
“I'd say almost never in production code. The only case I've seen defended is a one-liner in a REPL session. In a code review I'd always ask for a for loop instead — it makes the intent explicit and does not allocate a throwaway list.”

Memory hook

Like a pre-filled form shared by everyone, the mutable default argument accumulates everyone's input in one place — give each caller a fresh form by defaulting to None and creating the container inside.

If you remember only one thing:

Default mutable arguments to None and create them inside the function; use for loops for side effects, never comprehensions.

Self-check

  1. What is the fix for def f(lst=[]): lst.append(1); return lst?

    Use def f(lst=None): if lst is None: lst = []; lst.append(1); return lst.

  2. What does [print(x) for x in range(3)] return?

    [None, None, None] — print returns None.

  3. Why does x is None catch cases that x == None misses?

    == calls __eq__, which can be overridden; is tests identity unconditionally.

  4. Name two exceptions a except Exception catch swallows that you never want to swallow.

    KeyError (missing field access) and TypeError (wrong argument type) are both programming errors that should propagate.

  5. When is a one-letter variable name acceptable?

    In a list comprehension or a short lambda where the scope is one line and the type is obvious from context.

Prove it

Write a function extract_amounts(rows, errors=None) that appends the id of any row where amount is None to errors and returns rows where amount is not None. Call it twice with the same errors list. Confirm both calls share the list, then rewrite with errors=None and confirm they do not share. Paste the output of both versions.