Your ETL script runs on your laptop, passes in CI, and silently loads a different version of a dependency in production. This chapter shows you how Python resolves packages, how environments isolate them, and how to structure a project so any colleague — or a fresh CI runner — can reproduce exactly what you tested.
Difficulty
Priority
Show
Priority
Difficulty
Level
py-10-01Card 1 of 6
Python package reproducibility: pyproject, environments and resolution
Every Python job that runs in production depends on a dependency resolution decision made at install time. If that decision is not frozen, it can change silently between your QA build and your prod deploy, between today and next month, between your machine and the CI runner. In ETL pipelines the failure is rarely a crash — it is a silent schema change in a data-parsing library that drops one column, or an off-by-one in a date parser that shifts your aggregations by a day. Interviewers at product companies ask about reproducible builds because it predicts whether you understand that "it works on my machine" is a build-process failure, not a coincidence.
Explain like I'm 5
A recipe card says "200g butter, 3 eggs, flour". Anyone who follows it ends up with a slightly different cake depending on which brand of butter they grab. A lockfile is the version of the recipe that says "Amul unsalted 82% fat, Nandini size-L eggs, Pillsbury all-purpose". Every cook who follows the lockfile gets the same cake.
Core concept
Python package installs are controlled by three layers:
pyproject.toml declares what your code requires: "requests>=2.31.0". This is the recipe card — it names an acceptable range, not a single version.
A lockfile (uv.lock, poetry.lock, or a hashed requirements.txt) pins every direct and transitive dependency to an exact version and, optionally, a SHA-256 hash. This is the shopping receipt.
A virtual environment is where those pinned packages are installed in isolation from the system Python and every other project.
Reproducibility requires all three. A pyproject without a lockfile leaves resolution to the tool on every fresh install. A lockfile without a venv risks conflicts with the system or other projects. A venv without a lockfile is isolated but not deterministic across machines.
The decision rule: write ranges in pyproject.toml (so your package is composable with others), commit the lockfile (so every install is identical), and never commit the venv directory itself. The lockfile is the contract; the venv is a generated artefact. Re-generate the venv from the lockfile on every fresh machine.
CPython's import system does not know about lockfiles — that concept lives entirely in pip, uv and poetry. What CPython does know is sys.path: an ordered list of directories it searches for modules. pip install places packages into a site-packages directory, and the venv mechanism works by prepending the venv's site-packages to sys.path and pointing the python executable at a copy that has the site-packages path baked in. This means: if you activate the wrong venv, or no venv, Python silently finds a different site-packages and your pinned versions are irrelevant. The actual pinning enforcement happens at install time; at runtime, Python just takes whatever is on sys.path first.
Minimal example Parse a pyproject.toml with tomllib
PythonRead project metadata from a pyproject.toml stringruns
import tomllib, textwrap
src = textwrap.dedent("""
[project]
name = "etl-job"
version = "1.0.0"
requires-python = ">=3.12"
dependencies = [
"requests>=2.31.0",
"psycopg[binary]>=3.1",
]
""").strip().encode()
config = tomllib.loads(src.decode())
for dep in config["project"]["dependencies"]:
print(dep)
Output
requests>=2.31.0
psycopg[binary]>=3.1
tomllib (stdlib since Python 3.11) parses TOML. The >= in the declared deps is a range, not a pin — this is intentional in pyproject.toml so your library composes with others. The lockfile converts these ranges to exact versions.
In production Detect unpinned dependencies
This script reads a requirements.txt-style file and flags anything without an exact pin — useful as a pre-deploy gate in Jenkins or GitHub Actions.
PythonFlag unpinned deps in requirements.txt contentruns
import re
requirements ="""\
requests==2.31.0
boto3>=1.28.0
pandas
psycopg[binary]==3.1.9
"""defcheck_pinned(reqs: str) ->list[tuple[str, str]]:
results = []
for line in reqs.strip().splitlines():
line = line.strip()
ifnot line or line.startswith("#"):
continue
name = re.split(r"[>=<!;\[]", line)[0].strip()
pinned ="=="in line
results.append((name, "pinned"if pinned else"UNPINNED"))
return results
for name, status incheck_pinned(requirements):
print(f"{name:<30} {status}")
boto3>=1.28.0 and bare pandas are UNPINNED. A lockfile generated by uv lock or pip-compile would replace these with exact pinned versions and file hashes.
Scenario A
In your world
The situation
Your Jenkins job pulls settlements.csv from S3 and loads it into Postgres using psycopg. The requirements.txt in the repo says psycopg[binary]>=3.1. Jenkins ran this fine for six months.
What actually breaks
A new psycopg 3.2 minor release changed the default row_factory return type. Your SELECT queries now return Row objects instead of tuples, so row[0] still works but dict(row) raises TypeError: cannot convert Row to dict. The Jenkins build stays green because the INSERT step still runs — only the downstream reconciliation report silently produces wrong column names. Finance notices a week later.
What you'd do on Monday
Pin the exact version in a lockfile and add a hash so the CI runner cannot silently pick up a newer release:
ShellAdd uv to a project and generate a lockfile
uv init # creates pyproject.toml if absent
uv add "psycopg[binary]"# adds to [project.dependencies]
uv lock # writes uv.lock with exact versions and hashes
uv sync # installs from the lock, creates .venv/
Commit pyproject.toml and uv.lock. Add .venv/ to .gitignore. On CI, run uv sync --locked (fails if uv.lock is stale relative to pyproject.toml). --frozen skips the freshness check; use it only when the lock is known-good, e.g. inside a Docker layer build.
Scenario B
Imagine you work at Atlassian
200 Python services, 150 transitive deps each
The numbers
Illustrative: 200 microservices × 150 transitive deps = 30,000 package slots. PyPI releases roughly 200–400 new package versions per day across popular packages. If each service's 150 deps include 1 package that releases a new minor version per week on average, that is 200 services × 1 unexpected upgrade/week = 200 silent dependency drifts per week across the fleet. Any one of those can break a downstream consumer quietly.
What the Scenario-A answer does here
The Scenario A fix (pinning direct deps) still helps but does not close the gap. Direct-dep pins constrain only the first level; a direct dep that itself declares >= ranges lets pip pick a newer transitive dep on every fresh install. Only locking the full transitive graph eliminates all 30,000 drift vectors.
What flips at this scale
At this scale you adopt a lockfile-first policy: every service ships a uv.lock or poetry.lock, and the CI pipeline runs uv sync --locked or poetry install --no-root — both fail if the lockfile is out of sync with pyproject.toml, making "lockfile out of date" a build failure rather than a runtime surprise.
The trade-off you'd defend
I would defend committing the lockfile over leaving resolution to CI, trading a larger git diff on dependency bumps for the guarantee that every environment is byte-for-byte identical. The only cost is a deliberate uv lock --upgrade-package boto3 when you actually want to bump a dep.
Illustrative numbers chosen to make the arithmetic visible; not Atlassian-published figures.
Gotchas
Pin disappears after pip install -r requirements.txt on a clean machine
Symptom
The exact version you tested is gone; a newer version is installed without warning.
Cause
requirements.txt was written with ranges (>=) not exact pins (==). pip resolves the latest compatible version on every fresh install.
Fix
Regenerate the file with pip freeze > requirements.txt (captures the currently installed exact versions) or switch to uv lock / pip-compile which produce a separate lock file with hashes.
uv sync installs a different version than pip install
Symptom
uv installs version X, pip installs version Y from the same requirements.txt.
Cause
uv's resolver uses a different algorithm (PubGrub) than pip's backtracking resolver. They can reach different solutions for the same set of constraints.
Fix
Commit the lockfile and always install from it (uv sync --locked). Never mix pip and uv installs against the same environment.
Hash mismatch causes install to exit 1 with a mismatch error
Symptom
pip install --require-hashes -r requirements.txt exits 1 with THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE.
Cause
The hash in the file was recorded for a different artifact than pip fetched. This happens with pip-compile --generate-hashes output: a requirements file generated on macOS records macOS wheel hashes; on Linux, pip downloads the Linux wheel and the hash does not match. It also happens if a package was re-uploaded to PyPI with a different tarball under the same version string.
Fix
Use uv.lock (universal — records hashes for all platforms) instead of pip-compile output. If pip-compile is required, generate the requirements file on the same platform as deployment, or use uv export --frozen --no-hashes and pip install --only-binary :all:.
Tricky questions
Is pyproject.toml the lockfile?
Most people say
Yes — if you pin exact versions in pyproject.toml like "requests==2.31.0" it is reproducible.
Actually
pyproject.toml with pinned direct deps is still not a lockfile. It does not record transitive deps, and it does not record file hashes. A real lockfile (uv.lock, poetry.lock) pins the complete resolved graph with hashes. Pinning direct deps in pyproject.toml is also bad practice for a library, because it prevents users from combining it with other packages that need a different version.
Does committing requirements.txt from pip freeze give reproducibility?
Most people say
Yes, pip freeze pins everything.
Actually
pip freeze pins exact versions but not hashes. Without hashes, a malicious or corrupted PyPI mirror can serve a different tarball under the same version number. Use pip-compile --generate-hashes or uv lock to include SHA-256 hashes.
Scenario question
You're handed a Jenkins pipeline where the same requirements.txt produces different boto3 versions on Tuesday's build and Thursday's build. The file says boto3>=1.28.0. The team claims "it always worked before."
Model answer
I'd first confirm the root cause by running pip install --dry-run -r requirements.txt on both runner images and comparing the resolved boto3 version. If they differ, the AWS SDK released a minor version between Tuesday and Thursday and pip picked it up. My fix is to add uv lock to the CI pipeline, commit uv.lock, and change the CI install step to uv sync --locked. For the immediate regression I'd pin boto3==<last-known-good> in requirements.txt as a stop-gap while the lockfile infra is wired up. The trade-off is that we now need a deliberate uv lock --upgrade-package boto3 to pick up security patches instead of getting them automatically — I would defend that as the right default for a production ETL job.
Interview corner
CommonWhat is the difference between pyproject.toml and a lockfile?
“I'd say pyproject.toml declares your intent — what ranges you accept — and the lockfile records the decision: exactly which version was picked for every direct and transitive dep, plus hashes. You commit both, but only pyproject.toml is meant for other people's dependency solvers to consume.”
TrickIs it safe to pin all direct deps in pyproject.toml to exact versions?
“I'd say it depends on whether the project is a library or an application. For an application or a deployed service, exact pins in a lockfile are fine. For a library published to PyPI, pinning your own dependencies to exact versions in pyproject.toml prevents users from combining your library with others — they will hit version conflicts. Libraries should use ranges in pyproject.toml and only pin in a lockfile for their own CI.”
Experience checkYour ETL job loaded different row counts on two consecutive runs from the same S3 file. Environment could be the cause. How do you rule it in or out?
“I'd compare the output of pip list or uv pip list on both runs by adding it to the build log. If any data-parsing package version differs, that is my first suspect. Then I'd check the changelog for that package for any parsing or schema changes between the two versions.”
DesignHow would you enforce that every service in a 200-service Python monorepo uses reproducible builds in CI?
“I'd add a CI step that runs uv sync --locked and fails if the lockfile is out of sync with pyproject.toml. I'd also add a lint step that scans each service's pyproject.toml for unpinned direct deps and warns — not fails — so teams can migrate gradually. The trade-off is that bumping a dep now requires a deliberate uv lock commit, which some engineers see as friction. I think that friction is the point.”
Memory hook
A pyproject.toml is the recipe card; the lockfile is the shopping receipt. Commit both. The venv is the meal — throw it away and re-cook from the receipt any time.
If you remember only one thing:
Without a lockfile, your transitive dependencies quietly update themselves between every install.
Self-check
What does uv sync --locked do differently from uv sync?
--locked fails if uv.lock is out of sync with pyproject.toml — plain uv sync would silently re-resolve and overwrite the lockfile. --frozen skips the freshness check entirely; use it only when the lockfile is known-current, such as inside a Docker layer build.
Why is pinning all deps in pyproject.toml insufficient for full reproducibility?
It pins direct dependencies but not transitive ones, and it records no file hashes, so a different tarball could be silently served under the same version number.
Where does the venv directory belong in git?
It does not belong in git. The venv is a generated artefact derived from the lockfile. Commit the lockfile; list .venv/ in .gitignore.
What mechanism does Python use at runtime to find installed packages?
sys.path — an ordered list of directories searched for module names. Activating a venv prepends the venv's site-packages directory to sys.path.
What does tomllib do and since which Python version is it in the stdlib?
tomllib parses TOML files (read-only). It is in the stdlib since Python 3.11. For writing TOML you need a third-party library.
Prove it
Create a fresh directory, write a pyproject.toml that declares one direct dep with a range (requests>=2.31.0), run uv lock (or pip-compile --generate-hashes) to produce a lockfile, then delete the lockfile and change the range to a different constraint. Run the install command again and confirm that the two install runs resolve different versions. Capture the output of uv pip list or pip list from both runs side by side and paste it as the artefact.
Python has three distinct problems that look similar: isolating packages from the system, managing which packages a project needs, and locking those packages to reproducible versions. venv, uv and poetry each solve different subsets. Using the wrong tool for the job leads to polluted system interpreters, lockfile drift, or team disagreements about who owns the pyproject.toml. Interviewers ask this because "I use pip" is the answer of someone who has never shared a Python project with a colleague and had it break on their machine.
Explain like I'm 5
A venv is a parking-bay reservation: it reserves a dedicated space for this project's packages so they do not collide with the system Python or other projects. uv is a valet: it reserves the bay, fetches the packages at high speed, and writes a receipt (lockfile) so the next valet parks the exact same cars. Poetry is a concierge: it reserves the bay, fetches the packages, writes the receipt, and also manages the whole project — versions, publication, build.
Core concept
Each tool occupies a different position on the spectrum from isolation only to full project management:
venv creates an isolated Python environment (a directory with its own python and site-packages). It does nothing else — you still use pip to install packages. Use it when you need quick, zero-dependency isolation on a machine that already has pip.
uv (0.4+) creates envs, installs packages at 10-100× pip speed (Rust-based resolver + parallel downloads), and manages a uv.lock lockfile. It reads pyproject.toml as the source of truth. Use it for new projects where speed and lockfile reproducibility matter and there is no existing poetry setup.
poetry creates envs, installs packages, manages a poetry.lock lockfile, and handles version bumping and PyPI publishing. It pre-dates uv's lockfile support and many teams already have poetry.lock in their repos. Use it on projects already using poetry, or where publishing to PyPI from the same tool matters.
Decision rule: for a new project, use uv — it covers environments, lockfiles, and publishing via uv publish. For a project already on poetry, stay on poetry unless you have a compelling reason to migrate. Use bare venv only for throwaway scripts or when you cannot install uv or poetry on the machine.
Whichever tool you choose, the workflow is the same: the lockfile is committed, the venv is not, and CI installs from the lockfile with a flag that fails on drift.
Minimal example Read declared dependency groups from pyproject.toml
uv sync installs the [project.dependencies] group. uv sync --extra dev adds the dev group. Poetry calls these the same thing but uses [tool.poetry.dependencies] and [tool.poetry.dev-dependencies] in older versions.
In production Audit a realistic settlements-etl pyproject.toml for unpinned deps
PythonCheck all dependency groups for unpinned entriesruns
import tomllib, textwrap, re
content = textwrap.dedent("""
[project]
name = "settlements-etl"
version = "0.2.1"
requires-python = ">=3.12"
dependencies = [
"psycopg==3.1.9",
"boto3==1.28.62",
"pydantic==2.5.0",
]
[project.optional-dependencies]
dev = ["pytest==7.4.3", "ruff==0.1.6", "mypy==1.7.0"]
ci = ["pytest==7.4.3", "ruff==0.1.6", "mypy==1.7.0", "coverage==7.3.2"]
""").strip().encode()
cfg = tomllib.loads(content.decode())
runtime = cfg["project"]["dependencies"]
dev = cfg["project"]["optional-dependencies"]["dev"]
unpinned = [d for d in runtime + dev if"=="notin d]
print("unpinned deps:", unpinned if unpinned else"none - all pinned")
print("runtime count:", len(runtime))
print("dev count: ", len(dev))
Output
unpinned deps: none - all pinned
runtime count: 3
dev count: 3
All dependencies are exact-pinned here. In practice your pyproject.toml for a library would use ranges; the lockfile holds the exact pins. This audit script is useful as a CI gate on services (not libraries).
Scenario A
In your world
The situation
You clone a new colleague's ETL repo, run pip install -r requirements.txt, and start the job. It crashes immediately.
What actually breaks
The colleague used poetry and never committed a requirements.txt. There is a poetry.lock in the repo but you do not have poetry installed. You try pip install . — that reads pyproject.toml but ignores poetry.lock, so it resolves fresh and picks a newer psycopg that has a different binary wheel. The job crashes with a ImportError: cannot import name 'AsyncConnection' that was not in the old version.
What you'd do on Monday
Install the tool that the lockfile belongs to, then install from it:
ShellInstall from a poetry lockfile (requires poetry >=1.8)
pip install poetry # or: pipx install poetry
poetry install --no-root # reads poetry.lock, installs exact versions
poetry run python jobs/load.py # runs inside the venv poetry created
For uv-based projects the equivalent is uv sync (reads uv.lock). Never use bare pip install on a project that has a lockfile — you skip the lockfile and lose reproducibility.
Scenario B
Imagine you work at Postman
80 Python test-grid workers, fresh envs per run
The numbers
Illustrative: Postman's integration-test grid spins up 80 isolated Python workers per test suite run. Each worker creates a fresh venv with 40 dependencies. Without a pre-built wheel cache: each package takes ~2s to download and install = 80 × 40 × 2 = 6,400s of total install CPU per run. Workers run 10 in parallel, so wall-clock install time is 6,400 / 10 = 640s ≈ 11 minutes. With uv + pre-built wheels: ~0.15s per package = 80 × 40 × 0.15 / 10 = 48s ≈ 1 minute. That 10-minute difference per test run, at 30 runs/day, is 5 hours of CI feedback delay eliminated.
What the Scenario-A answer does here
The Scenario A fix (pick the right tool for your project) still applies. But at this scale, even the right tool matters less than the wheel cache strategy. uv's shared wheel cache across workers is where most of the gain comes from — without it, even uv downloads the same wheel 80 times.
What flips at this scale
At this scale you decouple "which tool manages the lockfile" from "how the CI installs packages". The CI worker image pre-bakes the wheel cache from the lockfile at image build time, so uv sync at test time is a cache hit for every package. That requires a uv export > requirements.txt && uv pip download step in the Docker build, not just a different install tool.
The trade-off you'd defend
I would defend baking the wheel cache into the CI image over fetching at runtime, trading a slightly larger image for guaranteed sub-10s install times regardless of network conditions. The only risk is cache staleness on lockfile bumps, which the image rebuild step catches.
Illustrative numbers chosen to show the arithmetic; not Postman-published CI figures.
venv vs uv vs poetry
Feature
venv
uv
poetry
Creates an isolated env
yes
yes
yes
Installs packages
no (use pip)
yes (uv add, uv sync)
yes (poetry add, poetry install)
Generates a lockfile
no
yes (uv.lock)
yes (poetry.lock)
Manages pyproject.toml
no
yes
yes
Install speed vs pip
1×
10–100×
~0.5–1×
Publishes to PyPI
no
yes (uv publish)
yes
Stable lockfile format
n/a
since uv 0.4
yes
When to use
throwaway scripts, no-extra-tools constraint
new projects; can publish via uv publish
existing poetry repos
Gotchas
Activating the wrong venv silently uses the wrong packages
Symptom
You run the script; it imports successfully but produces wrong results because it found the system-installed version of a library, not the venv version.
Cause
You opened a new terminal tab without running source .venv/bin/activate (or .venv\Scripts\activate on Windows). The python command resolves to the system interpreter.
Fix
Always check which python (Mac/Linux) or where python (Windows) before running. Use uv run or poetry run to ensure the venv is always selected without manual activation.
pip install inside a poetry-managed venv updates poetry.lock silently
Symptom
Running pip install some-package inside the venv "works" but poetry.lock still shows the old version. The next poetry install reverts the change.
Cause
pip and poetry manage separate metadata. pip writes to site-packages directly; poetry does not see that change.
Fix
Use poetry add some-package to update both pyproject.toml and poetry.lock together.
uv.lock checked in but ignored on CI
Symptom
CI runs pip install -r requirements.txt and resolves different versions than local.
Cause
Someone added a requirements.txt export for legacy CI but the CI step never got updated to uv sync --locked.
Fix
Replace the CI install step with uv sync --locked and delete the requirements.txt export from CI.
Tricky questions
Does python -m venv .venv also install packages?
Most people say
Yes — that is why the venv is useful.
Actually
No. venv creates the isolated directory and copies the interpreter. On Python 3.12+ it installs only pip — setuptools is no longer bundled by ensurepip and wheel was never installed by venv. The --upgrade-deps flag upgrades pip (plus setuptools on older Pythons). You still need pip install or uv add to put application packages inside it.
Is poetry's venv the same as a venv created by python -m venv?
Most people say
No — poetry uses a special internal format.
Actually
It is the same standard venv layout (a pyvenv.cfg file and Lib/site-packages). The difference is that poetry manages it automatically and stores it in a cache directory rather than inside the project. You can tell poetry to store it in .venv/ with poetry config virtualenvs.in-project true.
Scenario question
You're handed a team of five engineers who all work on the same Python ETL service. Two use macOS, two use Ubuntu, one uses Windows WSL. They keep seeing "works on my machine" failures. You have an hour to fix the environment discipline. What do you do?
Model answer
I'd switch the project to uv in that hour: run uv init if there is no pyproject.toml, uv add all the current direct deps, uv lock to generate uv.lock, and commit both files. I'd update the README to say: clone the repo, run uv sync, done. For CI I'd change the install step to uv sync --locked. The trade-off is that everyone now needs uv installed (pip install uv or curl -LsSf https://astral.sh/uv/install.sh | sh), which is a one-time cost. I'd resist switching to Docker as the "fix" unless the team already uses it, because Docker for local dev is a larger adoption barrier than uv.
Interview corner
CommonWhat does activating a venv do?
“I'd say it prepends the venv's bin/ directory to PATH, so python and pip resolve to the venv's copies instead of the system ones. It also sets VIRTUAL_ENV. Deactivating reverses that. Without activation, you can still use the venv explicitly with .venv/bin/python.”
TrickCan you have two venvs active at the same time?
“I'd say not cleanly. Activating a second one overwrites PATH and VIRTUAL_ENV with the new venv's values. The first venv is effectively deactivated. If you need isolation within a single shell session, use uv run or poetry run — they activate and deactivate per command.”
Experience checkYou need to install a package that is not on PyPI, only on a private S3-hosted index. Which tool handles that best?
“I'd use uv with the --extra-index-url flag or by setting [[tool.uv.index]] in pyproject.toml pointing at the S3 bucket (which has an index-style layout). The same lockfile approach applies — uv will lock the private package with its hash. Poetry also supports extra indices. pip-tools supports them too but requires more manual wiring.”
DesignYour 80-worker test grid creates a fresh venv per worker every run and install time dominates the feedback loop. How do you fix it?
“I'd bake the wheel cache into the CI image at image-build time: run uv export --frozen > requirements-ci.txt then uv pip download -r requirements-ci.txt --dest ./wheel-cache in the Docker build. At test time, uv sync --locked finds every wheel already on disk — a cache hit for all 40 deps across all 80 workers. The trade-off is a larger Docker image that must be rebuilt whenever the lockfile changes, but install time drops from 11 minutes to under a minute and the cache is always consistent with the lockfile.”
Memory hook
venv parks the car, uv parks it fast and writes the receipt, poetry parks it and also books the service appointment. Know which one the project is already using before you touch it.
If you remember only one thing:
Use uv sync --locked in CI — it fails if uv.lock is out of sync with pyproject.toml, closing the "works on my machine" gap.
Self-check
What does uv sync --locked do that plain uv sync does not?
--locked fails if uv.lock is out of sync with pyproject.toml. Plain uv sync would silently re-resolve and overwrite the lockfile. --frozen skips the freshness check and installs from the lockfile as-is.
If you run pip install requests inside a poetry-managed venv, does poetry.lock update?
No. pip and poetry maintain separate metadata. poetry.lock only updates when you use poetry add or poetry update.
What is the minimum set of files to commit for a uv-managed project to be reproducible?
pyproject.toml and uv.lock. The .venv/ directory must be in .gitignore.
What does python -m venv .venv actually install in the new environment?
On Python 3.12+, only pip. setuptools is no longer bundled by ensurepip and wheel was never installed by venv. Use --without-pip to skip pip too.
Prove it
On a machine with uv installed: create a new directory, run uv init, add one package (uv add httpx), lock it (uv lock), then delete the .venv/ and run uv sync --locked. Confirm the same version of httpx is installed by running uv pip show httpx. Paste the output of uv pip show httpx and the first 5 lines of uv.lock as the artefact.
"Module not found" and "AttributeError: module has no attribute X" are the two most common import-related errors in Python. Both trace back to the same misunderstanding: the difference between a module (a single .py file) and a package (a directory that Python treats as a namespace). If you do not know this distinction you cannot read an import error message correctly, cannot design a public API for your own code, and cannot explain your project layout to a colleague. This is a 100 % interview question because it is a prerequisite for every other packaging concept.
Explain like I'm 5
A module is a book — one file, everything inside it is in one place. A package is a library shelf — a folder that holds many books and a special index card (__init__.py) that says "yes, this shelf is a library". When you import from payments.charge import apply, Python walks: find the payments shelf, open it, find the charge book, get the apply function.
Core concept
A module is any .py file. import utils loads utils.py from sys.path. Every name at the top level of utils.py is accessible as utils.some_name.
A regular package is a directory that contains an __init__.py file. When you import payments, Python runs payments/__init__.py and the result is the payments module object. Submodules inside the directory are accessible as payments.charge, payments.refund, etc.
A namespace package (PEP 420, Python 3.3+) is a directory without __init__.py. Python still treats it as a package but its __file__ is None and it has no __init__ code. Namespace packages are used for splitting a single package across multiple installed directories (rare in application code; common in framework plugins).
__all__ is a list you define in __init__.py (or any module) to declare the public API: the names that from pkg import * will export. Names not in __all__ are still importable by explicit path; __all__ only controls star imports. Even without __all__, names starting with _ are excluded from star imports by convention.
The decision rule: if you have one utility file, use a module. If you have related modules that form a coherent unit with a public API, use a package with __init__.py and define __all__. Do not create packages just to group files — a flat module is easier to import.
Minimal example Package public surface via __all__
Python__all__ controls which names star-import exposesruns
public names after 'from payments import *': ['charge', 'refund']
_internal_state accessible? True
__all__ controls star imports only. _internal_state is hidden from from payments import * but still reachable as payments._internal_state — Python does not enforce true privacy; __all__ is a declaration of intent.
In production Realistic ETL package with relative imports
PythonBuild and use an etl package with three submodulesruns
Relative imports (.extract, .transform) are preferred inside a package: they work regardless of what name you give the package in sys.path, and they make the dependency direction explicit. Absolute imports (from etl.extract import read_csv) also work but break if you rename the package.
See it happen How Python resolves `from etl.pipeline import Pipeline`
Python walks sys.path left to right, looking for a directory named 'etl' with __init__.py or a file named etl.py.
The directory has __init__.py so it is a regular package. Python executes __init__.py to build the 'etl' module object.
Now Python looks for etl/pipeline.py (or etl/pipeline/__init__.py).
etl/pipeline.py is executed; 'Pipeline' is looked up in its namespace and bound in the caller.
A repeat 'from etl.pipeline import Pipeline' hits sys.modules['etl.pipeline'] immediately — the file is not re-executed.
sys.modules acts as the import cache. Module code runs exactly once per process.
Notice: the module code runs once and is cached; repeated imports return the same object, which is why module-level state is global for the lifetime of the process.
Scenario A
In your world
The situation
Your Jenkins ETL repo grows to ten Python files in the root directory. A colleague adds a file called utils.py. Your script already does from utils import parse_date. It imports fine locally.
What actually breaks
On the CI runner the system Python has a third-party utils package installed globally. The CI runner's sys.path finds the third-party utils first. Your parse_date function does not exist there: ImportError: cannot import name 'parse_date' from 'utils'. Local still works because the repo is on your local sys.path first.
What you'd do on Monday
Convert the root-level files into a proper package. Create a directory with your project name (e.g. etl_jobs/), add __init__.py, move the modules inside, and change all imports to from etl_jobs.utils import parse_date. A named package is far less likely to collide with a third-party name.
Illustrative: a 500 KLOC codebase split into ~50 modules. If 20 of those modules do from core import * and core/__init__.py exports 60 names without __all__, each module's namespace is polluted with 60 extra names. 20 modules × 60 names = 1,200 implicit bindings. One name conflict (say status in core and status as a local variable) produces a wrong result with no error.
What the Scenario-A answer does here
The Scenario A fix (use a package, avoid name collisions with stdlib) still works. But from core import * across 20 modules means a new name added to core can silently shadow a local variable in any of those modules — "it passed tests" because the test fixture happened to have the same value.
What flips at this scale
At this scale you enforce an explicit import policy: __all__ in every __init__.py limits the exported surface to the 3-5 names that other modules actually need. A linting rule (ruff: F401, F403) fails any from module import * in non-test code. This makes the dependency graph between modules visible and auditable.
The trade-off you'd defend
I would defend explicit __all__ and no star imports over convenience, trading slightly more verbose import lines for a codebase where every module's dependencies are visible in its import block. The one exception is test helpers, where from fixtures import * is a common pattern.
Illustrative numbers; not Freshworks-published codebase metrics.
module vs package vs namespace package
Feature
Module (.py file)
Regular package (dir + __init__.py)
Namespace package (dir, no __init__.py)
__file__
path to the .py file
path to __init__.py
None
Can contain submodules
no
yes
yes
Runs code on import
yes (the file body)
yes (__init__.py body)
no (nothing to run)
Can define __all__
yes
yes
no (no __init__.py)
Use when
one utility file
coherent group with a public API
split install across multiple dirs (plugins)
Gotchas
import mypackage imports nothing from submodules
Symptom
After import mypackage, mypackage.utils raises AttributeError: module 'mypackage' has no attribute 'utils'.
Cause
__init__.py is empty (or does not import utils). Python does not auto-import submodules unless __init__.py explicitly does so.
Fix
Either add from . import utils in __init__.py, or use from mypackage.utils import X at the call site — the explicit form is usually better because it does not force all submodules to load on every import of the package.
A file named the same as a stdlib module shadows the stdlib
Symptom
import json crashes with AttributeError: module 'json' has no attribute 'loads', or import csv works but csv.DictReader is missing.
Cause
You have a json.py or csv.py in your working directory, which is on sys.path before the stdlib.
Fix
Never name your files after stdlib modules. Rename the file and restart the interpreter (the old module may be cached in sys.modules).
Relative imports fail when the file is run as a script
Symptom
from .utils import X raises ImportError: attempted relative import with no known parent package.
Cause
Python sets __package__ = None for a script run with python myfile.py. Relative imports require a non-None __package__.
Does a namespace package (no __init__.py) import differently from a regular package?
Most people say
No — Python detects both the same way and they behave identically.
Actually
They differ in two observable ways. First, myns.__file__ is None for a namespace package (no __init__.py to point to). Second, a namespace package's __path__ is a _NamespacePath, not a plain list, and can span multiple directories. Behaviour that depends on __file__ being set (e.g. Path(__file__).parent) will crash on a namespace package.
Does __all__ prevent import mymodule._private?
Most people say
Yes — __all__ enforces privacy.
Actually
No. __all__ is documentation and controls star imports only. Any name, including those starting with _, is importable by its full path. Python has no access modifiers that enforce privacy at the module level.
Scenario question
You're handed a Flask API where from models import db works in the main app but raises ImportError in a standalone data-migration script run directly with python migrations/add_column.py.
Model answer
I'd check two things first: whether the script is inside a package (does migrations/ have an __init__.py?) and whether models is a relative or absolute import. If it is from models import db (absolute), Python will find models.py only if the project root is on sys.path. When you run a script directly, Python adds the script's own directory to sys.path[0], not the project root. My fix is to run the migration as a module (python -m migrations.add_column from the project root) so Python adds the project root to sys.path, or to add the project root to PYTHONPATH as a CI environment variable. The trade-off of PYTHONPATH is that it is implicit and breaks if someone moves the repo; python -m is explicit and portable.
Interview corner
CommonWhat is the difference between a module and a package?
“I'd say a module is a single .py file. A package is a directory that contains an __init__.py — that file is what marks it as a package rather than just a folder. When Python sees import payments, it looks for either payments.py or payments/__init__.py.”
CommonWhat does __all__ do?
“I'd say it declares the public API of a module: the list of names that from module import * will export. It does not prevent direct access to anything — it is documentation that also controls star imports. If __all__ is absent, star imports get everything that does not start with an underscore.”
TrickIf you name your file json.py, what happens to import json?
“I'd say it imports your file instead of the stdlib, because Python checks the current directory first in sys.path. json.loads will be missing and everything that depends on stdlib json will break. I've seen this happen with datetime.py and logging.py and it is always confusing to debug because the error points at the caller, not the file with the name collision.”
Experience checkYour test suite imports fine but the same import fails in a scheduled Jenkins job. Both use the same Python version. What is the first thing you check?
“I'd check sys.path in both environments — print it at the top of the failing script. The most common cause is that the test suite is run from the project root (which adds the root to sys.path) but the Jenkins job runs from a different working directory. Then I'd check whether the package is installed in the venv or just present on the file system.”
Memory hook
A module is a book; a package is a shelf with an index card. The index card (__init__.py) is what makes Python say "yes, this is a library" instead of "yes, this is a folder."
If you remember only one thing:
If your project directory has no __init__.py, Python might treat it as a namespace package or skip it entirely — always add __init__.py unless you know what a namespace package is.
Self-check
What file makes a directory into a regular Python package?
__init__.py. Without it, Python either treats the directory as a namespace package (3.3+) or skips it.
Does __all__ prevent someone from doing from mymodule import _private?
No. __all__ only controls star imports. Any name is still accessible by its full explicit path.
If you run python mypackage/script.py instead of python -m mypackage.script, which directory is added to sys.path[0]?
The directory of the script itself (mypackage/), not the project root. This breaks absolute imports of sibling modules in the project root.
What is the observable difference between a regular package and a namespace package?
A namespace package has __file__ == None and __path__ is a _NamespacePath rather than a list. Its __init__.py does not exist so no code runs on import of the top-level name.
What does from . import utils mean inside __init__.py?
It imports the submodule utils.py from the same package directory and binds it as utils in the package namespace, so callers can do from mypackage import utils.
Prove it
Create a directory myetl/ with an __init__.py that defines __all__ = ['Pipeline'], a pipeline.py with a Pipeline class, and a _helpers.py with a helper function. Run a Python snippet that does from myetl import * and prints dir(), then imports myetl._helpers explicitly and calls the helper. Paste the printed dir() output and the helper return value as the artefact, confirming that _helpers is absent from the star import but reachable by explicit path.
Circular imports are the most silent class of architectural mistake in a Python project. The code looks correct, the tests pass individually, and then one day a new function is added to a module that triggers the cycle to execute in a different order, and you get ImportError: cannot import name 'X' from partially initialized module 'Y'. This is an un-Googleable error message for engineers who do not know what "partially initialized" means. Interviewers ask about it because fixing a circular import usually requires understanding the entire dependency graph, not just the two files involved.
Explain like I'm 5
Two kitchen sous-chefs: Riya handles sauces, Arjun handles mains. Riya's mise-en-place requires Arjun's stock reduction (which is not ready yet). Arjun's mise-en-place requires Riya's base sauce (which is also not ready yet). They are both waiting for each other before they can start. Python's import system hits exactly this deadlock when models.py imports from utils.py and utils.py imports from models.py.
Core concept
When Python imports a module, it follows these steps:
Check sys.modules for a cached copy. If found, return it immediately.
Create a new, empty module object and put it in sys.modules under the module's name.
Execute the module's source code. The module object is now being filled in.
Return the completed module object.
A circular import happens when step 3 of module A triggers an import of module B, whose step 3 triggers an import of module A — but A is still in step 3 (partially initialized). Python finds A in sys.modules (it was added in step 2) and returns the partially initialized object. Any name that A's step 3 has not yet defined is missing from that object.
The result is ImportError: cannot import name 'X' from partially initialized module 'A' (most likely due to a circular import).
The fix follows one of three patterns, in order of preference:
Move shared code to a third module that neither A nor B imports from (most common).
Defer the import: put the import statement inside the function that needs it, not at the top level.
Use import module instead of from module import name: the module object reference is valid even when partially initialized; a name that does not yet exist is looked up at call time, not at import time.
The decision rule: if you reach for option 2 or 3, the real fix is option 1 — the cycle is a symptom of wrong layering. Draw the dependency arrow direction and make it acyclic.
Under the hood
Python's module cache (sys.modules) is the mechanism that makes circular imports possible rather than infinite-recursion fatal. When A starts importing B and B tries to import A, Python finds sys.modules['a'] — an incomplete module object — and returns it without re-entering A's source code. The partial object has whatever names A's __init__ or top-level code defined before the line that triggered the import of B. Names defined after that line do not exist yet. This means the error is positional: if AMOUNT = 100 appears on line 1 of models.py and from utils import format_amount appears on line 2, then AMOUNT exists in the partial object but format_amount does not yet.
Minimal example See the partially-initialized module error
PythonCircular import raises ImportError at the missing nameruns
ImportError: cannot import name 'AMOUNT' from partially initialized module 'models' (most likely due to a circular import)
AMOUNT is defined on line 2 of models.py, but models tries to import format_amount from utils on line 1. When utils imports back from models, models is half-executed and AMOUNT does not exist yet.
In production Fix the cycle by extracting shared constants
Move AMOUNT (and other shared constants) to a third module that neither models nor utils depends on. This breaks the cycle.
PythonExtract shared constants to break the circular importruns
constants.py has no imports from models or utils, so there is no cycle. The dependency arrows are: models → constants, utils → constants. Both are one-directional.
See it happen Why the partial-initialization error happens
Before executing models.py, Python registers an empty module object in sys.modules to handle re-entrant imports.
Line 1 is 'from utils import format_amount' — Python now imports utils.
Same pattern: an empty utils module is registered before its code runs.
Line 1 is 'from models import AMOUNT' — Python needs models, which is already in sys.modules.
Python finds models in sys.modules and returns it immediately — but models is only partially executed. AMOUNT has not been defined yet (it is on line 2).
utils tries to find AMOUNT in the partial models object. It is not there. ImportError is raised.
Notice: the partial module object returned from sys.modules is what exposes the problem — Python returns whatever has been defined so far, not a complete module.
Scenario A
In your world
The situation
Your ETL package has pipeline.py that imports from validators import validate_schema, and validators.py that imports from pipeline import SCHEMA_VERSION (a module-level constant) to build its error messages.
What actually breaks
The job starts fine for six months. A new engineer adds a function to pipeline.py above the SCHEMA_VERSION = 3 line. Now SCHEMA_VERSION is no longer defined when validators.py tries to import it on startup. The Jenkins job starts crashing with ImportError: cannot import name 'SCHEMA_VERSION' from partially initialized module 'pipeline'. The error points at validators.py but the real problem is the import cycle.
What you'd do on Monday
Move SCHEMA_VERSION and other shared constants to a dedicated config.py (or constants.py) module that neither pipeline.py nor validators.py imports from. Both import from config.py. The cycle is gone.
ShellExtract constants to break the cycle
# Create a config.py with shared constantsecho"SCHEMA_VERSION = 3"> etl/config.py
# In pipeline.py: replace 'from validators import ...' that caused the cycle# In validators.py: replace 'from pipeline import SCHEMA_VERSION'# with 'from config import SCHEMA_VERSION'
Scenario B
Imagine you work at PhonePe
30 Python modules across payment, risk and reporting domains
The numbers
Illustrative: 30 modules, each importing from an average of 4 others = 120 import edges. In a clean layered architecture (data → domain → service → api) all edges point downward. If 5% of edges point upward (6 edges), each creates a potential cycle: 6 cycles × average 2 modules per cycle = 12 modules that can produce partial-initialization errors on any startup order change.
What the Scenario-A answer does here
The Scenario A fix (move shared constants to a third module) works for individual cycles. But at 30 modules across three domains, fixing cycles one at a time without a map leads to playing whack-a-mole: extract AMOUNT to constants.py, then discover that constants.py now imports from validators.py to format error messages, creating a new cycle.
What flips at this scale
At this scale you enforce a dependency-direction rule at the architecture level: the data layer may not import from service, service may not import from api. A CI lint check (import-linter or a custom script that walks ast.Import nodes) fails any import that violates the layering. The rule converts a runtime error that appears months after the cycle is introduced into a CI failure on the PR that introduces it.
The trade-off you'd defend
I would defend a strict layering policy enforced by CI over ad-hoc cycle fixes, trading some design rigidity for the guarantee that cycles cannot silently accumulate. The one exception is test files, which legitimately import across layers.
Illustrative numbers; not PhonePe-published architecture metrics.
Gotchas
The import works in a test but fails in the deployed job
Symptom
pytest runs fine; python jobs/load.py crashes with the partial-initialization error.
Cause
pytest imports modules in alphabetical or collection order; the deployed script imports them in a different order. The cycle was always there but the import order in tests happened to avoid the partial-initialization window.
Fix
The cycle is the bug, not the import order. Fix the cycle using one of the three patterns in core.
Moving the import inside the function silences the error but hides the cycle
Symptom
After moving from models import AMOUNT inside the function body, the error disappears. Six months later a different engineer sees the unusual local import and moves it back to the top.
Cause
Deferred imports are a workaround, not a fix. They are not idiomatic and invite regression.
Fix
Use the deferred import only as a temporary measure with a # TODO: extract to constants.py comment. Track it as a tech-debt ticket.
from models import * hides cycles more than explicit imports
Symptom
Star imports mask the partial-initialization error; the star import silently imports zero names from the partial object instead of raising an error on the missing name.
Cause
from models import * imports whatever names exist at the moment — an empty partial object has none but raises no ImportError.
Fix
Never use star imports in production code. Use explicit imports so partial-initialization failures raise immediately.
Tricky questions
Does putting the import inside the function body fix the circular dependency?
Most people say
Yes — deferred imports are the recommended fix.
Actually
A deferred import avoids the error but does not fix the dependency. The cycle still exists; the deferred import just delays it until the function is called, at which point the module is fully initialized. The real fix is to extract the shared name to a third module. Deferred imports are a maintenance hazard because they are not visible in the import block.
If module A does import B (not from B import X), does a circular import still cause errors?
Most people say
No — import B only needs the module object, which exists in sys.modules even if partial.
Actually
It depends on when A uses names from B. If A only calls B.some_function() inside a function body, import B works fine — B is fully initialized by the time A calls it. But if A accesses B.CONSTANT at module level, and that constant is defined after the point where B imports A, A gets AttributeError: module 'B' has no attribute 'CONSTANT'.
Scenario question
You're handed a models.py and a services.py that import each other. The error only appears when you run the app from the command line, not in pytest. A colleague says "just add lazy imports". What do you do?
Model answer
I'd first draw the dependency graph — what does models.py actually need from services.py and vice versa. In my experience, models should not import from services at all: models define data structures, services use them. The cycle usually means one of the imports is in the wrong direction. I'd extract whatever models.py needs from services.py (probably a constant or a type) into a shared types.py or config.py. I'd accept lazy imports only if the dependency is genuinely bidirectional at runtime, and I'd document why. The trade-off of lazy imports is that the dependency is invisible at the module level and future engineers will keep trying to "clean it up."
Interview corner
CommonWhat does "partially initialized module" mean in a Python ImportError?
“I'd say it means Python started executing the module's source code — and added the module object to sys.modules — but the import of a dependency triggered a re-import of the first module before it finished. The returned module object only has the names defined up to the line that triggered the cycle. Any name after that line is missing.”
TrickCan you have a circular import that never raises an error?
“I'd say yes, if neither module accesses names from the other at module-level code. If A and B import each other but only call functions on each other inside function bodies, the cycle is resolved by the time any name is looked up and no error occurs. That is why cycles can hide for months. The fix is still to remove the cycle, because a future top-level use of a name will silently fail.”
Experience checkYou have a cycle between models.py and utils.py. Describe the exact sequence of events that causes the ImportError.
“I'd walk through it: Python starts executing models.py and hits 'from utils import X'. It registers an empty models object in sys.modules, then starts executing utils.py. utils.py hits 'from models import Y'. Python returns that partial models object — only names defined before the point where models.py triggered utils.py exist. If Y is after that point, it is missing and Python raises ImportError.”
DesignHow would you enforce that a 30-module Python package never develops circular imports?
“I'd add import-linter to CI: define layers (data, domain, service, api) and the allowed import directions. Any violating import fails the build, converting a cycle that silently grows over months into a PR-time failure. The trade-off is maintaining the layer definitions as the codebase evolves.”
Memory hook
Two sous-chefs waiting for each other before they start is a circular import — Python hands each one a half-filled prep station and the missing ingredient is the ImportError.
If you remember only one thing:
Move shared names that two modules both need into a third module with no imports of its own.
Self-check
What does Python put in sys.modules at the start of importing a module?
An empty (partially initialized) module object, before any of the module's source code runs.
Why does a circular import sometimes produce no error?
If neither module accesses names from the other at module-level (only inside function bodies), the cycle resolves before any name lookup and no ImportError is raised.
What are the three patterns for fixing a circular import, in order of preference?
Extract shared names to a third module. 2. Defer the import inside the function that needs it. 3. Use import module instead of from module import name and access names at call time.
What is the real error message Python raises for a circular import?
ImportError: cannot import name 'X' from partially initialized module 'Y' (most likely due to a circular import).
Why does the error appear only at module level, not inside function bodies?
Function bodies execute when called, at which point all modules in the cycle are fully initialized. Module-level code executes at import time, when some modules may still be partial.
Prove it
Create two files alpha.py and beta.py that import from each other. Run python -c "import alpha" and copy the full error traceback as the artefact. Then apply the fix (extract to shared.py) and confirm the same command exits 0. Paste both the error output and the successful python -c "import alpha; print(alpha.VALUE)" output.
"It works when I run it but crashes when I import it" is the failure mode of a Python file with top-level side effects: a database connection opened at import time, a print that runs on every import, or an argparse.parse_args() that exits non-zero when the test suite imports the module. These failures are preventable with a two-line pattern that every product-company engineer is expected to know. Interviewers ask about __main__ and entry points because they separate "script I wrote on my own machine" from "distributable tool that teams and customers can run."
Explain like I'm 5
__main__.py is the front door of a package. When you knock on the front door with python -m mypackage, Python opens __main__.py. When your code imports the package normally (import mypackage), Python walks in through the side entrance (__init__.py) and ignores the front door entirely. The if __name__ == '__main__': guard is the same idea on a single file: put it around the code that should only run when someone knocks on the front door.
Core concept
Python sets __name__ to the string '__main__' when a file is the entry point of execution: run directly with python file.py, run as a module with python -m package, or run as a package's __main__.py via python -m package. When the same file is imported by another module, __name__ is the module's dotted name, never '__main__'.
The if __name__ == '__main__': guard wraps any code that should only run at the top level: argparse, database connections, expensive setup, or main() calls. Everything outside that block is import-safe.
__main__.py extends this to packages: placing a __main__.py inside mypackage/ makes python -m mypackage a valid command. The file's __package__ is set to 'mypackage', so relative imports work correctly — unlike running python mypackage/somefile.py directly.
Console scripts (entry points in pyproject.toml) let you expose your package's entry point as a command-line name after pip install:
After installation, etl-run on the PATH calls main() in etl_jobs/__main__.py directly, without the user needing to know the package name or module path.
Decision rule: use if __name__ == '__main__': for any file that has both a useful import surface and a runnable main function. Use __main__.py when the package itself should be executable as a command. Use console scripts when you ship the package to end users or teammates via pip.
Under the hood
When Python executes python -m mypackage, it sets sys.argv[0] to the path of __main__.py, sets __name__ to '__main__', and sets __package__ to 'mypackage'. That last assignment is what makes relative imports inside __main__.py work. By contrast, python mypackage/__main__.py sets __package__ to None, breaking any relative import inside the file. This is the reason python -m is the canonical way to run a package: it correctly seeds the package context that the rest of the import machinery depends on.
Minimal example __name__ guard protects import-time side effects
Python__name__ is __main__ only when the file is the entry pointruns
defgreet(name: str) ->str:
returnf"hello, {name}"# Runs only when executed directly, not when imported.if__name__=="__main__":
print(greet("world"))
print("__package__:", __package__)
Output
hello, world
__package__: None
__package__ is None here because this snippet is run as a standalone script, not as part of a package. Running with python -m mypackage would print __package__: mypackage.
In production Package with __main__.py and argparse entry point
PythonPackage that runs as python -m etltool or via console_scriptsruns
The argparse.ArgumentParser call is inside main(), which is called at the bottom of __main__.py. If a test imports etltool, Python runs __init__.py only — __main__.py is never touched and parse_args() never executes.
Scenario A
In your world
The situation
You write a settlement reconciliation script reconcile.py that opens a database connection at module level and calls argparse.parse_args() at the top. Your senior engineer asks you to add unit tests.
What actually breaks
The first test that does import reconcile connects to the production database, prints the argparse help, and calls sys.exit(0) — terminating the entire test process. The test runner shows Process finished with exit code 0 and no tests ran.
What you'd do on Monday
Wrap all startup code in the if __name__ == '__main__': guard. Move the database connection inside the function that needs it (or accept it as a parameter for testability). The module is now import-safe: importing it loads the functions but executes nothing.
PythonImport-safe reconciliation module (needs a real DB connection)
import argparse
from pathlib importPathdefreconcile(settlements_path: Path, db_url: str) ->dict:
# opens connection here, inside the function
...
defmain() ->None:
parser = argparse.ArgumentParser()
parser.add_argument("--db-url", required=True)
parser.add_argument("input", type=Path)
args = parser.parse_args()
result =reconcile(args.input, args.db_url)
print(result)
if__name__=="__main__":
main()
Tests can now from reconcile import reconcile and pass a mock DB URL without triggering argparse or a real connection.
Scenario B
Imagine you work at Stripe
5 CLI tools, ~3 k monthly active users each
The numbers
Illustrative: Stripe ships ~5 Python CLI tools to developers via PyPI. Each tool has ~3,000 monthly active users. Without a console_scripts entry point: users run python -m stripe_tool --help, which requires knowing the internal module name. With console_scripts: users type stripe-tool --help. At 3,000 users × 5 tools × avg 4 invocations/day = 60,000 command invocations per day. If 2% of those fail because the user forgot the python -m prefix and got a ModuleNotFoundError instead: 1,200 confused invocations per day → support tickets.
What the Scenario-A answer does here
The Scenario A fix (use if __name__ == '__main__': to make the module import-safe) is necessary but not sufficient for an installed CLI. An import-safe module still requires the user to know to run python -m rather than just the tool name. Without a console_scripts entry point, the tool is not a first-class CLI.
What flips at this scale
At this scale you define the entry point in pyproject.toml, ship the tool via PyPI, and guarantee that stripe-tool is on PATH after pip install stripe-tool. The implementation details (module path, package name) are invisible to users. Versioned console scripts also mean pip install --upgrade updates the command transparently.
The trade-off you'd defend
I would defend console_scripts over documenting python -m for any tool used by people outside your team, trading a small pyproject.toml entry for a standard user experience. For internal scripts used by 2-3 engineers who know the module structure, python -m is fine.
Illustrative numbers chosen to make the arithmetic visible; not Stripe-published CLI usage figures.
Gotchas
python mypackage/__main__.py fails with a relative import error
Symptom
from .utils import X raises ImportError: attempted relative import with no known parent package.
Cause
Running a file directly with python path/to/__main__.py sets __package__ to None. Relative imports require __package__ to be set.
Fix
Always run packages with python -m mypackage, not python mypackage/__main__.py.
argparse.parse_args() at module level kills the test runner
Symptom
import mymodule causes the test process to exit 0 with no output.
Cause
argparse.parse_args() at module level reads sys.argv, finds --help or an unexpected argument (such as pytest's own flags), and calls sys.exit(0).
Fix
Move parse_args() inside main() and call main() only under if __name__ == '__main__':.
Installed console_scripts point to the wrong Python
Symptom
mycommand runs but uses a different Python or a different venv than expected.
Cause
pip install added the console script to the venv's bin/ directory, but the venv is not activated and the system PATH finds an old copy of the script first.
Fix
Activate the venv before running the command, or use pipx install for user-level CLI tools so the tool's dependencies are isolated from the project's venv.
Tricky questions
Does if __name__ == '__main__': prevent the code from running when the file is imported?
Most people say
No — Python still parses the whole file including that block.
Actually
Python parses and compiles the entire file, including the if __name__ == '__main__': block. The block is not executed because the condition is False at import time. The function definitions inside the block are also not defined until the block runs, so a function defined only inside if __name__ == '__main__': is not accessible to importers.
Is python -m mypackage the same as python mypackage/__main__.py?
Most people say
Yes — both run __main__.py.
Actually
Both execute __main__.py, but they set __package__ differently. python -m mypackage sets __package__ = 'mypackage' and adds the project root to sys.path. python mypackage/__main__.py sets __package__ = None and adds mypackage/ to sys.path[0]. Relative imports inside __main__.py work with -m but fail with the direct path.
Scenario question
You're handed a library that a colleague uses in two ways: import data_loader in their pipeline, and python data_loader.py --validate orders.csv from the command line. Adding a unit test breaks things. What structure do you impose?
Model answer
I'd impose a strict separation: all importable logic goes in functions at module level, and the command-line entry point goes in if __name__ == '__main__': at the bottom, calling a main() function that wires argparse to the logic functions. For the library use case, the test imports from data_loader import load, validate — no side effects. For the CLI use case, python data_loader.py --validate orders.csv hits the guard and calls main(). If the team later wants to install it as a command, I'd move the logic into a package and add a console_scripts entry in pyproject.toml. The trade-off is that a main() function that accepts parsed args (not argparse.Namespace directly) is more testable than one that calls parse_args() internally.
Interview corner
CommonWhat does if __name__ == '__main__': do?
“I'd say it guards code that should only run when the file is the top-level entry point. Python sets __name__ to '__main__' when a file is executed directly or via python -m. When imported, __name__ is the module's dotted name. The guard prevents argparse calls, database connections and main() from running every time a colleague imports your module.”
TrickWhere does __package__ get its value when you run python -m mypackage?
“I'd say Python sets __package__ to 'mypackage' — the name of the package containing __main__.py. That is why relative imports work under python -m but not when you run the same __main__.py as a direct script: in the direct case, __package__ is None and relative imports fail.”
Experience checkYour ETL entry point uses argparse.parse_args() at module level. A new engineer adds from etl import reconcile to a test. The test suite exits immediately. Diagnose and fix.
“I'd say parse_args() at module level reads sys.argv when the module is imported. During a pytest run, sys.argv contains pytest's own arguments, which argparse does not recognise, so it prints the usage message and calls sys.exit(2). The fix is to move parse_args() inside main() and call main() only under if __name__ == '__main__':.”
DesignWhen would you choose console_scripts over documenting python -m?
“I'd say console_scripts whenever the tool is used by people outside my immediate team or shipped as a package. It gives users a named command that works regardless of the Python installation details. I'd use python -m for internal scripts where everyone knows the project structure and a named command would add maintenance overhead for no user benefit.”
Memory hook
__main__.py is the front door of a package — Python only knocks on it when you run python -m, never when you import. Everything behind the if __name__ == '__main__': guard is the part that lives and dies by the front door.
If you remember only one thing:
Wrap argparse, database connections and main() calls in if __name__ == '__main__': so that importing the module never executes them.
Self-check
What value does Python give to __name__ when a file is imported?
The module's dotted name (e.g. 'etl.pipeline'), not '__main__'.
What is the difference in __package__ between python -m mypackage and python mypackage/__main__.py?
What happens to argparse.parse_args() when it is called at module level during a pytest run?
sys.argv contains pytest's arguments. argparse does not recognise them and calls sys.exit(2), which terminates the test process.
What must pyproject.toml contain to install a mycommand CLI command?
A [project.scripts] section: mycommand = "mypackage.__main__:main".
Prove it
Create a calc/ package with __init__.py exporting a add(a, b) function and __main__.py that uses argparse to accept two numbers and prints their sum. Verify that python -m calc 3 5 prints 8 and that python -c "from calc import add; print(add(3, 5))" also prints 8 without triggering argparse. Paste the output of both commands as the artefact.
Every environment you have ever tested in is contaminated: your laptop has years of pip install accumulation, your CI runner has cached layers from six months ago, and your colleague's machine has a different OS. The only way to know that your lockfile actually works is to destroy the environment and rebuild it from scratch — in the same Python version and on the same platform as production. Interviewers ask about this because "it worked in CI" followed by "it crashed in prod" is a failure of artifact verification, and product companies run this check automatically on every merge.
Explain like I'm 5
Testing a recipe by cooking it in your normal kitchen — where you already have last week's leftover stock on the stove — does not tell you whether the recipe works in a clean kitchen. A clean-install test deletes the stock, puts an empty pot on the stove, and follows the recipe from scratch. If the dish comes out wrong, the recipe is incomplete.
Core concept
A clean install test has four steps:
Delete the existing environment: remove .venv/, deactivate the current env, or use a fresh Docker container.
Reinstall from the lockfile only: uv sync --locked, poetry install --no-root, or pip install --require-hashes -r requirements.txt. No --upgrade, no implicit re-resolution.
Check platform constraints: Python version (requires-python), compiled extensions (C wheels have platform tags), and operating system. An extension built on macOS will not load on Amazon Linux.
Run a smoke test: import the main entry point, run the cheapest test suite, or execute the ETL job against a known fixture. Confirm exit code 0.
The platform trap is the most common clean-install failure in ETL. uv.lock is a universal, cross-platform lockfile that records wheels for every platform, so it is safe to commit from any machine. The trap is the deployment artifact: a developer runs pip install -t package/ on macOS and zips the result; the zip contains macOS .so files that fail on Lambda's Amazon Linux with an ImportError naming the compiled module.
The same trap applies to pip freeze and pip-compile output — those files record the wheel that was installed on the developer's machine, which is platform-specific.
The fix: build the deployment artifact for the target platform. Either build it inside a Linux container (e.g. python:3.12-slim or the public Lambda base image), or use uv pip install --python-platform x86_64-manylinux_2_28 --python-version 3.12 --only-binary :all: --target ./package.
Decision rule: every deployment artifact (Docker image, Lambda zip, wheel) must be built on the same platform as production. The lockfile is safe to generate on any machine; it is the artifact build that must be platform-correct.
Under the hood
Python wheels are tagged: psycopg_binary-3.1.9-cp312-cp312-manylinux_2_17_x86_64.whl. The tag has three parts: Python tag (cp312), ABI tag (cp312), platform tag (manylinux_2_17_x86_64). When pip or uv installs, it picks the wheel whose tags match the running interpreter and platform. uv.lock records wheels for all platforms, so uv sync --locked on Linux fetches the Linux wheel and on macOS fetches the macOS wheel — this is why uv.lock can be committed from any developer machine. If a requirements file or lock format does not include a matching wheel for the target platform, the tool falls back to an sdist (source build). This is why artifact-build failures show error: command 'gcc' failed: no pre-compiled wheel exists for the target platform, so pip attempted to compile from source but the required C headers were absent in the build environment.
Minimal example Check Python version against pyproject.toml requirement
PythonValidate the current interpreter against requires-pythonruns
import sys, re
defcheck_python_req(req: str) ->bool:
"""Parse '>=3.12' from requires-python and check sys.version_info."""
m = re.match(r">=(\d+)\.(\d+)", req)
ifnot m:
returnFalse
major, minor =int(m.group(1)), int(m.group(2))
return sys.version_info >= (major, minor)
print("Python:", sys.version.split()[0])
print("Satisfies >=3.12:", check_python_req(">=3.12"))
print("Satisfies >=3.13:", check_python_req(">=3.13"))
Use this check at the top of your CI script before the install step. Failing early with "Python 3.11 does not satisfy >=3.12" is clearer than a later SyntaxError from a 3.12-only syntax.
In production Environment consistency checker
This script reads pyproject.toml and reports whether the running environment satisfies the declared requirements. It uses only the stdlib (tomllib, importlib.metadata, re, sys).
PythonAudit installed packages against pyproject.toml declarationsruns
import sys, re, tomllib, importlib.metadata as meta, textwrap
# Inline pyproject.toml — in practice read from Path("pyproject.toml").read_bytes()
pyproject_src = textwrap.dedent("""
[project]
name = "etl-job"
requires-python = ">=3.12"
dependencies = [
"requests==2.31.0",
]
""").strip().encode()
cfg = tomllib.loads(pyproject_src.decode())
requires_python = cfg["project"].get("requires-python", "")
declared_deps = cfg["project"].get("dependencies", [])
defsatisfies_python(req: str) ->bool:
m = re.match(r">=(\d+)\.(\d+)", req)
return sys.version_info >= (int(m.group(1)), int(m.group(2))) if m elseTrue
results: list[tuple[str, str, str]] = []
py_ok =satisfies_python(requires_python)
results.append(("python", sys.version.split()[0], "OK"if py_ok else"FAIL"))
for dep in declared_deps:
pkg = re.split(r"[>=<!;\[]", dep)[0].strip()
req_ver = dep.split("==")[1] if"=="in dep elseNonetry:
installed = meta.version(pkg)
status ="OK"if (req_ver isNoneor installed == req_ver) elsef"MISMATCH ({installed})"except meta.PackageNotFoundError:
status ="MISSING"
results.append((pkg, req_ver or"any", status))
for name, version, status in results:
print(f"{name:<20} {version:<15} {status}")
Output
python 3.12.4 OK
requests 2.31.0 MISSING
requests shows MISSING because this runs in a stdlib-only environment — exactly what a clean Lambda cold start looks like before the layer is attached. The checker is the gate that would catch this before deployment.
Scenario A
In your world
The situation
Your ETL job runs fine in Jenkins. A new engineer deploys it to AWS Lambda for the first time. The Lambda function crashes on every invocation.
What actually breaks
The Lambda log shows ImportError: No module named 'psycopg._psycopg'. The engineer ran pip install -t package/ -r requirements.txt on their macOS laptop and zipped the result. The zip contains macOS arm64 .so files; Lambda runs on Amazon Linux x86_64 and cannot load them.
What you'd do on Monday
Build the deployment artifact for the Lambda platform, not the developer's laptop:
ShellBuild a Lambda-compatible deployment package with uv pip install
# Export locked requirements (no hashes needed for --target install)
uv export--frozen --no-hashes > requirements-lambda.txt
# Install wheels for the Lambda platform (Amazon Linux 2023 = manylinux_2_28+)
uv pip install \
--python-platform x86_64-manylinux_2_28 \
--python-version 3.12 \
--only-binary :all: \
--target ./package \
-r requirements-lambda.txt
zip -r function.zip package/ jobs/
uv.lock is a universal lockfile and is safe to generate and commit from any machine. The fix is the artifact build step, not the lockfile.
Scenario B
Imagine you work at Groww
8 Lambda functions, dev on macOS, prod on Amazon Linux
The numbers
Illustrative: 8 Lambda functions, each with ~20 Python dependencies, 4 of which are compiled extensions (psycopg-binary, lxml, cryptography, numpy). Without a CI-enforced artifact build step: 4 compiled deps × 8 functions = 32 macOS .so files zip-deployed to Lambda. 8 functions crash at cold start → 8 ImportError alerts at 9 AM on a trading day. Time to diagnose: 45 min (logs, deployment comparison, revert). Trade cost: Illustrative: 45 min × average Groww order volume → non-trivial.
What the Scenario-A answer does here
The naive approach: each developer runs pip install -t package/ -r requirements.txt on their laptop and zips the result. uv sync --locked in CI is correct for the test step; the failure is the separate artifact-packaging step that is not platform-aware.
What flips at this scale
At this scale you add a dedicated artifact-build job in CI that runs on ubuntu-latest (or inside python:3.12-slim) and uses uv pip install --python-platform x86_64-manylinux_2_28 --only-binary :all: --target ./package. Developers never build the Lambda ZIP locally — only CI does. The lockfile can be committed from any machine; the ZIP cannot.
The trade-off you'd defend
I would defend CI-only artifact packaging over trusting developer laptops, trading the inconvenience of "I cannot test the exact Lambda ZIP locally" for the guarantee that every deployed ZIP contains Linux binaries. For a trading platform where 9 AM crashes are measured in real money, that trade is obvious.
Illustrative numbers chosen to show the scale consequence; not Groww-published deployment metrics.
Gotchas
Clean install succeeds but the job crashes with a missing shared library at runtime
Symptom
pip install exits 0, but import psycopg raises ImportError: libpq.so.5: cannot open shared object file.
Cause
psycopg (the pure-Python distribution) was installed instead of psycopg[binary]. The pure package has no compiled extension but requires the system libpq to be present at runtime; a clean Lambda environment has no libpq.
Fix
Depend on psycopg[binary] in pyproject.toml, which installs a self-contained compiled wheel with no system-library dependency. Alternatively, install libpq in the Lambda layer or Docker image.
uv sync --locked fails in CI but lockfile was just committed
Symptom
CI reports error: lockfile is out of date even though the lockfile was committed 2 minutes ago.
Cause
pyproject.toml was edited — a dependency version or requires-python was bumped — without re-running uv lock to update the lockfile. The committed lockfile no longer matches the current pyproject.toml.
Fix
Run uv lock after every change to pyproject.toml, then commit both files together. Add a CI lint step: uv lock --check exits non-zero if the lockfile is stale.
The smoke test passes in CI but fails in the Lambda ZIP
Symptom
pytest exits 0 in CI; the Lambda function crashes with ModuleNotFoundError.
Cause
The smoke test ran against the CI venv, which is a full Python environment. The Lambda ZIP was built with a different install step that missed a transitive dependency.
Fix
Run the smoke test against the deployment artifact itself: unzip the Lambda ZIP into a temp directory and run python -c "from jobs import handler" pointing at that directory.
Tricky questions
Does pip install -r requirements.txt on a fresh machine guarantee reproducibility if the file was generated by pip freeze?
Most people say
Yes — pip freeze captures exact versions, so reinstalling from it is identical.
Actually
pip freeze captures exact versions but not hashes. On a different platform or Python micro-version, pip may install a different binary wheel for the same version string. pip-compile --generate-hashes adds hashes but its output is also platform-specific — generated on macOS, it records only macOS-compatible wheels. uv.lock is cross-platform: it records wheels for all platforms and lets the installer pick the correct one per environment. Only uv.lock (or a manually curated multi-platform requirements file) is fully reproducible across platforms.
If you run pip install --upgrade on a locked environment, what happens to the lockfile?
Most people say
The lockfile updates automatically.
Actually
The lockfile does not change. pip install --upgrade modifies site-packages silently, creating a divergence between the lockfile and the actual environment. The next uv sync --locked or poetry install will revert the upgrade by reinstalling from the lockfile.
Scenario question
You're handed a Lambda function that works in one AWS region (ap-south-1) but crashes with ImportError in a second region (us-east-1) where a new team is deploying. Both use the same deployment ZIP. Same Python version. What do you investigate?
Model answer
I'd check whether the two regions use the same Lambda runtime version — Amazon Linux 2 vs Amazon Linux 2023 have different glibc versions. I'd run python -c "import platform; print(platform.platform())" in both Lambdas. If the runtimes differ, the ZIP's compiled extensions only match one of them. Fix: rebuild the ZIP targeting the us-east-1 runtime with uv pip install --python-platform x86_64-manylinux_2_28 --only-binary :all:. The trade-off is a separate CI build step per runtime.
Interview corner
CommonWhat is the difference between a venv and a Docker container for environment reproducibility?
“I'd say a venv isolates Python packages but shares the system libraries and the Python interpreter version with the host. A Docker container also isolates the OS, system libraries and the Python interpreter — so it is a stronger form of isolation. For Lambda deployments where the runtime OS matters, a Docker build is the right tool. For local development and CI, a venv plus a locked requirements file is usually sufficient.”
TrickYou run uv sync --locked in CI and it exits 0. Is the environment reproducible?
“I'd say the venv is reproducible. uv.lock is universal — it records wheels for all platforms, so it is safe to generate from any machine. What uv sync --locked does not verify is the deployment artifact: if the Lambda ZIP was built with pip install -t package/ on a developer's laptop, that ZIP may carry macOS binaries regardless of how clean the venv is.”
Experience checkThe Jenkins build passes but the Lambda fails. How do you diagnose?
“I'd compare the installed packages in each environment: add a step that runs pip list > /tmp/packages.txt and uploads it as a build artifact. Diff the two lists. If they differ, the install steps are not equivalent. Then I'd check the platform tags on the wheels actually installed — pip show --files psycopg lists the wheel file name, which includes the platform tag.”
DesignHow would you ensure every Lambda ZIP contains Linux-compatible binaries across a team?
“I'd move artifact packaging into CI entirely. A dedicated job on ubuntu-latest runs uv pip install --python-platform x86_64-manylinux_2_28 --only-binary :all: --target ./package and produces the ZIP. Developers run uv sync --locked locally; they never build the ZIP. uv.lock is universal and can be committed from any machine. The trade-off is losing local ZIP reproduction, which is acceptable for a Linux-deployed service.”
Memory hook
A clean kitchen test is the only test that tells you whether the recipe actually works. Delete the venv, reinstall from the lockfile, build the artifact on the production platform, run the smoke test — that is what "reproducible" means.
If you remember only one thing:
Build the deployment artifact on the production platform — a zip packaged on macOS carries macOS binaries that crash on Linux.
Self-check
Why does a pip install -t package/ run on macOS produce a Lambda ZIP that crashes on Linux?
Because pip install selects wheels for the current platform. macOS installs arm64 .so files; Lambda runs on Linux x86_64 and cannot load them. uv.lock is universal — the lockfile is not the problem; the artifact packaging step must be platform-aware.
What is the correct way to build a Lambda deployment package that includes compiled extensions?
Use uv pip install --python-platform x86_64-manylinux_2_28 --python-version 3.12 --only-binary :all: --target ./package -r requirements-lambda.txt in a CI job running on Linux, or inside a Linux Docker container.
What is the failure mode of --only-binary :all: when no binary wheel exists?
pip install fails immediately with ERROR: Could not find a version ... that satisfies the requirement. This is the desired behaviour — fail fast rather than silently building a broken source wheel.
What causes error: lockfile is out of date in uv sync --locked?
pyproject.toml was edited — a dependency version or requires-python was changed — without re-running uv lock to update the lockfile. Fix: run uv lock and commit both files.
Prove it
Create a project with uv init, add psycopg-binary as a dependency, and run uv lock. Open uv.lock and search for psycopg. You will find wheel entries for multiple platforms — macOS arm64, Linux x86_64, and more — all in the same lockfile. This confirms that uv.lock is universal. Then run uv export --frozen --no-hashes > requirements.txt and inspect that file: it lists only the packages, not the wheels. Paste a few of the wheel lines from uv.lock as the artefact confirming the universal format.