Shelf · Python · Chapter 10

Chapter 10: Packaging, envs & project layout

Your ETL script runs on your laptop, passes in CI, and silently loads a different version of a dependency in production. This chapter shows you how Python resolves packages, how environments isolate them, and how to structure a project so any colleague — or a fresh CI runner — can reproduce exactly what you tested.

Difficulty
Priority
py-10-01Card 1 of 6

Python package reproducibility: pyproject, environments and resolution

MediumP2 · Very likelyKnow cold13 min
pyproject-tomlpinningdependency-resolutionreproducibilitylockfiles
Needs: nothing. Start here.

Why you care

Every Python job that runs in production depends on a dependency resolution decision made at install time. If that decision is not frozen, it can change silently between your QA build and your prod deploy, between today and next month, between your machine and the CI runner. In ETL pipelines the failure is rarely a crash — it is a silent schema change in a data-parsing library that drops one column, or an off-by-one in a date parser that shifts your aggregations by a day. Interviewers at product companies ask about reproducible builds because it predicts whether you understand that "it works on my machine" is a build-process failure, not a coincidence.

Explain like I'm 5

A recipe card says "200g butter, 3 eggs, flour". Anyone who follows it ends up with a slightly different cake depending on which brand of butter they grab. A lockfile is the version of the recipe that says "Amul unsalted 82% fat, Nandini size-L eggs, Pillsbury all-purpose". Every cook who follows the lockfile gets the same cake.

Core concept

Python package installs are controlled by three layers:

  • pyproject.toml declares what your code requires: "requests>=2.31.0". This is the recipe card — it names an acceptable range, not a single version.
  • A lockfile (uv.lock, poetry.lock, or a hashed requirements.txt) pins every direct and transitive dependency to an exact version and, optionally, a SHA-256 hash. This is the shopping receipt.
  • A virtual environment is where those pinned packages are installed in isolation from the system Python and every other project.

Reproducibility requires all three. A pyproject without a lockfile leaves resolution to the tool on every fresh install. A lockfile without a venv risks conflicts with the system or other projects. A venv without a lockfile is isolated but not deterministic across machines.

The decision rule: write ranges in pyproject.toml (so your package is composable with others), commit the lockfile (so every install is identical), and never commit the venv directory itself. The lockfile is the contract; the venv is a generated artefact. Re-generate the venv from the lockfile on every fresh machine.

Supply-chain hygiene (Supply chain: SBOM, lockfiles, provenance and hygiene) builds on this: hash-pinned lockfiles let you detect a tampered package before it reaches production.

Under the hood

CPython's import system does not know about lockfiles — that concept lives entirely in pip, uv and poetry. What CPython does know is sys.path: an ordered list of directories it searches for modules. pip install places packages into a site-packages directory, and the venv mechanism works by prepending the venv's site-packages to sys.path and pointing the python executable at a copy that has the site-packages path baked in. This means: if you activate the wrong venv, or no venv, Python silently finds a different site-packages and your pinned versions are irrelevant. The actual pinning enforcement happens at install time; at runtime, Python just takes whatever is on sys.path first.

Minimal example Parse a pyproject.toml with tomllib

PythonRead project metadata from a pyproject.toml stringruns
import tomllib, textwrap

src = textwrap.dedent("""
    [project]
    name = "etl-job"
    version = "1.0.0"
    requires-python = ">=3.12"
    dependencies = [
        "requests>=2.31.0",
        "psycopg[binary]>=3.1",
    ]
""").strip().encode()

config = tomllib.loads(src.decode())
for dep in config["project"]["dependencies"]:
    print(dep)
Output
requests>=2.31.0
psycopg[binary]>=3.1

tomllib (stdlib since Python 3.11) parses TOML. The >= in the declared deps is a range, not a pin — this is intentional in pyproject.toml so your library composes with others. The lockfile converts these ranges to exact versions.

In production Detect unpinned dependencies

This script reads a requirements.txt-style file and flags anything without an exact pin — useful as a pre-deploy gate in Jenkins or GitHub Actions.

PythonFlag unpinned deps in requirements.txt contentruns
import re

requirements = """\
requests==2.31.0
boto3>=1.28.0
pandas
psycopg[binary]==3.1.9
"""

def check_pinned(reqs: str) -> list[tuple[str, str]]:
    results = []
    for line in reqs.strip().splitlines():
        line = line.strip()
        if not line or line.startswith("#"):
            continue
        name = re.split(r"[>=<!;\[]", line)[0].strip()
        pinned = "==" in line
        results.append((name, "pinned" if pinned else "UNPINNED"))
    return results

for name, status in check_pinned(requirements):
    print(f"{name:<30} {status}")
Output
requests                       pinned
boto3                          UNPINNED
pandas                         UNPINNED
psycopg                        pinned

boto3>=1.28.0 and bare pandas are UNPINNED. A lockfile generated by uv lock or pip-compile would replace these with exact pinned versions and file hashes.

Scenario A

In your world

The situation

Your Jenkins job pulls settlements.csv from S3 and loads it into Postgres using psycopg. The requirements.txt in the repo says psycopg[binary]>=3.1. Jenkins ran this fine for six months.

What actually breaks

A new psycopg 3.2 minor release changed the default row_factory return type. Your SELECT queries now return Row objects instead of tuples, so row[0] still works but dict(row) raises TypeError: cannot convert Row to dict. The Jenkins build stays green because the INSERT step still runs — only the downstream reconciliation report silently produces wrong column names. Finance notices a week later.

What you'd do on Monday

Pin the exact version in a lockfile and add a hash so the CI runner cannot silently pick up a newer release:

ShellAdd uv to a project and generate a lockfile
uv init                   # creates pyproject.toml if absent
uv add "psycopg[binary]"  # adds to [project.dependencies]
uv lock                   # writes uv.lock with exact versions and hashes
uv sync                   # installs from the lock, creates .venv/

Commit pyproject.toml and uv.lock. Add .venv/ to .gitignore. On CI, run uv sync --locked (fails if uv.lock is stale relative to pyproject.toml). --frozen skips the freshness check; use it only when the lock is known-good, e.g. inside a Docker layer build.

Scenario B

Imagine you work at Atlassian

200 Python services, 150 transitive deps each

The numbers

Illustrative: 200 microservices × 150 transitive deps = 30,000 package slots. PyPI releases roughly 200–400 new package versions per day across popular packages. If each service's 150 deps include 1 package that releases a new minor version per week on average, that is 200 services × 1 unexpected upgrade/week = 200 silent dependency drifts per week across the fleet. Any one of those can break a downstream consumer quietly.

What the Scenario-A answer does here

The Scenario A fix (pinning direct deps) still helps but does not close the gap. Direct-dep pins constrain only the first level; a direct dep that itself declares >= ranges lets pip pick a newer transitive dep on every fresh install. Only locking the full transitive graph eliminates all 30,000 drift vectors.

What flips at this scale

At this scale you adopt a lockfile-first policy: every service ships a uv.lock or poetry.lock, and the CI pipeline runs uv sync --locked or poetry install --no-root — both fail if the lockfile is out of sync with pyproject.toml, making "lockfile out of date" a build failure rather than a runtime surprise.

The trade-off you'd defend

I would defend committing the lockfile over leaving resolution to CI, trading a larger git diff on dependency bumps for the guarantee that every environment is byte-for-byte identical. The only cost is a deliberate uv lock --upgrade-package boto3 when you actually want to bump a dep.

Illustrative numbers chosen to make the arithmetic visible; not Atlassian-published figures.

Gotchas

Pin disappears after pip install -r requirements.txt on a clean machine

Symptom

The exact version you tested is gone; a newer version is installed without warning.

Cause

requirements.txt was written with ranges (>=) not exact pins (==). pip resolves the latest compatible version on every fresh install.

Fix

Regenerate the file with pip freeze > requirements.txt (captures the currently installed exact versions) or switch to uv lock / pip-compile which produce a separate lock file with hashes.

uv sync installs a different version than pip install

Symptom

uv installs version X, pip installs version Y from the same requirements.txt.

Cause

uv's resolver uses a different algorithm (PubGrub) than pip's backtracking resolver. They can reach different solutions for the same set of constraints.

Fix

Commit the lockfile and always install from it (uv sync --locked). Never mix pip and uv installs against the same environment.

Hash mismatch causes install to exit 1 with a mismatch error

Symptom

pip install --require-hashes -r requirements.txt exits 1 with THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE.

Cause

The hash in the file was recorded for a different artifact than pip fetched. This happens with pip-compile --generate-hashes output: a requirements file generated on macOS records macOS wheel hashes; on Linux, pip downloads the Linux wheel and the hash does not match. It also happens if a package was re-uploaded to PyPI with a different tarball under the same version string.

Fix

Use uv.lock (universal — records hashes for all platforms) instead of pip-compile output. If pip-compile is required, generate the requirements file on the same platform as deployment, or use uv export --frozen --no-hashes and pip install --only-binary :all:.

Tricky questions

Is pyproject.toml the lockfile?
Most people say

Yes — if you pin exact versions in pyproject.toml like "requests==2.31.0" it is reproducible.

Actually

pyproject.toml with pinned direct deps is still not a lockfile. It does not record transitive deps, and it does not record file hashes. A real lockfile (uv.lock, poetry.lock) pins the complete resolved graph with hashes. Pinning direct deps in pyproject.toml is also bad practice for a library, because it prevents users from combining it with other packages that need a different version.

Does committing requirements.txt from pip freeze give reproducibility?
Most people say

Yes, pip freeze pins everything.

Actually

pip freeze pins exact versions but not hashes. Without hashes, a malicious or corrupted PyPI mirror can serve a different tarball under the same version number. Use pip-compile --generate-hashes or uv lock to include SHA-256 hashes.

Scenario question

You're handed a Jenkins pipeline where the same requirements.txt produces different boto3 versions on Tuesday's build and Thursday's build. The file says boto3>=1.28.0. The team claims "it always worked before."

Model answer

I'd first confirm the root cause by running pip install --dry-run -r requirements.txt on both runner images and comparing the resolved boto3 version. If they differ, the AWS SDK released a minor version between Tuesday and Thursday and pip picked it up. My fix is to add uv lock to the CI pipeline, commit uv.lock, and change the CI install step to uv sync --locked. For the immediate regression I'd pin boto3==<last-known-good> in requirements.txt as a stop-gap while the lockfile infra is wired up. The trade-off is that we now need a deliberate uv lock --upgrade-package boto3 to pick up security patches instead of getting them automatically — I would defend that as the right default for a production ETL job.

Interview corner

CommonWhat is the difference between pyproject.toml and a lockfile?
“I'd say pyproject.toml declares your intent — what ranges you accept — and the lockfile records the decision: exactly which version was picked for every direct and transitive dep, plus hashes. You commit both, but only pyproject.toml is meant for other people's dependency solvers to consume.”
TrickIs it safe to pin all direct deps in pyproject.toml to exact versions?
“I'd say it depends on whether the project is a library or an application. For an application or a deployed service, exact pins in a lockfile are fine. For a library published to PyPI, pinning your own dependencies to exact versions in pyproject.toml prevents users from combining your library with others — they will hit version conflicts. Libraries should use ranges in pyproject.toml and only pin in a lockfile for their own CI.”
Experience checkYour ETL job loaded different row counts on two consecutive runs from the same S3 file. Environment could be the cause. How do you rule it in or out?
“I'd compare the output of pip list or uv pip list on both runs by adding it to the build log. If any data-parsing package version differs, that is my first suspect. Then I'd check the changelog for that package for any parsing or schema changes between the two versions.”
DesignHow would you enforce that every service in a 200-service Python monorepo uses reproducible builds in CI?
“I'd add a CI step that runs uv sync --locked and fails if the lockfile is out of sync with pyproject.toml. I'd also add a lint step that scans each service's pyproject.toml for unpinned direct deps and warns — not fails — so teams can migrate gradually. The trade-off is that bumping a dep now requires a deliberate uv lock commit, which some engineers see as friction. I think that friction is the point.”

Memory hook

A pyproject.toml is the recipe card; the lockfile is the shopping receipt. Commit both. The venv is the meal — throw it away and re-cook from the receipt any time.

If you remember only one thing:

Without a lockfile, your transitive dependencies quietly update themselves between every install.

Self-check

  1. What does uv sync --locked do differently from uv sync?

    --locked fails if uv.lock is out of sync with pyproject.toml — plain uv sync would silently re-resolve and overwrite the lockfile. --frozen skips the freshness check entirely; use it only when the lockfile is known-current, such as inside a Docker layer build.

  2. Why is pinning all deps in pyproject.toml insufficient for full reproducibility?

    It pins direct dependencies but not transitive ones, and it records no file hashes, so a different tarball could be silently served under the same version number.

  3. Where does the venv directory belong in git?

    It does not belong in git. The venv is a generated artefact derived from the lockfile. Commit the lockfile; list .venv/ in .gitignore.

  4. What mechanism does Python use at runtime to find installed packages?

    sys.path — an ordered list of directories searched for module names. Activating a venv prepends the venv's site-packages directory to sys.path.

  5. What does tomllib do and since which Python version is it in the stdlib?

    tomllib parses TOML files (read-only). It is in the stdlib since Python 3.11. For writing TOML you need a third-party library.

Prove it

Create a fresh directory, write a pyproject.toml that declares one direct dep with a range (requests>=2.31.0), run uv lock (or pip-compile --generate-hashes) to produce a lockfile, then delete the lockfile and change the range to a different constraint. Run the install command again and confirm that the two install runs resolve different versions. Capture the output of uv pip list or pip list from both runs side by side and paste it as the artefact.

py-10-02Card 2 of 6

Choose venv, uv or poetry for a project

EasyP2 · Very likelyKnow cold12 min
venvuvpoetryenvironment-isolationdependency-management

Why you care

Python has three distinct problems that look similar: isolating packages from the system, managing which packages a project needs, and locking those packages to reproducible versions. venv, uv and poetry each solve different subsets. Using the wrong tool for the job leads to polluted system interpreters, lockfile drift, or team disagreements about who owns the pyproject.toml. Interviewers ask this because "I use pip" is the answer of someone who has never shared a Python project with a colleague and had it break on their machine.

Explain like I'm 5

A venv is a parking-bay reservation: it reserves a dedicated space for this project's packages so they do not collide with the system Python or other projects. uv is a valet: it reserves the bay, fetches the packages at high speed, and writes a receipt (lockfile) so the next valet parks the exact same cars. Poetry is a concierge: it reserves the bay, fetches the packages, writes the receipt, and also manages the whole project — versions, publication, build.

Core concept

Each tool occupies a different position on the spectrum from isolation only to full project management:

  • venv creates an isolated Python environment (a directory with its own python and site-packages). It does nothing else — you still use pip to install packages. Use it when you need quick, zero-dependency isolation on a machine that already has pip.
  • uv (0.4+) creates envs, installs packages at 10-100× pip speed (Rust-based resolver + parallel downloads), and manages a uv.lock lockfile. It reads pyproject.toml as the source of truth. Use it for new projects where speed and lockfile reproducibility matter and there is no existing poetry setup.
  • poetry creates envs, installs packages, manages a poetry.lock lockfile, and handles version bumping and PyPI publishing. It pre-dates uv's lockfile support and many teams already have poetry.lock in their repos. Use it on projects already using poetry, or where publishing to PyPI from the same tool matters.

Decision rule: for a new project, use uv — it covers environments, lockfiles, and publishing via uv publish. For a project already on poetry, stay on poetry unless you have a compelling reason to migrate. Use bare venv only for throwaway scripts or when you cannot install uv or poetry on the machine.

Whichever tool you choose, the workflow is the same: the lockfile is committed, the venv is not, and CI installs from the lockfile with a flag that fails on drift.

Minimal example Read declared dependency groups from pyproject.toml

PythonParse runtime and dev dependency groupsruns
import tomllib, textwrap

content = textwrap.dedent("""
    [project]
    name = "data-service"
    requires-python = ">=3.12"
    dependencies = ["requests==2.31.0", "pydantic==2.5.0"]

    [project.optional-dependencies]
    dev = ["pytest==7.4.3", "ruff==0.1.6"]
    ci  = ["pytest==7.4.3", "ruff==0.1.6", "coverage==7.3.2"]
""").strip().encode()

cfg = tomllib.loads(content.decode())
print("runtime:", cfg["project"]["dependencies"])
print("dev:    ", cfg["project"]["optional-dependencies"]["dev"])
Output
runtime: ['requests==2.31.0', 'pydantic==2.5.0']
dev:     ['pytest==7.4.3', 'ruff==0.1.6']

uv sync installs the [project.dependencies] group. uv sync --extra dev adds the dev group. Poetry calls these the same thing but uses [tool.poetry.dependencies] and [tool.poetry.dev-dependencies] in older versions.

In production Audit a realistic settlements-etl pyproject.toml for unpinned deps

PythonCheck all dependency groups for unpinned entriesruns
import tomllib, textwrap, re

content = textwrap.dedent("""
    [project]
    name = "settlements-etl"
    version = "0.2.1"
    requires-python = ">=3.12"
    dependencies = [
        "psycopg==3.1.9",
        "boto3==1.28.62",
        "pydantic==2.5.0",
    ]

    [project.optional-dependencies]
    dev = ["pytest==7.4.3", "ruff==0.1.6", "mypy==1.7.0"]
    ci  = ["pytest==7.4.3", "ruff==0.1.6", "mypy==1.7.0", "coverage==7.3.2"]
""").strip().encode()

cfg = tomllib.loads(content.decode())
runtime = cfg["project"]["dependencies"]
dev = cfg["project"]["optional-dependencies"]["dev"]

unpinned = [d for d in runtime + dev if "==" not in d]
print("unpinned deps:", unpinned if unpinned else "none - all pinned")
print("runtime count:", len(runtime))
print("dev count:    ", len(dev))
Output
unpinned deps: none - all pinned
runtime count: 3
dev count:     3

All dependencies are exact-pinned here. In practice your pyproject.toml for a library would use ranges; the lockfile holds the exact pins. This audit script is useful as a CI gate on services (not libraries).

Scenario A

In your world

The situation

You clone a new colleague's ETL repo, run pip install -r requirements.txt, and start the job. It crashes immediately.

What actually breaks

The colleague used poetry and never committed a requirements.txt. There is a poetry.lock in the repo but you do not have poetry installed. You try pip install . — that reads pyproject.toml but ignores poetry.lock, so it resolves fresh and picks a newer psycopg that has a different binary wheel. The job crashes with a ImportError: cannot import name 'AsyncConnection' that was not in the old version.

What you'd do on Monday

Install the tool that the lockfile belongs to, then install from it:

ShellInstall from a poetry lockfile (requires poetry >=1.8)
pip install poetry              # or: pipx install poetry
poetry install --no-root        # reads poetry.lock, installs exact versions
poetry run python jobs/load.py  # runs inside the venv poetry created

For uv-based projects the equivalent is uv sync (reads uv.lock). Never use bare pip install on a project that has a lockfile — you skip the lockfile and lose reproducibility.

Scenario B

Imagine you work at Postman

80 Python test-grid workers, fresh envs per run

The numbers

Illustrative: Postman's integration-test grid spins up 80 isolated Python workers per test suite run. Each worker creates a fresh venv with 40 dependencies. Without a pre-built wheel cache: each package takes ~2s to download and install = 80 × 40 × 2 = 6,400s of total install CPU per run. Workers run 10 in parallel, so wall-clock install time is 6,400 / 10 = 640s ≈ 11 minutes. With uv + pre-built wheels: ~0.15s per package = 80 × 40 × 0.15 / 10 = 48s ≈ 1 minute. That 10-minute difference per test run, at 30 runs/day, is 5 hours of CI feedback delay eliminated.

What the Scenario-A answer does here

The Scenario A fix (pick the right tool for your project) still applies. But at this scale, even the right tool matters less than the wheel cache strategy. uv's shared wheel cache across workers is where most of the gain comes from — without it, even uv downloads the same wheel 80 times.

What flips at this scale

At this scale you decouple "which tool manages the lockfile" from "how the CI installs packages". The CI worker image pre-bakes the wheel cache from the lockfile at image build time, so uv sync at test time is a cache hit for every package. That requires a uv export > requirements.txt && uv pip download step in the Docker build, not just a different install tool.

The trade-off you'd defend

I would defend baking the wheel cache into the CI image over fetching at runtime, trading a slightly larger image for guaranteed sub-10s install times regardless of network conditions. The only risk is cache staleness on lockfile bumps, which the image rebuild step catches.

Illustrative numbers chosen to show the arithmetic; not Postman-published CI figures.

venv vs uv vs poetry

Featurevenvuvpoetry
Creates an isolated envyesyesyes
Installs packagesno (use pip)yes (uv add, uv sync)yes (poetry add, poetry install)
Generates a lockfilenoyes (uv.lock)yes (poetry.lock)
Manages pyproject.tomlnoyesyes
Install speed vs pip1×10–100×~0.5–1×
Publishes to PyPInoyes (uv publish)yes
Stable lockfile formatn/asince uv 0.4yes
When to usethrowaway scripts, no-extra-tools constraintnew projects; can publish via uv publishexisting poetry repos

Gotchas

Activating the wrong venv silently uses the wrong packages

Symptom

You run the script; it imports successfully but produces wrong results because it found the system-installed version of a library, not the venv version.

Cause

You opened a new terminal tab without running source .venv/bin/activate (or .venv\Scripts\activate on Windows). The python command resolves to the system interpreter.

Fix

Always check which python (Mac/Linux) or where python (Windows) before running. Use uv run or poetry run to ensure the venv is always selected without manual activation.

pip install inside a poetry-managed venv updates poetry.lock silently

Symptom

Running pip install some-package inside the venv "works" but poetry.lock still shows the old version. The next poetry install reverts the change.

Cause

pip and poetry manage separate metadata. pip writes to site-packages directly; poetry does not see that change.

Fix

Use poetry add some-package to update both pyproject.toml and poetry.lock together.

uv.lock checked in but ignored on CI

Symptom

CI runs pip install -r requirements.txt and resolves different versions than local.

Cause

Someone added a requirements.txt export for legacy CI but the CI step never got updated to uv sync --locked.

Fix

Replace the CI install step with uv sync --locked and delete the requirements.txt export from CI.

Tricky questions

Does python -m venv .venv also install packages?
Most people say

Yes — that is why the venv is useful.

Actually

No. venv creates the isolated directory and copies the interpreter. On Python 3.12+ it installs only pip — setuptools is no longer bundled by ensurepip and wheel was never installed by venv. The --upgrade-deps flag upgrades pip (plus setuptools on older Pythons). You still need pip install or uv add to put application packages inside it.

Is poetry's venv the same as a venv created by python -m venv?
Most people say

No — poetry uses a special internal format.

Actually

It is the same standard venv layout (a pyvenv.cfg file and Lib/site-packages). The difference is that poetry manages it automatically and stores it in a cache directory rather than inside the project. You can tell poetry to store it in .venv/ with poetry config virtualenvs.in-project true.

Scenario question

You're handed a team of five engineers who all work on the same Python ETL service. Two use macOS, two use Ubuntu, one uses Windows WSL. They keep seeing "works on my machine" failures. You have an hour to fix the environment discipline. What do you do?

Model answer

I'd switch the project to uv in that hour: run uv init if there is no pyproject.toml, uv add all the current direct deps, uv lock to generate uv.lock, and commit both files. I'd update the README to say: clone the repo, run uv sync, done. For CI I'd change the install step to uv sync --locked. The trade-off is that everyone now needs uv installed (pip install uv or curl -LsSf https://astral.sh/uv/install.sh | sh), which is a one-time cost. I'd resist switching to Docker as the "fix" unless the team already uses it, because Docker for local dev is a larger adoption barrier than uv.

Interview corner

CommonWhat does activating a venv do?
“I'd say it prepends the venv's bin/ directory to PATH, so python and pip resolve to the venv's copies instead of the system ones. It also sets VIRTUAL_ENV. Deactivating reverses that. Without activation, you can still use the venv explicitly with .venv/bin/python.”
TrickCan you have two venvs active at the same time?
“I'd say not cleanly. Activating a second one overwrites PATH and VIRTUAL_ENV with the new venv's values. The first venv is effectively deactivated. If you need isolation within a single shell session, use uv run or poetry run — they activate and deactivate per command.”
Experience checkYou need to install a package that is not on PyPI, only on a private S3-hosted index. Which tool handles that best?
“I'd use uv with the --extra-index-url flag or by setting [[tool.uv.index]] in pyproject.toml pointing at the S3 bucket (which has an index-style layout). The same lockfile approach applies — uv will lock the private package with its hash. Poetry also supports extra indices. pip-tools supports them too but requires more manual wiring.”
DesignYour 80-worker test grid creates a fresh venv per worker every run and install time dominates the feedback loop. How do you fix it?
“I'd bake the wheel cache into the CI image at image-build time: run uv export --frozen > requirements-ci.txt then uv pip download -r requirements-ci.txt --dest ./wheel-cache in the Docker build. At test time, uv sync --locked finds every wheel already on disk — a cache hit for all 40 deps across all 80 workers. The trade-off is a larger Docker image that must be rebuilt whenever the lockfile changes, but install time drops from 11 minutes to under a minute and the cache is always consistent with the lockfile.”

Memory hook

venv parks the car, uv parks it fast and writes the receipt, poetry parks it and also books the service appointment. Know which one the project is already using before you touch it.

If you remember only one thing:

Use uv sync --locked in CI — it fails if uv.lock is out of sync with pyproject.toml, closing the "works on my machine" gap.

Self-check

  1. What does uv sync --locked do that plain uv sync does not?

    --locked fails if uv.lock is out of sync with pyproject.toml. Plain uv sync would silently re-resolve and overwrite the lockfile. --frozen skips the freshness check and installs from the lockfile as-is.

  2. If you run pip install requests inside a poetry-managed venv, does poetry.lock update?

    No. pip and poetry maintain separate metadata. poetry.lock only updates when you use poetry add or poetry update.

  3. What is the minimum set of files to commit for a uv-managed project to be reproducible?

    pyproject.toml and uv.lock. The .venv/ directory must be in .gitignore.

  4. What does python -m venv .venv actually install in the new environment?

    On Python 3.12+, only pip. setuptools is no longer bundled by ensurepip and wheel was never installed by venv. Use --without-pip to skip pip too.

Prove it

On a machine with uv installed: create a new directory, run uv init, add one package (uv add httpx), lock it (uv lock), then delete the .venv/ and run uv sync --locked. Confirm the same version of httpx is installed by running uv pip show httpx. Paste the output of uv pip show httpx and the first 5 lines of uv.lock as the artefact.

py-10-03Card 3 of 6

Distinguish modules and packages

EasyP1 · Must knowKnow cold13 min
modulespackagesinit-pydunder-allimport-surfaceproject-layout
Needs: nothing. Start here.

Why you care

"Module not found" and "AttributeError: module has no attribute X" are the two most common import-related errors in Python. Both trace back to the same misunderstanding: the difference between a module (a single .py file) and a package (a directory that Python treats as a namespace). If you do not know this distinction you cannot read an import error message correctly, cannot design a public API for your own code, and cannot explain your project layout to a colleague. This is a 100 % interview question because it is a prerequisite for every other packaging concept.

Explain like I'm 5

A module is a book — one file, everything inside it is in one place. A package is a library shelf — a folder that holds many books and a special index card (__init__.py) that says "yes, this shelf is a library". When you import from payments.charge import apply, Python walks: find the payments shelf, open it, find the charge book, get the apply function.

Core concept

A module is any .py file. import utils loads utils.py from sys.path. Every name at the top level of utils.py is accessible as utils.some_name.

A regular package is a directory that contains an __init__.py file. When you import payments, Python runs payments/__init__.py and the result is the payments module object. Submodules inside the directory are accessible as payments.charge, payments.refund, etc.

A namespace package (PEP 420, Python 3.3+) is a directory without __init__.py. Python still treats it as a package but its __file__ is None and it has no __init__ code. Namespace packages are used for splitting a single package across multiple installed directories (rare in application code; common in framework plugins).

__all__ is a list you define in __init__.py (or any module) to declare the public API: the names that from pkg import * will export. Names not in __all__ are still importable by explicit path; __all__ only controls star imports. Even without __all__, names starting with _ are excluded from star imports by convention.

The decision rule: if you have one utility file, use a module. If you have related modules that form a coherent unit with a public API, use a package with __init__.py and define __all__. Do not create packages just to group files — a flat module is easier to import.

Minimal example Package public surface via __all__

Python__all__ controls which names star-import exposesruns
import sys, tempfile
from pathlib import Path

td = Path(tempfile.mkdtemp())
pkg = td / "payments"
pkg.mkdir()
(pkg / "__init__.py").write_text(
    "__all__ = ['charge', 'refund']\n"
    "charge = lambda amt: f'charged {amt}'\n"
    "refund = lambda amt: f'refunded {amt}'\n"
    "_internal_state = {}\n"
)

sys.path.insert(0, str(td))
import payments
ns = {}
exec("from payments import *", ns)
public = sorted(k for k in ns if not k.startswith("_"))
print("public names after 'from payments import *':", public)
print("_internal_state accessible?", hasattr(payments, "_internal_state"))
sys.path.pop(0)
import shutil; shutil.rmtree(td)
Output
public names after 'from payments import *': ['charge', 'refund']
_internal_state accessible? True

__all__ controls star imports only. _internal_state is hidden from from payments import * but still reachable as payments._internal_state — Python does not enforce true privacy; __all__ is a declaration of intent.

In production Realistic ETL package with relative imports

PythonBuild and use an etl package with three submodulesruns
import sys, tempfile
from pathlib import Path

td = Path(tempfile.mkdtemp())
pkg = td / "etl"
pkg.mkdir()
(pkg / "__init__.py").write_text('__all__ = ["Pipeline"]\n')
(pkg / "extract.py").write_text(
    "def read_csv(path: str) -> list[dict]:\n"
    "    # real version opens S3; sqlite3 stands in here\n"
    "    return [{\"id\": 1, \"amount\": 250}, {\"id\": 2, \"amount\": 90}]\n"
)
(pkg / "transform.py").write_text(
    "def enrich(rows: list[dict]) -> list[dict]:\n"
    "    return [{**r, \"tax\": round(r[\"amount\"] * 0.18, 2)} for r in rows]\n"
)
(pkg / "load.py").write_text(
    "def write_summary(rows: list[dict]) -> dict:\n"
    "    return {\"count\": len(rows), \"total\": sum(r[\"amount\"] for r in rows)}\n"
)
(pkg / "pipeline.py").write_text(
    "from .extract import read_csv\n"
    "from .transform import enrich\n"
    "from .load import write_summary\n"
    "\n"
    "class Pipeline:\n"
    "    def run(self, path: str) -> dict:\n"
    "        rows = read_csv(path)\n"
    "        rows = enrich(rows)\n"
    "        return write_summary(rows)\n"
)

sys.path.insert(0, str(td))
from etl.pipeline import Pipeline

result = Pipeline().run("orders.csv")
print("count:", result["count"])
print("total:", result["total"])
sys.path.pop(0)
import shutil; shutil.rmtree(td)
Output
count: 2
total: 340

Relative imports (.extract, .transform) are preferred inside a package: they work regardless of what name you give the package in sys.path, and they make the dependency direction explicit. Absolute imports (from etl.extract import read_csv) also work but break if you rename the package.

See it happen How Python resolves `from etl.pipeline import Pipeline`

import statementsys.pathetl/__init__.pyetl/pipeline.pysearch for 'etl'found etl/ with __init__.pysearch for 'pipeline' inside 'etl'Pipeline classsecond import: already cachedcached module object
  1. Python walks sys.path left to right, looking for a directory named 'etl' with __init__.py or a file named etl.py.
  2. The directory has __init__.py so it is a regular package. Python executes __init__.py to build the 'etl' module object.
  3. Now Python looks for etl/pipeline.py (or etl/pipeline/__init__.py).
  4. etl/pipeline.py is executed; 'Pipeline' is looked up in its namespace and bound in the caller.
  5. A repeat 'from etl.pipeline import Pipeline' hits sys.modules['etl.pipeline'] immediately — the file is not re-executed.
  6. sys.modules acts as the import cache. Module code runs exactly once per process.
Notice: the module code runs once and is cached; repeated imports return the same object, which is why module-level state is global for the lifetime of the process.
Scenario A

In your world

The situation

Your Jenkins ETL repo grows to ten Python files in the root directory. A colleague adds a file called utils.py. Your script already does from utils import parse_date. It imports fine locally.

What actually breaks

On the CI runner the system Python has a third-party utils package installed globally. The CI runner's sys.path finds the third-party utils first. Your parse_date function does not exist there: ImportError: cannot import name 'parse_date' from 'utils'. Local still works because the repo is on your local sys.path first.

What you'd do on Monday

Convert the root-level files into a proper package. Create a directory with your project name (e.g. etl_jobs/), add __init__.py, move the modules inside, and change all imports to from etl_jobs.utils import parse_date. A named package is far less likely to collide with a third-party name.

ShellConvert loose files to a package
mkdir etl_jobs
mv utils.py extract.py load.py etl_jobs/
touch etl_jobs/__init__.py
# Update imports: 'from utils import X' → 'from etl_jobs.utils import X'
Scenario B

Imagine you work at Freshworks

500 KLOC Python CRM monolith

The numbers

Illustrative: a 500 KLOC codebase split into ~50 modules. If 20 of those modules do from core import * and core/__init__.py exports 60 names without __all__, each module's namespace is polluted with 60 extra names. 20 modules × 60 names = 1,200 implicit bindings. One name conflict (say status in core and status as a local variable) produces a wrong result with no error.

What the Scenario-A answer does here

The Scenario A fix (use a package, avoid name collisions with stdlib) still works. But from core import * across 20 modules means a new name added to core can silently shadow a local variable in any of those modules — "it passed tests" because the test fixture happened to have the same value.

What flips at this scale

At this scale you enforce an explicit import policy: __all__ in every __init__.py limits the exported surface to the 3-5 names that other modules actually need. A linting rule (ruff: F401, F403) fails any from module import * in non-test code. This makes the dependency graph between modules visible and auditable.

The trade-off you'd defend

I would defend explicit __all__ and no star imports over convenience, trading slightly more verbose import lines for a codebase where every module's dependencies are visible in its import block. The one exception is test helpers, where from fixtures import * is a common pattern.

Illustrative numbers; not Freshworks-published codebase metrics.

module vs package vs namespace package

FeatureModule (.py file)Regular package (dir + __init__.py)Namespace package (dir, no __init__.py)
__file__path to the .py filepath to __init__.pyNone
Can contain submodulesnoyesyes
Runs code on importyes (the file body)yes (__init__.py body)no (nothing to run)
Can define __all__yesyesno (no __init__.py)
Use whenone utility filecoherent group with a public APIsplit install across multiple dirs (plugins)

Gotchas

import mypackage imports nothing from submodules

Symptom

After import mypackage, mypackage.utils raises AttributeError: module 'mypackage' has no attribute 'utils'.

Cause

__init__.py is empty (or does not import utils). Python does not auto-import submodules unless __init__.py explicitly does so.

Fix

Either add from . import utils in __init__.py, or use from mypackage.utils import X at the call site — the explicit form is usually better because it does not force all submodules to load on every import of the package.

A file named the same as a stdlib module shadows the stdlib

Symptom

import json crashes with AttributeError: module 'json' has no attribute 'loads', or import csv works but csv.DictReader is missing.

Cause

You have a json.py or csv.py in your working directory, which is on sys.path before the stdlib.

Fix

Never name your files after stdlib modules. Rename the file and restart the interpreter (the old module may be cached in sys.modules).

Relative imports fail when the file is run as a script

Symptom

from .utils import X raises ImportError: attempted relative import with no known parent package.

Cause

Python sets __package__ = None for a script run with python myfile.py. Relative imports require a non-None __package__.

Fix

Run the file as a module with python -m mypackage.myfile, or use __main__.py (Give a package an explicit entry point).

Tricky questions

Does a namespace package (no __init__.py) import differently from a regular package?
Most people say

No — Python detects both the same way and they behave identically.

Actually

They differ in two observable ways. First, myns.__file__ is None for a namespace package (no __init__.py to point to). Second, a namespace package's __path__ is a _NamespacePath, not a plain list, and can span multiple directories. Behaviour that depends on __file__ being set (e.g. Path(__file__).parent) will crash on a namespace package.

Does __all__ prevent import mymodule._private?
Most people say

Yes — __all__ enforces privacy.

Actually

No. __all__ is documentation and controls star imports only. Any name, including those starting with _, is importable by its full path. Python has no access modifiers that enforce privacy at the module level.

Scenario question

You're handed a Flask API where from models import db works in the main app but raises ImportError in a standalone data-migration script run directly with python migrations/add_column.py.

Model answer

I'd check two things first: whether the script is inside a package (does migrations/ have an __init__.py?) and whether models is a relative or absolute import. If it is from models import db (absolute), Python will find models.py only if the project root is on sys.path. When you run a script directly, Python adds the script's own directory to sys.path[0], not the project root. My fix is to run the migration as a module (python -m migrations.add_column from the project root) so Python adds the project root to sys.path, or to add the project root to PYTHONPATH as a CI environment variable. The trade-off of PYTHONPATH is that it is implicit and breaks if someone moves the repo; python -m is explicit and portable.

Interview corner

CommonWhat is the difference between a module and a package?
“I'd say a module is a single .py file. A package is a directory that contains an __init__.py — that file is what marks it as a package rather than just a folder. When Python sees import payments, it looks for either payments.py or payments/__init__.py.”
CommonWhat does __all__ do?
“I'd say it declares the public API of a module: the list of names that from module import * will export. It does not prevent direct access to anything — it is documentation that also controls star imports. If __all__ is absent, star imports get everything that does not start with an underscore.”
TrickIf you name your file json.py, what happens to import json?
“I'd say it imports your file instead of the stdlib, because Python checks the current directory first in sys.path. json.loads will be missing and everything that depends on stdlib json will break. I've seen this happen with datetime.py and logging.py and it is always confusing to debug because the error points at the caller, not the file with the name collision.”
Experience checkYour test suite imports fine but the same import fails in a scheduled Jenkins job. Both use the same Python version. What is the first thing you check?
“I'd check sys.path in both environments — print it at the top of the failing script. The most common cause is that the test suite is run from the project root (which adds the root to sys.path) but the Jenkins job runs from a different working directory. Then I'd check whether the package is installed in the venv or just present on the file system.”

Memory hook

A module is a book; a package is a shelf with an index card. The index card (__init__.py) is what makes Python say "yes, this is a library" instead of "yes, this is a folder."

If you remember only one thing:

If your project directory has no __init__.py, Python might treat it as a namespace package or skip it entirely — always add __init__.py unless you know what a namespace package is.

Self-check

  1. What file makes a directory into a regular Python package?

    __init__.py. Without it, Python either treats the directory as a namespace package (3.3+) or skips it.

  2. Does __all__ prevent someone from doing from mymodule import _private?

    No. __all__ only controls star imports. Any name is still accessible by its full explicit path.

  3. If you run python mypackage/script.py instead of python -m mypackage.script, which directory is added to sys.path[0]?

    The directory of the script itself (mypackage/), not the project root. This breaks absolute imports of sibling modules in the project root.

  4. What is the observable difference between a regular package and a namespace package?

    A namespace package has __file__ == None and __path__ is a _NamespacePath rather than a list. Its __init__.py does not exist so no code runs on import of the top-level name.

  5. What does from . import utils mean inside __init__.py?

    It imports the submodule utils.py from the same package directory and binds it as utils in the package namespace, so callers can do from mypackage import utils.

Prove it

Create a directory myetl/ with an __init__.py that defines __all__ = ['Pipeline'], a pipeline.py with a Pipeline class, and a _helpers.py with a helper function. Run a Python snippet that does from myetl import * and prints dir(), then imports myetl._helpers explicitly and calls the helper. Paste the printed dir() output and the helper return value as the artefact, confirming that _helpers is absent from the star import but reachable by explicit path.

py-10-04Card 4 of 6

Untangle imports and circular dependencies

MediumP2 · Very likelyKnow cold13 min
circular-importsimporterrorpartially-initialized-moduledependency-direction

Why you care

Circular imports are the most silent class of architectural mistake in a Python project. The code looks correct, the tests pass individually, and then one day a new function is added to a module that triggers the cycle to execute in a different order, and you get ImportError: cannot import name 'X' from partially initialized module 'Y'. This is an un-Googleable error message for engineers who do not know what "partially initialized" means. Interviewers ask about it because fixing a circular import usually requires understanding the entire dependency graph, not just the two files involved.

Explain like I'm 5

Two kitchen sous-chefs: Riya handles sauces, Arjun handles mains. Riya's mise-en-place requires Arjun's stock reduction (which is not ready yet). Arjun's mise-en-place requires Riya's base sauce (which is also not ready yet). They are both waiting for each other before they can start. Python's import system hits exactly this deadlock when models.py imports from utils.py and utils.py imports from models.py.

Core concept

When Python imports a module, it follows these steps:

  1. Check sys.modules for a cached copy. If found, return it immediately.
  2. Create a new, empty module object and put it in sys.modules under the module's name.
  3. Execute the module's source code. The module object is now being filled in.
  4. Return the completed module object.

A circular import happens when step 3 of module A triggers an import of module B, whose step 3 triggers an import of module A — but A is still in step 3 (partially initialized). Python finds A in sys.modules (it was added in step 2) and returns the partially initialized object. Any name that A's step 3 has not yet defined is missing from that object.

The result is ImportError: cannot import name 'X' from partially initialized module 'A' (most likely due to a circular import).

The fix follows one of three patterns, in order of preference:

  1. Move shared code to a third module that neither A nor B imports from (most common).
  2. Defer the import: put the import statement inside the function that needs it, not at the top level.
  3. Use import module instead of from module import name: the module object reference is valid even when partially initialized; a name that does not yet exist is looked up at call time, not at import time.

The decision rule: if you reach for option 2 or 3, the real fix is option 1 — the cycle is a symptom of wrong layering. Draw the dependency arrow direction and make it acyclic.

Under the hood

Python's module cache (sys.modules) is the mechanism that makes circular imports possible rather than infinite-recursion fatal. When A starts importing B and B tries to import A, Python finds sys.modules['a'] — an incomplete module object — and returns it without re-entering A's source code. The partial object has whatever names A's __init__ or top-level code defined before the line that triggered the import of B. Names defined after that line do not exist yet. This means the error is positional: if AMOUNT = 100 appears on line 1 of models.py and from utils import format_amount appears on line 2, then AMOUNT exists in the partial object but format_amount does not yet.

Minimal example See the partially-initialized module error

PythonCircular import raises ImportError at the missing nameruns
import sys, tempfile
from pathlib import Path

td = Path(tempfile.mkdtemp())
(td / "models.py").write_text("from utils import format_amount\nAMOUNT = 100\n")
(td / "utils.py").write_text(
    "from models import AMOUNT\ndef format_amount(x): return f'Rs{x}'\n"
)

sys.path.insert(0, str(td))
try:
    import models
except ImportError as e:
    msg = str(e)
    if msg.rfind(" (") > 0:
        msg = msg[: msg.rfind(" (")]
    print(type(e).__name__ + ":", msg)
finally:
    for mod in ("models", "utils"):
        sys.modules.pop(mod, None)
    sys.path.pop(0)
    import shutil; shutil.rmtree(td)
Output
ImportError: cannot import name 'AMOUNT' from partially initialized module 'models' (most likely due to a circular import)

AMOUNT is defined on line 2 of models.py, but models tries to import format_amount from utils on line 1. When utils imports back from models, models is half-executed and AMOUNT does not exist yet.

In production Fix the cycle by extracting shared constants

Move AMOUNT (and other shared constants) to a third module that neither models nor utils depends on. This breaks the cycle.

PythonExtract shared constants to break the circular importruns
import sys, tempfile
from pathlib import Path

td = Path(tempfile.mkdtemp())
(td / "constants.py").write_text("AMOUNT = 100\n")
(td / "models.py").write_text("from constants import AMOUNT\n")
(td / "utils.py").write_text(
    "from constants import AMOUNT\n"
    "def format_amount(x: int) -> str:\n"
    "    return f'Rs{x}'\n"
)

sys.path.insert(0, str(td))
import models, utils

print("models.AMOUNT:", models.AMOUNT)
print("utils.format_amount(250):", utils.format_amount(250))

sys.path.pop(0)
for mod in ("models", "utils", "constants"):
    sys.modules.pop(mod, None)
import shutil; shutil.rmtree(td)
Output
models.AMOUNT: 100
utils.format_amount(250): Rs250

constants.py has no imports from models or utils, so there is no cycle. The dependency arrows are: models → constants, utils → constants. Both are one-directional.

See it happen Why the partial-initialization error happens

Pythonmodels.pyutils.pysys.modulesadd empty 'models' objectstart executing line 1add empty 'utils' objectstart executing line 1return partial models objectImportError: cannot import name 'AMOUNT'
  1. Before executing models.py, Python registers an empty module object in sys.modules to handle re-entrant imports.
  2. Line 1 is 'from utils import format_amount' — Python now imports utils.
  3. Same pattern: an empty utils module is registered before its code runs.
  4. Line 1 is 'from models import AMOUNT' — Python needs models, which is already in sys.modules.
  5. Python finds models in sys.modules and returns it immediately — but models is only partially executed. AMOUNT has not been defined yet (it is on line 2).
  6. utils tries to find AMOUNT in the partial models object. It is not there. ImportError is raised.
Notice: the partial module object returned from sys.modules is what exposes the problem — Python returns whatever has been defined so far, not a complete module.
Scenario A

In your world

The situation

Your ETL package has pipeline.py that imports from validators import validate_schema, and validators.py that imports from pipeline import SCHEMA_VERSION (a module-level constant) to build its error messages.

What actually breaks

The job starts fine for six months. A new engineer adds a function to pipeline.py above the SCHEMA_VERSION = 3 line. Now SCHEMA_VERSION is no longer defined when validators.py tries to import it on startup. The Jenkins job starts crashing with ImportError: cannot import name 'SCHEMA_VERSION' from partially initialized module 'pipeline'. The error points at validators.py but the real problem is the import cycle.

What you'd do on Monday

Move SCHEMA_VERSION and other shared constants to a dedicated config.py (or constants.py) module that neither pipeline.py nor validators.py imports from. Both import from config.py. The cycle is gone.

ShellExtract constants to break the cycle
# Create a config.py with shared constants
echo "SCHEMA_VERSION = 3" > etl/config.py
# In pipeline.py: replace 'from validators import ...' that caused the cycle
# In validators.py: replace 'from pipeline import SCHEMA_VERSION'
#                   with 'from config import SCHEMA_VERSION'
Scenario B

Imagine you work at PhonePe

30 Python modules across payment, risk and reporting domains

The numbers

Illustrative: 30 modules, each importing from an average of 4 others = 120 import edges. In a clean layered architecture (data → domain → service → api) all edges point downward. If 5% of edges point upward (6 edges), each creates a potential cycle: 6 cycles × average 2 modules per cycle = 12 modules that can produce partial-initialization errors on any startup order change.

What the Scenario-A answer does here

The Scenario A fix (move shared constants to a third module) works for individual cycles. But at 30 modules across three domains, fixing cycles one at a time without a map leads to playing whack-a-mole: extract AMOUNT to constants.py, then discover that constants.py now imports from validators.py to format error messages, creating a new cycle.

What flips at this scale

At this scale you enforce a dependency-direction rule at the architecture level: the data layer may not import from service, service may not import from api. A CI lint check (import-linter or a custom script that walks ast.Import nodes) fails any import that violates the layering. The rule converts a runtime error that appears months after the cycle is introduced into a CI failure on the PR that introduces it.

The trade-off you'd defend

I would defend a strict layering policy enforced by CI over ad-hoc cycle fixes, trading some design rigidity for the guarantee that cycles cannot silently accumulate. The one exception is test files, which legitimately import across layers.

Illustrative numbers; not PhonePe-published architecture metrics.

Gotchas

The import works in a test but fails in the deployed job

Symptom

pytest runs fine; python jobs/load.py crashes with the partial-initialization error.

Cause

pytest imports modules in alphabetical or collection order; the deployed script imports them in a different order. The cycle was always there but the import order in tests happened to avoid the partial-initialization window.

Fix

The cycle is the bug, not the import order. Fix the cycle using one of the three patterns in core.

Moving the import inside the function silences the error but hides the cycle

Symptom

After moving from models import AMOUNT inside the function body, the error disappears. Six months later a different engineer sees the unusual local import and moves it back to the top.

Cause

Deferred imports are a workaround, not a fix. They are not idiomatic and invite regression.

Fix

Use the deferred import only as a temporary measure with a # TODO: extract to constants.py comment. Track it as a tech-debt ticket.

from models import * hides cycles more than explicit imports

Symptom

Star imports mask the partial-initialization error; the star import silently imports zero names from the partial object instead of raising an error on the missing name.

Cause

from models import * imports whatever names exist at the moment — an empty partial object has none but raises no ImportError.

Fix

Never use star imports in production code. Use explicit imports so partial-initialization failures raise immediately.

Tricky questions

Does putting the import inside the function body fix the circular dependency?
Most people say

Yes — deferred imports are the recommended fix.

Actually

A deferred import avoids the error but does not fix the dependency. The cycle still exists; the deferred import just delays it until the function is called, at which point the module is fully initialized. The real fix is to extract the shared name to a third module. Deferred imports are a maintenance hazard because they are not visible in the import block.

If module A does import B (not from B import X), does a circular import still cause errors?
Most people say

No — import B only needs the module object, which exists in sys.modules even if partial.

Actually

It depends on when A uses names from B. If A only calls B.some_function() inside a function body, import B works fine — B is fully initialized by the time A calls it. But if A accesses B.CONSTANT at module level, and that constant is defined after the point where B imports A, A gets AttributeError: module 'B' has no attribute 'CONSTANT'.

Scenario question

You're handed a models.py and a services.py that import each other. The error only appears when you run the app from the command line, not in pytest. A colleague says "just add lazy imports". What do you do?

Model answer

I'd first draw the dependency graph — what does models.py actually need from services.py and vice versa. In my experience, models should not import from services at all: models define data structures, services use them. The cycle usually means one of the imports is in the wrong direction. I'd extract whatever models.py needs from services.py (probably a constant or a type) into a shared types.py or config.py. I'd accept lazy imports only if the dependency is genuinely bidirectional at runtime, and I'd document why. The trade-off of lazy imports is that the dependency is invisible at the module level and future engineers will keep trying to "clean it up."

Interview corner

CommonWhat does "partially initialized module" mean in a Python ImportError?
“I'd say it means Python started executing the module's source code — and added the module object to sys.modules — but the import of a dependency triggered a re-import of the first module before it finished. The returned module object only has the names defined up to the line that triggered the cycle. Any name after that line is missing.”
TrickCan you have a circular import that never raises an error?
“I'd say yes, if neither module accesses names from the other at module-level code. If A and B import each other but only call functions on each other inside function bodies, the cycle is resolved by the time any name is looked up and no error occurs. That is why cycles can hide for months. The fix is still to remove the cycle, because a future top-level use of a name will silently fail.”
Experience checkYou have a cycle between models.py and utils.py. Describe the exact sequence of events that causes the ImportError.
“I'd walk through it: Python starts executing models.py and hits 'from utils import X'. It registers an empty models object in sys.modules, then starts executing utils.py. utils.py hits 'from models import Y'. Python returns that partial models object — only names defined before the point where models.py triggered utils.py exist. If Y is after that point, it is missing and Python raises ImportError.”
DesignHow would you enforce that a 30-module Python package never develops circular imports?
“I'd add import-linter to CI: define layers (data, domain, service, api) and the allowed import directions. Any violating import fails the build, converting a cycle that silently grows over months into a PR-time failure. The trade-off is maintaining the layer definitions as the codebase evolves.”

Memory hook

Two sous-chefs waiting for each other before they start is a circular import — Python hands each one a half-filled prep station and the missing ingredient is the ImportError.

If you remember only one thing:

Move shared names that two modules both need into a third module with no imports of its own.

Self-check

  1. What does Python put in sys.modules at the start of importing a module?

    An empty (partially initialized) module object, before any of the module's source code runs.

  2. Why does a circular import sometimes produce no error?

    If neither module accesses names from the other at module-level (only inside function bodies), the cycle resolves before any name lookup and no ImportError is raised.

  3. What are the three patterns for fixing a circular import, in order of preference?
    1. Extract shared names to a third module. 2. Defer the import inside the function that needs it. 3. Use import module instead of from module import name and access names at call time.
  4. What is the real error message Python raises for a circular import?

    ImportError: cannot import name 'X' from partially initialized module 'Y' (most likely due to a circular import).

  5. Why does the error appear only at module level, not inside function bodies?

    Function bodies execute when called, at which point all modules in the cycle are fully initialized. Module-level code executes at import time, when some modules may still be partial.

Prove it

Create two files alpha.py and beta.py that import from each other. Run python -c "import alpha" and copy the full error traceback as the artefact. Then apply the fix (extract to shared.py) and confirm the same command exits 0. Paste both the error output and the successful python -c "import alpha; print(alpha.VALUE)" output.

py-10-05Card 5 of 6

Give a package an explicit entry point

MediumP3 · Role-dependentKnow cold12 min
main-pyif-name-mainentry-pointspyproject-toml-scriptsimport-safe-execution

Why you care

"It works when I run it but crashes when I import it" is the failure mode of a Python file with top-level side effects: a database connection opened at import time, a print that runs on every import, or an argparse.parse_args() that exits non-zero when the test suite imports the module. These failures are preventable with a two-line pattern that every product-company engineer is expected to know. Interviewers ask about __main__ and entry points because they separate "script I wrote on my own machine" from "distributable tool that teams and customers can run."

Explain like I'm 5

__main__.py is the front door of a package. When you knock on the front door with python -m mypackage, Python opens __main__.py. When your code imports the package normally (import mypackage), Python walks in through the side entrance (__init__.py) and ignores the front door entirely. The if __name__ == '__main__': guard is the same idea on a single file: put it around the code that should only run when someone knocks on the front door.

Core concept

Python sets __name__ to the string '__main__' when a file is the entry point of execution: run directly with python file.py, run as a module with python -m package, or run as a package's __main__.py via python -m package. When the same file is imported by another module, __name__ is the module's dotted name, never '__main__'.

The if __name__ == '__main__': guard wraps any code that should only run at the top level: argparse, database connections, expensive setup, or main() calls. Everything outside that block is import-safe.

__main__.py extends this to packages: placing a __main__.py inside mypackage/ makes python -m mypackage a valid command. The file's __package__ is set to 'mypackage', so relative imports work correctly — unlike running python mypackage/somefile.py directly.

Console scripts (entry points in pyproject.toml) let you expose your package's entry point as a command-line name after pip install:

Textpyproject.toml console_scripts entry point
[project.scripts]
etl-run = "etl_jobs.__main__:main"

After installation, etl-run on the PATH calls main() in etl_jobs/__main__.py directly, without the user needing to know the package name or module path.

Decision rule: use if __name__ == '__main__': for any file that has both a useful import surface and a runnable main function. Use __main__.py when the package itself should be executable as a command. Use console scripts when you ship the package to end users or teammates via pip.

Under the hood

When Python executes python -m mypackage, it sets sys.argv[0] to the path of __main__.py, sets __name__ to '__main__', and sets __package__ to 'mypackage'. That last assignment is what makes relative imports inside __main__.py work. By contrast, python mypackage/__main__.py sets __package__ to None, breaking any relative import inside the file. This is the reason python -m is the canonical way to run a package: it correctly seeds the package context that the rest of the import machinery depends on.

Minimal example __name__ guard protects import-time side effects

Python__name__ is __main__ only when the file is the entry pointruns
def greet(name: str) -> str:
    return f"hello, {name}"

# Runs only when executed directly, not when imported.
if __name__ == "__main__":
    print(greet("world"))
    print("__package__:", __package__)
Output
hello, world
__package__: None

__package__ is None here because this snippet is run as a standalone script, not as part of a package. Running with python -m mypackage would print __package__: mypackage.

In production Package with __main__.py and argparse entry point

PythonPackage that runs as python -m etltool or via console_scriptsruns
import sys, tempfile, subprocess
from pathlib import Path

td = Path(tempfile.mkdtemp())
pkg = td / "etltool"
pkg.mkdir()
(pkg / "__init__.py").write_text('VERSION = "1.0.0"\n')
(pkg / "__main__.py").write_text(
    "import argparse\n"
    "from etltool import VERSION\n"
    "\n"
    "def main() -> None:\n"
    "    parser = argparse.ArgumentParser(prog='etltool')\n"
    "    parser.add_argument('--version', action='store_true')\n"
    "    parser.add_argument('--dry-run', action='store_true')\n"
    "    args = parser.parse_args()\n"
    "    if args.version:\n"
    "        print(f'etltool {VERSION}')\n"
    "    elif args.dry_run:\n"
    "        print('dry-run: no changes made')\n"
    "\n"
    "main()\n"
)

result = subprocess.run(
    [sys.executable, "-m", "etltool", "--version"],
    cwd=str(td),
    capture_output=True,
    text=True,
)
print(result.stdout.strip())
print("returncode:", result.returncode)

import shutil; shutil.rmtree(td)
Output
etltool 1.0.0
returncode: 0

The argparse.ArgumentParser call is inside main(), which is called at the bottom of __main__.py. If a test imports etltool, Python runs __init__.py only — __main__.py is never touched and parse_args() never executes.

Scenario A

In your world

The situation

You write a settlement reconciliation script reconcile.py that opens a database connection at module level and calls argparse.parse_args() at the top. Your senior engineer asks you to add unit tests.

What actually breaks

The first test that does import reconcile connects to the production database, prints the argparse help, and calls sys.exit(0) — terminating the entire test process. The test runner shows Process finished with exit code 0 and no tests ran.

What you'd do on Monday

Wrap all startup code in the if __name__ == '__main__': guard. Move the database connection inside the function that needs it (or accept it as a parameter for testability). The module is now import-safe: importing it loads the functions but executes nothing.

PythonImport-safe reconciliation module (needs a real DB connection)
import argparse
from pathlib import Path

def reconcile(settlements_path: Path, db_url: str) -> dict:
    # opens connection here, inside the function
    ...

def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--db-url", required=True)
    parser.add_argument("input", type=Path)
    args = parser.parse_args()
    result = reconcile(args.input, args.db_url)
    print(result)

if __name__ == "__main__":
    main()

Tests can now from reconcile import reconcile and pass a mock DB URL without triggering argparse or a real connection.

Scenario B

Imagine you work at Stripe

5 CLI tools, ~3 k monthly active users each

The numbers

Illustrative: Stripe ships ~5 Python CLI tools to developers via PyPI. Each tool has ~3,000 monthly active users. Without a console_scripts entry point: users run python -m stripe_tool --help, which requires knowing the internal module name. With console_scripts: users type stripe-tool --help. At 3,000 users × 5 tools × avg 4 invocations/day = 60,000 command invocations per day. If 2% of those fail because the user forgot the python -m prefix and got a ModuleNotFoundError instead: 1,200 confused invocations per day → support tickets.

What the Scenario-A answer does here

The Scenario A fix (use if __name__ == '__main__': to make the module import-safe) is necessary but not sufficient for an installed CLI. An import-safe module still requires the user to know to run python -m rather than just the tool name. Without a console_scripts entry point, the tool is not a first-class CLI.

What flips at this scale

At this scale you define the entry point in pyproject.toml, ship the tool via PyPI, and guarantee that stripe-tool is on PATH after pip install stripe-tool. The implementation details (module path, package name) are invisible to users. Versioned console scripts also mean pip install --upgrade updates the command transparently.

The trade-off you'd defend

I would defend console_scripts over documenting python -m for any tool used by people outside your team, trading a small pyproject.toml entry for a standard user experience. For internal scripts used by 2-3 engineers who know the module structure, python -m is fine.

Illustrative numbers chosen to make the arithmetic visible; not Stripe-published CLI usage figures.

Gotchas

python mypackage/__main__.py fails with a relative import error

Symptom

from .utils import X raises ImportError: attempted relative import with no known parent package.

Cause

Running a file directly with python path/to/__main__.py sets __package__ to None. Relative imports require __package__ to be set.

Fix

Always run packages with python -m mypackage, not python mypackage/__main__.py.

argparse.parse_args() at module level kills the test runner

Symptom

import mymodule causes the test process to exit 0 with no output.

Cause

argparse.parse_args() at module level reads sys.argv, finds --help or an unexpected argument (such as pytest's own flags), and calls sys.exit(0).

Fix

Move parse_args() inside main() and call main() only under if __name__ == '__main__':.

Installed console_scripts point to the wrong Python

Symptom

mycommand runs but uses a different Python or a different venv than expected.

Cause

pip install added the console script to the venv's bin/ directory, but the venv is not activated and the system PATH finds an old copy of the script first.

Fix

Activate the venv before running the command, or use pipx install for user-level CLI tools so the tool's dependencies are isolated from the project's venv.

Tricky questions

Does if __name__ == '__main__': prevent the code from running when the file is imported?
Most people say

No — Python still parses the whole file including that block.

Actually

Python parses and compiles the entire file, including the if __name__ == '__main__': block. The block is not executed because the condition is False at import time. The function definitions inside the block are also not defined until the block runs, so a function defined only inside if __name__ == '__main__': is not accessible to importers.

Is python -m mypackage the same as python mypackage/__main__.py?
Most people say

Yes — both run __main__.py.

Actually

Both execute __main__.py, but they set __package__ differently. python -m mypackage sets __package__ = 'mypackage' and adds the project root to sys.path. python mypackage/__main__.py sets __package__ = None and adds mypackage/ to sys.path[0]. Relative imports inside __main__.py work with -m but fail with the direct path.

Scenario question

You're handed a library that a colleague uses in two ways: import data_loader in their pipeline, and python data_loader.py --validate orders.csv from the command line. Adding a unit test breaks things. What structure do you impose?

Model answer

I'd impose a strict separation: all importable logic goes in functions at module level, and the command-line entry point goes in if __name__ == '__main__': at the bottom, calling a main() function that wires argparse to the logic functions. For the library use case, the test imports from data_loader import load, validate — no side effects. For the CLI use case, python data_loader.py --validate orders.csv hits the guard and calls main(). If the team later wants to install it as a command, I'd move the logic into a package and add a console_scripts entry in pyproject.toml. The trade-off is that a main() function that accepts parsed args (not argparse.Namespace directly) is more testable than one that calls parse_args() internally.

Interview corner

CommonWhat does if __name__ == '__main__': do?
“I'd say it guards code that should only run when the file is the top-level entry point. Python sets __name__ to '__main__' when a file is executed directly or via python -m. When imported, __name__ is the module's dotted name. The guard prevents argparse calls, database connections and main() from running every time a colleague imports your module.”
TrickWhere does __package__ get its value when you run python -m mypackage?
“I'd say Python sets __package__ to 'mypackage' — the name of the package containing __main__.py. That is why relative imports work under python -m but not when you run the same __main__.py as a direct script: in the direct case, __package__ is None and relative imports fail.”
Experience checkYour ETL entry point uses argparse.parse_args() at module level. A new engineer adds from etl import reconcile to a test. The test suite exits immediately. Diagnose and fix.
“I'd say parse_args() at module level reads sys.argv when the module is imported. During a pytest run, sys.argv contains pytest's own arguments, which argparse does not recognise, so it prints the usage message and calls sys.exit(2). The fix is to move parse_args() inside main() and call main() only under if __name__ == '__main__':.”
DesignWhen would you choose console_scripts over documenting python -m?
“I'd say console_scripts whenever the tool is used by people outside my immediate team or shipped as a package. It gives users a named command that works regardless of the Python installation details. I'd use python -m for internal scripts where everyone knows the project structure and a named command would add maintenance overhead for no user benefit.”

Memory hook

__main__.py is the front door of a package — Python only knocks on it when you run python -m, never when you import. Everything behind the if __name__ == '__main__': guard is the part that lives and dies by the front door.

If you remember only one thing:

Wrap argparse, database connections and main() calls in if __name__ == '__main__': so that importing the module never executes them.

Self-check

  1. What value does Python give to __name__ when a file is imported?

    The module's dotted name (e.g. 'etl.pipeline'), not '__main__'.

  2. What is the difference in __package__ between python -m mypackage and python mypackage/__main__.py?

    python -m mypackage sets __package__ = 'mypackage'; python mypackage/__main__.py sets __package__ = None.

  3. What happens to argparse.parse_args() when it is called at module level during a pytest run?

    sys.argv contains pytest's arguments. argparse does not recognise them and calls sys.exit(2), which terminates the test process.

  4. What must pyproject.toml contain to install a mycommand CLI command?

    A [project.scripts] section: mycommand = "mypackage.__main__:main".

Prove it

Create a calc/ package with __init__.py exporting a add(a, b) function and __main__.py that uses argparse to accept two numbers and prints their sum. Verify that python -m calc 3 5 prints 8 and that python -c "from calc import add; print(add(3, 5))" also prints 8 without triggering argparse. Paste the output of both commands as the artefact.

py-10-06Card 6 of 6

Rebuild the environment and test the artifact

MediumP3 · Role-dependentTalk about it13 min
clean-installreproducible-buildplatform-constraintsartifact-verificationci

Why you care

Every environment you have ever tested in is contaminated: your laptop has years of pip install accumulation, your CI runner has cached layers from six months ago, and your colleague's machine has a different OS. The only way to know that your lockfile actually works is to destroy the environment and rebuild it from scratch — in the same Python version and on the same platform as production. Interviewers ask about this because "it worked in CI" followed by "it crashed in prod" is a failure of artifact verification, and product companies run this check automatically on every merge.

Explain like I'm 5

Testing a recipe by cooking it in your normal kitchen — where you already have last week's leftover stock on the stove — does not tell you whether the recipe works in a clean kitchen. A clean-install test deletes the stock, puts an empty pot on the stove, and follows the recipe from scratch. If the dish comes out wrong, the recipe is incomplete.

Core concept

A clean install test has four steps:

  1. Delete the existing environment: remove .venv/, deactivate the current env, or use a fresh Docker container.
  2. Reinstall from the lockfile only: uv sync --locked, poetry install --no-root, or pip install --require-hashes -r requirements.txt. No --upgrade, no implicit re-resolution.
  3. Check platform constraints: Python version (requires-python), compiled extensions (C wheels have platform tags), and operating system. An extension built on macOS will not load on Amazon Linux.
  4. Run a smoke test: import the main entry point, run the cheapest test suite, or execute the ETL job against a known fixture. Confirm exit code 0.

The platform trap is the most common clean-install failure in ETL. uv.lock is a universal, cross-platform lockfile that records wheels for every platform, so it is safe to commit from any machine. The trap is the deployment artifact: a developer runs pip install -t package/ on macOS and zips the result; the zip contains macOS .so files that fail on Lambda's Amazon Linux with an ImportError naming the compiled module.

The same trap applies to pip freeze and pip-compile output — those files record the wheel that was installed on the developer's machine, which is platform-specific.

The fix: build the deployment artifact for the target platform. Either build it inside a Linux container (e.g. python:3.12-slim or the public Lambda base image), or use uv pip install --python-platform x86_64-manylinux_2_28 --python-version 3.12 --only-binary :all: --target ./package.

Decision rule: every deployment artifact (Docker image, Lambda zip, wheel) must be built on the same platform as production. The lockfile is safe to generate on any machine; it is the artifact build that must be platform-correct.

Under the hood

Python wheels are tagged: psycopg_binary-3.1.9-cp312-cp312-manylinux_2_17_x86_64.whl. The tag has three parts: Python tag (cp312), ABI tag (cp312), platform tag (manylinux_2_17_x86_64). When pip or uv installs, it picks the wheel whose tags match the running interpreter and platform. uv.lock records wheels for all platforms, so uv sync --locked on Linux fetches the Linux wheel and on macOS fetches the macOS wheel — this is why uv.lock can be committed from any developer machine. If a requirements file or lock format does not include a matching wheel for the target platform, the tool falls back to an sdist (source build). This is why artifact-build failures show error: command 'gcc' failed: no pre-compiled wheel exists for the target platform, so pip attempted to compile from source but the required C headers were absent in the build environment.

Minimal example Check Python version against pyproject.toml requirement

PythonValidate the current interpreter against requires-pythonruns
import sys, re

def check_python_req(req: str) -> bool:
    """Parse '>=3.12' from requires-python and check sys.version_info."""
    m = re.match(r">=(\d+)\.(\d+)", req)
    if not m:
        return False
    major, minor = int(m.group(1)), int(m.group(2))
    return sys.version_info >= (major, minor)

print("Python:", sys.version.split()[0])
print("Satisfies >=3.12:", check_python_req(">=3.12"))
print("Satisfies >=3.13:", check_python_req(">=3.13"))
Output
Python: 3.12.4
Satisfies >=3.12: True
Satisfies >=3.13: False

Use this check at the top of your CI script before the install step. Failing early with "Python 3.11 does not satisfy >=3.12" is clearer than a later SyntaxError from a 3.12-only syntax.

In production Environment consistency checker

This script reads pyproject.toml and reports whether the running environment satisfies the declared requirements. It uses only the stdlib (tomllib, importlib.metadata, re, sys).

PythonAudit installed packages against pyproject.toml declarationsruns
import sys, re, tomllib, importlib.metadata as meta, textwrap

# Inline pyproject.toml — in practice read from Path("pyproject.toml").read_bytes()
pyproject_src = textwrap.dedent("""
    [project]
    name = "etl-job"
    requires-python = ">=3.12"
    dependencies = [
        "requests==2.31.0",
    ]
""").strip().encode()

cfg = tomllib.loads(pyproject_src.decode())
requires_python = cfg["project"].get("requires-python", "")
declared_deps = cfg["project"].get("dependencies", [])

def satisfies_python(req: str) -> bool:
    m = re.match(r">=(\d+)\.(\d+)", req)
    return sys.version_info >= (int(m.group(1)), int(m.group(2))) if m else True

results: list[tuple[str, str, str]] = []
py_ok = satisfies_python(requires_python)
results.append(("python", sys.version.split()[0], "OK" if py_ok else "FAIL"))

for dep in declared_deps:
    pkg = re.split(r"[>=<!;\[]", dep)[0].strip()
    req_ver = dep.split("==")[1] if "==" in dep else None
    try:
        installed = meta.version(pkg)
        status = "OK" if (req_ver is None or installed == req_ver) else f"MISMATCH ({installed})"
    except meta.PackageNotFoundError:
        status = "MISSING"
    results.append((pkg, req_ver or "any", status))

for name, version, status in results:
    print(f"{name:<20} {version:<15} {status}")
Output
python               3.12.4          OK
requests             2.31.0          MISSING

requests shows MISSING because this runs in a stdlib-only environment — exactly what a clean Lambda cold start looks like before the layer is attached. The checker is the gate that would catch this before deployment.

Scenario A

In your world

The situation

Your ETL job runs fine in Jenkins. A new engineer deploys it to AWS Lambda for the first time. The Lambda function crashes on every invocation.

What actually breaks

The Lambda log shows ImportError: No module named 'psycopg._psycopg'. The engineer ran pip install -t package/ -r requirements.txt on their macOS laptop and zipped the result. The zip contains macOS arm64 .so files; Lambda runs on Amazon Linux x86_64 and cannot load them.

What you'd do on Monday

Build the deployment artifact for the Lambda platform, not the developer's laptop:

ShellBuild a Lambda-compatible deployment package with uv pip install
# Export locked requirements (no hashes needed for --target install)
uv export --frozen --no-hashes > requirements-lambda.txt

# Install wheels for the Lambda platform (Amazon Linux 2023 = manylinux_2_28+)
uv pip install \
    --python-platform x86_64-manylinux_2_28 \
    --python-version 3.12 \
    --only-binary :all: \
    --target ./package \
    -r requirements-lambda.txt

zip -r function.zip package/ jobs/

uv.lock is a universal lockfile and is safe to generate and commit from any machine. The fix is the artifact build step, not the lockfile.

Scenario B

Imagine you work at Groww

8 Lambda functions, dev on macOS, prod on Amazon Linux

The numbers

Illustrative: 8 Lambda functions, each with ~20 Python dependencies, 4 of which are compiled extensions (psycopg-binary, lxml, cryptography, numpy). Without a CI-enforced artifact build step: 4 compiled deps × 8 functions = 32 macOS .so files zip-deployed to Lambda. 8 functions crash at cold start → 8 ImportError alerts at 9 AM on a trading day. Time to diagnose: 45 min (logs, deployment comparison, revert). Trade cost: Illustrative: 45 min × average Groww order volume → non-trivial.

What the Scenario-A answer does here

The naive approach: each developer runs pip install -t package/ -r requirements.txt on their laptop and zips the result. uv sync --locked in CI is correct for the test step; the failure is the separate artifact-packaging step that is not platform-aware.

What flips at this scale

At this scale you add a dedicated artifact-build job in CI that runs on ubuntu-latest (or inside python:3.12-slim) and uses uv pip install --python-platform x86_64-manylinux_2_28 --only-binary :all: --target ./package. Developers never build the Lambda ZIP locally — only CI does. The lockfile can be committed from any machine; the ZIP cannot.

The trade-off you'd defend

I would defend CI-only artifact packaging over trusting developer laptops, trading the inconvenience of "I cannot test the exact Lambda ZIP locally" for the guarantee that every deployed ZIP contains Linux binaries. For a trading platform where 9 AM crashes are measured in real money, that trade is obvious.

Illustrative numbers chosen to show the scale consequence; not Groww-published deployment metrics.

Gotchas

Clean install succeeds but the job crashes with a missing shared library at runtime

Symptom

pip install exits 0, but import psycopg raises ImportError: libpq.so.5: cannot open shared object file.

Cause

psycopg (the pure-Python distribution) was installed instead of psycopg[binary]. The pure package has no compiled extension but requires the system libpq to be present at runtime; a clean Lambda environment has no libpq.

Fix

Depend on psycopg[binary] in pyproject.toml, which installs a self-contained compiled wheel with no system-library dependency. Alternatively, install libpq in the Lambda layer or Docker image.

uv sync --locked fails in CI but lockfile was just committed

Symptom

CI reports error: lockfile is out of date even though the lockfile was committed 2 minutes ago.

Cause

pyproject.toml was edited — a dependency version or requires-python was bumped — without re-running uv lock to update the lockfile. The committed lockfile no longer matches the current pyproject.toml.

Fix

Run uv lock after every change to pyproject.toml, then commit both files together. Add a CI lint step: uv lock --check exits non-zero if the lockfile is stale.

The smoke test passes in CI but fails in the Lambda ZIP

Symptom

pytest exits 0 in CI; the Lambda function crashes with ModuleNotFoundError.

Cause

The smoke test ran against the CI venv, which is a full Python environment. The Lambda ZIP was built with a different install step that missed a transitive dependency.

Fix

Run the smoke test against the deployment artifact itself: unzip the Lambda ZIP into a temp directory and run python -c "from jobs import handler" pointing at that directory.

Tricky questions

Does pip install -r requirements.txt on a fresh machine guarantee reproducibility if the file was generated by pip freeze?
Most people say

Yes — pip freeze captures exact versions, so reinstalling from it is identical.

Actually

pip freeze captures exact versions but not hashes. On a different platform or Python micro-version, pip may install a different binary wheel for the same version string. pip-compile --generate-hashes adds hashes but its output is also platform-specific — generated on macOS, it records only macOS-compatible wheels. uv.lock is cross-platform: it records wheels for all platforms and lets the installer pick the correct one per environment. Only uv.lock (or a manually curated multi-platform requirements file) is fully reproducible across platforms.

If you run pip install --upgrade on a locked environment, what happens to the lockfile?
Most people say

The lockfile updates automatically.

Actually

The lockfile does not change. pip install --upgrade modifies site-packages silently, creating a divergence between the lockfile and the actual environment. The next uv sync --locked or poetry install will revert the upgrade by reinstalling from the lockfile.

Scenario question

You're handed a Lambda function that works in one AWS region (ap-south-1) but crashes with ImportError in a second region (us-east-1) where a new team is deploying. Both use the same deployment ZIP. Same Python version. What do you investigate?

Model answer

I'd check whether the two regions use the same Lambda runtime version — Amazon Linux 2 vs Amazon Linux 2023 have different glibc versions. I'd run python -c "import platform; print(platform.platform())" in both Lambdas. If the runtimes differ, the ZIP's compiled extensions only match one of them. Fix: rebuild the ZIP targeting the us-east-1 runtime with uv pip install --python-platform x86_64-manylinux_2_28 --only-binary :all:. The trade-off is a separate CI build step per runtime.

Interview corner

CommonWhat is the difference between a venv and a Docker container for environment reproducibility?
“I'd say a venv isolates Python packages but shares the system libraries and the Python interpreter version with the host. A Docker container also isolates the OS, system libraries and the Python interpreter — so it is a stronger form of isolation. For Lambda deployments where the runtime OS matters, a Docker build is the right tool. For local development and CI, a venv plus a locked requirements file is usually sufficient.”
TrickYou run uv sync --locked in CI and it exits 0. Is the environment reproducible?
“I'd say the venv is reproducible. uv.lock is universal — it records wheels for all platforms, so it is safe to generate from any machine. What uv sync --locked does not verify is the deployment artifact: if the Lambda ZIP was built with pip install -t package/ on a developer's laptop, that ZIP may carry macOS binaries regardless of how clean the venv is.”
Experience checkThe Jenkins build passes but the Lambda fails. How do you diagnose?
“I'd compare the installed packages in each environment: add a step that runs pip list > /tmp/packages.txt and uploads it as a build artifact. Diff the two lists. If they differ, the install steps are not equivalent. Then I'd check the platform tags on the wheels actually installed — pip show --files psycopg lists the wheel file name, which includes the platform tag.”
DesignHow would you ensure every Lambda ZIP contains Linux-compatible binaries across a team?
“I'd move artifact packaging into CI entirely. A dedicated job on ubuntu-latest runs uv pip install --python-platform x86_64-manylinux_2_28 --only-binary :all: --target ./package and produces the ZIP. Developers run uv sync --locked locally; they never build the ZIP. uv.lock is universal and can be committed from any machine. The trade-off is losing local ZIP reproduction, which is acceptable for a Linux-deployed service.”

Memory hook

A clean kitchen test is the only test that tells you whether the recipe actually works. Delete the venv, reinstall from the lockfile, build the artifact on the production platform, run the smoke test — that is what "reproducible" means.

If you remember only one thing:

Build the deployment artifact on the production platform — a zip packaged on macOS carries macOS binaries that crash on Linux.

Self-check

  1. Why does a pip install -t package/ run on macOS produce a Lambda ZIP that crashes on Linux?

    Because pip install selects wheels for the current platform. macOS installs arm64 .so files; Lambda runs on Linux x86_64 and cannot load them. uv.lock is universal — the lockfile is not the problem; the artifact packaging step must be platform-aware.

  2. What is the correct way to build a Lambda deployment package that includes compiled extensions?

    Use uv pip install --python-platform x86_64-manylinux_2_28 --python-version 3.12 --only-binary :all: --target ./package -r requirements-lambda.txt in a CI job running on Linux, or inside a Linux Docker container.

  3. What is the failure mode of --only-binary :all: when no binary wheel exists?

    pip install fails immediately with ERROR: Could not find a version ... that satisfies the requirement. This is the desired behaviour — fail fast rather than silently building a broken source wheel.

  4. What causes error: lockfile is out of date in uv sync --locked?

    pyproject.toml was edited — a dependency version or requires-python was changed — without re-running uv lock to update the lockfile. Fix: run uv lock and commit both files.

Prove it

Create a project with uv init, add psycopg-binary as a dependency, and run uv lock. Open uv.lock and search for psycopg. You will find wheel entries for multiple platforms — macOS arm64, Linux x86_64, and more — all in the same lockfile. This confirms that uv.lock is universal. Then run uv export --frozen --no-hashes > requirements.txt and inspect that file: it lists only the packages, not the wheels. Paste a few of the wheel lines from uv.lock as the artefact confirming the universal format.