Shelf · Python · Chapter 11

Chapter 11: Testing in Python

Every ETL job you have written runs nightly, succeeds in Jenkins, and may be delivering zero rows — silently, with no exception raised. This chapter gives you the mechanics to catch that before it reaches finance, from pytest fixture lifecycles through mocking, clock injection, and invariant testing.

Difficulty
Priority
py-11-01Card 1 of 6

Python test mechanics: pytest fixtures, assertions and lifecycle

MediumP2 · Very likelyKnow cold12 min
testingpytestfixtureslifecycleassertionsunittest

Why you care

Every production bug that passes CI undetected passed because the test either did not exist, or existed but was ordered to depend on state set by a previous test — and nobody noticed until the run happened in a different order. Product company engineers are expected to maintain test suites that could run in any sequence. Interviewers ask about fixture scope because the answer tells them whether you have been bitten by shared mutable state, which is the most common source of order-dependent test failures in Python projects. This card owns the mechanics — lifecycle, scope, assertions, and teardown guarantees. The ETL-specific application is in Test an ETL job through failure and replay.

Explain like I'm 5

A test suite is like a hotel that runs a banquet for many guests. The kitchen is stocked once before any guests arrive (setUpClass). Each table gets its own clean tablecloth before that course begins (setUp), the food is served (the test body runs), and the tablecloth is cleared away (tearDown). After the last guest leaves, the kitchen is shut down (tearDownClass). If one guest leaves crumbs on the tablecloth and setUp does not replace it, the next guest finds yesterday's mess — and the failure report blames the wrong course.

Core concept

Python has two test frameworks you will encounter: unittest (stdlib, ships with Python) and pytest (third-party, industry standard). Both use the same arrange-act-assert mental model.

The unittest lifecycle runs in this fixed order:

  • setUpModule once before any class in the module
  • setUpClass once per class (decorated @classmethod)
  • setUp once before each individual test method
  • the test method itself
  • tearDown once after each test method, even if the test raised
  • tearDownClass once after all methods in the class
  • tearDownModule once after all classes

Teardown guarantee: tearDown runs even when the test body raises. setUpClass teardown (tearDownClass) is skipped only if setUpClass itself raises — which means resources allocated there are not released on setup failure.

pytest fixtures replace the setUp pattern: any test function that names a fixture as a parameter receives its return value. The scope argument controls how often the fixture is created: "function" (default, one per test), "class", "module", or "session" (once per pytest process). A yield fixture replaces tearDown: code after yield runs on cleanup.

Decision rule: use setUp/tearDown (or function-scoped fixtures) for state that must be fresh per test — database connections, mutable containers, temporary files. Use setUpClass (or class/module-scoped fixtures) only for expensive read-only resources — a compiled regex, a loaded static file, an in-memory SQLite schema that no test mutates. Never share mutable objects across tests unless every test reinitialises the relevant fields in setUp.

Minimal example Lifecycle order logged to a list

PythonsetUpClass runs once; setUp runs per testruns
import unittest
import io

log = []

class T(unittest.TestCase):
    @classmethod
    def setUpClass(cls):
        log.append('setUpClass')

    def setUp(self):
        log.append('setUp')

    def test_a(self):
        log.append('test_a')

    def test_b(self):
        log.append('test_b')

    def tearDown(self):
        log.append('tearDown')

    @classmethod
    def tearDownClass(cls):
        log.append('tearDownClass')

suite = unittest.TestLoader().loadTestsFromTestCase(T)
unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite)
print(log)
Output
['setUpClass', 'setUp', 'test_a', 'tearDown', 'setUp', 'test_b', 'tearDown', 'tearDownClass']

In production Settlement loader with per-test fresh database

PythonRealistic setUp pattern: shared CSV fixture, isolated SQLite per testruns
import sqlite3
import csv
import io
import unittest

FIXTURE_CSV = 'settlement_id,amount\nS001,10.50\nS002,200.00\n'


def parse_paise(s):
    return round(float(s) * 100)


def load_settlements(csv_body, conn):
    conn.execute(
        'CREATE TABLE IF NOT EXISTS settlements (id TEXT, paise INT)'
    )
    for row in csv.DictReader(io.StringIO(csv_body)):
        conn.execute(
            'INSERT INTO settlements VALUES (?,?)',
            (row['settlement_id'], parse_paise(row['amount'])),
        )
    conn.commit()


class SettlementLoaderTests(unittest.TestCase):
    @classmethod
    def setUpClass(cls):
        # Compiled once: each test re-reads this string, not the S3 file.
        cls.csv_body = FIXTURE_CSV

    def setUp(self):
        # Fresh DB per test: mutations in one test cannot bleed into another.
        self.conn = sqlite3.connect(':memory:')

    def tearDown(self):
        self.conn.close()

    def test_row_count(self):
        load_settlements(self.csv_body, self.conn)
        n = self.conn.execute(
            'SELECT COUNT(*) FROM settlements'
        ).fetchone()[0]
        self.assertEqual(n, 2)

    def test_paise_conversion(self):
        load_settlements(self.csv_body, self.conn)
        row = self.conn.execute(
            'SELECT paise FROM settlements WHERE id=?', ('S001',)
        ).fetchone()
        self.assertEqual(row[0], 1050)


buf = io.StringIO()
runner = unittest.TextTestRunner(stream=buf, verbosity=2)
suite = unittest.TestLoader().loadTestsFromTestCase(SettlementLoaderTests)
result = runner.run(suite)
print('tests run:', result.testsRun, 'failures:', len(result.failures))
Output
tests run: 2 failures: 0

See it happen setUp and tearDown wrap every individual test

test runnersetUpClasssetUp / tearDowntest bodysetUpClass()donesetUp()test_a runsdone (or raised)tearDown()setUp()test_b runs
  1. Called once before any test method in the class runs.
  2. setUpClass finishes; shared class-level state is now available via cls.*.
  3. Called before test_a. A fresh SQLite connection is created here.
  4. The test exercises the loader; it mutates its own isolated DB.
  5. Control returns regardless of whether the test passed or raised.
  6. Connection closed. Any state from test_a is gone.
  7. Called again before test_b — a clean connection, no memory of test_a.
  8. test_b sees only the state it creates in this setUp call.
Notice: setUp and tearDown bracket every individual test, so even a test that mutates state cannot affect its neighbour.
Scenario A

In your world

The situation

Your Jenkins job runs a suite of three test methods. Two of them share a module-level list that the first test appends to. The second and third tests pass locally when run in alphabetical order but fail on CI because CI runs them in a different order.

What actually breaks

The build is green one day and red the next with no code change. The failure message is AssertionError: 3 != 2 — a count is wrong, but the count changes by run order, not by logic.

What you'd do on Monday

Move the list into setUp. Each test gets its own fresh copy. If it must be shared, make it read-only or reset it at the start of each setUp.

PythonsetUp resets state per test; order no longer mattersruns
import unittest
import io

class OrderSafeTests(unittest.TestCase):
    def setUp(self):
        self.log = []   # fresh list every test

    def test_first(self):
        self.log.append('first')
        self.assertEqual(len(self.log), 1)

    def test_second(self):
        self.log.append('second')
        self.assertEqual(len(self.log), 1)

buf = io.StringIO()
result = unittest.TextTestRunner(stream=buf, verbosity=0).run(
    unittest.TestLoader().loadTestsFromTestCase(OrderSafeTests)
)
print('tests run:', result.testsRun, 'failures:', len(result.failures))
Output
tests run: 2 failures: 0
Scenario B

Imagine you work at Shopify

50k tests per CI run

The numbers

50,000 tests × 80 ms average each = 50,000 × 0.08 s = 4,000 s ≈ 67 min wall-clock with a single worker. With 8 parallel workers: 4,000 s ÷ 8 = 500 s ≈ 8 min.

What the Scenario-A answer does here

When fixtures at class or module scope hold mutable shared state — a DB seeded once per class, a cache populated in setUpClass — parallel workers can interleave reads and writes. Parallelism makes hidden order-dependency crash tests that ran fine sequentially.

What flips at this scale

Every mutable fixture that parallel tests touch must be scoped to the function level (one instance per test), or the mutation must be behind a transaction that is rolled back in teardown. The expensive setup stays at the class level but must produce read-only objects. That contract — expensive shared setup, isolated mutable teardown — is what forces a clean separation between seed data and working data.

The trade-off you'd defend

I would defend function-scoped mutable fixtures over class-scoped shared ones, trading a small amount of setup overhead for the ability to run the full suite in any order with any concurrency setting.

Illustrative; Shopify's internal CI architecture is not published.

pytest fixtures vs unittest setUp

pytest fixtureunittest setUp / tearDown
Scope controlfunction / class / module / sessionclass or module level only
Teardownyield in the fixture bodytearDown method
Sharing between testsimplicit via parameter nameexplicit method call
Stdlibno (requires pip install pytest)yes
Run commandpytestpython -m unittest

Gotchas

Shared mutable fixture causes order-dependent failures

Symptom

AssertionError: 5 != 2 on CI; passes locally. The number changes when the test runner changes the execution order.

Cause

A list or dict initialised in setUpClass is appended to by setUp or the test body, so each subsequent test inherits the previous test's mutations.

Fix

Move the mutable object into setUp so it is reinitialised before each test.

tearDownClass skipped on setUpClass failure

Symptom

A resource allocated at the start of setUpClass (a temp directory, a socket) is never released if setUpClass raises halfway through.

Cause

CPython only calls tearDownClass if setUpClass succeeds. Any partial allocation made before the error leaks.

Fix

Use a try/finally inside setUpClass to release partial allocations, or use addClassCleanup (available from Python 3.8+) which always fires.

assertRaises only checks the exception type, not the message

Symptom

The test passes even though the function raised for a completely different reason than expected.

Cause

self.assertRaises(ValueError, func, arg) (and the with form) only checks that the named type was raised; it does not inspect the message. Two different code paths that both raise ValueError are indistinguishable to assertRaises alone.

Fix

Use with self.assertRaises(ValueError) as ctx: and then check self.assertIn('expected text', str(ctx.exception)) after the block, or use self.assertRaisesRegex(ValueError, r'expected text', func, arg) to assert both type and message in one call.

Tricky questions

Does tearDown run if the test body raises?
Most people say

No — an unhandled exception in the test skips teardown, which is why resources leak.

Actually

tearDown always runs after the test body, whether the test passed, failed an assertion, or raised an unexpected exception. The only lifecycle method that skips teardown when it itself fails is setUpClass: if setUpClass raises, tearDownClass is not called.

If setUp raises, does the test method run?
Most people say

Yes — the test runner calls setUp and then the test method regardless.

Actually

If setUp raises, the test method is not called and neither is tearDown. The test is marked as an error, not a failure. This means any resource allocated in setUp before the exception is leaked unless you use addCleanup.

Scenario question

You're handed a test suite of 400 tests that passes reliably when run with python -m unittest discover, but fails intermittently with pytest --randomly-seed=12345. Fix it without changing how the tests are structured.

Model answer

I'd start by running pytest -v --randomly-seed=12345 and collecting the first failing test. I'd then ask: what global or class-level state could test N-1 have mutated that test N depends on? The typical culprits are class-level lists, dicts, or database tables seeded in setUpClass. I'd grep for assignments to cls.* that happen in any method other than setUpClass. The fix is to move the mutable object into setUp, which runs once per test. The trade-off is a small overhead increase for cheap fixtures — acceptable for deterministic results.

Interview corner

CommonWhat is the difference between setUpClass and setUp?
“I'd say setUpClass runs once for the entire class and shares state across all tests in it, while setUp runs before every individual test method. I use setUpClass for expensive read-only setup — loading a fixture file, compiling a regex — and setUp for anything mutable, like a fresh database connection, because tests must not share mutable state.”
TrickIf setUp raises, is tearDown called?
“I've been caught by this: no, tearDown is not called if setUp raises. That means anything I allocate in setUp before the error leaks. The safe pattern is addCleanup — it registers a cleanup callable before anything can go wrong, and it always fires.”
CommonWhat does assertRaises actually check?
“I'd say assertRaises only checks the exception type, not the message. Two different code paths can both raise ValueError for completely different reasons, and assertRaises passes either way. I always follow it with self.assertIn('expected text', str(ctx.exception)), or I use assertRaisesRegex to check both type and message in one assertion.”
Experience checkHow would you structure fixtures for tests that share an expensive S3 download?
“I'd put the download in setUpClass and store the result on cls as a read-only string — no test method modifies it. Each setUp creates a fresh in-memory SQLite connection that processes a subset of that string. The expensive work happens once; isolation is enforced by making every mutable resource function-scoped.”

Memory hook

A test suite is like a hotel banquet: the kitchen is stocked once (setUpClass), each table gets a clean cloth before every course (setUp), the meal is served (the test), and the cloth is cleared away (tearDown). If setUp does not replace the cloth, the next test inherits the previous guest's mess.

If you remember only one thing:

Put every mutable fixture in setUp — never setUpClass — so tests can run in any order.

Self-check

  1. When is setUpClass called relative to setUp?

    setUpClass is called once before the first test method in the class; setUp is called before every test method.

  2. What happens if setUp raises?

    The test method is not called and tearDown is not called either. The test is reported as an error, not a failure.

  3. Can tearDown release resources safely even when the test fails?

    Yes. tearDown runs after every test method regardless of whether the test passed, failed an assertion, or raised an unexpected exception.

  4. What assertion method should you use to verify an exception message?

    Use with self.assertRaises(ExcType) as ctx: and then self.assertIn('expected text', str(ctx.exception)).

  5. What does iter(tc) is tc indicate about a test case?

    Nothing — that identity check applies to iterators (see Iterator protocol, exhaustion and pull model). A test case is not an iterator.

Prove it

Write a test class with a class-level mutable list initialised in setUpClass. Add two tests that each append to that list and assert len == 1. Run with python -m unittest (alphabetical order). Then swap the test method names so they run in reverse order. Paste the failure output showing which assertion breaks and at which count. Then move the list to setUp and confirm both orderings pass.

py-11-02Card 2 of 6

Parametrize meaningful boundary cases

EasyP2 · Very likelyKnow cold10 min
testingparametrizeboundarysubtestpytest

Why you care

A function that parses a payment amount from a CSV string can break in at least six distinct ways: empty string, whitespace-only, negative value, value with three decimal places, value that overflows int, and locale comma as decimal separator. If you write one test method per case your test file becomes 200 lines of near-identical boilerplate. If you write one test with if/elif branches you lose individual failure isolation — one failure hides the rest. Interviewers ask about parametrize specifically because it separates test logic from test data, which is the design discipline that makes boundary coverage tractable at scale.

Explain like I'm 5

Imagine a customs officer who has one rulebook and a queue of 50 passengers to check. Writing a separate rulebook for each passenger is absurd — the rulebook stays the same, only the passport changes. parametrize is the queue: you write the rulebook once and hand it a table of passports. Each row in the table becomes a separate test run, with its own pass or fail.

Core concept

pytest.mark.parametrize takes a parameter name (or comma-separated names) and a list of values (or tuples). Each entry becomes a separate test case. On failure, pytest reports the parameter values that failed, so you get one failure row, not one failed test that hides the rest.

Pythonneeds pytest
import pytest

@pytest.mark.parametrize('raw,expected', [
    ('',       0),
    ('0.00',   0),
    ('10.50',  1050),
    ('99.99',  9999),
])
def test_to_paise(raw, expected):
    assert to_paise(raw) == expected

unittest.subTest is the stdlib equivalent. It does not produce separate test items, but on failure it continues running the remaining cases and lists all failures at the end.

When to use which: use parametrize when you want each case to be a named, independently re-runnable test item (useful for CI that re-runs only failing tests). Use subTest when you are constrained to stdlib or need the cases to share setUp/tearDown state.

Boundary strategy — before writing test data, enumerate the regions around each decision point:

  • The boundary value itself (0, max)
  • One below (−1, max−1)
  • One above (1, max+1)
  • Empty input
  • Malformed input (wrong type, wrong format)
  • Null / None

Decision rule: if your function has N branches, you need at least N tests. If it parses external data, add boundary tests for every format the data can violate.

Minimal example subTest covers five boundary cases in one method

PythonEach case reported independently on failureruns
import unittest
import io


def to_paise(s):
    if not s:
        return 0
    try:
        return round(float(s) * 100)
    except ValueError:
        raise ValueError(f'invalid amount: {s!r}')


class AmountBoundaryTests(unittest.TestCase):
    def test_boundaries(self):
        cases = [
            ('',       0),
            ('0.00',   0),
            ('0.01',   1),
            ('10.50',  1050),
            ('99.99',  9999),
        ]
        for raw, expected in cases:
            with self.subTest(raw=raw):
                self.assertEqual(to_paise(raw), expected)

    def test_malformed_raises(self):
        with self.assertRaises(ValueError) as ctx:
            to_paise('abc')
        self.assertIn('invalid amount', str(ctx.exception))


buf = io.StringIO()
runner = unittest.TextTestRunner(stream=buf, verbosity=2)
suite = unittest.TestLoader().loadTestsFromTestCase(AmountBoundaryTests)
result = runner.run(suite)
print('tests run:', result.testsRun, 'failures:', len(result.failures))
Output
tests run: 2 failures: 0

In production CRED-style bill amount parser with boundary suite

PythonBoundary cases for Indian credit card bill amounts (paise)runs
import unittest
import io


def parse_bill_amount(raw):
    """
    Parse a credit card bill amount string to paise (int).
    Accepts '10.50', '1,200.00' (Indian locale commas removed).
    Returns 0 for empty strings. Raises ValueError for anything else.
    """
    if raw is None or not str(raw).strip():
        return 0
    cleaned = str(raw).replace(',', '')
    try:
        paise = round(float(cleaned) * 100)
    except ValueError:
        raise ValueError(f'unparseable bill amount: {raw!r}')
    if paise < 0:
        raise ValueError(f'negative bill amount: {raw!r}')
    return paise


class BillAmountTests(unittest.TestCase):
    _cases = [
        # (raw_input,          expected_paise,  label)
        (None,                  0,               'none'),
        ('',                    0,               'empty string'),
        ('   ',                 0,               'whitespace only'),
        ('0.00',                0,               'zero'),
        ('0.01',                1,               'one paise'),
        ('10.50',               1050,            'typical amount'),
        ('1,200.00',            120000,          'Indian locale comma'),
        ('99999999.99',         9999999999,      'large amount'),
    ]

    def test_all_valid_cases(self):
        for raw, expected, label in self._cases:
            with self.subTest(label=label, raw=raw):
                self.assertEqual(parse_bill_amount(raw), expected)

    def test_negative_raises(self):
        with self.assertRaises(ValueError) as ctx:
            parse_bill_amount('-10.00')
        self.assertIn('negative', str(ctx.exception))

    def test_malformed_raises(self):
        for bad in ['abc', '10.50.25', '1e308']:
            with self.subTest(bad=bad):
                with self.assertRaises((ValueError, OverflowError)):
                    parse_bill_amount(bad)


buf = io.StringIO()
runner = unittest.TextTestRunner(stream=buf, verbosity=2)
suite = unittest.TestLoader().loadTestsFromTestCase(BillAmountTests)
result = runner.run(suite)
print('tests run:', result.testsRun, 'failures:', len(result.failures))
Output
tests run: 3 failures: 0
Scenario A

In your world

The situation

You add a parse_bill_amount function to a payment pipeline. You write two tests: test_happy_path and test_empty. Both pass. Finance reports that bills with a comma-formatted amount (1,200.00) are silently loading as 1.20 rather than 1,200.00.

What actually breaks

The missing boundary case is the Indian locale number format. The function calls float('1,200.00') which raises ValueError, which is caught and returns 0, so the row is skipped entirely — no exception in CI.

What you'd do on Monday

Add raw.replace(',', '') before the float() call. Add a subTest case with '1,200.00' to the boundary table so the format is locked in and a regression reintroduces it immediately.

Scenario B

Imagine you work at CRED

12M bill payments per month

The numbers

12M payments ÷ 30 days ≈ 400,000/day; each payment carries one amount field. 200 boundary test cases × 5 ms each = 200 × 0.005 s = 1,000 ms = 1 s per full boundary-test run.

What the Scenario-A answer does here

With three hand-written test methods covering happy path, empty, and one error path, 6 distinct format variants go untested. At 400,000 payments per day, a 0.01% silent parse failure rate drops ~40 payments per day with no exception logged — green CI, wrong data in the DB.

What flips at this scale

Systematic boundary enumeration before writing code (negative, zero, one paise, locale formats, max value, malformed) turns a 6-item format table into 12 subTest cases. The test table becomes the spec: anyone adding a new input format must add a row to the table, and CI enforces it.

The trade-off you'd defend

I would defend a parametrized boundary table over a property-based generator (Introduce property-based testing) for this case, trading exhaustive random coverage for explicit, reviewable, and domain-meaningful examples that double as specification.

Illustrative; not CRED-published.

Gotchas

subTest continues after failure — not all failures are shown in short output

Symptom

You see one failure in the test summary but suspect more cases broke. Running in verbose mode shows 3 sub-test failures.

Cause

subTest collects all failures inside the with block but the short output only shows the first message at the top of the method.

Fix

Run python -m unittest -v or pytest -v to see all sub-test failures listed individually.

pytest.mark.parametrize on a class method requires the class decorator

Symptom

@pytest.mark.parametrize on a method inside a class does nothing — the class setUp runs but all values are ignored.

Cause

parametrize on a class applies to all methods. On an instance method it works only in pytest-collected class tests, but if the class also inherits from unittest.TestCase the parametrize decorator is silently ignored.

Fix

In pytest, use plain functions with fixtures rather than TestCase subclasses when you want parametrize. In unittest, use subTest.

Missing the expected exception type in assertRaises

Symptom

assertRaises(ValueError) passes even though the function raised TypeError.

Cause

assertRaises re-raises any exception not matching the named type; if the test catches Exception broadly elsewhere, mismatches slip through.

Fix

Keep the with assertRaises(...) block as tight as possible — only the one call that should raise, nothing else.

Tricky questions

Does subTest isolate failures from each other?
Most people say

Yes — if one subTest raises, the others do not run.

Actually

subTest is designed so that a failure inside the with block is recorded, and execution continues with the next iteration. All cases in the loop run; the test method is marked failed after collecting all sub-failures.

Can pytest.mark.parametrize pass multiple arguments?
Most people say

No — you can only pass one parameter at a time; multiple parameters need multiple decorators.

Actually

@pytest.mark.parametrize('a,b,expected', [(1, 2, 3), (0, 0, 0)]) passes a tuple per case; pytest unpacks it into the named arguments.

Scenario question

You're handed a discount_rate(tier, days_since_signup) function with four tier values and a days threshold of 90. You are asked to write a test suite that gives the team confidence before deploying a pricing change.

Model answer

I'd start by drawing the decision table: 4 tiers × 3 day-regions (< 90, exactly 90, > 90) = 12 base cases, plus one case for each invalid input (negative days, unknown tier). That gives 16 cases in a parametrized table. I'd implement them as subTest rows so every case is individually reported. The trade-off is that a table this specific is brittle — if the pricing rule changes, the table must be updated. That is the point: the table is the living specification of the pricing logic.

Interview corner

CommonWhat is the difference between subTest and writing separate test methods?
“I'd say subTest lets me express a table of inputs and expected outputs in a few lines and get independent failure reports for each row, while separate test methods repeat the same assertion code N times. The catch is that subTest failures all appear under one test method name in the summary, so I run in verbose mode to see which rows failed.”
TrickWill a pytest.mark.parametrize decorator work on a unittest.TestCase method?
“I've tried this: no. pytest collects TestCase classes but silently ignores the parametrize decorator on their methods. If I need parametrize, I write plain functions with fixtures. If I'm in a TestCase, I use subTest.”
CommonHow do you test that a function raises the right exception with the right message?
“I use with self.assertRaises(ValueError) as ctx: and then after the block I call self.assertIn('expected text', str(ctx.exception)). Just checking the type is not enough — two different code paths can raise the same type for different reasons.”

Memory hook

A customs officer checks 50 passports with one rulebook, not 50 rulebooks. parametrize (or subTest) is the queue: write the check once, hand it a table of cases, and every row gets its own pass or fail.

If you remember only one thing:

Put your boundary inputs in a table and loop them with subTest — one method, all edge cases, each failure reported separately.

Self-check

  1. What does subTest do when a case fails?

    It records the failure and continues running the remaining cases in the loop.

  2. How do you pass two parameters per case to pytest.mark.parametrize?

    Use a comma-separated string as the first argument and a list of tuples as the second: @pytest.mark.parametrize('a,b', [(1, 2), (3, 4)]).

  3. What are the six boundary regions to check for a parsed number field?

    Empty input, whitespace-only, zero, one unit, the maximum, and malformed (wrong type or format).

  4. Does pytest.mark.parametrize work on unittest.TestCase methods?

    No — pytest silently ignores the decorator on TestCase methods. Use subTest inside the class, or plain functions outside it.

Prove it

Write parse_bill_amount from the production block above. Add one intentionally wrong boundary case (e.g. ('1,200.00', 120) instead of 120000). Run python -m unittest -v. Paste the output showing the sub-test failure with the label and both values. Then fix the case and run again to confirm zero failures.

py-11-03Card 3 of 6

Mock the dependency where it is looked up

MediumP2 · Very likelyKnow cold12 min
testingmockingpatchwhere-to-patchunittest-mock

Why you care

Any test that calls a real database, a real S3 bucket, or a real HTTP endpoint is slow, unreliable, and cannot be run without network access. Removing that dependency is the point of mocking. The mistake that ruins half the mocks written by developers who have not studied the lookup rule is patching the original definition instead of the name the subject module actually uses. The result is a mock that does nothing — the real dependency executes, the test flaps, and nobody understands why. Interviewers ask "where do you patch?" because it is the single most diagnostic question for Python testing experience.

Explain like I'm 5

Imagine a stunt double: the director replaces an actor with the double, but only the actor's trailer. If the director only changes the sign on the actor's home studio three cities away, the on-set crew never meets the double. Python's patch replaces a name in a specific module's namespace — if you patch the wrong module, the subject code reads from its own copy of the name and never sees the replacement.

Core concept

unittest.mock.patch(target) replaces the object at target for the duration of the with block (or the decorated function). target is a dot-separated string: 'module.name'.

The rule that everything else follows: patch the name where it is looked up, not where it is defined. This rule is strict only for from x import name; with import x, patching the original also works because the attribute is looked up dynamically at call time.

  • If payments.py does import os and calls os.path.exists(...), patching either 'os.path.exists' or 'payments.os' works: payments.os is the same os module object, so replacing os.path.exists is visible at call time.
  • If payments.py does from os.path import exists and calls exists(...), you must patch 'payments.exists'. Patching 'os.path.exists' does nothing: from ... import bound the function object directly into payments.exists at import time. The patch replaces the name on os.path, but payments.exists still points to the original function.
  • The safe habit: always patch '<subject_module>.<local_name>'. It works for both import styles and never misfires.

unittest.mock.MagicMock auto-creates attributes and return values on access. assert_called_once_with(...) and call_args let you assert how the mock was called.

Dependency injection is cleaner than patching when you control the subject: if the function accepts s3_client=None as a parameter, passing a MagicMock() in the test requires no patch at all.

Decision rule: prefer dependency injection for new code; use patch when you cannot modify the subject.

Minimal example Patching the wrong location vs the right location

PythonWrong patch leaves real dependency active; right patch intercepts itruns
import types
import sys
from unittest.mock import patch

# checker.py does: from os.path import exists
checker = types.ModuleType('checker')
exec(
    'from os.path import exists\n'
    'def file_ok(p): return exists(p)',
    checker.__dict__,
)
sys.modules['checker'] = checker

# Wrong: the 'from ... import' already bound checker.exists to the real function.
# Patching os.path.exists after that binding is too late.
with patch('os.path.exists', return_value=True):
    result = checker.file_ok('/does/not/exist')
print('wrong patch (real function runs):', result)

# Right: patch the name in checker's own namespace.
with patch.object(checker, 'exists', return_value=True):
    result = checker.file_ok('/does/not/exist')
print('right patch (mock intercepts):', result)
Output
wrong patch (real function runs): False
right patch (mock intercepts): True

In production S3 download count via dependency injection

PythonNo patch needed when the dependency is injected as a parameterruns
import io
import unittest
from unittest.mock import MagicMock


def download_csv(s3_client, bucket, key):
    """Download an S3 object and return its body as a string."""
    obj = s3_client.get_object(Bucket=bucket, Key=key)
    return obj['Body'].read().decode()


def count_settlement_rows(s3_client, bucket, key):
    """Returns the number of data rows (header excluded) in a CSV object."""
    body = download_csv(s3_client, bucket, key)
    lines = [l for l in body.strip().splitlines() if l]
    return max(0, len(lines) - 1)


class S3IngestionTests(unittest.TestCase):
    def _make_s3_mock(self, csv_body):
        m = MagicMock()
        m.get_object.return_value = {
            'Body': MagicMock(read=lambda: csv_body.encode()),
        }
        return m

    def test_count_rows(self):
        fake_csv = 'id,amount\nS001,10.50\nS002,200.00\nS003,5.00\n'
        s3 = self._make_s3_mock(fake_csv)
        n = count_settlement_rows(s3, 'raw-data', 'settlements/2024-06-15.csv')
        self.assertEqual(n, 3)
        s3.get_object.assert_called_once_with(
            Bucket='raw-data', Key='settlements/2024-06-15.csv'
        )

    def test_empty_file(self):
        fake_csv = 'id,amount\n'
        s3 = self._make_s3_mock(fake_csv)
        n = count_settlement_rows(s3, 'raw-data', 'settlements/empty.csv')
        self.assertEqual(n, 0)


buf = io.StringIO()
runner = unittest.TextTestRunner(stream=buf, verbosity=2)
suite = unittest.TestLoader().loadTestsFromTestCase(S3IngestionTests)
result = runner.run(suite)
print('tests run:', result.testsRun, 'failures:', len(result.failures))
Output
tests run: 2 failures: 0
Scenario A

In your world

The situation

You add a test for a function that sends a Slack alert when a settlement file is missing. The function does from requests import post at the top of the module. You patch requests.post and assert it was called.

What actually breaks

The test passes locally but in CI the real HTTP request to api.slack.com is made instead of the mock. The mock assertion also passes because requests.post was replaced — but the subject's local post name was already bound to the real function at import time.

What you'd do on Monday

Patch 'your_module.post' — the name post inside the module where the function lives — not 'requests.post'. The rule: find the import statement in the subject file; the target is '<subject_module>.<local_name>'.

PythonPatch the module-local name, not the definitionruns
import types
import sys
from unittest.mock import patch, MagicMock

# Install a fake requests BEFORE exec so the 'from requests import post' succeeds.
fake_requests = types.ModuleType('requests')
fake_requests.post = lambda url, **kw: None
sys.modules['requests'] = fake_requests

# Simulate alert_module: from requests import post
alert_module = types.ModuleType('alert_module')
exec(
    'from requests import post\n'
    'def send_alert(msg): return post("http://slack/", json={"text": msg}).status_code',
    alert_module.__dict__,
)
sys.modules['alert_module'] = alert_module

# Patch the local name 'post' in alert_module's namespace
with patch.object(alert_module, 'post') as mock_post:
    mock_post.return_value = MagicMock(status_code=200)
    code = alert_module.send_alert('settlement file missing')
print('status code returned:', code)
print('post called with text:', mock_post.call_args.kwargs['json']['text'])
Output
status code returned: 200
post called with text: settlement file missing
Scenario B

Imagine you work at Paytm

800 unit tests against a payment gateway

The numbers

800 tests × 450 ms average per real HTTP call to the payment gateway = 800 × 0.45 s = 360 s = 6 min per CI run. With mocked HTTP: 800 × 3 ms = 2,400 ms = 2.4 s.

What the Scenario-A answer does here

Each test hits the real sandbox endpoint. Beyond the 6-minute penalty, the sandbox rate-limits after ~100 calls per minute, so the CI run begins failing with HTTP 429 after the first 100 tests — even though the function under test is correct. Green or red is now determined by the sandbox, not the code.

What flips at this scale

The payment-gateway call is extracted behind an injectable gateway_client parameter. Tests pass in a MagicMock(). Only the integration test (a single test, run on a separate nightly schedule, not in PR CI) touches the real sandbox. The design change is forced by the scale — you cannot afford 800 real HTTP calls in every developer's CI run.

The trade-off you'd defend

I would defend dependency injection over patch-based mocking for new code, trading a small design discipline (every external call is an explicit parameter) for tests that are faster, more portable, and require no knowledge of the patch-location rule.

Illustrative; not Paytm-published.

Gotchas

Patching from module import name at the wrong target

Symptom

The patch decorator is applied but the real function executes — you can tell because the assertion mock.assert_called_once() passes but the real side effect (a file written, an HTTP call made) also happens.

Cause

from module import name binds the function object directly into the subject module's namespace. Patching the original module after that binding has no effect on the subject.

Fix

Patch 'subject_module.name', not 'original_module.name'.

patcher.start() in setUp without addCleanup leaks the mock across tests

Symptom

Every test in the class — including tests that should not touch the mock — sees the patched version. After the class finishes, tests in other classes also see it.

Cause

with patch(...) always restores in __exit__ even if the test raises — that is not the trap. The trap is patcher.start() in setUp without a matching stop. If you call start() and tearDown is never reached (or is simply missing the stop call), the patch remains active for the rest of the process.

Fix

Pair every patcher.start() with self.addCleanup(patcher.stop) on the very next line. addCleanup runs unconditionally after every test, even if setUp raises partway through.

PythonaddCleanup guarantees patcher.stop() even if setUp raises
def setUp(self):
    patcher = patch('mymodule.boto3')
    self.mock_boto3 = patcher.start()
    self.addCleanup(patcher.stop)   # runs unconditionally

Asserting the wrong call signature

Symptom

assert_called_once_with(bucket, key) raises AssertionError even though you saw the function called in the debugger.

Cause

The mock was called with keyword arguments and you asserted positional, or vice versa. call('a', key='b') does not equal call('a', 'b').

Fix

Inspect mock.call_args and mock.call_args_list before writing the assertion.

Tricky questions

If you patch os.path.exists at the top level of a test module, does it affect all tests?
Most people say

Yes — a module-level patch applies to all tests in the file.

Actually

Only if you called patcher.start() without a matching patcher.stop(). A with patch(...) block or a test decorator scopes the patch to that block or function. A module-level patcher.start() without addCleanup(patcher.stop) leaks the patch and affects every subsequent test in the process.

Can you use patch as a decorator and also get the mock object in setUp?
Most people say

No — decorators are defined before setUp runs, so the mock is not available there.

Actually

Use self.patcher = patch('module.name') in setUp, call self.mock_obj = self.patcher.start(), and call self.patcher.stop() in tearDown. That makes the mock available as an instance attribute throughout the test.

Scenario question

You're handed a function fetch_orders(s3_key) that creates a boto3 client internally and calls s3.get_object. You need to write a test that verifies get_object is called with the exact bucket and key. You cannot modify fetch_orders. How do you write the test?

Model answer

I'd start by reading the import statement in fetch_orders's module. If it does import boto3 and calls boto3.client('s3'), I patch '<module>.boto3' and configure mock_boto3.client.return_value.get_object.return_value to a fake body. That chain is four attribute accesses — .client, .return_value, .get_object, .return_value — but MagicMock builds each child lazily, so the assignment reads like normal attribute access. The trade-off is that this mock mirrors the internal call structure of fetch_orders; if the function switches to a different client factory, the chain breaks and the test must be updated.

Interview corner

CommonWhere should you patch a dependency in Python?
“I'd say: patch it where it is looked up, not where it is defined. If the subject module does from os.path import exists, I patch 'subject_module.exists'. Patching 'os.path.exists' after a from-import has already bound the name is too late.”
TrickA test patches requests.get but the real HTTP call still fires. What went wrong?
“I've hit this: the subject module did from requests import get, which bound the name get directly. The patch on requests.get changed the attribute on the requests module, but the subject never reads from requests.get again — it reads from its own local name. The fix is to patch 'subject_module.get'.”
Experience checkHow do you write a reusable mock for an S3 client that multiple tests share?
“I'd put it in setUp: self.mock_s3 = MagicMock() and configure self.mock_s3.get_object.return_value with a factory helper. If setUp needs to reset it per test, I call self.mock_s3.reset_mock() at the top of setUp. I prefer dependency injection so no patch is needed at all.”
DesignWhen would you choose dependency injection over patch?
“I'd choose injection for any new code I'm writing — it makes the dependency explicit in the signature, requires no knowledge of the patch-location rule, and the test reads like normal function calls. I use patch when I cannot modify the subject code, or when the dependency is buried three call levels deep and injecting it would require threading a parameter through unrelated functions.”

Memory hook

A stunt double only works if hired for the right on-set crew, not the actor's home studio. Patch the name in the module that calls it, not the module that defines it.

If you remember only one thing:

Patch 'subject_module.local_name' — always the import site, never the definition site.

Self-check

  1. A module does from json import loads. Where do you patch loads?

    At 'the_module.loads', not 'json.loads'.

  2. What does patch.object(module, 'name') do that patch('module.name') also does?

    Both replace the named attribute on the module object for the duration of the context. patch.object takes the module object directly; patch resolves the dotted string. They are equivalent in effect.

  3. What is mock.assert_called_once_with(Bucket='b', Key='k') checking?

    That the mock was called exactly once, with no positional arguments, and the keyword arguments Bucket and Key had those exact values.

  4. What does MagicMock() return when you access an attribute that was never set?

    A new MagicMock instance. Every attribute access on a MagicMock auto-creates a child mock.

Prove it

Write a module notifier.py containing from os.path import exists and a function log_if_missing(path) that returns 'missing' if exists(path) returns False. Write two tests: one that patches os.path.exists (the wrong location) and one that patches notifier.exists (the right location). Run both. Paste the output showing that the first mock has no effect and the second does. Time both runs: they should differ by at most a few milliseconds, confirming no real filesystem call occurs in the second test.

py-11-04Card 4 of 6

Control time and interpret coverage

MediumP3 · Role-dependentKnow cold11 min
testingtimeclock-injectioncoveragefreezegun

Why you care

Any function that calls datetime.now() internally cannot be tested with a fixed expected output: the result changes every second. The usual workaround — patching datetime.datetime.now — works but is fragile because datetime is a C-extension type and some patch targets do not survive module caching. The robust solution is clock injection: pass the time source as a parameter. This card also covers coverage: running tests that exercise every line is not the goal; running tests that exercise every meaningful decision is. Interviewers ask about time in tests because flaky date-dependent tests are a recurrent CI headache, and knowing the fix separates engineers who have maintained test suites from those who have not.

Explain like I'm 5

Imagine a film director who wants to test how actors behave at midnight. The director does not wait for midnight — they bring a clock prop and set it to 12:00. Clock injection is that prop: you hand the function a fake clock that says whatever the test needs, and the function never knows the difference.

Core concept

Clock injection is the simplest approach: instead of datetime.now() inside the function, accept a clock callable as a parameter with a sensible default.

Pythonclock parameter with stdlib default
from datetime import datetime, timezone

def days_overdue(due_date, clock=None):
    if clock is None:
        clock = lambda: datetime.now(timezone.utc)
    return max(0, (clock() - due_date).days)

In tests, pass clock=lambda: fixed_dt — no patch required, deterministic, and readable.

Patching datetime via unittest.mock.patch works for datetime.datetime but not for datetime.datetime.now directly, because datetime.datetime is implemented in C and its methods are not replaceable as plain attributes. The safe target is the module binding: patch('mymodule.datetime').

freezegun (pip install freezegun) is a third-party library that patches all datetime, date, time, time.time() and time.localtime() calls globally. It is the most ergonomic option but requires installation. Examples in this card use norun for freezegun.

Coverage measures which lines executed during a test run. coverage run -m pytest then coverage report show the percentage and the uncovered line numbers. A high coverage percentage is not a goal — it is a proxy. You can have 100% line coverage with tests that make no assertions. What matters is branch coverage: every if/else path, every except branch, and every return None path is exercised. Run coverage report --branch (or coverage html) to see which branches are uncovered.

Decision rule: use clock injection for new code. Use freezegun for large existing codebases where injection would require threading a parameter through many layers.

Minimal example Clock injection gives deterministic output

PythonFixed clock makes the result identical every runruns
from datetime import datetime, timezone


def days_overdue(due_date, clock=None):
    if clock is None:
        clock = lambda: datetime.now(timezone.utc)
    delta = (clock() - due_date).days
    return delta if delta > 0 else 0


# Fixed 'today': 2024-06-15
# due:           2024-05-10
# May 10 -> May 31 = 21 days; June 1 -> June 15 = 15 days; total = 36 days
fixed_now = datetime(2024, 6, 15, tzinfo=timezone.utc)
due_may   = datetime(2024, 5, 10, tzinfo=timezone.utc)

print(days_overdue(due_may, clock=lambda: fixed_now))

# Not yet due
due_future = datetime(2024, 7, 1, tzinfo=timezone.utc)
print(days_overdue(due_future, clock=lambda: fixed_now))
Output
36
0

In production Meesho EMI due-date reminder with injected clock

PythonAll four branches exercised; output is deterministicruns
import unittest
import io
from datetime import datetime, timezone, timedelta


def emi_reminder(emi_due_date, clock=None):
    """
    Returns a reminder severity string based on days until EMI is due.
    clock: callable -> datetime (UTC-aware). Defaults to datetime.now.
    """
    if clock is None:
        clock = lambda: datetime.now(timezone.utc)
    today = clock()
    days_left = (emi_due_date - today).days
    if days_left < 0:
        return f'OVERDUE: {-days_left} day(s)'
    if days_left == 0:
        return 'DUE TODAY'
    if days_left <= 3:
        return f'DUE SOON: {days_left} day(s)'
    return f'OK: {days_left} day(s)'


class EmiReminderTests(unittest.TestCase):
    def setUp(self):
        self.now = datetime(2024, 3, 15, 9, 0, 0, tzinfo=timezone.utc)
        self.clock = lambda: self.now

    def test_overdue(self):
        due = self.now - timedelta(days=5)
        self.assertEqual(emi_reminder(due, self.clock), 'OVERDUE: 5 day(s)')

    def test_due_today(self):
        self.assertEqual(emi_reminder(self.now, self.clock), 'DUE TODAY')

    def test_due_soon(self):
        due = self.now + timedelta(days=2)
        self.assertEqual(emi_reminder(due, self.clock), 'DUE SOON: 2 day(s)')

    def test_ok(self):
        due = self.now + timedelta(days=10)
        self.assertEqual(emi_reminder(due, self.clock), 'OK: 10 day(s)')


buf = io.StringIO()
runner = unittest.TextTestRunner(stream=buf, verbosity=2)
suite = unittest.TestLoader().loadTestsFromTestCase(EmiReminderTests)
result = runner.run(suite)
print('tests run:', result.testsRun, 'failures:', len(result.failures))
Output
tests run: 4 failures: 0
Scenario A

In your world

The situation

You have a cron job that sends an SMS reminder to customers whose EMI is due within three days. It calls datetime.now() directly. You write three tests covering DUE TODAY, DUE SOON, and OK. They pass when you run them on March 15. On March 31 two of them fail; on February 28 in a non-leap year, one more fails.

What actually breaks

The tests use datetime.today() + timedelta(days=2) to build the expected due date relative to the real system clock. When the real clock changes (month boundary, year end, leap year), the delta arithmetic lands in the wrong branch.

What you'd do on Monday

Extract the now call behind a clock parameter. In tests, pass clock=lambda: datetime(2024, 3, 15, tzinfo=timezone.utc). The test is now frozen to a specific date and will produce the same output in 2024, 2025, and 2034.

Scenario B

Imagine you work at Meesho

5M EMI schedules in flight

The numbers

5M active EMI schedules × 1 due-date check per night = 5M evaluations per run. 120 unit tests covering the reminder logic × 10 ms each = 1,200 ms = 1.2 s per test run.

What the Scenario-A answer does here

Without clock injection, the 120 tests pass on most days but fail on the last day of a 28-day February (off-by-one in days_left calculation) and on December 31 (year rollover in date arithmetic). CI runs at midnight UTC, which is 5:30 AM IST — close to the daily batch start. A flaky test on those days is silently retried, masking a real bug in the reminder logic that sends wrong-severity SMSs to 50,000 customers on the affected day.

What flips at this scale

Clock injection makes every test date explicit. A boundary suite adds cases for 2024-02-28, 2024-02-29 (leap), 2024-03-01, and 2024-12-31. The flip is not a performance change — it is a correctness change: the test suite can now say "this function is correct on every date we care about" rather than "this function was correct on the day we ran tests".

The trade-off you'd defend

I would defend clock injection over freezegun, trading the ergonomic convenience of a global time freeze for a dependency-free approach that works the same way as mocking any other external dependency.

Illustrative; not Meesho-published.

Clock injection vs freezegun vs patch

Clock injectionunittest.mock.patchfreezegun
Stdlibyesyesno (third-party)
Patches all datetime calls globallynowith careyes
Test reads likeregular function calldecorator/contextdecorator/context
Requires design change to subjectyesnono
Reliable on C-extension datetimeyesfragileyes
Best fornew codelegacy, single call sitelegacy, many call sites

Gotchas

Patching datetime.datetime.now directly fails silently

Symptom

patch('datetime.datetime.now', return_value=fixed) is applied, but the function still returns the real current time.

Cause

datetime.datetime is a C-extension type. Its methods cannot be replaced by the default patch mechanism in Python 3.12; the attribute assignment silently fails or is overridden by the C implementation.

Fix

Patch the module binding instead: patch('mymodule.datetime', wraps=...). Or use clock injection. Or use freezegun.

Coverage 100% with no assertions — the green lie

Symptom

coverage report shows 100% but a function has a silent wrong-return path.

Cause

Coverage measures line execution, not assertion correctness. A test that calls every line but never asserts on the return value can achieve 100% while exercising nothing meaningful.

Fix

Run coverage report --branch to find uncovered branches. Write assertions, not just calls. A test without an assertion is a smoke test, not a unit test.

Timezone-naive and timezone-aware datetimes mixed in arithmetic

Symptom

TypeError: can't subtract offset-naive and offset-aware datetimes at runtime on production, not in tests.

Cause

Tests construct fixtures with datetime(2024, 1, 1) (naive) but production passes datetime.now(timezone.utc) (aware). Arithmetic between the two raises at runtime.

Fix

Always construct test datetimes with tzinfo=timezone.utc when the subject is timezone-aware. Add a test case that passes a naive datetime and asserts a TypeError is raised.

Tricky questions

Does 100% line coverage mean your tests will catch a regression?
Most people say

Yes — if every line runs, every bug is covered.

Actually

No. Coverage measures execution, not correctness. A function can return the wrong value on every call and still achieve 100% line coverage if no test asserts on the return value. Branch coverage is more useful than line coverage, and meaningful assertions are more useful than either.

Will with patch('datetime.datetime', ...) work in Python 3.12?
Most people say

No — datetime.datetime is a C type, so any patch target on it fails silently.

Actually

Patching the module-level binding ('mymodule.datetime') works because you are replacing the name in the subject module's namespace, not the C type itself. The subject then calls self.datetime.now() against the mock. Clock injection avoids the whole issue.

Scenario question

You're handed a billing function compute_late_fee(invoice_date) that calls datetime.date.today() internally. You cannot modify the function. How do you write a test that fixes the date to 2024-06-30?

Model answer

I'd use patch('billing_module.datetime') and configure the mock so mock_datetime.date.today.return_value = date(2024, 6, 30). I'd also set mock_datetime.date.side_effect = datetime.date in case the function uses datetime.date(year, month, day) constructors internally — without that, any direct datetime.date() call on the mock returns another mock instead of a real date. The trade-off is fragility: if the module imports from datetime import date, I need to patch 'billing_module.date' instead, following the patch-location rule from Mock the dependency where it is looked up.

Interview corner

CommonHow do you test a function that calls datetime.now() internally?
“I'd extract the call behind a clock parameter — the function accepts clock=None, defaults to datetime.now, and every test passes a fixed lambda. No patch needed, no fragility with C extensions. If I can't modify the function, I use freezegun or patch the module binding.”
TrickWhat does 100% coverage tell you about the quality of a test suite?
“I'd say it tells you which lines executed, not which behaviours are verified. I've seen suites with 100% coverage that had zero meaningful assertions — every line ran, nothing was checked. I care more about branch coverage (every if/else path exercised) and assertion density (every test has at least one assertEqual or assertRaises).”
Experience checkYour test suite has a test that passes every day except the last day of February. What is the root cause?
“I've debugged this: the test constructs a relative date (today + timedelta(30)) using the real system clock. On Feb 28 or 29, adding 30 days lands in a different month than the test expects. The fix is clock injection — freeze the test date to a specific day and make the 28/29/31-day boundary cases explicit subTest rows.”

Memory hook

A film director does not wait for midnight to test the midnight scene — they bring a clock prop. Hand the function a clock parameter, pass a fixed lambda in tests, and the result is the same in every timezone and every year.

If you remember only one thing:

Accept a clock parameter; in tests, pass lambda: datetime(2024, 3, 15, tzinfo=timezone.utc).

Self-check

  1. Why does patching datetime.datetime.now fail in Python 3.12?

    Because datetime.datetime is a C-extension type; its method attributes cannot be replaced by the normal attribute-assignment patch mechanism.

  2. What is the difference between line coverage and branch coverage?

    Line coverage counts which lines executed. Branch coverage counts which decision paths (both sides of every if/else) executed. A function with an else branch that returns wrong data has full line coverage but only 50% branch coverage.

  3. What is the minimum change to make a date-dependent function testable without patch or freezegun?

    Add a clock=None parameter; inside the function, default it to lambda: datetime.now(timezone.utc). Tests pass a fixed lambda.

  4. Does a test without an assertion count toward branch coverage?

    Yes — lines execute whether or not assertions follow. But a test without assertions does not verify behaviour; it is a smoke test.

Prove it

Write emi_reminder from the production block. Run python -m coverage run -m unittest (install coverage first: pip install coverage). Then run python -m coverage report --branch. Paste the output. Identify one uncovered branch (if any). Add the missing test case and confirm coverage improves.

py-11-05Card 5 of 6

Test an ETL job through failure and replay

HardP2 · Very likelyKnow cold14 min
testingetlpure-transformidempotencyintegration-boundaryreplay

Why you care

An ETL job that has no tests is not untested — it is tested manually, by finance, after midnight. The specific failure mode for ETL is silent partial load: 48,000 rows processed, then a crash, 24,000 rows in the database, no exception in Jenkins, no audit entry flagging the gap. A test that catches this must model the ETL as two separable things: a pure transform (input CSV → list of typed rows, no I/O) and a loader (typed rows → database, no transformation). The pure transform is trivially testable with plain Python and no mocks. The loader boundary is tested with sqlite3 as a stand-in for Postgres. This card is hard because it requires decomposing code that most ETL scripts do not separate. Interviewers ask this because ETL testing is the largest quality gap in backend engineers coming from scripting backgrounds.

Explain like I'm 5

A crash-test engineer does not drive the car into a wall while the passengers are inside. They build a sled that carries a dummy, run it into a controlled barrier, and measure the result. The ETL equivalent: the "sled" is transform_settlements, which takes a CSV string and returns a list — no I/O, no crash risk. The "barrier" test is the loader, which accepts the list and writes to sqlite3. Each component is tested independently; the crash is simulated by injecting a failure mid-load and verifying the database ends up in the expected state.

Core concept

Decompose into pure transform + loader:

  1. transform(csv_body) -> list[tuple] — reads CSV, parses fields, validates, returns typed rows. No database, no S3. Completely deterministic. Testable with assertEqual on the output list.
  2. load(rows, conn) -> int — accepts the list, writes to database. No parsing. Testable with sqlite3 in memory.

Idempotency means running the load twice with the same input leaves the database in the same state as running it once. For ETL this is the replay guarantee: if the job crashes and restarts, it must not duplicate rows. Use INSERT OR REPLACE (SQLite) or INSERT ... ON CONFLICT DO UPDATE (Postgres) with a primary key on the natural identifier.

Integration boundary is where you mock. The S3 download is the only external dependency. If the function accepts an s3_client parameter, you pass a MagicMock in the test without patching. The database is replaced by sqlite3.connect(':memory:').

Pure transform tests are the highest-value tests in an ETL suite: they run in microseconds, need no mocks, and cover every data validation rule. Test them first.

Decision rule: if a function does both I/O and transformation, split it. transform functions that call open() or boto3 cannot be tested without mocks; functions that accept a string or list can be tested with a one-liner.

Under the hood

When sqlite3.executemany(sql, rows) runs, CPython passes all rows to the C-level SQLite library in one call: the GIL is held for the duration but no Python frame is entered per row. This is why executemany is dramatically faster than execute in a Python loop for bulk inserts — 1,000 execute calls = 1,000 Python/C boundary crossings; one executemany call = 1. For the test, this means your in-memory sqlite3 load is not meaningfully slower than a real Postgres COPY for small fixture sets.

The other CPython detail that matters for ETL testing: csv.DictReader is a lazy iterator (Iterator protocol, exhaustion and pull model). Calling list() on it inside transform is intentional — it materialises all rows before returning, which means the transform either succeeds completely or raises before any row is returned. The loader never receives a partial list.

Minimal example Pure transform + idempotent loader, all in memory

PythonTransform first, load second; second load leaves the same row countruns
import csv
import io
import sqlite3


def transform_settlements(csv_body):
    """Pure: CSV string -> list of (settlement_id, paise) tuples."""
    rows = []
    for row in csv.DictReader(io.StringIO(csv_body)):
        sid = (row.get('settlement_id') or '').strip()
        if not sid:
            continue
        rows.append((sid, round(float(row['amount']) * 100)))
    return rows


def load_rows(rows, conn):
    """Idempotent: INSERT OR REPLACE. Returns number of rows written."""
    conn.execute(
        'CREATE TABLE IF NOT EXISTS settlements '
        '(settlement_id TEXT PRIMARY KEY, paise INT)'
    )
    conn.executemany(
        'INSERT OR REPLACE INTO settlements VALUES (?,?)', rows
    )
    conn.commit()
    return len(rows)


CSV = 'settlement_id,amount\nS001,10.50\nS002,200.00\nS003,0.75\n'

conn = sqlite3.connect(':memory:')
rows = transform_settlements(CSV)
print('transformed:', rows)

n1 = load_rows(rows, conn)
n2 = load_rows(rows, conn)   # idempotency: second run
count = conn.execute('SELECT COUNT(*) FROM settlements').fetchone()[0]
print('loaded first:', n1, 'loaded second:', n2, 'db count:', count)
Output
transformed: [('S001', 1050), ('S002', 20000), ('S003', 75)]
loaded first: 3 loaded second: 3 db count: 3

In production Full ETL test suite with partial-failure case

PythonFour tests: happy path, idempotency, missing id, bad amountruns
import csv
import io
import sqlite3
import unittest


def parse_paise(s):
    if not s or not s.strip():
        raise ValueError(f'empty amount: {s!r}')
    return round(float(s) * 100)


def transform_settlements(csv_body):
    """Pure transform. Raises ValueError on malformed amount."""
    rows = []
    for row in csv.DictReader(io.StringIO(csv_body)):
        sid = (row.get('settlement_id') or '').strip()
        if not sid:
            continue
        rows.append((sid, parse_paise(row.get('amount', ''))))
    return rows


def load_rows(rows, conn):
    """Idempotent loader. Returns number of rows written."""
    conn.execute(
        'CREATE TABLE IF NOT EXISTS settlements '
        '(settlement_id TEXT PRIMARY KEY, paise INT)'
    )
    conn.executemany(
        'INSERT OR REPLACE INTO settlements VALUES (?,?)', rows
    )
    conn.commit()
    return len(rows)


class ETLTests(unittest.TestCase):
    CSV_GOOD = (
        'settlement_id,amount\n'
        'S001,10.50\n'
        'S002,200.00\n'
        'S003,0.75\n'
    )
    CSV_GAP = (
        'settlement_id,amount\n'
        'S001,10.50\n'
        ',200.00\n'
        'S003,0.75\n'
    )

    def setUp(self):
        self.conn = sqlite3.connect(':memory:')

    def tearDown(self):
        self.conn.close()

    def test_happy_path(self):
        rows = transform_settlements(self.CSV_GOOD)
        n = load_rows(rows, self.conn)
        self.assertEqual(n, 3)
        paise = self.conn.execute(
            'SELECT paise FROM settlements WHERE settlement_id=?', ('S001',)
        ).fetchone()[0]
        self.assertEqual(paise, 1050)

    def test_idempotent(self):
        rows = transform_settlements(self.CSV_GOOD)
        load_rows(rows, self.conn)
        load_rows(rows, self.conn)
        count = self.conn.execute(
            'SELECT COUNT(*) FROM settlements'
        ).fetchone()[0]
        self.assertEqual(count, 3)  # no duplicates after replay

    def test_skips_missing_id(self):
        rows = transform_settlements(self.CSV_GAP)
        self.assertEqual(len(rows), 2)

    def test_bad_amount_raises(self):
        with self.assertRaises(ValueError) as ctx:
            transform_settlements('settlement_id,amount\nS001,\n')
        self.assertIn('empty amount', str(ctx.exception))


buf = io.StringIO()
runner = unittest.TextTestRunner(stream=buf, verbosity=2)
suite = unittest.TestLoader().loadTestsFromTestCase(ETLTests)
result = runner.run(suite)
print('tests run:', result.testsRun, 'failures:', len(result.failures))
Output
tests run: 4 failures: 0

See it happen ETL decomposed into testable layers

real S3test mockcsv_body (str)rows (list)executemanytest connS3 CSV filedownload_csv(s3_client, bucket, key)transform_settlements(csv_body) -> listload_rows(rows, conn) -> intPostgres / SQLiteMagicMocks3_client':memory:'conn
  1. In production, S3 supplies the CSV body via download_csv.
  2. In the unit test, download_csv is bypassed: transform receives a fixture string directly.
  3. The load test injects a MagicMock for s3_client and ':memory:' for the DB connection.
  4. The idempotency test calls load_rows twice with the same rows list and checks that the DB count does not grow.
Notice: the transform function and the loader function each have exactly one test dependency — a string and a DB connection respectively. Neither requires a live S3 bucket or Postgres instance.
Scenario A

In your world

The situation

Your Jenkins job processes 48,000 settlement rows. It reads from S3 in one loop and inserts into Postgres in the same loop. If an insert fails at row 24,000, the job exits, the partial data is in Postgres, and Jenkins marks the run green because the exception was caught and logged.

What actually breaks

Finance asks why yesterday's settlement total is half the expected amount. The audit table shows the run completed. No test exists to verify that either the full data is loaded or none of it is.

What you'd do on Monday

Split into transform + load. Wrap the load in a transaction: if it raises, roll back entirely. Add a test that asserts: after a simulated crash mid-load (pass only half the rows), the DB either contains all rows (complete) or zero rows (rolled back).

PythonTransaction rollback on partial failureruns
import sqlite3


def load_rows_atomic(rows, conn):
    """Load all rows or none: atomic transaction."""
    conn.execute(
        'CREATE TABLE IF NOT EXISTS settlements '
        '(settlement_id TEXT PRIMARY KEY, paise INT)'
    )
    try:
        conn.executemany(
            'INSERT OR REPLACE INTO settlements VALUES (?,?)', rows
        )
        conn.commit()
        return len(rows)
    except Exception:
        conn.rollback()
        raise


conn = sqlite3.connect(':memory:')
good_rows = [('S001', 1050), ('S002', 20000), ('S003', 75)]
print('loaded:', load_rows_atomic(good_rows, conn))

# Bad batch: third row has wrong arity — executemany raises ProgrammingError
# mid-batch; the except clause rolls back all three rows.
conn2 = sqlite3.connect(':memory:')
bad_rows = [('S001', 1050), ('S002', 20000), ('S003',)]   # S003 missing paise
try:
    load_rows_atomic(bad_rows, conn2)
except Exception as e:
    count = conn2.execute('SELECT COUNT(*) FROM settlements').fetchone()[0]
    print(f'rollback triggered ({type(e).__name__}): rows in DB = {count}')
Output
loaded: 3
rollback triggered (ProgrammingError): rows in DB = 0
Scenario B

Imagine you work at Chargebee

200k invoice lines per run

The numbers

200,000 invoice lines × 250 bytes/row = 200,000 × 250 = 50,000,000 bytes = 50 MB per batch. With a 1% partial-failure rate (network glitch mid-load): 0.01 × 200,000 = 2,000 rows in an inconsistent state per incident, undiscoverable without an idempotency test.

What the Scenario-A answer does here

The transform + load split and the idempotent loader both work at this scale. What breaks is the test assumption that :memory: is equivalent to Postgres: sqlite3 does not enforce the same type coercions, does not run triggers or check constraints as strictly, and does not reproduce connection pool exhaustion. A test suite that passes in sqlite3 can still miss Postgres-specific failures.

What flips at this scale

At 50 MB per batch, the integration test must run against a real Postgres instance (Docker in CI), not sqlite3. The unit tests (pure transform) stay in-memory and fast. The integration test runs once per PR, not per commit. The flip is not in the ETL logic — it is in where the test boundary sits: unit tests own correctness, integration tests own compatibility.

The trade-off you'd defend

I would defend running unit tests (pure transform) in every CI run and integration tests (Postgres) in a separate CI stage, trading a slightly slower PR pipeline for the confidence that type coercions and constraint violations are actually tested.

Illustrative; not Chargebee-published.

Gotchas

Transform and load mixed in one function makes replay untestable

Symptom

To test the ETL, you need a real S3 bucket with a specific file. The test suite cannot run offline.

Cause

The function calls boto3.client('s3').get_object(...) and psycopg2.connect(...) inside the same loop.

Fix

Extract transform(csv_body) as a pure function. Test it with a fixture string. Mock only the I/O layer.

sqlite3 passes tests that Postgres rejects

Symptom

All tests pass in CI but the job crashes on Postgres with DataError: invalid input syntax for type integer.

Cause

sqlite3 accepts any value in an INT column (dynamic typing). Postgres enforces the column type strictly.

Fix

Use float or int Python types when constructing rows, not strings. Or run the integration test against a real Postgres instance (Docker in CI).

INSERT without ON CONFLICT duplicates rows on replay

Symptom

After a crash and replay, the settlement count is double the expected value.

Cause

A plain INSERT does not check for existing rows with the same primary key.

Fix

Use INSERT OR REPLACE (SQLite) or INSERT INTO ... ON CONFLICT (settlement_id) DO UPDATE SET paise = EXCLUDED.paise (Postgres). Add an idempotency test that loads the same data twice and asserts the count equals the unique-row count.

Tricky questions

Is an ETL job idempotent by default?
Most people say

Yes — inserting the same rows twice just re-inserts the same data, which is harmless.

Actually

No. A plain INSERT raises a unique constraint violation if the row already exists. Even if the database is lenient, inserting the same settlement row twice doubles the amount in an aggregate. Idempotency requires explicit ON CONFLICT handling.

Should the transform function validate data types or just parse them?
Most people say

Validation belongs in the loader, which knows the DB schema.

Actually

Validation belongs in the transform. The transform is the last point where the original data is available and the error message is meaningful ('invalid amount: "abc" at row 3'). By the time the loader receives the row, the original CSV context is gone.

Scenario question

You're handed a 300-line ETL script that reads from S3 and writes to Postgres inside a single for loop. You are asked to add tests without refactoring the job. What do you do?

Model answer

I'd start by extracting the transform logic into a function I can call with a string: even without changing the production function, I can write a wrapper that takes csv_body as a parameter and returns the rows that would be inserted. That one extraction makes the data validation rules testable without S3 or Postgres. For the load path, I'd add a test that calls the original function with a mock psycopg2.connect (patching 'etl_module.psycopg2') and a pre-populated cursor.fetchall return. The trade-off is that this is harder to maintain than a clean split — the mock mirrors the internal structure of the function, so refactoring breaks the test. I'd document the intention to split the function in the next sprint.

Interview corner

CommonHow do you test an ETL function that reads from S3 and writes to Postgres?
“I split it into a pure transform and a loader. I test the transform with a fixture string — no mocks, no I/O. I test the loader with sqlite3 in memory and a MagicMock for the S3 client. That gives me two fast unit tests and one integration test that touches real Postgres in CI.”
Experience checkWhat is the most common ETL testing gap you've seen in Python codebases?
“In my experience it is silent partial load: the job runs, writes 60% of the rows, then crashes, and nothing in the code or the test suite catches the gap. The fix is an idempotency test: load the same data twice, assert the DB count equals the unique-row count, and wrap the load in a transaction with a rollback on failure.”
DesignWhen would you run ETL tests against a real Postgres instance vs sqlite3?
“I'd use sqlite3 for pure-transform unit tests — they run in milliseconds and need no infrastructure. I'd add a Postgres integration test in CI when the transform produces types or constraints that sqlite3 does not enforce, like NUMERIC(10,2) or partial indexes. I run the integration test per PR, not per commit, to keep the fast feedback loop.”
TrickYour idempotency test inserts the same 3 rows twice and asserts count == 3. The test passes. Is the job actually idempotent?
“I'd check whether the assertion is on the count or on the values. If the second insert silently replaces the rows rather than updating them, the count is still 3 but the values may have changed. The full idempotency check is: insert twice with the same data; assert count == 3 AND each row has the same value as after the first insert.”

Memory hook

A crash-test engineer runs the dummy into the barrier, not the passengers. transform_settlements is the dummy: pure, no I/O, runs in microseconds, and tells you exactly what the ETL would load before any row touches the database.

If you remember only one thing:

Extract a pure transform(csv_body) -> list function; test it with a fixture string; test the loader with sqlite3 in memory.

Self-check

  1. Why does a pure transform function not need mocks?

    Because it accepts a string and returns a list — no I/O, no external dependencies.

  2. What SQL keyword makes a loader idempotent on a primary key?

    INSERT OR REPLACE (SQLite) or INSERT ... ON CONFLICT DO UPDATE (Postgres).

  3. What is the CPython reason executemany is faster than a loop of execute?

    executemany passes all rows to the C-level library in one call; execute in a loop crosses the Python/C boundary once per row.

  4. What does sqlite3 allow that Postgres rejects?

    sqlite3 accepts any value in any column regardless of the declared type (dynamic typing). Postgres enforces types strictly.

Prove it

Copy the ETLTests class from the production block. Add a fifth test: test_transaction_rollback. Inject a row with a non-parseable amount into the middle of the CSV, call transform_settlements, and assert that ValueError is raised and the database table is empty (because the transform raised before any load happened). Run python -m unittest -v. Paste the output showing all five tests passing.

py-11-06Card 6 of 6

Introduce property-based testing

MediumP4 · DifferentiatorTalk about it11 min
testingproperty-basedhypothesisinvariantsshrinking

Why you care

Example-based tests (parametrize, subTest) tell you "the function is correct for these N inputs I thought of". Property-based testing tells you "the function satisfies this invariant for every input a generator could construct in N tries". The distinction matters when the edge cases you did not think of are also the cases that exist in production data — amounts with unusual Unicode characters, time zones you did not anticipate, empty batches on holiday schedules. Interviewers ask about property-based testing at the conversational level: you are not expected to have hypothesis in production, but you are expected to know what an invariant is and why example tests cannot always find the bugs that matter.

Explain like I'm 5

Imagine a stress-testing robot that throws random objects at a vending machine for eight hours. You wrote the test cases for a coin, a crisp note, and a crumpled note. The robot finds a folded note with a corner torn off. Your example tests never found that — the robot did, because it generated 50,000 inputs, not 3. Property-based testing is that robot: you describe what the machine should always do (return correct change), and the library generates the objects.

Core concept

A property is an assertion that must hold for all valid inputs, not just specific values:

  • Round-trip: decode(encode(s)) == s for any string s
  • Idempotency: sort(sort(L)) == sort(L) for any list L
  • Commutativity: add(a, b) == add(b, a) for any integers a, b
  • Partition invariant: sum(split(total)) == total for any positive integer total

Hypothesis (pip install hypothesis) is the standard Python property-based library. You annotate a test function with @given(strategies) and Hypothesis generates hundreds of examples, then on failure shrinks the failing input to the smallest counterexample it can find.

Pythonneeds hypothesis
from hypothesis import given, strategies as st

@given(st.text())
def test_encode_roundtrip(s):
    assert decode(encode(s)) == s

Shrinking is what makes property-based testing useful in practice. If the failure is found at a random 200-character string full of Unicode, Hypothesis reduces it to the shortest string that still fails — often a single character or empty string. Without shrinking, diagnosing the failure would require reading a 200-character random string.

Manual property testing requires no library: write a loop over random.seed(N) and generated inputs, assert the invariant in each iteration, and collect or raise on the first failure. It is less ergonomic than Hypothesis but works with stdlib only.

When property-based testing adds most value:

  • Parsing functions that accept structured text (CSV, JSON, date strings)
  • Mathematical invariants (splits, rounding, aggregates)
  • Serialisation round-trips
  • Sorting and ordering algorithms

When it adds least value:

  • Functions that implement a business rule with a small, enumerable domain (use parametrize instead)
  • Side-effect-heavy functions (I/O, database writes) where each generated input requires significant setup

Decision rule: use example tests as the foundation; add property tests for invariants that span an unbounded input space.

Minimal example Manual property test for a round-trip invariant

Python500 random ASCII strings, zero round-trip failuresruns
import base64
import random


def encode_decode_roundtrip(s):
    return base64.b64decode(base64.b64encode(s.encode())).decode()


random.seed(42)
failures = []
for _ in range(500):
    length = random.randint(0, 30)
    s = ''.join(chr(random.randint(32, 126)) for _ in range(length))
    if encode_decode_roundtrip(s) != s:
        failures.append(s)

print(f'500 random ASCII strings tested, {len(failures)} failures')
Output
500 random ASCII strings tested, 0 failures

In production Payment routing invariant — splits always sum to total

PythonPartition invariant: 500 random totals and weight distributionsruns
import random


def split_payment(total_paise, gateway_weights):
    """
    Distribute total_paise across gateways proportional to integer weights.
    Returns dict gateway -> paise.
    Invariant: sum(result.values()) == total_paise.
    """
    if not gateway_weights:
        raise ValueError('no gateways configured')
    total_weight = sum(gateway_weights.values())
    if total_weight <= 0:
        raise ValueError('weights must be positive')

    splits = {}
    remainder = total_paise
    gateways = sorted(gateway_weights)  # deterministic order
    for gw in gateways[:-1]:
        share = int(total_paise * gateway_weights[gw] / total_weight)
        splits[gw] = share
        remainder -= share
    splits[gateways[-1]] = remainder   # last gateway gets the rounding remainder
    return splits


# Property: sum of splits always equals the original total
random.seed(99)
failures = 0
for _ in range(500):
    total = random.randint(1, 1_000_000)
    n_gw = random.randint(1, 5)
    weights = {f'gw{i}': random.randint(1, 100) for i in range(n_gw)}
    result = split_payment(total, weights)
    if sum(result.values()) != total:
        failures += 1
        print('FAIL total:', total, 'weights:', weights, 'result:', result)

print(f'500 random inputs tested, {failures} invariant failures')
# Illustrative example
print('example:', split_payment(10000, {'A': 1, 'B': 2, 'C': 7}))
Output
500 random inputs tested, 0 invariant failures
example: {'A': 1000, 'B': 2000, 'C': 7000}
Scenario A

In your world

The situation

You have a split_payment function used in the payment router. A colleague runs the manual property test above and finds zero failures. You ship it. A week later, Ops reports that on a payment of Rs 1 (100 paise) with weights {A: 1, B: 1, C: 1}, gateway A receives 33 paise, B receives 33 paise, and C receives 34 paise — but the system treats the split as (33, 33, 33) because of a display bug that shows int(paise / 100) for each gateway separately, and int(33/100) + int(33/100) + int(33/100) = 0 + 0 + 0.

What actually breaks

The invariant at the split_payment level is satisfied. The bug is in the display layer, not the router. The property test for split_payment correctly passes. Without a property test on the display function, the visual rounding error goes undetected.

What you'd do on Monday

Write a property test for the display function: sum(display_amount(p) for p in split.values()) approximately equals display_amount(total). A manual property loop finds the failure on total=100, weights={A:1, B:1, C:1} within the first few iterations.

Scenario B

Imagine you work at Juspay

100M+ transactions per year

The numbers

100,000,000 transactions per year ÷ 365 days ≈ 273,972 transactions per day ≈ 3 per second average, with higher peak rates. Each transaction routes through 3-5 gateways. A routing invariant failure on 0.001% of transactions = 0.00001 × 100,000,000 = 1,000 incorrect splits per year.

What the Scenario-A answer does here

Example tests cover the cases the developer thought of: (100, {A:1, B:1}), (1000, {A:3, B:7}). They do not cover total=1, n_gateways=5 (all gateways receive 0 paise, sum=0≠1) or total=7, weights={A:1, B:1, B:2} (duplicate key overwrites previous weight — Python dict silently keeps the last value).

What flips at this scale

A manual property loop over 1,000 random.randint(1, 100) totals with random.randint(1, 10) weights per gateway will find both failure cases within the first 100 iterations. When Hypothesis is available, it finds and shrinks both to a minimal example in under 1 second. The design change the scale forces is: property tests belong in the same CI run as example tests, not in a separate performance suite.

The trade-off you'd defend

I would defend manual property tests with a fixed seed for new ETL code, trading Hypothesis's shrinking capability and richer strategies for zero dependencies and a test that reads exactly like the invariant it checks.

Illustrative; not Juspay-published.

Gotchas

Random seed not fixed — property test is non-deterministic

Symptom

The test passes on one CI run and fails on the next with no code change.

Cause

random.seed() was not called before the test loop, so different seeds produce different inputs on each run.

Fix

Call random.seed(42) (or any fixed integer) at the top of the test function. Document the seed in the test so that a regression can be reproduced by running with the same seed.

Property test passes but example tests are missing

Symptom

500 random inputs all pass the invariant. You skip the boundary examples (total=0, total=1, n_gateways=1). A bug exists exactly at total=0.

Cause

random.randint(1, ...) never generates 0; the boundary is excluded from the domain.

Fix

Add explicit boundary values to the random range or add separate example tests for zero, one, and max-int inputs. Property tests and example tests are complementary, not alternatives.

Hypothesis test is slow because the strategy generates too much state

Symptom

@given(st.lists(st.integers())) generates 100 examples, each a list of up to 100 integers, and the test takes 10 seconds.

Cause

The default Hypothesis settings allow large examples. The function under test may be doing O(n²) work.

Fix

Constrain the strategy: st.lists(st.integers(), max_size=20). Use @settings(max_examples=50) for a quick smoke pass in CI and a larger run overnight.

Tricky questions

Does a property test replace example tests?
Most people say

Yes — if the invariant holds for 500 random inputs, the function is correct.

Actually

No. Property tests check invariants; example tests check specific expected outputs. A round-trip property test on a base64 encoder does not tell you that encode('') returns b'' specifically. Both are needed: properties for coverage of the input space, examples for pinning known correct outputs.

Can Hypothesis find all bugs?
Most people say

Yes — with enough examples, every bug is eventually triggered.

Actually

No. Hypothesis generates plausible inputs within the defined strategy. If the strategy does not include valid negative amounts because the strategy is st.integers(min_value=0), Hypothesis will never find a bug triggered by a negative amount. The strategy defines the test space.

Scenario question

You're handed a generate_invoice_id(date, sequence) function. You are asked to add a property test. What property do you choose and why?

Model answer

I'd pick uniqueness: for any two distinct (date, sequence) pairs, the generated IDs must be different. I'd run a loop over 1,000 random pairs, collect all generated IDs into a set, and assert len(ids) == 1000. The second property I'd add is round-trip: if the ID format encodes the date and sequence, I'd write a parser and assert parse(generate_invoice_id(d, s)) == (d, s). The trade-off between manual and Hypothesis here is that Hypothesis would also find cases where two different inputs accidentally collide — something the manual loop might miss if the random distribution does not explore the collision range. I'd start with the manual loop and add Hypothesis if the domain is large enough to warrant it.

Interview corner

CommonWhat is property-based testing and when would you use it?
“I'd describe it as: instead of writing specific input/output pairs, you define an invariant that must hold for all inputs — like "sort(sort(L)) equals sort(L)" — and the library generates hundreds of examples and finds a counterexample if one exists. I'd use it for parsing functions, round-trips, and mathematical invariants like payment splits. For business rules with a small domain, I'd use a parametrized example table instead.”
TrickWhat is shrinking in the context of Hypothesis?
“I'd say Hypothesis automatically reduces a failing input to the smallest counterexample that still fails. If the failure was found on a 200-character random string, shrinking might reduce it to a single space or an empty string. That makes the failure message actionable instead of a wall of random characters.”
TrickA property test finds no failures on 500 random inputs. Does that mean the function is correct?
“I'd say no — and that is the important part. The inputs were generated from a strategy I defined, and that strategy may not include the failure case. If I used randint(1, 1000), a bug triggered by 0 or by 1001 will never be found. Property tests give confidence within the space the strategy explores, not outside it.”

Memory hook

A stress-testing robot throws 50,000 random objects at the vending machine while you were testing with a coin, a crisp note, and a crumpled note. Property-based testing is that robot: you describe the invariant, the library generates the objects.

If you remember only one thing:

Describe one invariant your function must always satisfy, then write a loop that tests it with 500 seeded random inputs.

Self-check

  1. What is an invariant in the context of property-based testing?

    A condition that must hold for all valid inputs — for example, that the sum of payment splits equals the original total.

  2. Why is a fixed random seed important for a manual property test?

    So that a failing test produces the same failure on every run, making it reproducible and debuggable.

  3. Does a property test replace example-based boundary tests?

    No. Property tests check invariants across the input space; example tests pin specific expected outputs. Both are needed.

  4. What does Hypothesis do after it finds a failing input?

    It shrinks the input to the smallest counterexample that still causes the failure.

Prove it

Implement split_payment from the production block. Introduce a bug: instead of giving the last gateway the remainder, give it int(total_paise / len(gateway_weights)) (equal split, ignoring weights and rounding). Run the manual property loop with random.seed(99). Paste the first failing iteration output (total, weights, actual sum vs expected). Then fix the bug and confirm zero failures.