Shelf · Core Eng · Chapter 5

Chapter 5: Testing strategy & quality

A green build tells you almost nothing on its own; this chapter is about deciding what to test where, why suites go flaky or lie about coverage, and how to prove a data pipeline survives bad data and a mid-batch crash. Pytest mechanics (fixtures, parametrize, patching, coverage tooling) live in the Python testing chapter — this is the strategy underneath them.

Difficulty
Priority
ce-05-01Card 1 of 6

Testing strategy: unit, integration, contract and e2e

MediumP2 · Very likelyTalk about it11 min
test-pyramidunit-testingintegration-testingcontract-tests
Needs: nothing. Start here.

Why you care

Every backend job description says "strong testing practices," and every interview asks some version of "walk me through your test pyramid." The useful skill is not naming four layers on a whiteboard; it's knowing which layer would have caught last week's incident, and why it didn't. Call something a "unit test" when it actually opens a real socket, and you get a suite that is slow, flaky, and still cannot answer "is this API safe to deploy." Interviewers ask this because it is the fastest way to tell whether you have operated a CI pipeline or only written tests inside one repo in isolation.

Explain like I'm 5

Think of proving a bridge is safe before opening it. A unit test is checking one bolt in a vice: fast, and it tells you that bolt is fine, but nothing about the bridge. An integration test is bolting two real beams together and loading them: slower, and it tells you the joint holds. A contract test is checking that the blueprints of two independently built sections agree, without building either section. An end-to-end test is driving a real truck across the finished bridge: the most convincing signal, and the one you can afford to run the least often.

Core concept

Four boundaries, each answering a different question:

  • Unit test: one function or class, every collaborator faked or absent. Answers "is this logic correct in isolation." Milliseconds. Tells you nothing about wiring.
  • Integration test: real collaborators (a real database, a real queue, or a close-enough fake). Answers "do these pieces actually talk to each other correctly." Seconds, sometimes more.
  • Contract test: checks the API or schema shape two independently deployed services agree on, without booting the other service. Answers "will my change break a consumer I don't control." Runs in each side's own pipeline.
  • End-to-end (e2e) test: drives the real, deployed system through a real user flow. Answers "does the whole thing work," at the highest cost and the least stability.

The classic test pyramid (many unit, fewer integration, fewest e2e) is a bet about where most bugs live and how expensive feedback is at each layer, not a mandated ratio. Its critics point out real counter-cases: a service that is mostly thin glue between other services needs more integration tests than unit tests (sometimes drawn as a "honeycomb" or "trophy" shape), and a monorepo with many independently deployed consumers needs contract tests as a first-class layer, not an afterthought squeezed between integration and e2e.

The decision rule: pick the layer that answers the specific question you need answered, at the cheapest cost that still tells the truth. Use unit tests for logic with many input combinations. Use integration tests to prove your own wiring against a real dependency. Use contract tests at every boundary you do not fully control end to end. Reserve e2e for the handful of flows where "the whole system worked, for real" is the only acceptable evidence. Pytest fixtures, assertions and parametrize mechanics for actually writing any of these are covered separately in the Python testing chapter (start there for syntax); this card is about which boundary to test, not how to call assert.

Minimal example Same test count, very different CI time

PythonPyramid-shaped versus inverted, same "coverage confidence budget"runs
UNIT_S, INTEGRATION_S, E2E_S = 0.02, 2.0, 90.0

pyramid = {"unit": 700, "integration": 150, "e2e": 10}
inverted = {"unit": 10, "integration": 150, "e2e": 700}


def suite_time(mix):
    return mix["unit"] * UNIT_S + mix["integration"] * INTEGRATION_S + mix["e2e"] * E2E_S


for name, mix in [("pyramid-shaped", pyramid), ("inverted (e2e-heavy)", inverted)]:
    total = suite_time(mix)
    print(f"{name}: {sum(mix.values())} tests, {total:,.0f}s ({total / 60:.1f} min)")
Output
pyramid-shaped: 860 tests, 1,214s (20.2 min)
inverted (e2e-heavy): 860 tests, 63,300s (1055.0 min)

In production Partition a suite by boundary, not by folder name

PythonTag tests by what they touch, then run only what a fast gate needsruns
import unittest


def tag(*labels):
    def decorator(fn):
        fn.tags = labels
        return fn
    return decorator


class SettlementTests(unittest.TestCase):
    @tag("unit")
    def test_fee_calculation_is_pure(self):
        self.assertEqual(round(1000 * 0.02), 20)

    @tag("integration")
    def test_insert_and_read_back_from_real_db(self):
        import sqlite3
        conn = sqlite3.connect(":memory:")
        conn.execute("CREATE TABLE t (id INTEGER)")
        conn.execute("INSERT INTO t VALUES (1)")
        self.assertEqual(conn.execute("SELECT COUNT(*) FROM t").fetchone()[0], 1)

    @tag("e2e")
    def test_full_settlement_flow_through_the_api(self):
        self.skipTest("would call a deployed staging environment")


def suite_with_tags(test_case_cls, wanted):
    loader = unittest.TestLoader()
    suite = unittest.TestSuite()
    for name in loader.getTestCaseNames(test_case_cls):
        method = getattr(test_case_cls, name)
        if wanted & set(getattr(method, "tags", ())):
            suite.addTest(test_case_cls(name))
    return suite


fast_gate = suite_with_tags(SettlementTests, {"unit"})
full_suite = suite_with_tags(SettlementTests, {"unit", "integration", "e2e"})

print("fast pre-commit gate runs:", fast_gate.countTestCases(), "test(s)")
print("full nightly suite runs:", full_suite.countTestCases(), "test(s)")

result = unittest.TextTestRunner(verbosity=0).run(fast_gate)
print("fast gate failures:", len(result.failures) + len(result.errors))
Output
fast pre-commit gate runs: 1 test(s)
full nightly suite runs: 3 test(s)
fast gate failures: 0

See it happen Speed and confidence trade off in opposite directions

more confidencemore confidencemore speedunit700 tests, 0.02s eachintegration150 tests, 2s eachcontractruns without the other service700 tests, 90s each (inverted shape)
  1. Unit tests check one function or class in isolation: fast, but they say nothing about how pieces talk to each other.
  2. Integration tests exercise real collaborators to prove the wiring works, at roughly 100 times the cost per test.
  3. Contract tests check the API shape both independently deployed sides agree on, without booting the other service, so they scale across many teams.
  4. End-to-end tests drive the real deployed system through a real flow: the most convincing signal, and the slowest and most brittle.
  5. Speed and confidence trade off in opposite directions as you move right; nothing here is both instant and fully convincing.
  6. A suite that is mostly e2e does not buy more confidence for the same bugs; it buys a much slower, flakier signal.
Notice: move up the diagram for confidence and right for realism, but every step in either direction also costs more time; the pyramid shape is a bet on where most of your bugs actually live, not a rule.
Scenario A

In your world

The situation

Your team's Jenkins job has one stage called "unit tests." Forty of its test methods do conn = psycopg2.connect(TEST_DSN) in setUp, hit a shared test database, and take about 3 seconds each once connection overhead is counted.

What actually breaks

The "unit" stage takes 10 minutes. Nobody runs it before pushing because 10 minutes feels like a coffee break, not a sanity check, so regressions in that code land on main and get caught only when the nightly build fails, a day later than they should have.

What you'd do on Monday

Reclassify by what each test touches, not by its folder name. Anything opening a real socket or connection is integration, not unit, and it moves to a stage that runs on merge, not on every save:

PythonSplit by boundary crossed, not by test nameruns
def touches_network_or_db(test_body_uses_connect):
    return "integration" if test_body_uses_connect else "unit"


tests = [
    ("test_fee_calculation", False),
    ("test_insert_then_read_from_db", True),
    ("test_parse_settlement_row", False),
]
for name, uses_connect in tests:
    print(name, "->", touches_network_or_db(uses_connect))
Output
test_fee_calculation -> unit
test_insert_then_read_from_db -> integration
test_parse_settlement_row -> unit
Scenario B

Imagine you work at Google

5,430 tests, 60 minutes serial

The numbers

Illustrative: a service with 5,000 fast tests at 20ms, 400 integration tests at 2s, and 30 e2e tests at 90s, run serially, costs 5,000×0.02 + 400×2 + 30×90 = 100 + 800 + 2,700 = 3,600s, about 60 minutes of feedback for a single commit.

What the Scenario-A answer does here

The Scenario A fix (classify by boundary) still holds, but "run the whole reclassified suite on every commit anyway" does not: 60 minutes of feedback per push, multiplied across hundreds of commits a day, either queues commits for hours or burns a proportional amount of CI compute just waiting.

What flips at this scale

At this scale you split the gate by timing, not just by boundary: a pre-submit gate runs only the sub-2-second layer (roughly 100s here) and blocks the merge; the integration and e2e layers run post-submit or nightly, sharded across workers, and a failure there pages an owner instead of blocking every other commit behind it.

The trade-off you'd defend

I would defend a fast, mandatory pre-submit gate plus a slower, parallel post-submit gate over one serial "everything, every time" pipeline, trading immediate blocking on integration/e2e regressions for a feedback loop developers will actually wait for.

Illustrative figures chosen to make the arithmetic visible; not Google-published numbers.

Gotchas

"Unit" tests that open a real connection

Symptom

A stage named "unit tests" takes minutes, not seconds, and developers skip running it locally.

Cause

The tests touch a real database, queue or network call, which makes them integration tests wearing a unit-test label.

Fix

Reclassify by what the test body actually touches, and move real-collaborator tests to their own, less-frequent stage.

A contract change breaks a consumer nobody tested against

Symptom

Every layer's tests pass, the deploy goes green, and a downstream service still starts 500ing.

Cause

No contract test exists between two independently deployed services, so a renamed or retyped field is invisible to both pipelines.

Fix

Add a provider/consumer contract test at that boundary instead of relying on integration or e2e coverage to notice.

A flaky e2e test gets re-run until it's green

Symptom

The e2e stage is red about one run in ten, and the habit is to click "re-run" without reading why.

Cause

The layer meant to give the most confidence is instead training the team to distrust red builds.

Fix

Treat e2e flakiness as a defect to fix or quarantine with an owner, not noise to route around.

Tricky questions

Does more unit tests always mean better coverage of real risk?
Most people say

Yes, unit tests are cheap, so write as many as possible.

Actually

A service that is mostly thin calls to other services has little pure logic to unit test; it needs more integration and contract coverage than unit tests, which is exactly the "honeycomb" shape critics of the strict pyramid point to.

Is a contract test just a smaller integration test?
Most people say

Yes, it's an integration test with less setup.

Actually

A contract test can run without the other service ever running, against a recorded or shared schema. That is the point: it decouples two teams' pipelines, which an integration test — which needs the real collaborator up — cannot do.

Scenario question

You're handed a service where "tests" means one enormous e2e suite that clicks through a staging environment, takes 40 minutes, and fails about twice a week for reasons nobody has time to chase. Where do you start?

Model answer

I would not add more e2e tests. I would first sort the 40-minute suite's coverage by what it is actually proving, and for every check that is really "does this one function compute the right number," pull it down into a fast unit test. For every check that is "do our own two services agree on a call," I would add an integration test against a real (or disposable) dependency. Only the handful of checks that need the full, deployed stack to mean anything would stay e2e. The trade-off is upfront work to triage 40 minutes of tests, but the payoff is a pre-submit gate that runs in seconds and an e2e suite small enough that failures get investigated instead of re-run.

Interview corner

CommonExplain the test pyramid and when it doesn't apply.
“I'd describe it as unit tests at the base for speed, integration in the middle for real wiring, and e2e at the top for full-system confidence, in decreasing count and increasing cost. It doesn't apply cleanly to a thin orchestration service with little logic of its own, or to a system with many independently deployed consumers, where contract tests matter more than the ratio.”
TrickIs 90% unit test coverage on a thin API gateway a good sign?
“Not necessarily. If the service is mostly routing and calling other services, unit tests are cheap to write but don't prove much; I'd want to see integration and contract tests at the boundaries it actually depends on.”
Experience checkTell me about a test suite that was slow for a bad reason.
“I've seen a 'unit' stage take ten minutes because every test opened a real database connection in setUp. Reclassifying by what each test touched, not its filename, got the real unit layer down to seconds and moved the DB-touching tests to a stage that ran less often.”
DesignHow would you structure CI stages for a growing monorepo?
“I'd gate merges on a fast layer, under a couple of minutes, covering unit and contract tests, then run integration and e2e post-merge or nightly, sharded across workers, with failures routed to an owner instead of blocking the next commit.”

Memory hook

A unit test is one bolt in a vice; an e2e test is driving the truck across the finished bridge.

If you remember only one thing:

Pick the layer that answers your actual question, at the cheapest cost that still tells the truth.

Self-check

  1. What question does a contract test answer that an integration test cannot answer cheaply?

    "Will my change break a consumer I don't control," without needing that consumer's service running at all.

  2. Why is the test pyramid a bet, not a rule?

    It assumes most bugs live in isolated logic and integration is expensive; a thin, glue-heavy service breaks that assumption.

  3. In the production example, why does the fast gate run only one test?

    Because only test_fee_calculation_is_pure is tagged "unit"; the other two touch a real database or a deployed environment.

Prove it

List the last 15 tests you wrote or touched. For each, name the boundary it actually crosses (unit, integration, contract, e2e) based on what it touches, not its file name. Report how many were mislabeled, and re-time your suite split into a "fast" group and everything else.

ce-05-02Card 2 of 6

Use red-green-refactor to test a behavior

MediumP3 · Role-dependentKnow cold10 min
tddred-green-refactorregression-tests

Why you care

You will rarely be required to practice strict TDD at work, but "walk me through how you'd test-drive this" is a common interview exercise, and the habit behind it — write a test that fails for the right reason before writing the fix — is what actually prevents a bug from coming back silently. Skip the "red" step and you cannot tell whether your new test proves anything at all; it might pass because the code already worked, or because the test is coupled to the wrong detail. Interviewers ask this to see whether you can demonstrate a failure before claiming a fix, which is the same discipline as writing a regression test for a production bug.

Explain like I'm 5

Installing a smoke detector: first you make sure it actually beeps in smoke (red), because an untested detector that never beeps is worse than none, since you'd trust it. Then you fix whatever was smoking (green). Then you tidy the wiring behind the wall (refactor), and you're not worried about breaking the detector while you do it, because you already know it beeps when it should.

Core concept

Three steps, repeated in a tight loop, usually minutes not days:

  • Red: write one small test for a behavior that does not exist yet, run it, and confirm it fails for the reason you expect (the assertion, not an import error or a typo). If it doesn't fail, either the behavior already exists or the test isn't exercising the new code path.
  • Green: write the minimum code that makes that one test pass. Resist adding anything the test doesn't require yet.
  • Refactor: improve names, remove duplication, extract structure, with no change in behavior, then rerun the full suite, not just the new test, to prove nothing else broke.

The safety net only works if assertions check observable outcomes, not implementation detail. A test that asserts mock.call_count == 3 breaks on every harmless refactor that changes how many times an internal helper is called, even when the output is identical; that kind of coupling defeats the reason you did the refactor step at all. Use TDD for logic with a clear expected behavior: parsers, fee calculations, boundary rules, bug fixes with a repro. It fits less well when the design itself is still unstable, such as early UI exploration; there, stabilize the shape first and add tests once behavior is worth locking down.

Minimal example Red then green, one function

PythonThe same test, unchanged, fails then passesruns
import unittest


class FeeTest(unittest.TestCase):
    def test_flat_fee_under_1000(self):
        self.assertEqual(fee(100), 2)

    def test_percent_fee_over_1000(self):
        self.assertEqual(fee(2000), 40)


def run_against(target):
    global fee
    fee = target
    result = unittest.TextTestRunner(verbosity=0).run(
        unittest.TestLoader().loadTestsFromTestCase(FeeTest)
    )
    return len(result.failures) + len(result.errors)


def fee_v0(amount):
    return 0  # not implemented yet


def fee_v1(amount):
    return 2 if amount < 1000 else round(amount * 0.02)


print("red phase failures:", run_against(fee_v0))
print("green phase failures:", run_against(fee_v1))
Output
red phase failures: 2
green phase failures: 0

In production A refactor safety net catches a boundary bug

PythonAdding a boundary test during refactor catches an off-by-oneruns
import unittest


def fee_v1(amount):
    return 2 if amount < 1000 else round(amount * 0.02)


def fee_v2_refactored(amount):
    # attempted refactor: table-driven tiers instead of if/else
    tiers = [(1000, lambda a: 2), (float("inf"), lambda a: round(a * 0.02))]
    for limit, charge in tiers:
        if amount <= limit:            # bug: should be amount < limit
            return charge(amount)


class FeeTest(unittest.TestCase):
    def test_flat_fee_under_1000(self):
        self.assertEqual(fee(100), 2)

    def test_percent_fee_over_1000(self):
        self.assertEqual(fee(2000), 40)

    def test_boundary_at_1000(self):
        self.assertEqual(fee(1000), 20)  # added while refactoring


def run_against(target):
    global fee
    fee = target
    result = unittest.TextTestRunner(verbosity=0).run(
        unittest.TestLoader().loadTestsFromTestCase(FeeTest)
    )
    return len(result.failures) + len(result.errors)


print("fee_v1 failures:", run_against(fee_v1))
print("fee_v2_refactored failures:", run_against(fee_v2_refactored))
Output
fee_v1 failures: 0
fee_v2_refactored failures: 1

See it happen The loop, one commit at a time

developertest suitewrite failing testRED: AssertionErrorwrite minimum coderun full suiteGREEN: all passrefactor structurerun full suite againGREEN: still passing
  1. Write one small test for behavior that doesn't exist yet.
  2. Run it immediately: it must fail, and for the right reason.
  3. Write just enough implementation to satisfy that one test.
  4. Run everything, not just the new test.
  5. Every test passes, including the old ones; nothing broke.
  6. Improve names, remove duplication, no behavior change.
  7. Rerun everything after refactor, before starting the next behavior.
  8. The same green suite proves the refactor didn't change behavior.
Notice: refactor only happens inside an already-green suite; if it isn't fully green first, you don't have a safety net yet, you have a hope.
Scenario A

In your world

The situation

A bug report says "settlement fee is wrong for exactly 1000." You open fee() and see the obvious fix.

What actually breaks

You change the boundary condition directly and manually check the one number from the ticket in a REPL. It looks right, so you ship it. Two months later someone "cleans up" the function into a table-driven version, gets the same boundary subtly wrong in the opposite direction, and nothing catches it because the only evidence the original bug existed was a REPL session, not a test.

What you'd do on Monday

Write the test from the bug report first, watch it fail the way the ticket describes, then fix it:

PythonReproduce the bug as a test before fixing itruns
def fee_buggy(amount):
    return 2 if amount <= 1000 else round(amount * 0.02)  # bug: should be <


result = fee_buggy(1000)
try:
    assert result == 20, f"amount=1000 expected 20, got {result}"
except AssertionError as e:
    print("reproduced the reported bug:", e)


def fee_fixed(amount):
    return 2 if amount < 1000 else round(amount * 0.02)


assert fee_fixed(1000) == 20
print("fixed: fee_fixed(1000) ==", fee_fixed(1000))
Output
reproduced the reported bug: amount=1000 expected 20, got 2
fixed: fee_fixed(1000) == 20
Scenario B

Imagine you work at Spotify

1% canary, 50,000 requests/sec

The numbers

Illustrative: a ranking change ships to 1% of a hypothetical 500M-user base, about 5M users, serving roughly 50,000 requests/second; if a division-by-zero regression is caught by a 2-second local test instead of the canary, the alternative is 10 minutes of live traffic at that rate, about 30M affected responses before anyone notices and rolls back.

What the Scenario-A answer does here

The Scenario A cycle (reproduce, fix, refactor under green) still works for a feature branch, but "I'll add the test after this same-day hotfix ships" does not remove the risk: a hotfix that skips red-green still reaches the same canary with no regression net, and the missing test doesn't announce itself until something breaks it again.

What flips at this scale

At this scale TDD stops being a personal habit and becomes a gate: changes to ranking logic require a named regression test for the class of bug being fixed (a boundary, a division, an off-by-one) before merge, so the 2-second unit test catches it for free instead of the canary catching it after it has already served millions of responses.

The trade-off you'd defend

I would defend requiring a regression test for every ranking bug fix, even a one-line hotfix, trading a few extra minutes per fix for not relying on a canary, which serves real users before it tells you anything, as the first line of defense.

Illustrative figures for the arithmetic; not Spotify-published numbers.

Gotchas

The failing test didn't fail for the right reason

Symptom

A new test is red, but on a typo or an import error, not the assertion you care about.

Cause

The test was never actually exercising the missing behavior in the first place.

Fix

Read the failure message during the red step, not just the color; fix the test itself if the failure isn't the one you expected.

Assertions coupled to implementation

Symptom

A previously green test breaks on a harmless refactor even though the output didn't change.

Cause

The test asserts an internal detail (a call count, a private attribute) instead of observable behavior.

Fix

Assert on outputs; reserve interaction assertions for side effects you must guarantee actually happened, like "an email was sent."

Tests added after the fact test the bug too

Symptom

Coverage looks fine, but the same class of bug ships again later.

Cause

Writing tests after the code tests what the code does, including its mistakes, not what it was supposed to do.

Fix

Write the test from the spec or the bug report first, and confirm it fails the way the report describes, before changing the implementation.

Tricky questions

Does TDD mean writing all your tests before any code?
Most people say

Yes, you write the full test plan first, then implement everything.

Actually

It's a tight loop of one small failing test, minimal code, refactor, repeat. Each cycle is usually minutes; a big upfront test plan is a different practice (and not a bad one, just not this one).

If a new test passes on the first try, did you do it correctly?
Most people say

Yes, first-try green just means you wrote good code.

Actually

A genuinely new-behavior test should fail first. If it passes immediately, either it's testing something that already worked, or it isn't exercising the new code path at all.

Scenario question

You're handed a bug report: "settlement fee is wrong for exactly 1000." Before touching the fee function, what do you do?

Model answer

I'd write a test asserting the exact expected fee at amount=1000 first, run it, and confirm it fails the way the bug report describes, not for an unrelated reason like an import error. Then I'd change the minimum code needed to make that one test pass, likely a boundary condition, without also refactoring surrounding code in the same step. Only once it's green would I refactor for clarity, rerunning the full suite after each small change. The trade-off is speed: writing the test first costs a couple of minutes I could spend "just fixing it," but it's also the regression test that stops this exact bug from coming back silently.

Interview corner

CommonWhat is red-green-refactor?
“I'd describe it as a tight loop: write one small failing test, write the minimum code to pass it, then refactor with the suite green, rerunning everything after each step. It's usually minutes per cycle, not a big upfront test plan.”
TrickIf your new test passes immediately, is that a problem?
“I'd be suspicious. Either the behavior already existed, or my test isn't actually exercising the new code path. I'd check by temporarily breaking the implementation and confirming the test then fails.”
Experience checkGive an example of a test that broke on every refactor even though behavior didn't change.
“I've seen tests asserting a mock's call count instead of the function's output, so any refactor that changed how many times an internal helper ran broke the test for no real reason. Switching the assertion to the observable result fixed it.”
DesignWould you mandate TDD for a team that's never done it?
“Not wholesale. I'd start with logic that has a clear expected behavior, like boundary rules and bug fixes, where writing the failing test first is cheap and the payoff is obvious, and leave exploratory or UI-heavy work until the design settles.”

Memory hook

Make the smoke detector beep before you trust it, fix the smoke, then tidy the wiring without worrying.

If you remember only one thing:

If the test didn't fail first, you don't know it's testing anything.

Self-check

  1. What's the risk of writing the test and the code together instead of test-first?

    The test may pass immediately without proving anything, or get coupled to the implementation you just wrote.

  2. Why should refactor-step assertions avoid checking call counts or private attributes?

    Because they then break on harmless refactors, which defeats the point of having a safety net.

  3. In the production example, what specific bug did the extra boundary test catch?

    An off-by-one in the table-driven refactor: amount <= limit charged the flat fee at exactly 1000 instead of the percentage.

Prove it

Pick one real bug fix you've done recently, or a small function you own. Write a failing test that reproduces it verbatim and paste the failure. Fix it minimally and paste green. Refactor once, rerun, and paste green again.

ce-05-03Card 3 of 6

Choose test doubles, fixtures and data builders

MediumP2 · Very likelyKnow cold10 min
test-doublesmockingfixturesdata-builders

Why you care

"Mock or stub?" sounds like trivia, but the real question behind it is "what does this test actually prove?" Mock your own database layer and a refactor that silently breaks a real query still shows green. Use a fake instead, and it exercises real logic paths, like rejecting a duplicate key, that a mock can never fail on because a mock only ever tells you what you told it to say. Interviewers ask this to find out whether you understand test isolation as a cost with a purpose, not a checkbox you tick with unittest.mock on every collaborator.

Explain like I'm 5

Stunt doubles, by role. A dummy is a mannequin passed to fill a seat; it never acts. A stub gives a canned answer to whatever it's asked, and doesn't remember being asked. A fake is a working, lightweight stand-in, like an in-memory database that really enforces its own rules. A spy watches a real or fake performer and records what happened so you can check later. A mock is pre-programmed with an expected scene; if the wrong line is delivered, the mock fails the take.

Core concept

  • Dummy: satisfies a parameter's type, never used. Stub: returns canned data, no behavior verification. Fake: a real, working, simplified implementation, like a dict-backed repository, which exercises actual logic. Spy: records calls on a real or fake object for later inspection. Mock: asserts how it was called, not just what it returned.
  • Sociable vs. solitary: a sociable test keeps some real (or fake) collaborators, so it also proves how units talk to each other; a solitary test doubles every collaborator, so a failure always points at exactly one unit. Sociable buys integration confidence; solitary buys precise failure localization. Neither is universally correct.
  • Fixtures and data builders: a fixture is reusable setup a test needs (a connection, a populated table); a data builder is a function or class that constructs a valid test object with sensible defaults, letting a test state only what's different (make_order(amount=500)) instead of repeating every field everywhere. Fixture lifecycle, scope and teardown mechanics belong to the pytest chapter; this card is about what you build with them.
  • Isolate the boundaries that make tests nondeterministic: inject the clock instead of calling datetime.now(), inject or seed randomness, and replace network calls at the edge of the process, not deep inside your own logic.

Decision rule: mock only what you don't own, or what's genuinely expensive or nondeterministic to run for real (a payment gateway, a third-party API). Fake what you do own, especially a data-access layer, because a fake exercises the real logic paths a mock structurally cannot. Use a stub when you just need canned data and don't care how it's used. Reserve mocks for confirming a real side effect happened exactly once, like "did we call send_email."

Minimal example A permissive mock hides a bug a fake catches

PythonMock accepts anything; a fake enforces real rulesruns
from unittest.mock import Mock


class FakeGateway:
    """A lightweight but real implementation: it actually validates."""
    def charge(self, amount):
        if amount <= 0:
            raise ValueError(f"invalid charge amount {amount}")
        return {"status": "ok", "amount": amount}


def charge_customer(gateway, amount):
    return gateway.charge(amount)


mock_gateway = Mock()
mock_gateway.charge.return_value = {"status": "ok"}
print("mock accepts a bad amount:", charge_customer(mock_gateway, -50))

fake_gateway = FakeGateway()
try:
    charge_customer(fake_gateway, -50)
except ValueError as e:
    print("fake catches it:", e)
Output
mock accepts a bad amount: {'status': 'ok'}
fake catches it: invalid charge amount -50

In production A data builder plus an injected clock

PythonTests state only what differs; time is injected, not realruns
from dataclasses import dataclass, replace
from datetime import datetime, timezone


@dataclass
class Order:
    order_id: str
    amount: int
    currency: str
    placed_at: datetime


def make_order(**overrides):
    """Builder: sensible defaults, tests only state what differs."""
    base = Order(order_id="O-1", amount=500, currency="INR",
                 placed_at=datetime(2026, 1, 1, tzinfo=timezone.utc))
    return replace(base, **overrides)


def settle(order, clock):
    """clock is injected so tests never depend on the real wall clock."""
    age_days = (clock() - order.placed_at).days
    return {"order_id": order.order_id, "late": age_days > 30}


def fixed_clock():
    return datetime(2026, 1, 15, tzinfo=timezone.utc)  # 14 days after Jan 1


fresh = make_order()
stale = make_order(order_id="O-2", placed_at=datetime(2025, 1, 1, tzinfo=timezone.utc))

print(settle(fresh, fixed_clock))
print(settle(stale, fixed_clock))
Output
{'order_id': 'O-1', 'late': False}
{'order_id': 'O-2', 'late': True}
Scenario A

In your world

The situation

Your OrderRepository is patched to Mock() in every test that touches it, including tests for a code path that inserts an order and then re-inserts it on a retry.

What actually breaks

Someone renames OrderRepository.insert to OrderRepository.save and updates most callers, but misses one retry path. Every test still passes, because Mock() happily accepts a call to insert that no longer exists on the real class. Production 500s the first time that retry path runs.

What you'd do on Monday

Replace Mock() for your own repository with either a fake (an in-memory implementation of the same class) or a mock built with spec=OrderRepository, so an unknown method name fails the test immediately:

PythonA spec'd mock rejects a call to a method that no longer existsruns
from unittest.mock import Mock


class OrderRepository:
    def save(self, order_id, amount):
        return True


bare_mock = Mock()
bare_mock.insert("O-1", 500)          # silently "succeeds" on a renamed method
print("bare mock accepted a dead method call")

spec_mock = Mock(spec=OrderRepository)
try:
    spec_mock.insert("O-1", 500)
except AttributeError as e:
    print("spec'd mock caught it:", e)
Output
bare mock accepted a dead method call
spec'd mock caught it: Mock object has no attribute 'insert'
Scenario B

Imagine you work at Atlassian

12 internal service collaborators

The numbers

Illustrative: a service integrates with 12 internal collaborators; a fully solitary test suite doubles all 12 in every test, so a rename in one collaborator's interface is invisible until a human notices, while a suite with even 3 of those 12 replaced by fakes instead of mocks would have caught roughly a quarter of "renamed method, tests still green" incidents in a typical quarter of that kind.

What the Scenario-A answer does here

The Scenario A fix (spec'd mocks or a fake for your own repository) still works one collaborator at a time, but manually spec'ing or faking all 12 collaborators for every test file is a maintenance tax nobody signs up to pay, so teams default back to bare Mock() everywhere, which is exactly what let the rename through.

What flips at this scale

At this scale you standardize: a small shared library of fakes and spec'd-mock factories for the handful of collaborators everyone tests against, so any team writing a new test gets a safe default instead of a bare Mock(), and a renamed method fails fast in whichever test happens to use the shared factory, not just the ones someone remembered to spec by hand.

The trade-off you'd defend

I would defend investing in shared fakes/spec factories for the most-mocked internal collaborators over leaving every team to hand-roll Mock(), trading upfront library work for catching interface drift automatically instead of by luck.

Illustrative figures for the arithmetic; not Atlassian-published numbers.

Gotchas

A mocked boundary stays green after the real thing changes

Symptom

Tests pass; production fails calling a renamed or retyped method.

Cause

A bare Mock() accepts any attribute or call, so it never notices the real class no longer matches.

Fix

Use spec= or autospec=True, or replace the mock with a fake for anything you own.

Shared mock state leaks between tests

Symptom

A test passes alone but fails, or passes differently, depending on which test ran before it.

Cause

A module-level or fixture-scoped mock's call history wasn't reset between tests.

Fix

Create doubles per test at function scope, or call reset_mock() explicitly before each test.

A test depends on the real clock or real randomness

Symptom

A test fails intermittently, or only near midnight, or only on specific dates.

Cause

The code under test reads datetime.now() or unseeded random directly instead of through an injectable seam.

Fix

Inject a clock function or a seeded RNG, as shown in the production example, so the test controls every input.

Tricky questions

Is Mock() with autospec=True basically the same as writing a fake?
Most people say

Yes, they both stop you from calling something that doesn't exist.

Actually

autospec=True only checks the signature matches; it still returns whatever you told it to, so it never exercises real logic like a duplicate-key rejection. A fake actually runs code and can be wrong in the same ways the real thing is wrong.

Are sociable tests just slower unit tests?
Most people say

Yes, they're the same test with more setup.

Actually

Sociable tests deliberately keep some real collaborators to verify how units communicate, message shapes, ordering, side effects; solitary tests double everything so a failure always points at one unit. The trade-off is integration confidence versus failure localization, not just speed.

Scenario question

You're handed a test suite where every repository call is Mock()'d, including your own OrderRepository. A refactor renamed a method the repository still has, but callers didn't update, and every test stayed green while production 500'd. What do you change?

Model answer

I'd replace the bare Mock() for OrderRepository with either a fake, an in-memory dict-backed implementation of the same class, or a mock built with spec=OrderRepository so it fails on unknown attributes. I'd keep plain Mock() only for boundaries I genuinely don't own, like the external payment gateway. The trade-off is that a fake costs more to write and maintain than Mock(), but it's my own repository, so I'm paying a small one-time cost for a test that actually exercises my logic instead of my assumptions about it.

Interview corner

CommonWhat's the difference between a mock and a stub?
“A stub returns canned data and nobody checks how it was called. A mock asserts the interaction itself, like 'this was called exactly once with these arguments.' I reach for a stub when I just need data, and a mock when the side effect is the thing I need to prove happened.”
TrickWhy would you use a fake instead of mocking your own database layer?
“Because a mock only ever tells me what I told it to say; it can't fail the way a real query fails. A fake, even a simple in-memory one, actually runs logic like uniqueness checks, so it catches bugs a mock structurally can't.”
Experience checkWhere has an over-mocked test suite bitten you?
“I've seen a renamed repository method ship because every test mocked that repository with a bare Mock, which accepted the old method name forever. Switching to spec'd mocks and a fake for that repository caught the next one immediately.”
DesignHow do you decide sociable versus solitary for a new service test?
“I lean solitary when I need to pinpoint exactly which unit broke, and sociable when the risk is in how units talk to each other, like message shape or ordering. For a new service I usually start solitary for logic and add a handful of sociable tests at the boundaries I'm least sure about.”

Memory hook

Dummy sits, stub answers, spy watches, mock expects a scene, fake actually works.

If you remember only one thing:

Mock only what you don't own; fake what you do.

Self-check

  1. Why does a bare Mock() fail to catch a renamed method?

    It accepts any attribute or call by default; nothing checks it against the real class's actual interface.

  2. What's the difference between a fixture and a data builder?

    A fixture is reusable setup a test needs, like a connection; a data builder constructs a valid test object with defaults, letting a test state only what's different.

  3. Why does the production example inject a clock function instead of calling datetime.now()?

    So the test's outcome depends only on the fixed value it controls, not on when the test happens to run.

Prove it

Take one function in your own codebase. Write a solitary test using Mock(spec=...) for every collaborator, and a sociable test using real or fake collaborators. Break one collaborator's method signature deliberately and rerun both; paste which test suite caught it.

ce-05-04Card 4 of 6

Test real infrastructure and service contracts

MediumP3 · Role-dependentTalk about it10 min
testcontainersintegration-testingcontract-tests

Why you care

A fake in-memory repository can be structurally wrong in ways that never show up until the real database disagrees, like a unique constraint you forgot to model, or a foreign key you assumed cascades. Testing against a real, disposable engine, and testing your API's shape against what a real consumer expects, catches exactly the class of bug a hand-written double cannot. Interviewers ask "how do you test the database layer" or "how do you avoid a full deploy just to test two services together" because both questions predict whether you've operated a pipeline that promotes real artifacts, not just run pytest against fakes.

Explain like I'm 5

A fake is reading a script alone in your room. Standing up a real, disposable database for a test is a dress rehearsal on a real stage, with real props, that gets struck the moment the scene ends. A contract test is neither: it's checking that your script and your co-star's script agree on the lines at the handoff, without either of you being in the room.

Core concept

  • Testcontainers-style testing spins up a real dependency, most often Postgres, inside a throwaway container for the test run, then destroys it. You get real constraint violations, real SQL error codes, real transaction semantics, not a mock's guess at them. In Python, from testcontainers.postgres import PostgresContainer, then with PostgresContainer("postgres:16") as pg: and pg.get_connection_url() for a real connection string; the container is torn down automatically when the block exits.
  • Readiness and isolation: the library waits for the container to actually be ready (a log line, an open port) before handing back control, not a fixed sleep; give each test or test session its own container, or its own database inside a shared one, so tests never share state.
  • This is still not production: a disposable single-node container has no realistic data volume, no replicas, no real network partition. It proves your queries and migrations are structurally sound, not that they perform at scale; staging and production checks still matter for that.
  • Provider/consumer contract tests solve a different problem: instead of booting the other service, the consumer records what fields and types it depends on from the provider's API, and the provider replays that recorded expectation in its own pipeline before it deploys. Neither service needs the other running.

Decision rule: use a real, disposable engine for anything you own, your own schema, migrations, queries and constraints. Use a contract test at the boundary to a service you don't own, or that's deployed independently of you. Reserve full multi-service integration environments for the rare case a contract test genuinely can't express, and never write test data into a live production system.

Minimal example A dict fake misses a constraint a real engine enforces

PythonA fake silently overwrites; sqlite3 (a real engine) rejects itruns
import sqlite3


class FakeOrderRepo:
    """In-memory fake: no real constraints."""
    def __init__(self):
        self.rows = {}

    def insert(self, order_id, amount):
        self.rows[order_id] = amount  # duplicate key: silently overwritten


fake = FakeOrderRepo()
fake.insert("O-1", 500)
fake.insert("O-1", 999)
print("fake rows:", fake.rows)

conn = sqlite3.connect(":memory:")
conn.execute("CREATE TABLE orders (order_id TEXT PRIMARY KEY, amount INTEGER)")
conn.execute("INSERT INTO orders VALUES ('O-1', 500)")
try:
    conn.execute("INSERT INTO orders VALUES ('O-1', 999)")
except sqlite3.IntegrityError as e:
    print("real engine rejects it:", e)
Output
fake rows: {'O-1': 999}
real engine rejects it: UNIQUE constraint failed: orders.order_id

In production A provider replays the consumer's recorded expectations

PythonA tiny contract test catches a "harmless" field renameruns
import json
from pathlib import Path

# The consumer records what it depends on: a small, shared "pact."
# Types are named as strings because the pact is a JSON file, not Python objects.
Path("order_service.pact.json").write_text(json.dumps({
    "consumer": "billing-service",
    "expects": {"order_id": "str", "amount": "int", "currency": "str"},
}))

TYPES_BY_NAME = {"str": str, "int": int, "float": float}


def get_order(order_id):
    # stand-in for the real provider endpoint or database row
    return {"order_id": order_id, "amount": 500, "currency": "INR"}


def get_order_v2_broken(order_id):
    # a "harmless" rename that breaks every consumer silently
    return {"order_id": order_id, "amt": 500, "currency": "INR"}


def check_pact(provider_fn, pact_path):
    pact = json.loads(Path(pact_path).read_text())
    sample = provider_fn("O-1")
    return [f for f, t in pact["expects"].items()
            if f not in sample or not isinstance(sample[f], TYPES_BY_NAME[t])]


print("v1 missing fields:", check_pact(get_order, "order_service.pact.json"))
print("v2 missing fields:", check_pact(get_order_v2_broken, "order_service.pact.json"))
Output
v1 missing fields: []
v2 missing fields: ['amount']
Scenario A

In your world

The situation

Your OrderRepository is tested entirely against a hand-rolled, dict-based fake. Migrations only get exercised for real once, against a shared staging Postgres that three teams write to.

What actually breaks

A migration adds a NOT NULL column with no default. The fake repo has never heard of NOT NULL, so its tests stay green. The migration fails against staging at 2am because a concurrent job inserted a row before the migration finished, and staging's data is now half-migrated for three teams at once.

What you'd do on Monday

Add a test that runs the actual migration against a disposable, real Postgres before it ever touches shared staging:

PythonRun the real migration against a disposable Postgres (needs Docker + testcontainers)
from testcontainers.postgres import PostgresContainer
import psycopg2

with PostgresContainer("postgres:16") as pg:
    conn = psycopg2.connect(pg.get_connection_url().replace("+psycopg2", ""))
    with conn, conn.cursor() as cur:
        cur.execute("CREATE TABLE orders (id INT PRIMARY KEY, amount INT)")
        cur.execute("INSERT INTO orders VALUES (1, 500)")
        cur.execute("ALTER TABLE orders ADD COLUMN currency TEXT NOT NULL")  # fails: existing row has no value
Scenario B

Imagine you work at Flipkart

a catalogue API with 40 downstream consumer teams

The numbers

Illustrative: a catalogue service has 40 consumer teams; a full-service integration test that boots 5 real dependencies (catalogue, pricing, inventory, search, cart) at about 90 seconds each serially costs 450 seconds per run, so 40 teams each running that before every PR is 40 × 450s = 18,000s, 5 hours of aggregate CI compute a day.

What the Scenario-A answer does here

The Scenario A fix, a real disposable Postgres for your own service, still works for one service's own dependency, but "boot all 5 real dependent services for every consumer's PR" does not scale: 40 teams paying that 450-second boot cost, and one flaky dependent service anywhere fails everyone's pipeline.

What flips at this scale

At this scale you replace "boot the real other service" with a contract test per pair: each consumer records and publishes its pact against the catalogue API's schema in its own fast pipeline, with no other service running, and the catalogue team replays every published pact in a single job before it deploys, one team's boot cost, not 40.

The trade-off you'd defend

I would defend contract tests over full multi-service integration for cross-team boundaries, trading a slightly higher chance of missing an emergent multi-service behavior for turning a 5-hour aggregate CI cost into two fast, independently owned jobs.

Illustrative figures for the arithmetic; not Flipkart-published numbers.

Gotchas

A test asserts before the container is actually ready

Symptom

"connection refused" against a real testcontainer, intermittently, especially on a slower CI runner.

Cause

The test raced the container's startup instead of using the library's readiness wait.

Fix

Rely on the library's built-in wait strategy (a log line, an open port), not a fixed sleep, and confirm the CI runner actually supports containers.

Two test files' containers collide on the same port

Symptom

Parallel test runs fail with "address already in use" on 5432.

Cause

A fixed host port was hardcoded instead of letting the library assign an ephemeral one.

Fix

Read the assigned connection string back from the library (get_connection_url()); never hardcode the container's host port.

A contract test never catches the breaking change it exists for

Symptom

A provider ships a breaking field rename and no pipeline flags it.

Cause

The provider verified against a stale, previously recorded pact instead of the consumer's latest published expectations.

Fix

Run provider verification against the latest published consumer pacts as part of the provider's own pipeline on every deploy, not once when the contract was written.

Tricky questions

Is testing against a real, disposable Postgres basically testing in production?
Most people say

Yes, so you can skip staging checks once this passes.

Actually

It's a single-node, unseeded, disposable container. It proves your queries and migrations are structurally sound, not that they perform under real data volume, replicas, or load; staging and production checks are still a separate, necessary guarantee.

Is a contract test just an integration test with a smaller scope?
Most people say

Yes, it's an integration test that only checks part of the API.

Actually

A contract test can run with the other service never running at all, against a recorded or shared schema. That decoupling, not the scope, is the actual point.

Scenario question

You're handed two independently deployed services, orders and billing, whose only test coverage is manual QA clicking through staging before every release. What do you add first, and why not full end-to-end automation?

Model answer

I'd add a consumer-driven contract test first: billing records what fields and types it needs from orders' API, and orders' pipeline replays that pact before every deploy. It's cheap, no second service has to be running, and it catches exactly the failure mode manual QA is standing in for, a breaking API change. Full e2e would need both services deployed together, be slower, and still let orders redeploy without billing ever testing against the new version. I'd add full e2e only for the one or two flows that must be provably correct across the whole stack, not as the first line of defense.

Interview corner

CommonWhy not just mock the database in every test?
“A mock never enforces a real constraint, like a unique key or a foreign key, so it can't catch a bug that only shows up against real engine behavior. I'd test my own schema and queries against a real, disposable database instead.”
TrickDoesn't a real disposable container make your tests as slow and flaky as hitting shared staging?
“Less than you'd think, because it's isolated and torn down every run, so there's no shared state to go stale. It is slower than a fake, which is exactly why I reserve it for my own data layer, not for every collaborator.”
Experience checkTell me about a bug a real-engine test caught that a mock wouldn't.
“A dict-based fake happily overwrote a duplicate order id; a real database's unique constraint rejected it immediately with an IntegrityError, which is the exact failure we needed to catch before it hit production.”
DesignHow would you test two independently deployed services without booting both?
“I'd reach for a consumer-driven contract test: the consumer publishes what it expects from the provider's API, and the provider replays every published pact in its own pipeline before deploying. Neither pipeline needs the other service running.”

Memory hook

A fake is reading the script alone; a real disposable container is a dress rehearsal on a real stage that gets struck afterward.

If you remember only one thing:

Test your own infrastructure with the real engine, disposable; test someone else's API with a contract, not a live boot.

Self-check

  1. What does a real, disposable Postgres container catch that an in-memory fake cannot?

    Real constraint enforcement (unique keys, foreign keys, types), real transaction semantics and real SQL error codes.

  2. Why can a contract test run without the provider service ever starting?

    Because it checks the consumer's recorded expectations against a schema or a replay, not against a live running collaborator.

  3. Why is passing a real-container test still not equivalent to a staging check?

    The container is unseeded and single-node; it proves structural correctness, not behavior under real data volume or load.

Prove it

Add a real-engine test, testcontainers if Docker is available, sqlite3 as a stand-in otherwise, that runs your service's actual migration DDL and inserts a row that violates a real constraint. Paste the exact IntegrityError message caught.

ce-05-05Card 5 of 6

Investigate flakiness and misleading coverage

MediumP2 · Very likelyKnow cold9 min
flaky-teststest-coverageci

Why you care

A test that "sometimes fails" trains a team to click re-run, and a rerun-until-green habit makes a real, intermittent regression statistically indistinguishable from noise. Meanwhile a coverage percentage tells you a line executed, not that anything checked its result, so "92% coverage" can sit right next to a customer-reported bug in code a test technically ran. Both are trust problems, not tooling problems, and interviewers probe them because "make CI green again, for real" is a senior engineer's daily job, not a one-time setup task.

Explain like I'm 5

A flaky test is a smoke alarm that beeps at random, with no smoke. Eventually someone pulls the battery, silencing it exactly when there's a real fire. Coverage without real assertions is a metal detector with no battery: it sweeps over the ground (the line executes) but never actually detects anything, because nothing was checked.

Core concept

  • Flaky root causes: shared mutable state (a global, a class attribute, a fixture with the wrong scope leaking between tests), timing races (a sleep-based wait instead of an explicit signal), reachability of a real external dependency inside a supposed "unit" test, and unseeded randomness or unordered results.
  • Deterministic reproduction before quarantine: run the failing test alone, many times; run the suite with a fixed order or seed; find which other test poisons it via shared state, rather than guessing. Only after that, quarantine deliberately, with a ticket, an owner and a removal date, never a silent @skip that nobody tracks.
  • Why coverage lies: line and statement coverage record that code executed, not that any assertion checked its result. Branch coverage is stronger (did both sides of an if run) but still not sufficient. Mutation testing, deliberately breaking the code and checking whether a test then fails, is the sharper measure of assertion quality; it's expensive, so use it selectively on the logic that actually matters.
  • Decision rule: treat a coverage percentage as a map of what's untested, never a quality target. Treat a flaky test as a bug, in the test or in the code it tests, never as noise to route around with a retry.

Minimal example Retrying a flaky test statistically hides a real regression

PythonThe same math applies to a flaky test and a real intermittent bugruns
def pass_prob_after_retries(fail_rate, retries):
    """Probability at least one of (retries + 1) independent attempts passes."""
    attempts = retries + 1
    return 1 - fail_rate ** attempts


for retries in (0, 1, 2, 3):
    p = pass_prob_after_retries(0.20, retries)
    print(f"20% failure rate, {retries} retries -> reported green {p:.1%} of the time")
Output
20% failure rate, 0 retries -> reported green 80.0% of the time
20% failure rate, 1 retries -> reported green 96.0% of the time
20% failure rate, 2 retries -> reported green 99.2% of the time
20% failure rate, 3 retries -> reported green 99.8% of the time

In production Executing every branch is not the same as asserting the result

Python100% branch coverage, then a mutation the weak test never catchesruns
def classify_amount(amount):
    if amount < 0:
        return "invalid"
    elif amount < 1000:
        return "small"
    else:
        return "large"


def classify_amount_bug(amount):
    if amount < 0:
        return "invalid"
    elif amount < 1000:
        return "small"
    else:
        return "small"          # mutation: the "large" branch is broken


def weak_test(fn):
    """Executes every branch, 100% branch coverage, but asserts nothing."""
    for amount in (-5, 500, 5000):
        fn(amount)               # branch executed, result discarded
    return "ran with 100% branch coverage, asserted nothing"


def strong_test(fn):
    result = fn(-5)
    assert result == "invalid", f"amount=-5 expected 'invalid' got {result!r}"
    result = fn(500)
    assert result == "small", f"amount=500 expected 'small' got {result!r}"
    result = fn(5000)
    assert result == "large", f"amount=5000 expected 'large' got {result!r}"
    return "passed"


print(weak_test(classify_amount_bug))
try:
    print(strong_test(classify_amount_bug))
except AssertionError as e:
    print("strong_test caught the mutation:", e)
Output
ran with 100% branch coverage, asserted nothing
strong_test caught the mutation: amount=5000 expected 'large' got 'small'
Scenario A

In your world

The situation

test_reconciliation_job calls time.sleep(0.5) then asserts a background thread finished. It fails about one run in twenty, and the habit is to click "re-run failed jobs" in Jenkins.

What actually breaks

A refactor introduces a genuine race in the reconciliation writer. The build goes red, gets re-run once, and goes green, because the team already treats this test's redness as noise. The race ships.

What you'd do on Monday

Stop guessing with sleep; wait on an explicit completion signal with a real, bounded timeout that fails loudly if it's never set:

PythonPoll for a real completion signal instead of guessing a sleepruns
import threading
import time


def start_reconciliation(done_event):
    def work():
        time.sleep(0.05)          # stand-in for real work
        done_event.set()
    threading.Thread(target=work, daemon=True).start()


done = threading.Event()
start_reconciliation(done)
finished = done.wait(timeout=2)   # bounded, explicit, no guessing
print("finished before timeout:", finished)
Output
finished before timeout: True
Scenario B

Imagine you work at Meesho

a 3,200-test nightly suite

The numbers

Illustrative: 3,200 nightly tests include 40 known-flaky tests (1.25%) averaging an 8% individual failure rate; with no retries that's roughly 40 × 0.08 ≈ 3.2 spurious failures per run, meaning 3 to 4 unrelated re-triggers across about 15 daily merges, each rerun costing about 15 minutes, so 45 to 60 CI-minutes a day spent re-running tests that were never wrong about the code.

What the Scenario-A answer does here

The Scenario A fix, polling on a real signal, removes that specific timing race, but a blanket "auto-retry on any failure" policy across 3,200 tests still cannot tell an 8%-flaky test from a rare, real regression that also happens to fail about 8% of the time on certain input data.

What flips at this scale

At this scale flaky tests get a dashboard, not a shrug: track each test's failure rate over its last N runs, auto-quarantine anything crossing a threshold (say, 2% over 200 runs) with a ticket and an owner, so the main suite stays a trustworthy, effectively 0%-flaky signal, and quarantined tests get a weekly triage instead of infinite silent retries.

The trade-off you'd defend

I would defend a quarantine dashboard with an ownership deadline over blanket retries, trading a short-term coverage gap on quarantined tests, since they aren't protecting anyone while quarantined, for keeping "red means something changed" true for everything else.

Illustrative figures for the arithmetic; not Meesho-published numbers.

Gotchas

A test fails only when run after a specific other test

Symptom

A test passes alone but fails only inside the full suite, depending on run order.

Cause

Shared module-level or class-level state, a global cache, a singleton, a wrongly-scoped fixture, leaks between tests.

Fix

Rebuild or reset that state per test; use function-scoped fixtures, never rely on suite order for correctness.

Failure rate correlates with time of day or day of week

Symptom

CI is flakier on Fridays, or at specific hours.

Cause

The test depends on the real wall clock, or on an external dependency whose load varies by day.

Fix

Inject a fixed clock and stub or bound the external dependency's timing deterministically.

100% coverage sits next to a real bug in "tested" code

Symptom

Coverage reports 100% on a function a customer just reported broken.

Cause

The covering test executed the line but asserted something trivial, or nothing about the actual result.

Fix

Audit assertions on critical paths, and run a targeted mutation check, break the logic on purpose, confirm the test then fails, instead of trusting the percentage.

Tricky questions

If a flaky test passes on rerun, is the code fine?
Most people say

Yes, if it eventually passes, the code works.

Actually

Passing after a rerun only means timing, order or randomness happened to align that time. It says nothing about the code, and a real intermittent bug looks statistically identical.

Coverage went from 80% to 95%; is the suite meaningfully better?
Most people say

Yes, higher coverage always means fewer bugs slip through.

Actually

Coverage counts executed lines, not verified outcomes. The new tests may execute new code while asserting something trivial or nothing at all; check what they actually assert before trusting the number.

Scenario question

You're handed a suite reporting 92% line coverage and a production incident in a function that had a test covering it. How do you explain that, and what do you check first?

Model answer

I'd explain that coverage measures execution, not verification: a line running inside a test proves nothing about whether the test's assertions would catch a wrong result. I'd open that specific test first and check whether its assertions actually pin down the value that broke, versus asserting something trivial like "no exception was raised." If the assertion is weak, I'd add one that would have failed against the buggy code, a quick mutation check, temporarily reintroduce the bug and confirm the test now fails, and treat the 92% number as a map of what's untested, not a quality score.

Interview corner

CommonWhat causes a flaky test, and how do you find the cause?
“Usually shared mutable state, timing races, or a reachable external dependency inside what should be an isolated test. I find it by running the failing test alone repeatedly, then with a fixed run order, to see whether another test is poisoning it.”
TrickIs 100% coverage a good goal?
“It's a good floor for finding untested code, not a quality target. I'd rather have 70% coverage where every assertion checks a real result than 100% where half the tests execute code and assert nothing.”
Experience checkTell me about a flaky test that turned out to hide a real bug.
“I had a sleep-based wait in a reconciliation test that flaked about one run in twenty; we re-ran past it, and eventually a genuine race in the writer shipped behind that same 'known flaky' test. Replacing the sleep with an explicit wait-for-signal caught the real bug the next time.”
DesignHow would you manage flaky tests across a suite of thousands?
“I'd track each test's failure rate over its recent runs, auto-quarantine anything crossing a threshold with a ticket and an owner, and keep the main suite's signal trustworthy by removing quarantined tests from the merge gate until they're fixed.”

Memory hook

A flaky test is a smoke alarm that beeps at random, until someone pulls the battery, right when there's a real fire.

If you remember only one thing:

Retrying doesn't fix flaky; it just makes real regressions look flaky too.

Self-check

  1. Why can't blanket retries distinguish a flaky test from a real intermittent bug?

    The math is identical: both look green after enough retries, because the retry policy doesn't know why either one fails.

  2. What does 100% line coverage actually guarantee?

    Only that those lines executed during some test run; nothing about whether any assertion checked the result.

  3. Name two common causes of test flakiness.

    Shared mutable state leaking between tests, and dependence on real timing (sleeps, the wall clock) instead of an explicit signal.

Prove it

Take a test with a time.sleep()-based wait in your own code, or write one. Run it 30 times back to back and record the failure count. Replace the sleep with an explicit wait-for-signal, rerun 30 times, and paste both failure counts.

ce-05-06Card 6 of 6

Prove a data pipeline survives bad data and reruns

HardP2 · Very likelyKnow cold11 min
data-pipeline-testingrerunsidempotencyreconciliation

Why you care

A pipeline "passing its tests" usually means the happy-path sample CSV loaded once, on a clean day. It says nothing about what happens when a column disappears, a batch arrives twice, or the job dies at row 40,000 and gets rerun from the CLI, and those are exactly the failures that happen in production. This is precisely what separates a junior test suite ("it works on my sample file") from a senior one, and it is why interviewers ask "how do you test a pipeline," not "how do you test a function": it reveals whether you have operated one, not just written one.

Explain like I'm 5

Crash-testing a car. You don't only check that it drives on a smooth road, the happy path. You deliberately crash it into missing airbags (nulls in required fields), send it a duplicate delivery of the same car (a resent batch), and pull the plug halfway through assembly and restart the line (a crash mid-batch, then a rerun), to prove nobody ends up with two cars, or half a car.

Core concept

Each pipeline failure mode needs its own deliberate test and its own assertion, not a bigger sample file:

  • Schema drift: a column is renamed, removed or retyped upstream. Assert the job either adapts, for a documented optional column, or fails loudly at ingestion, not silently three stages later at the load step.
  • Missing or null records: assert required-field nulls are counted and quarantined, not silently dropped or coerced to 0 or empty string.
  • Counts and reconciliation: assert rows-in equals rows-out plus rejected, exactly, on every run, not "looks about right."
  • Duplicates: assert a resent batch, or a retried call, doesn't double the target; this is the idempotent-write pattern in practice.
  • Late data: assert a record arriving after its processing window closed is handled by an explicit policy, reprocess the window or route to a correction bucket, not silently dropped by a cutoff comparison.
  • Partial failure, restart and rerun: kill the job mid-batch on purpose and rerun it; assert the final state matches "ran once," not "ran once plus a partial extra."

Correctness and freshness are different guarantees. A pipeline can be perfectly correct, every row reconciled, no duplicates, while breaching a freshness SLO, or perfectly on time while quietly wrong. Test both, separately; a green freshness dashboard is not evidence of correctness.

Decision rule: for a new pipeline stage, write the rerun-after-partial-failure test before the happy-path test. If you cannot answer "what happens when this job dies at row 40,000 and reruns," you do not know the pipeline yet.

Under the hood

A transaction in Postgres, or SQLite, is atomic at the storage-engine level: a reader sees either every row of a committed batch or none of them, never half. That single fact is what makes "kill the job mid-batch and rerun" a meaningful, deterministic test in the first place. There are really only two states a rerun needs to handle: "died before commit" (the batch is entirely missing, safe to reprocess) and "died after commit, before the checkpoint was durably recorded." If the checkpoint update is written in the same transaction as the batch insert, those two states collapse into one: either both happened or neither did, and "resume from the last checkpoint" is provably safe. Test at that transaction boundary, not at "row 40,000 out of 100,000," which is not a state the database can actually be caught in.

Minimal example Rows in must equal rows out plus rejected

PythonReconciliation as an assertion, not a glance at a dashboardruns
def process_batch(rows):
    good, rejected = [], []
    for r in rows:
        if r.get("amount") is None:
            rejected.append(r)
        else:
            good.append(r)
    return good, rejected


batch = [{"id": 1, "amount": 100}, {"id": 2, "amount": None}, {"id": 3, "amount": 50}]
good, rejected = process_batch(batch)
assert len(good) + len(rejected) == len(batch), "reconciliation failed: rows vanished"
print("in:", len(batch), "out:", len(good), "rejected:", len(rejected))
Output
in: 3 out: 2 rejected: 1

In production Kill mid-batch, rerun from checkpoint, no duplicates

PythonA crash right after a commit, then a rerun that resumes safelyruns
import sqlite3
from itertools import batched

conn = sqlite3.connect(":memory:")
conn.execute("CREATE TABLE orders (id INTEGER PRIMARY KEY, amount INTEGER)")
conn.execute("CREATE TABLE checkpoint (last_batch INTEGER)")
conn.execute("INSERT INTO checkpoint VALUES (0)")
conn.commit()

rows = [(i, i * 10) for i in range(1, 11)]  # 10 rows, batch size 3


def load(rows, batch_size, crash_after_batch=None):
    last = conn.execute("SELECT last_batch FROM checkpoint").fetchone()[0]
    for n, batch in enumerate(batched(rows, batch_size), start=1):
        if n <= last:
            continue                       # already committed: skip on rerun
        with conn:                         # rows + checkpoint, one transaction
            conn.executemany("INSERT OR IGNORE INTO orders VALUES (?, ?)", batch)
            conn.execute("UPDATE checkpoint SET last_batch = ?", (n,))
        if crash_after_batch == n:
            raise RuntimeError(f"simulated crash after batch {n}")


try:
    load(rows, batch_size=3, crash_after_batch=2)
except RuntimeError as e:
    print("crashed:", e)

load(rows, batch_size=3)  # rerun: resumes from the checkpoint

total = conn.execute("SELECT COUNT(*) FROM orders").fetchone()[0]
print("rows landed after crash + rerun:", total, "expected:", len(rows))
Output
crashed: simulated crash after batch 2
rows landed after crash + rerun: 10 expected: 10

See it happen What a crash-and-rerun test actually proves

batches 1-4donedonedonedonecheckpoint4rows landed10
  1. Job starts fresh: checkpoint is 0, all four batches are pending.
  2. Batch 1 commits atomically with the checkpoint update, in the same transaction.
  3. Batch 2 commits the same way; the process then crashes right after this commit.
  4. Crash: nothing half-committed exists. The database holds exactly two full batches, never one and a half.
  5. Rerun starts: the checkpoint says 2, so batches 1 and 2 are skipped, not re-inserted.
  6. Batch 3 commits; checkpoint becomes 3.
  7. Batch 4 commits; checkpoint becomes 4. Ten rows landed, each exactly once.
Notice: every commit updates the checkpoint in the same transaction, so a rerun can never duplicate a batch or skip one; it only needs to know the last number that made it into the database.
Scenario A

In your world

The situation

Your nightly ETL loads a vendor CSV into Postgres. It has been "tested" with one sample file that always has clean data and never crashes mid-run.

What actually breaks

The vendor ships a file where 20,000 of 60,000 rows have a null amount, all clustered near the end (a broken export on their side). Rows 1 through 40,000 are clean and load fine; int(row["amount"]) raises on the first null at row 40,001, the process dies with no checkpoint, and the on-call reruns the whole job from the CLI, which re-inserts the first 40,000 already-loaded rows because there's no dedupe key, doubling that morning's settlement totals.

What you'd do on Monday

Write the "does a rerun duplicate anything" assertion before the loader is called done:

PythonTest: rerunning the loader must not double-countruns
def load_once(rows, already_loaded_ids):
    inserted = 0
    for r in rows:
        if r["id"] in already_loaded_ids:
            continue                      # idempotent: skip what's already there
        already_loaded_ids.add(r["id"])
        inserted += 1
    return inserted


loaded_ids = set()
first_run = load_once([{"id": 1}, {"id": 2}, {"id": 3}], loaded_ids)
second_run = load_once([{"id": 1}, {"id": 2}, {"id": 3}, {"id": 4}], loaded_ids)
print("first run inserted:", first_run, "second run inserted:", second_run)
assert first_run + second_run == 4, "rerun duplicated or dropped rows"
Output
first run inserted: 3 second run inserted: 1
Scenario B

Imagine you work at Netflix

a 24-partition viewing-events pipeline

The numbers

Illustrative: a viewing-events pipeline processes 900M events/day, about 10,400 events/second across 24 partitions, roughly 433 events/second/partition; a single partition's hourly batch is about 1.5M events, so a 45-minute mid-batch crash affecting just one of the 24 partitions could double-count up to roughly 1.5M events, about 0.17% of daily volume, before reconciliation catches it.

What the Scenario-A answer does here

The Scenario A fix (idempotent skip-by-id, reconciled per batch) still works per partition, but "just check the total row count matches" does not scale to 24 independently rerunning partitions: a global sum can show the "right" total while one partition over-counts and another under-counts, and neither error is visible in the aggregate.

What flips at this scale

At this scale reconciliation has to be per-partition and per-batch-window, not one global sum, and each partition's rerun test asserts its own checkpoint independently. Late-arriving events need the same treatment as a crash rerun, because both are "the same event shows up again, later," and both need a tested, explicit policy.

The trade-off you'd defend

I would defend per-partition reconciliation with independent checkpoints over one global count, accepting more bookkeeping, 24 checkpoints instead of one, for the ability to catch a single bad partition instead of only a suspiciously round total.

Illustrative figures for the arithmetic; not Netflix-published numbers.

Gotchas

A global reconciliation total hides a local error

Symptom

The dashboard says counts match; a support ticket says a specific customer was billed twice.

Cause

Reconciliation summed across partitions or batches instead of checking each one, so an over-count in one canceled an under-count in another.

Fix

Reconcile per partition and per batch window, not only the grand total.

A rerun after a schema change produces different output than the original run

Symptom

The same source file loads differently before and after an incident's mid-fix.

Cause

An upstream schema change landed between the crash and the rerun, so the rerun processes rows under a different schema than the original run assumed.

Fix

Record the schema version with each batch's checkpoint, and fail the rerun loudly if it doesn't match, rather than silently reprocessing under a new schema.

A late-arriving record is silently missing from every report

Symptom

A record that is visibly present in the source never appears in any downstream aggregate.

Cause

The pipeline's window cutoff (event_time < window_end) treats "late" as "doesn't exist" instead of routing it somewhere.

Fix

Assert an explicit, tested policy exists for late records: reprocess the affected window, or land them in a correction bucket.

Tricky questions

If the nightly DAG runs green against a sample file every night, is the pipeline well tested?
Most people say

Yes, a nightly green run on real infrastructure is strong evidence.

Actually

A clean, small sample only exercises the happy path. It proves nothing about surviving a crash mid-batch, a duplicate delivery, or a null in a required column; those need their own deliberate tests, not a bigger sample.

If the freshness dashboard is green, is the data correct?
Most people say

Yes, on-time delivery implies the data made it through correctly.

Actually

Freshness (arrived on time) and correctness (the values are right) are independent guarantees. A job can finish exactly on schedule while having silently dropped or duplicated rows.

Scenario question

You're handed a pipeline that reconciles "rows read from the source" against "rows in the warehouse table," and the counts always match, yet finance found a client billed twice last month. What test did the team not write, and what do you add?

Model answer

I'd suspect the reconciliation checks a count, not identity, so a duplicate delivery, an at-least-once redelivery or a rerun after a crash, inserted the same event twice under a new generated row id, while something else was under-counted enough that the total still looked right. "Count matches source" is not the same claim as "no duplicate business keys." I'd add a test that runs the same batch through the pipeline twice and asserts the target table's distinct business-key count stays flat, not its row count, and I'd make the write itself idempotent on that key rather than relying on the test to catch it after the fact.

Interview corner

CommonHow do you test an ETL pipeline beyond the happy path?
“I'd deliberately test schema drift, null and missing records, duplicate delivery, late data, and a crash mid-batch followed by a rerun, each with its own assertion, reconciliation counts, dedupe on business key, an explicit late-data policy, rather than trusting one clean sample file.”
TrickIf your freshness SLO is green, is the data correct?
“Not necessarily. Freshness tells me it arrived on time; correctness tells me the values are right. I test both separately, because a job can be perfectly on schedule while silently dropping or duplicating rows.”
Experience checkTell me about a rerun that went wrong.
“A crash partway through a load with no checkpoint meant the on-call reran the whole job from the start, and because there was no dedupe key, it doubled everything already committed. Now I always test that rerunning a batch twice produces the same result as running it once.”
DesignHow would you test that a pipeline survives a crash mid-batch?
“I'd deliberately raise an exception partway through a multi-batch load, rerun the loader, and assert two things: the final row count matches the source, and no business key appears twice in the target. That's the same test whether the crash was real or simulated.”

Memory hook

Crash-test the pipeline: kill it mid-batch on purpose, rerun it, and prove nobody got two rows or half a row.

If you remember only one thing:

If you can't say what happens when this job dies at row 40,000, you don't know the pipeline yet.

Self-check

  1. Why is a single global row-count reconciliation not enough at scale?

    It can mask offsetting errors, one partition over-counting while another under-counts, that cancel out in the total.

  2. What's the difference between correctness and freshness for a pipeline?

    Correctness is whether the data is right; freshness is whether it arrived on time. They are independent guarantees.

  3. Why does committing the checkpoint in the same transaction as the batch make rerun-testing meaningful?

    It collapses every possible crash into exactly two states, batch fully there or fully absent, instead of infinite partial states.

Prove it

Write a loader over a 10-batch file, deliberately raise an exception inside batch 6, and rerun the loader. Assert the final row count equals the source count and no business key appears twice in the target. Paste both counts.