Shelf · Core Eng · Chapter 11

Chapter 11: Working like a senior

Writing an ETL script that runs is a different skill from being trusted with decisions nobody double-checks line by line. This chapter is six of those trust-building habits: reviewing for real risk, recording decisions so they outlive you, debugging unfamiliar code by method, estimating and scoping honestly, arguing for fixes with numbers, and running an incident people actually trust.

Difficulty
Priority
ce-11-01Card 1 of 6

Senior engineering practice: review, ADRs, debugging and scoping

MediumP2 · Very likelyTalk about it11 min
code-reviewdecision-recordsdebuggingscopingsenior-practice

Why you care

"Walk me through how you review a pull request" is a standard senior-level interview question because the answer predicts whether you catch the bug that matters or the one that doesn't. A junior reviewer comments on variable names; a senior reviewer traces what happens when the network call times out, whether the new endpoint checks that the caller owns the row it's reading, and whether the PR's claim ("tests pass") has evidence attached. On the job this is the difference between a review that is a formality and a review that is the actual safety net between a bug and production. This card is the index for the chapter: the other five cards (writing decisions down, debugging unfamiliar code, estimating and scoping, arguing for fixes, running incidents) are the deeper, specific skills a reviewer and reviewee both lean on.

Explain like I'm 5

A building inspector and an interior decorator can walk through the same house. The decorator comments on paint color and furniture placement. The inspector checks the load-bearing walls, the wiring behind the switch plates, and whether the exits actually open. Both kinds of comments are real, but only one kind stops the house from collapsing. A senior review is an inspection first; the paint comments can come after, clearly labelled as paint comments.

Core concept

Giving a review well means working in a fixed order, not reading top to bottom and reacting line by line:

  • Read for intent first. What problem is this PR solving, and does the diff actually solve it? You cannot spot a wrong fix if you never confirmed what "right" means here.
  • Check correctness and security risk explicitly: authentication/authorization on the new code path, injection or unvalidated input, resource cleanup on the failure path, and what happens when a dependency times out or returns something unexpected. This is a checklist, not a vibe; Application threat boundaries: SQLi, SSRF and IDOR and Make PRs and hooks useful safeguards own the specific attack and hook mechanics this check leans on.
  • Request evidence, not claims. "Tests pass" is a claim; a pasted test run or a link to a green CI check with the relevant test named is evidence. If the PR touches a failure path, ask for the output that shows the failure path was actually exercised.
  • Separate blocking issues from suggestions, and say which is which. A comment with no label reads as a demand. Decide: does leaving this unresolved risk correctness, security, or data loss (blocking), or is it a style/readability preference (suggestion, non-blocking)?
  • Respond to disagreement with rationale, not re-assertion. If the author pushes back, either you find the concrete risk you were pointing at (and say what it is), or you concede and move on. "I still think it's wrong" without a reason is not a review comment.

The decision rule: label a comment blocking only if you can name the concrete, observable failure it prevents (an error, a wrong count, a leaked credential, an unauthorized read). If you can't name the failure, it's a suggestion, and it should read like one.

Minimal example Blocking or suggestion, by rule not vibe

PythonClassify review comments by a concrete risk signalruns
def classify(comment):
    blocking_signals = ("auth", "sql injection", "secrets", "no error handling", "data loss", "race condition")
    text = comment.lower()
    if any(sig in text for sig in blocking_signals):
        return "blocking"
    return "suggestion"


comments = [
    "rename this variable to something clearer",
    "this swallows the exception with no error handling, we lose failed rows silently",
    "could extract this into a helper function",
    "this query builds SQL by string concatenation, looks like a sql injection risk",
]
for c in comments:
    print(classify(c), "-", c[:40])
Output
suggestion - rename this variable to something cleare
blocking - this swallows the exception with no erro
suggestion - could extract this into a helper functio
blocking - this query builds SQL by string concaten

In production Scan a diff for the patterns worth blocking on

A real review tool would use an AST or a linter; this stands in with plain text scanning over a unified diff's added lines, which is enough to show the two most common blocking patterns: a swallowed exception and string-built SQL.

PythonFlag risky added lines before reading a single commentruns
from pathlib import Path

diff_text = (
    "+def process_payment(order):\n"
    "+    try:\n"
    "+        charge(order)\n"
    "+    except Exception:\n"
    "+        pass\n"
    "+\n"
    "+def run_query(user_input):\n"
    '+    cur.execute("SELECT * FROM orders WHERE id = " + user_input)\n'
    "+    return cur.fetchone()\n"
)
Path("pr.diff").write_text(diff_text)

added = [ln[1:] for ln in Path("pr.diff").read_text().splitlines() if ln.startswith("+")]
flags = []
for i, line in enumerate(added):
    stripped = line.strip()
    if stripped == "pass" and i > 0 and "except" in added[i - 1]:
        flags.append((i, "exception swallowed with no logging or re-raise"))
    if "execute(" in line and " + " in line:
        flags.append((i, "string-built SQL: possible injection risk"))

for lineno, reason in flags:
    print(f"line {lineno}: {reason}")
print("blocking issues found before reading a single human comment:", len(flags))
Output
line 4: exception swallowed with no logging or re-raise
line 7: string-built SQL: possible injection risk
blocking issues found before reading a single human comment: 2
Scenario A

In your world

The situation

A teammate opens a PR that adds a retry wrapper around the Postgres load step in your nightly settlement job. CI is green, the diff is 60 lines, and you have 10 minutes before standup.

What actually breaks

You skim it, leave two comments about variable names, and approve. Three weeks later the retry wrapper's except Exception: pass inside the retry loop quietly eats a psycopg2.OperationalError on connection loss, the job "succeeds" having loaded half the file, and nobody notices until the finance reconciliation flags a short month.

What you'd do on Monday

Before approving anything that touches a failure path, name the two or three concrete things that could go wrong and check the diff against them specifically:

PythonA two-minute risk checklist for a retry-wrapper PRruns
risk_checklist = [
    "does every except branch log or re-raise, with no bare pass?",
    "does a retry have a cap, so a permanent failure doesn't loop forever?",
    "if the retry exhausts, does the job fail loudly instead of continuing with partial data?",
]
diff_summary = "except Exception: pass inside the retry loop, no retry cap, continues after exhausting retries"

for item in risk_checklist:
    hit = "pass" in diff_summary and "except Exception" in diff_summary
    print(f"- {item}  ->  {'FAILS, blocking' if hit else 'ok'}")
Output
- does every except branch log or re-raise, with no bare pass?  ->  FAILS, blocking
- does a retry have a cap, so a permanent failure doesn't loop forever?  ->  FAILS, blocking
- if the retry exhausts, does the job fail loudly instead of continuing with partial data?  ->  FAILS, blocking
Scenario B

Imagine you work at GitHub

4,000 pull requests merged per day across the org

The numbers

Illustrative: 4,000 PRs/day across an 8-hour review window is 500 PRs/hour. If one careful senior reviewer gives even 15 minutes of real attention per PR, they cover 4 PRs/hour; 50 such reviewers cover 200/hour, less than half the merge rate.

What the Scenario-A answer does here

The Scenario A checklist still works per PR, but "every senior reads every diff by eye for the risk patterns" doesn't scale: at 500 merges/hour, human attention runs out long before the PRs do, and the ones that get rubber-stamped are indistinguishable from the ones that got a real review.

What flips at this scale

At this scale the mechanical half of the checklist (missing error handling, raw SQL concatenation, uncapped retries, required-check status) moves into automated checks that block merge, required via branch protection and CODEOWNERS; a human reviewer's time is spent only on the judgment calls automation can't make, like whether the change actually solves the right problem.

The trade-off you'd defend

I would defend automating the pattern-matching half of review and reserving humans for intent and judgment, trading some false negatives in the automated checks for enough reviewer bandwidth to read for intent at all.

Illustrative figures to make the scale concrete; not GitHub-published numbers.

Gotchas

Stylistic nitpicking blocks a merge

Symptom

A PR sits unmerged for days over naming or formatting comments, while an actual missing authorization check went uncommented.

Cause

No shared convention for labelling comments as blocking versus a suggestion, so every comment reads as a demand.

Fix

Explicitly label comments ("blocking:" / "nit:") and agree as a team that only blocking comments gate the merge.

"LGTM" with no evidence attached

Symptom

A reviewer approves a PR that touches a failure path with no test output or log shown for that path.

Cause

The reviewer assumed CI being green covers the risky branch, which it often doesn't for error handling.

Fix

Ask for the specific evidence (test name, log line, or a local run) that exercises the risky path before approving it.

Comments silently marked resolved

Symptom

An author resolves a reviewer's comment with no reply, and the same bug reappears in a later PR.

Cause

Resolving a comment is easier than engaging with it, especially under deadline pressure.

Fix

Require a reply with a reason (fixed how, or why it's not actually a risk) before a comment can be marked resolved.

Tricky questions

Does zero comments on a PR mean it's clean?
Most people say

Yes, an approval with no comments means the reviewer found nothing wrong.

Actually

It often means the reviewer skimmed. A genuinely thorough review of anything touching money, auth, or a failure path almost always produces at least one question, because those paths are where real systems actually break.

Is "I disagree" a complete review response?
Most people say

Yes, if the reviewer feels strongly, that's enough to keep blocking.

Actually

A blocking comment needs a nameable failure. "I disagree" without naming what breaks is a preference dressed up as a veto; the reviewer either names the concrete risk or downgrades the comment to a suggestion.

Scenario question

You're handed a PR that adds a retry wrapper around a payment call: 400 lines, all CI checks green, and two reviewers already approved with zero comments. What do you do before adding your own approval?

Model answer

I'd treat the clean approvals as a signal to look harder, not softer, especially on a payment path. I'd check the three things that matter here: does every except branch log or re-raise instead of swallowing, is there a retry cap, and does exhausting the retries fail loudly instead of continuing with partial state. I'd ask for the specific test or log evidence that the failure path was exercised rather than trust the green check. If I find a real gap I'd label it blocking and name the concrete failure it causes; if I don't, I'd say explicitly what I checked, so my approval carries information instead of just adding a third rubber stamp.

Interview corner

CommonWhat do you look for first when reviewing a PR?
“I'd start by confirming what problem the PR is solving and whether the diff actually solves it, then specifically check the failure paths: error handling, authorization on new endpoints, and resource cleanup. Style comes last, and I label it as a suggestion, not a blocker.”
TrickTwo reviewers already approved a 400-line PR with zero comments. Do you just approve too?
“No, that's actually a reason to look harder. Clean approvals on a large diff usually mean it was skimmed, so I'd check the failure paths myself and ask for evidence the risky branch was tested, rather than add a third rubber stamp.”
Experience checkTell me about a review comment you made that caught a real production bug.
“I once flagged a bare except around a retry loop that swallowed a connection error; without the comment the job would have 'succeeded' with half the rows loaded. I named the concrete failure, the author added logging and a retry cap, and we added a regression test for it.”
DesignHow do you keep review quality high when a team is merging dozens of PRs a day?
“I'd push the mechanical checks, like missing error handling or an uncapped retry, into automated required checks, and keep human review for intent and judgment calls automation can't make. That way reviewer attention goes where it actually matters instead of running out on volume.”

Memory hook

A review is a building inspection first; the paint comments can come after, clearly labelled as paint.

If you remember only one thing:

Label every comment blocking or suggestion by naming the concrete failure it prevents, or it isn't a blocking comment.

Self-check

  1. What makes a review comment "blocking" rather than a "suggestion"?

    You can name the concrete, observable failure (an error, a wrong count, a leaked credential, an unauthorized read) that happens if it isn't fixed. If you can't name one, it's a suggestion.

  2. Why is "tests pass" not the same as evidence a risky path works?

    A green CI check doesn't tell you which branch of the code actually ran; the reviewer needs the specific test name or log output that exercised the failure path in question.

  3. Why should clean, comment-free approvals on a large diff make you more suspicious, not less?

    Large diffs with zero comments usually mean the reviewer skimmed rather than traced the failure paths; a careful review of risky code almost always produces at least one question.

Prove it

Take a real PR you've reviewed (yours or a teammate's) that touched error handling or an external call. Rewrite your review from scratch using the five-step order in this card, labelling every comment blocking or suggestion with the concrete failure it names. Paste the before (your original comments) and after (the relabelled, ordered review).

ce-11-02Card 2 of 6

Write a design doc and preserve the decision in an ADR

MediumP3 · Role-dependentTalk about it11 min
adrdesign-docsdecision-recordsarchitecture

Why you care

"Why did we pick Postgres over Mongo for this service" is a question that gets asked again and again, usually by someone who wasn't in the room the first time. Teams that answer it with "ask Priya, she remembers" lose that answer the day Priya changes teams. Interviewers ask about design docs and decision records because the answer reveals whether you've worked somewhere decisions outlive the people who made them, or somewhere every revisit relitigates the same argument from scratch. This card is the artifact, not a new design method: the actual trade-off analysis is Design interview method: the eight-step flow's job; this is how you make it survive.

Explain like I'm 5

A courtroom keeps a transcript. Months later, nobody has to rely on whoever argued loudest or remembers best; they read what was actually decided and why. A design doc is the trial: the arguments, the alternatives, the evidence. An Architecture Decision Record (ADR) is the one-paragraph verdict that goes in the permanent record after the trial ends, so the next person doesn't have to re-run the whole case to find out what the court decided.

Core concept

Two different documents, for two different moments:

  • A design doc comes before a hard-to-reverse decision. It states the problem and constraints, lays out at least two real alternatives (including "do nothing"), compares them on the axes that matter (cost, risk, latency, operational burden), names the chosen approach, and describes the rollout and how you'll verify it worked.
  • An ADR (Architecture Decision Record) is the compact record written after the decision is made, whether or not a design doc preceded it. The lightweight, widely used shape is four parts: Title and status (proposed, accepted, superseded), Context (what forced this decision), Decision (what you chose, one or two sentences), and Consequences (what gets easier, what gets harder, what you're now committed to).

Decision rule: write a design doc for anything cross-team, expensive to reverse, or genuinely contested among competent people. Write an ADR for every decision with lasting architectural consequence, even a small one, because the lightweight record (a paragraph, five minutes) is what survives when the design doc (or the Slack thread) doesn't get reread. ADRs are numbered and kept together, so "why is it like this" becomes a search, not an archaeology dig. When a later decision reverses an earlier one, mark the old ADR superseded, with a link to the new one; never silently delete it, because the old context is exactly what the next person needs to understand why the new decision was necessary.

Minimal example The four parts of an ADR, as a function

PythonA decision record has a fixed shaperuns
def render_adr(number, title, status, context, decision, consequences):
    return (
        f"# ADR-{number:04d}: {title}\n"
        f"Status: {status}\n"
        f"## Context\n{context}\n"
        f"## Decision\n{decision}\n"
        f"## Consequences\n{consequences}"
    )


doc = render_adr(
    7,
    "Batch size for nightly settlement load",
    "accepted",
    "Nightly load timed out at 1000-row batches once volume passed 2M rows.",
    "Increase batch size to 5000 rows per commit for the settlement loader.",
    "Fewer commits, more rows held per transaction; monitor lock time.",
)
print(doc)
Output
# ADR-0007: Batch size for nightly settlement load
Status: accepted
## Context
Nightly load timed out at 1000-row batches once volume passed 2M rows.
## Decision
Increase batch size to 5000 rows per commit for the settlement loader.
## Consequences
Fewer commits, more rows held per transaction; monitor lock time.

In production A numbered ADR log with supersession

Each new ADR gets the next number automatically, and superseding an old decision marks it rather than deleting it, so the record of why the old choice was made stays searchable.

PythonA small ADR index that tracks supersessionruns
from pathlib import Path


def render_adr(number, title, status, context, decision, consequences):
    return (
        f"# ADR-{number:04d}: {title}\n"
        f"Status: {status}\n"
        f"## Context\n{context}\n"
        f"## Decision\n{decision}\n"
        f"## Consequences\n{consequences}"
    )


adr_dir = Path("adr")
adr_dir.mkdir()


def next_number():
    existing = sorted(adr_dir.glob("[0-9][0-9][0-9][0-9]-*.md"))
    return int(existing[-1].name[:4]) + 1 if existing else 1


def write_adr(title, status, context, decision, consequences, supersedes=None):
    number = next_number()
    slug = title.lower().replace(" ", "-")
    path = adr_dir / f"{number:04d}-{slug}.md"
    path.write_text(render_adr(number, title, status, context, decision, consequences))
    if supersedes:
        old = next(adr_dir.glob(f"{supersedes:04d}-*.md"))
        old.write_text(old.read_text().replace("Status: accepted", f"Status: superseded by ADR-{number:04d}"))
    return path


write_adr(
    "Use Redis cache for session lookups",
    "accepted",
    "Session lookups hit Postgres directly and p99 latency crept past 200ms.",
    "Add a Redis cache in front of session reads.",
    "Extra infra to operate; faster reads.",
)
write_adr(
    "Replace Redis session cache with a materialized view",
    "accepted",
    "ADR-0001's Redis cache added on-call burden with no measured latency win.",
    "Drop Redis; use a Postgres materialized view refreshed every 5 minutes.",
    "One less moving part; a 5-minute staleness window.",
    supersedes=1,
)

for p in sorted(adr_dir.iterdir()):
    status_line = [ln for ln in p.read_text().splitlines() if ln.startswith("Status")][0]
    print(p.name, "->", status_line)
Output
0001-use-redis-cache-for-session-lookups.md -> Status: superseded by ADR-0002
0002-replace-redis-session-cache-with-a-materialized-view.md -> Status: accepted
Scenario A

In your world

The situation

Three engineers decide in a Slack thread to change the nightly ETL batch size from 1,000 to 5,000 rows, to fix a timeout. No doc, no ticket, just the thread.

What actually breaks

Three months later a new teammate, chasing a lock-contention complaint, reverts the batch size back to 1,000 "to be safe," re-introducing the exact timeout the original change fixed. Nobody can point to why 5,000 was chosen, because the Slack thread has scrolled out of anyone's memory and search.

What you'd do on Monday

Write the ADR the day the decision is made, not after, and require config changes above a certain blast radius to reference an ADR id in the PR description:

PythonA five-minute ADR, written the day the decision happensruns
def render_adr(number, title, status, context, decision, consequences):
    return (
        f"# ADR-{number:04d}: {title}\n"
        f"Status: {status}\n"
        f"## Context\n{context}\n"
        f"## Decision\n{decision}\n"
        f"## Consequences\n{consequences}"
    )


print(render_adr(
    12,
    "Nightly settlement batch size",
    "accepted",
    "1000-row batches timed out once nightly volume passed 2M rows (see incident #884).",
    "Raise batch size to 5000 rows per commit.",
    "Longer-held transactions per batch; reduced the timeout at current volume. Revisit if volume grows past 10M rows.",
))
Output
# ADR-0012: Nightly settlement batch size
Status: accepted
## Context
1000-row batches timed out once nightly volume passed 2M rows (see incident #884).
## Decision
Raise batch size to 5000 rows per commit.
## Consequences
Longer-held transactions per batch; reduced the timeout at current volume. Revisit if volume grows past 10M rows.
Scenario B

Imagine you work at CRED

36 architecturally significant decisions a month across 12 squads

The numbers

Illustrative: 12 squads making roughly 3 decisions a month each with lasting architectural consequence is 36 decisions/month. Without a shared, searchable record, "why did we pick X" means grepping Slack across 12 channels, each with tens of thousands of messages: effectively unsearchable within a sprint.

What the Scenario-A answer does here

Scenario A's one-ADR-per-decision habit still works, but "write an ADR when you remember to" collapses at this volume: if even a third of decisions get written down, 24 of 36 decisions a month have no record at all, and the gap grows every month.

What flips at this scale

At this scale an ADR becomes a required PR field for anything touching schema, dependencies, or cross-service contracts, with one index per squad linked from a root doc; a missing ADR for an architectural change becomes a blocking review comment, the same severity as a missing test, per Senior engineering practice: review, ADRs, debugging and scoping's decision rule.

The trade-off you'd defend

I would defend making ADRs a required PR field over a wiki page nobody updates, trading a few minutes of friction per architectural PR for an actual answer to "why" six months later.

Illustrative figures to make the arithmetic concrete; not CRED-published numbers.

Gotchas

The ADR is written after the argument is forgotten

Symptom

An ADR written months later says "we chose X because Y," but Y doesn't match what people actually remember discussing at the time.

Cause

No record was kept during the decision itself, so the retroactive write-up reconstructs a cleaner story than what happened.

Fix

Open the ADR as a draft the day the decision is actively being discussed, not after it's already shipped.

A design doc with no real alternatives

Symptom

The doc states "we will do X" with no "why not Y," and reads like a sign-off request rather than a review.

Cause

The writer wanted approval, not scrutiny, so the alternatives section is skipped or token.

Fix

Require at least two real alternatives, including an honest "do nothing" baseline, before anyone signs off.

Two ADRs quietly contradict each other

Symptom

Two accepted ADRs make incompatible claims, and whoever reads them first "wins" by accident.

Cause

No process marks an old ADR as superseded when a new decision replaces it.

Fix

When a new decision replaces an old one, update the old ADR's status to superseded with a link to the new ADR number.

Tricky questions

Is a design doc just a longer version of an ADR?
Most people say

Yes, same content, different length.

Actually

They answer different questions. A design doc explores the decision before it's made, with alternatives and trade-offs at length, for anyone who needs to weigh in. An ADR is the compact record of what was decided, written for someone six months later who just needs the "why," not the whole debate.

Does every decision need a full design doc?
Most people say

Yes, otherwise it isn't a "real" decision.

Actually

No. Small, easily reversible decisions get an ADR alone, often just a paragraph. Full design docs are for decisions that are expensive to reverse or genuinely contested; writing one for every config tweak would make people stop writing them at all.

Scenario question

You're handed a decision made two months ago in a Slack thread by three engineers: drop a Redis cache layer in favor of a Postgres materialized view. There's no doc. A new teammate wants to revert it and bring Redis back, arguing it's "obviously faster," with no record to argue against. What do you do?

Model answer

I'd reconstruct the context from the thread and write the ADR now, marked accepted (retroactive), with the actual reasoning (on-call burden versus measured latency win) and a link to the original thread as evidence. That gives the new teammate something concrete to argue against instead of institutional memory versus a hunch. If they still want to revert it, that's a new decision, and it gets its own ADR that explicitly supersedes the first one, with the measured latency numbers that justify reversing it. I wouldn't block the revert on authority; I'd block it on evidence, the same way Senior engineering practice: review, ADRs, debugging and scoping treats a review comment.

Interview corner

CommonWhat's the difference between a design doc and an ADR?
“I'd say a design doc comes before a hard-to-reverse decision and explores alternatives at length; an ADR is the compact record of what was decided, written afterward, with context, decision, and consequences. Not every decision needs the former, but any decision with lasting architectural consequence should get the latter.”
TrickDoes writing an ADR after the fact, from memory, still count?
“It's better than nothing, but I'd be honest about the risk: a retroactive ADR tends to tell a cleaner story than what actually happened, so I mark it accepted (retroactive) and link to whatever evidence exists, like the original thread, rather than presenting it as a contemporaneous record.”
Experience checkTell me about a decision that would have gone differently with an ADR.
“I've seen a batch size changed to fix a timeout with no record beyond a Slack thread; a new teammate reverted it months later, chasing an unrelated issue, and reintroduced the original timeout. One five-minute ADR would have stopped that revert from shipping.”
DesignHow do you keep an ADR index useful as a team grows?
“I'd keep one numbered index per team, linked from a root doc, and make referencing an ADR id required in the PR description for schema, dependency, or contract changes. A missing ADR for that kind of change becomes a blocking review comment, not a nice-to-have.”

Memory hook

A design doc is the trial; an ADR is the one-paragraph verdict that stays in the record after everyone forgets the argument.

If you remember only one thing:

Write the ADR the day the decision happens, four parts: context, decision, consequences, status, or it won't survive the next revert.

Self-check

  1. What are the four parts of a lightweight ADR?

    Title and status, context, decision, consequences.

  2. Why mark an old ADR "superseded" instead of deleting it?

    The old context is exactly what the next person needs to understand why the new decision was necessary; deleting it loses that history.

  3. When should you write a full design doc instead of just an ADR?

    When the decision is cross-team, expensive to reverse, or genuinely contested among competent people; small reversible decisions get an ADR alone.

Prove it

Pick a real decision your team made in the last few months with no written record (a config value, a library choice, a schema change). Write the ADR for it now, in the four-part shape, including an honest "Context" section that states what actually forced the decision. Paste the ADR.

ce-11-03Card 3 of 6

Read an unfamiliar codebase and debug by hypothesis

HardP2 · Very likelyKnow cold12 min
debuggingcode-readinghypothesis-testingexception-chaining

Why you care

"Walk me through how you'd debug a bug in a codebase you've never seen" is close to the single most common senior-signal question, because the wrong answer ("I'd add print statements everywhere and poke around") predicts exactly how you'll behave on your first on-call incident at a new job: slow, and prone to shipping a fix that didn't actually fix anything. A repeatable method turns "I've never seen this service" from a blocker into a non-issue, because the method doesn't depend on familiarity, only on evidence. It also gives you something concrete to say in an interview instead of "it depends."

Explain like I'm 5

A detective doesn't walk into a room and accuse the first suspect who looks nervous. They secure the scene so it can be examined the same way twice, write down only what's definitely true versus what's merely suspected, form one testable theory at a time, and run the cheapest check that would prove or kill that theory before accusing anyone. Guessing-and-checking without that discipline convicts the wrong suspect just as often as it convicts the right one, and you never find out which happened.

Core concept

Reading an unfamiliar codebase fast means not starting at main(). Start at the entry point for the specific feature you need: the route handler, the queue consumer, the cron job's entry script. From there, trace ownership: who writes the table or queue this code touches, and who else reads it. Find the state transitions: what makes a record move from pending to done or failed, because that's where most real bugs live. Read the tests before the comments; tests describe intended behavior in a form you can actually run, while comments drift out of date silently.

Debugging by hypothesis is a fixed loop, not a sequence of guesses:

  1. Reproduce. Get a deterministic, minimal repro: the same input produces the same failure every time. A bug you can't reproduce on demand can't be confirmed fixed either.
  2. Observe. Write down only directly observable facts: the exact error text, the exact input, the exact timestamp. Not "it's probably the cache," which is a hypothesis wearing an observation's clothes.
  3. Hypothesize. Form exactly one falsifiable guess at a time. Two guesses tested together means you won't know which one mattered when the symptom disappears.
  4. Discriminate. Run the cheapest experiment that would prove or kill that specific hypothesis, before touching the fix.
  5. Fix. Change only the code the confirmed hypothesis justifies.
  6. Regression-test. Lock the fix in with a test that fails before the fix and passes after, so the bug can't come back silently after an unrelated refactor.

Decision rule: when a bug is described as "intermittent," do not start hypothesizing yet. Make it reproducible first (fix the input, the seed, the timing), because an unreproducible "fix" is unverifiable, and you'll find out it didn't work the same week it recurs in production.

Under the hood

A hypothesis-driven debugger reads tracebacks constantly, and Python's exception chaining is where that reading goes wrong most often. When you raise NewError(...) from original, Python sets NewError.__cause__ to original, and the traceback prints "The above exception was the direct cause of the following exception." But if you instead write a bare except X: raise NewError(...) with no from, Python still records the original exception on __context__ (that's what produces "During handling of the above exception, another exception occurred"), but __cause__ stays None. Both tracebacks look similar at a glance, and both preserve the real original error, but only the explicit from form tells a reader this is the direct cause, not just this happened while handling something else. If your logging only prints str(e), you lose the chain entirely either way; reading e.__cause__ and e.__context__ directly is often the fastest way to find the actual root cause in an unfamiliar service's logs.

Minimal example `__cause__` versus `__context__`

PythonExplicit "from" sets __cause__; a bare re-raise doesn'truns
def load_row_explicit(row):
    try:
        return int(row["amount"])
    except KeyError as err:
        raise ValueError(f"bad row: {row}") from err


def load_row_implicit(row):
    try:
        return int(row["amount"])
    except KeyError:
        raise ValueError(f"bad row: {row}")


for fn in (load_row_explicit, load_row_implicit):
    try:
        fn({"id": 1})
    except ValueError as e:
        print(fn.__name__, "cause:", repr(e.__cause__), "context:", type(e.__context__).__name__)
Output
load_row_explicit cause: KeyError('amount') context: KeyError
load_row_implicit cause: None context: KeyError

In production Find a resume bug by hypothesis, then regression-test it

A nightly job resumes from a checkpoint last_id. Something is reprocessing one row every run. The script reproduces it, states one hypothesis, runs the cheapest check that confirms it, fixes it, and locks the fix with a regression test.

PythonReproduce, observe, confirm one hypothesis, fix, regression-testruns
rows = [{"id": i} for i in range(1, 11)]


def fetch_since_buggy(rows, last_id):
    return [r for r in rows if r["id"] >= last_id]  # bug: should be strictly greater


def run_job(fetch, rows, last_id, batch_size):
    batch = fetch(rows, last_id)[:batch_size]
    new_last_id = batch[-1]["id"] if batch else last_id
    return batch, new_last_id


# reproduce: two consecutive runs with the real checkpoint value
batch1, checkpoint = run_job(fetch_since_buggy, rows, last_id=1, batch_size=4)
batch2, _ = run_job(fetch_since_buggy, rows, last_id=checkpoint, batch_size=4)
print("batch1 ids:", [r["id"] for r in batch1])
print("batch2 ids:", [r["id"] for r in batch2])

# observe: a fact, not a guess
overlap = {r["id"] for r in batch1} & {r["id"] for r in batch2}
print("observed overlap:", overlap)

# hypothesize + discriminate: does the resume filter re-include the checkpoint row itself?
hypothesis_confirmed = checkpoint in {r["id"] for r in batch2}
print("hypothesis 'resume re-includes the checkpoint row' confirmed:", hypothesis_confirmed)


def fetch_since_fixed(rows, last_id):
    return [r for r in rows if r["id"] > last_id]


# fix + regression test
fixed_batch1, fixed_checkpoint = run_job(fetch_since_fixed, rows, last_id=1, batch_size=4)
fixed_batch2, _ = run_job(fetch_since_fixed, rows, last_id=fixed_checkpoint, batch_size=4)
fixed_overlap = {r["id"] for r in fixed_batch1} & {r["id"] for r in fixed_batch2}
print("fixed overlap:", fixed_overlap)
assert not fixed_overlap, "regression: resume must not reprocess the checkpoint row"
print("regression test: PASS")
Output
batch1 ids: [1, 2, 3, 4]
batch2 ids: [4, 5, 6, 7]
observed overlap: {4}
hypothesis 'resume re-includes the checkpoint row' confirmed: True
fixed overlap: set()
regression test: PASS

See it happen The loop, and what skipping two steps costs

next bugnaive: skip straight to a fixReproduceObserveHypothesizeDiscriminateFixRegression test
  1. Reproduce first: a deterministic repro, same input, same failure, every time.
  2. Observe only directly checkable facts: exact error, exact input, exact timestamp.
  3. Form exactly one falsifiable hypothesis from those facts, not three at once.
  4. Discriminate: run the cheapest experiment that would prove or kill that one hypothesis.
  5. Only a confirmed hypothesis earns a code change: fix exactly what it justifies.
  6. Lock it in with a regression test, so the same bug can't return silently after a refactor.
  7. The naive path jumps from a guess straight to a fix, skipping discriminate and the regression test: it 'works' until the next refactor brings the bug back.
Notice: the two steps the naive version skips, discriminate and regression-test, are exactly the two that turn "it works now" into "it's actually fixed."
Scenario A

In your world

The situation

You're handed settlement_reconciler.py, a service you've never opened, with a Jenkins job that fails intermittently, roughly once a week, with KeyError: 'ref_id'.

What actually breaks

Under deadline pressure, someone wraps the parsing loop in except KeyError: continue, the job goes green again, and everyone moves on. Three weeks later finance notices about 40 records a week are quietly missing, with no error anywhere, because the except block is swallowing exactly the rows that should have raised a loud failure.

What you'd do on Monday

Reproduce with the actual failing vendor file instead of guessing, observe the exact rows that trigger it, and discriminate between competing explanations before patching anything:

PythonDiscriminate before patching: is it corrupt data or an expected case?runs
rows = [
    {"id": 1, "type": "payment", "ref_id": "R100"},
    {"id": 2, "type": "refund"},                      # refunds have no ref_id
    {"id": 3, "type": "payment", "ref_id": "R101"},
]

missing_ref_id = [r for r in rows if "ref_id" not in r]
all_missing_are_refunds = all(r["type"] == "refund" for r in missing_ref_id)
print("rows missing ref_id:", [r["id"] for r in missing_ref_id])
print("hypothesis 'only refunds lack ref_id' confirmed:", all_missing_are_refunds)
Output
rows missing ref_id: [2]
hypothesis 'only refunds lack ref_id' confirmed: True
Scenario B

Imagine you work at Groww

300 microservices, one on-call engineer per week

The numbers

Illustrative: with 300 services, even a modest 15 minutes spent just locating the right repository, owner, and dashboard before debugging starts is 75 hours of pure "which service" time a week if ownership isn't documented anywhere. One on-call engineer cannot hold a mental map of 300 unfamiliar codebases.

What the Scenario-A answer does here

The Scenario A method still applies once you're in the right service, but "read the whole repo to find the entry point" doesn't scale across 300 services: with that many, you need an ownership map before you open a debugger, or the clock runs out finding the file, not fixing the bug.

What flips at this scale

At this scale the method needs a maintained ownership and runbook index (the mechanics belong to Prepare runbooks and prove on-call readiness) that gets an on-call engineer from "alert fired" to "the right repo, the right owner, the right dashboard" in minutes, so the reproduce/observe/hypothesize loop can start immediately instead of after a scavenger hunt through unfamiliar services.

The trade-off you'd defend

I would defend investing in a kept-current ownership index over trusting tribal memory, trading the overhead of maintaining it for cutting time-to-first-hypothesis across hundreds of services from hours to minutes.

Illustrative figures to make the arithmetic concrete; not Groww-published numbers.

Gotchas

Multiple hypotheses tested at once

Symptom

You change three things, the bug disappears, and it comes back a month later under slightly different conditions.

Cause

Changing several variables at once means you don't actually know which change mattered.

Fix

Test one hypothesis at a time; if you must change several things under time pressure, record which one you believe fixed it and verify that belief afterward.

Reading a wrapped exception's outer message as the root cause

Symptom

A traceback shows a generic failure in a shared utility function, but the real cause happened several calls earlier.

Cause

A bare except re-raised a new exception without from, so the chain exists on __context__ but doesn't read as a direct cause.

Fix

Read the full __cause__/__context__ chain, not just the outer message, and always re-raise with from when you translate an error.

"Fixed" with no regression test

Symptom

The exact same bug reappears months later after an unrelated refactor.

Cause

The fix patched the symptom at the call site without a test proving the specific failure can't recur.

Fix

Write a test that fails before the fix and passes after, using the real reproduction input, not a simplified stand-in.

Tricky questions

If adding a print statement makes an intermittent bug disappear, is it fixed?
Most people say

Yes, remove the debug print and ship, since the behavior is correct now.

Actually

That's a classic sign of a timing-dependent bug: the print statement changed timing, not logic. Shipping without a confirmed hypothesis about ordering or concurrency means the bug is still there, waiting for different timing.

Is "I'd add logging and see what happens" a debugging method?
Most people say

Yes, more visibility always helps find the bug faster.

Actually

Logging is only useful once you know what fact you're trying to observe. Logging everywhere without a hypothesis just produces more noise to read through; decide what you're testing before adding instrumentation.

Scenario question

You're handed a service you've never opened, with one ticket: "orders stuck in PENDING for over an hour, 240 affected." Walk me through your first 15 minutes.

Model answer

First I'd find the entry point for order state changes, not the whole repo: the handler or job that's supposed to move an order out of PENDING, and the table or queue it reads. I'd pull a handful of the 240 stuck orders and look for what they share, which is an observation, not a guess yet. I'd check the tests around that state transition to understand intended behavior quickly. Once I have a specific, shared fact (say, all 240 share a null field, or all arrived after a particular deploy), I'd form one hypothesis and find the cheapest check that would confirm or kill it before touching any code, because fixing based on a guess under incident pressure is how you ship a second bug on top of the first.

Interview corner

CommonHow do you approach debugging a codebase you've never seen?
“I'd find the entry point for the specific feature, not the whole repo, trace ownership and the state transitions involved, and read the tests before the comments. Then I'd apply the same reproduce, observe, hypothesize, discriminate, fix, regression-test loop I'd use anywhere, since the method doesn't depend on familiarity.”
TrickA bug is intermittent and you can't reproduce it reliably. What's your first move?
“I wouldn't start guessing fixes yet. I'd work on making it reproducible first, fixing the input, the seed, or the timing, because an unreproducible fix is unverifiable, and you find out it didn't work the next time it recurs in production.”
Experience checkTell me about a bug where the obvious explanation was wrong.
“I had a job fail intermittently with a KeyError on a vendor field; the quick fix someone proposed was to swallow the error. Observing the actual rows showed the missing field only happened on refund records, which legitimately don't have it, so my fix was explicit handling for that case, not suppression.”
DesignHow do you make debugging scale across hundreds of services?
“Individual debugging skill doesn't scale by itself; I'd push for a maintained ownership and runbook index so finding the right repo and owner takes minutes, not a scavenger hunt, which is what actually eats time once you have more services than any one person can hold in their head.”

Memory hook

Secure the scene, write down only what's true, test one theory at a time, and don't accuse anyone until the cheapest check confirms it.

If you remember only one thing:

Reproduce before you hypothesize, and regression-test before you call it fixed, or you've only patched the symptom.

Self-check

  1. What's the difference between __cause__ and __context__ on a chained exception?

    __cause__ is set only by an explicit raise ... from err, and prints as "the direct cause." __context__ is set automatically by any exception raised while handling another, even without from, and prints as "during handling of the above exception."

  2. Why should "intermittent" bugs be made reproducible before you hypothesize?

    A hypothesis you can't test on demand can't be confirmed or ruled out, and a "fix" for a bug you can't reproduce can't be verified either, so you find out it's still broken only when it recurs.

  3. What are the two steps a naive guess-and-patch loop usually skips?

    Discriminate (testing the hypothesis cheaply before fixing) and regression-test (locking the fix in so it can't silently return).

Prove it

Take a script with a genuine off-by-one or state-transition bug (write one deliberately if you don't have one handy). Debug it using the six-step method, writing down your reproduction, your one observed fact, your hypothesis, and the discriminating check, before you touch the fix. Paste the hypothesis log and the final regression test's output.

ce-11-04Card 4 of 6

Estimate work, control scope and say no explicitly

MediumP2 · Very likelyTalk about it11 min
estimationscopingschedulingsaying-no

Why you care

"How long will this take" is asked of every engineer constantly, and a bad answer costs more than being wrong: a bare number with no stated assumptions gets treated as a promise, and when reality includes something you didn't know about, you look unreliable instead of merely uninformed. Interviewers probe estimation and scoping because the answer reveals whether you distinguish "I don't know yet" from "I'm guessing confidently," and whether you can say no to a request without just going quiet and missing the deadline instead.

Explain like I'm 5

A weather forecast says "70% chance of rain, give or take a few hours," not "it will rain at 3:14pm." Nobody treats a forecast as a broken promise when it's off by an hour, because it was stated as a forecast with a range and named uncertainty. A single confident number with no range is a guarantee, and guarantees get judged as broken the moment anything unexpected happens, even something nobody could have known in advance.

Core concept

Estimating well means three things, not one number:

  • Break the work into subtasks, and give each one a range, not a point: optimistic, likely, pessimistic.
  • Name the assumptions and dependencies the estimate depends on ("assumes the partner's API docs are accurate," "assumes no schema change needed").
  • Carve out unknowns as explicit discovery tasks with their own small time-box, instead of folding guesswork into the main number. "I need half a day to find out how their auth works, then I can estimate the integration" is more honest and more useful than a padded guess.

Scoping means defining three things before you start: the smallest valuable deliverable (the smallest slice that is itself shippable and testable on its own), the acceptance evidence (how anyone will know it's actually done, not just coded), and the excluded work, named explicitly as deferred, not silently dropped.

Saying no is a structured trade-off, never a bare refusal: name the cost or risk of saying yes to everything, and offer a concrete alternative. "I can ship A by Friday with test evidence. A and B together need discovery time I don't have this sprint. I'd propose A now, B as a scoped follow-up" gives the asker something to decide on. A bare "no" or a silent slipped deadline gives them nothing.

Decision rule: if you can't name at least one assumption your estimate depends on, you don't have an estimate yet, you have a guess wearing an estimate's clothes.

Minimal example A range beats a point estimate

PythonPERT: expected time and uncertainty from three guesses, not oneruns
def pert(optimistic, likely, pessimistic):
    expected = (optimistic + 4 * likely + pessimistic) / 6
    stdev = (pessimistic - optimistic) / 6
    return round(expected, 2), round(stdev, 2)


tasks = {
    "partner auth setup": (1, 2, 5),
    "field mapping doc": (0.5, 1, 2),
    "export job + tests": (1, 1.5, 3),
}

total_expected = 0.0
total_var = 0.0
for name, (o, l, p) in tasks.items():
    expected, stdev = pert(o, l, p)
    print(f"{name}: {expected}d expected, +/-{stdev}d")
    total_expected += expected
    total_var += stdev ** 2

print(f"total: {round(total_expected, 1)}d expected, +/-{round(total_var ** 0.5, 1)}d")
Output
partner auth setup: 2.33d expected, +/-0.67d
field mapping doc: 1.08d expected, +/-0.25d
export job + tests: 1.67d expected, +/-0.33d
total: 5.1d expected, +/-0.8d

In production Scope the smallest valuable slice against a real deadline

Given more requested features than remaining capacity, pick the slice that delivers the most value per day of cost, and name what's deferred and why, instead of quietly dropping it.

PythonSmallest valuable deliverable, deferred work named explicitlyruns
features = [
    {"name": "CSV export", "value": 9, "cost_days": 1.5, "risk": "low, existing export pattern"},
    {"name": "audit logging", "value": 6, "cost_days": 3.0, "risk": "medium, needs a schema change"},
    {"name": "GraphQL support", "value": 4, "cost_days": 4.0, "risk": "high, new dependency, no spike done"},
]

remaining_capacity = 3.0  # days left in the sprint
ranked = sorted(features, key=lambda f: f["value"] / f["cost_days"], reverse=True)

accepted, deferred = [], []
budget = remaining_capacity
for f in ranked:
    if f["cost_days"] <= budget:
        accepted.append(f)
        budget -= f["cost_days"]
    else:
        deferred.append(f)

print("accepted this sprint (smallest valuable slice):")
for f in accepted:
    print(f"  - {f['name']} ({f['cost_days']}d, risk: {f['risk']})")
print(f"capacity left after accepted work: {budget}d")

print("deferred, with a reason, not silently dropped:")
for f in deferred:
    print(f"  - {f['name']}: needs {f['cost_days']}d, over the {remaining_capacity}d budget; risk: {f['risk']}")
Output
accepted this sprint (smallest valuable slice):
  - CSV export (1.5d, risk: low, existing export pattern)
capacity left after accepted work: 1.5d
deferred, with a reason, not silently dropped:
  - audit logging: needs 3.0d, over the 3.0d budget; risk: medium, needs a schema change
  - GraphQL support: needs 4.0d, over the 3.0d budget; risk: high, new dependency, no spike done
Scenario A

In your world

The situation

A PM asks, in a hallway conversation, "how long for exporting settlement data to a new partner's format?" You say "2 days" off the top of your head, with no breakdown.

What actually breaks

The partner's API turns out to need a custom auth handshake nobody documented, and the field mapping doesn't match your schema one-to-one. The 2-day estimate becomes 9 days. The PM doesn't just lose trust in this estimate; every future estimate you give gets mentally tripled, which means your genuinely quick 2-day tasks now get scheduled as 6-day tasks too.

What you'd do on Monday

Give a range tied to named assumptions, and carve the actual unknown (the partner's auth) into its own discovery task before committing to the integration estimate, as the minimal example above shows for exactly this situation: 5.1 days expected, give or take 0.8, with "partner auth setup" named as the biggest source of uncertainty, not folded invisibly into "2 days."

Scenario B

Imagine you work at Delhivery

40 new partner integrations queued this quarter

The numbers

Illustrative: if each of those 40 integrations hits a typical unknown-unknown costing about 1.5 extra days to resolve, that's 40 × 1.5 ≈ 60 unplanned days across the quarter; spread across a 10-person integration team, that's 60 ÷ 10 = 6 days per person of capacity nobody budgeted for.

What the Scenario-A answer does here

Scenario A's per-task range and discovery task still work for one integration, but "pad every ticket by the same multiplier" doesn't scale: some integrations really are 2-day jobs, so blanket padding wastes real capacity on them, while the genuinely risky integrations still blow past even the padded number because the padding wasn't targeted at the actual unknown.

What flips at this scale

At this scale the flip is a standard, time-boxed discovery spike (say, half a day) as its own ticket before the main estimate for every new integration, so uncertainty gets measured per integration instead of guessed once for all of them, and the backlog shows discovery and build as separately trackable items with their own evidence.

The trade-off you'd defend

I would defend a mandatory short discovery spike over either optimistic single estimates or blanket padding, trading an upfront half-day per integration for estimates the business can actually plan releases around.

Illustrative figures to make the arithmetic concrete; not Delhivery-published numbers.

Gotchas

A point estimate reads as a promise

Symptom

"It'll take 3 days" becomes an unmissable deadline the moment it's said out loud, with no room for the unexpected.

Cause

A single number carries no information about uncertainty, so it gets treated as a commitment rather than a forecast.

Fix

Always give a range tied to named assumptions, and explicitly revise it when an assumption turns out false.

Unknowns absorbed silently into padding

Symptom

Estimates balloon over time and nobody trusts them anymore, even for genuinely simple tasks.

Cause

Discovery work gets hidden inside the main number instead of being named as its own task.

Fix

Carve out explicit, time-boxed discovery tasks for the specific unknowns, instead of inflating every estimate equally.

A bare "no" reads as unhelpful

Symptom

Pushback on scope escalates straight to a director, instead of getting resolved at your level.

Cause

A refusal with no alternative gives the asker nothing to decide between, so they escalate to get a decision made.

Fix

Pair every "no" with a concrete trade-off and alternative: what you can do by when, and what the larger ask would cost.

Tricky questions

Does doubling every estimate fix chronic underestimation?
Most people say

Yes, if estimates always run over, just multiply by two and move on.

Actually

Blanket padding hides which tasks are actually risky, wastes real capacity on tasks that were accurate, and still fails on the one task whose real unknown is bigger than the multiplier. Named discovery tasks target the actual risk instead of taxing every task equally.

Is naming assumptions just covering yourself if you're wrong?
Most people say

Yes, it's a way to have an excuse ready.

Actually

Named assumptions are what make an estimate revisable. When an assumption turns out false, you update the estimate with a reason everyone can see, instead of either silently blowing the deadline or silently padding future estimates out of distrust.

Scenario question

You're handed a request from a PM for "CSV export, GraphQL support, and audit logging, all by end of sprint," with 3 days of capacity left. What do you say?

Model answer

I wouldn't just say no. I'd scope the smallest valuable slice I can actually finish with evidence: CSV export, fully done and tested, by Friday, since it fits the existing export pattern in about 1.5 days. The other two need their own estimate, and audit logging needs a schema change I haven't scoped yet, so I'd propose it as a follow-up ticket with a discovery task first rather than a number I'm not confident in. GraphQL support is the riskiest of the three, new dependency, no spike done, so I'd flag that it probably doesn't fit this sprint at all and ask whether it's actually more valuable than the other two before committing any time to it.

Interview corner

CommonHow do you estimate a task you've never done before?
“I break it into subtasks, give each a range instead of a point estimate, and name the assumptions it depends on. If there's a genuine unknown, like an undocumented partner API, I carve that out as its own small discovery task rather than guessing it into the main number.”
TrickYour manager wants a single number, not a range, for a status report. What do you do?
“I'd give the expected value from my range as the headline number, but I'd still state the range and the biggest assumption behind it in the same breath, so if that assumption turns out false, the number can move without it looking like I was wrong rather than informed.”
Experience checkTell me about an estimate that went badly, and what you changed.
“I gave a 2-day estimate off the top of my head for a partner export; it became 9 days because of an undocumented auth handshake. Now I carve out a short discovery task for anything involving a system I don't control before I commit to the main number.”
DesignHow do you say no to a scope request without just stalling?
“I name the cost or risk of saying yes to everything, and I always pair it with a concrete alternative: what I can deliver by when, and what the fuller ask would need in time or information I don't have yet. A no with no alternative just gets escalated past me.”

Memory hook

A single number is a promise; a range tied to a named assumption is a forecast, and forecasts are allowed to be revised when the assumption changes.

If you remember only one thing:

If you can't name one assumption your estimate depends on, it's a guess, not an estimate yet.

Self-check

  1. What three things does a real estimate need, beyond a single number?

    A range (optimistic/likely/pessimistic), named assumptions and dependencies, and explicit, time-boxed discovery tasks for genuine unknowns.

  2. Why does blanket padding (multiplying every estimate by the same factor) fail?

    It wastes capacity on tasks that were actually accurate, while the genuinely risky tasks can still exceed even the padded number, because the padding isn't targeted at the real unknown.

  3. What makes "no" land as helpful instead of obstructive?

    Pairing it with a concrete, named trade-off and an alternative the asker can actually choose, instead of a bare refusal with nothing to decide between.

Prove it

Take a real task you're about to start (or one you recently finished). Write a PERT-style range estimate for it with at least two subtasks, name one assumption each subtask depends on, and name one piece of scope you would defer if capacity were tighter, with the reason. Paste the estimate and the deferred item.

ce-11-05Card 5 of 6

Argue for tech debt with measured costs

MediumP4 · DifferentiatorTalk about it10 min
tech-debtcost-benefitmeasurementprioritization

Why you care

"This code is bad, we should refactor it" loses every planning meeting against a feature with a visible number next to it, and it should: a feeling isn't a competing bid for the same engineering time. Interviewers ask how you'd argue for paying down debt because it reveals whether you can turn "this is ugly" into a number that competes on the same terms as a feature's business case, or whether you'll just keep losing that argument and let the debt compound. This is a selective-exposure skill, not everyday work: most tickets don't need this treatment, but the ones that keep recurring do.

Explain like I'm 5

A leaking roof that you keep mopping up after costs something real: your time, every single time it rains, invisibly, a little at a time. Calling a roofer costs one visible number, once. The pitch for fixing the roof isn't "mopping is annoying," it's comparing the running total of all that mopping against the one-time repair bill. Nobody approves a roofer based on how annoying mopping feels; they approve it when the two numbers are side by side.

Core concept

Arguing for tech debt with numbers means three steps, in order:

  • Baseline the recurring cost, measured over a real window: hours per week on manual reruns or workarounds, on-call pages per month traceable to this component, the failure rate (percentage of deploys touching this area that need a hotfix), and any lead-time impact (extra days a typical feature takes because it has to work around this code).
  • Estimate the fix cost and the expected benefit, with the uncertainty stated honestly: engineer-days to build the fix, and your best estimate of how much the baseline number should drop, not a fabricated ROI percentage pulled out of thin air.
  • Propose a small, bounded, measurable change, not "rewrite the service." Define upfront how you'll re-measure the baseline after the fix ships, so the pitch is falsifiable instead of a one-time persuasion exercise.

Decision rule: if you can't name a number that will go down after the fix, you don't have a tech-debt argument yet, you have a style preference, and it will lose to a feature with a real business case every time, which is often the correct outcome.

Minimal example Toil cost versus a one-time fix, as a payback period

PythonMonthly toil cost compared to a one-time fix costruns
def toil_cost(pages_per_month, minutes_per_page, hourly_rate):
    hours = pages_per_month * minutes_per_page / 60
    return round(hours, 2), round(hours * hourly_rate, 2)


hours, cost = toil_cost(pages_per_month=8, minutes_per_page=30, hourly_rate=60)
fix_cost = 3 * 8 * 60  # 3 engineer-days, 8h/day, $60/h blended rate

print(f"toil: {hours}h/month, ${cost}/month")
print(f"one-time fix cost: ${fix_cost}")
print(f"payback period: {round(fix_cost / cost, 1)} months")
Output
toil: 4.0h/month, $240.0/month
one-time fix cost: $1440
payback period: 6.0 months

In production Aggregate incidents by root cause before pitching a fix

A single service's pages for a shared bug rarely clear a reasonable payback bar; the real argument is the aggregated cost of the same root cause across every service that copy-pasted it.

PythonAggregate by cause, then check whether the fix clears a payback thresholdruns
from collections import defaultdict

incidents = [
    {"service": "orders-api", "cause": "unbounded-retry", "minutes": 30},
    {"service": "orders-api", "cause": "unbounded-retry", "minutes": 45},
    {"service": "payments-api", "cause": "unbounded-retry", "minutes": 20},
    {"service": "payments-api", "cause": "disk-full", "minutes": 60},
    {"service": "ledger-sync", "cause": "unbounded-retry", "minutes": 35},
]

by_cause = defaultdict(float)
for inc in incidents:
    by_cause[inc["cause"]] += inc["minutes"] / 60


def recommend(hours_per_month, hourly_rate, fix_cost, threshold_months=3):
    monthly_cost = hours_per_month * hourly_rate
    payback = fix_cost / monthly_cost
    return round(payback, 1), payback <= threshold_months


for cause in sorted(by_cause, key=lambda c: -by_cause[c]):
    hours = round(by_cause[cause], 2)
    payback, worth_it = recommend(hours, hourly_rate=60, fix_cost=1440)
    print(f"{cause}: {hours}h/month across these services, payback {payback} months, propose fix now: {worth_it}")
Output
unbounded-retry: 2.17h/month across these services, payback 11.1 months, propose fix now: False
disk-full: 1.0h/month across these services, payback 24.0 months, propose fix now: False
Scenario A

In your world

The situation

Your shared retry_wrapper.py has no backoff. Every transient network blip pages whoever is on-call, at any hour. It's happened 8 times this past month, about 30 minutes each.

What actually breaks

Nobody fixes it because "there's no time," and twice this month a page that looked like the usual transient blip actually masked a worse downstream failure, since the unbounded retry kept hammering a dependency that was genuinely down, turning a 5-minute non-issue into a 3-hour incident. The toil is invisible in planning because it never shows up as a single big number, just a string of 30-minute interruptions nobody adds up.

What you'd do on Monday

Add up the actual cost, as the minimal example does: 4 hours/month of pages at a $240/month cost, against a 3-day, $1,440 fix, a 6-month payback. That's a real number to bring to planning, not "it's annoying."

Scenario B

Imagine you work at Coinbase

15 services sharing one copy-pasted retry pattern

The numbers

Illustrative: 15 services averaging 4 pages/month each from the same unbounded-retry pattern, at 30 minutes/page, is roughly 30 hours/month of on-call time company-wide for one root cause; at a $60/hr blended rate that's about $1,800/month, against an estimated 3 engineer-days (8h/day at $60/hr, the same rate as the minimal example: 3 × 8 × 60 = $1,440) to build one shared, tested backoff library.

What the Scenario-A answer does here

Scenario A's single-service toil calculation still works, and the production example shows exactly why it isn't enough on its own: fixing service 1's retry doesn't reduce pages on the other 14 that copy-pasted the same bad pattern, so no individual service's number alone clears a reasonable payback bar.

What flips at this scale

At this scale the ask flips from "fix a bug in this service" to "fix the pattern once, in one shared library, and migrate 15 call sites," with the payback measured against the combined 30 hours/month across all of them, which is also the only way the fix's cost is justified at all.

The trade-off you'd defend

I would defend one shared, tested backoff library with a migration ticket per service over 15 local patches, trading migration coordination overhead for a payback period under 1 month ($1,440 ÷ $1,800/month ≈ 0.8 months) on the combined toil number, instead of a 6-11 month payback on any single service's slice of it.

Illustrative figures to make the arithmetic concrete; not Coinbase-published numbers.

Gotchas

Debt argued as "it's ugly"

Symptom

A ticket titled "refactor for cleanliness" gets deprioritized every single sprint, indefinitely.

Cause

No number connects the ugliness to pages, hotfixes, or lead time, so it has nothing to compete with a feature's business case on.

Fix

Find the metric first (pages, hotfix rate, cycle time) before writing the ticket; drop the aesthetic framing entirely from the pitch.

An invented ROI percentage

Symptom

A slide claims "this saves 40% of engineering time" with no source anyone can point to.

Cause

A number was picked for persuasion rather than measured from real data.

Fix

State the measured baseline and the honest uncertainty in the expected benefit, and define how you'll re-measure after the fix ships.

A rewrite proposed as the fix

Symptom

A tech-debt ticket balloons into "replace the whole service" and gets rejected outright.

Cause

Fear of the problem recurring gets framed as scope, instead of being scoped down.

Fix

Propose the smallest change that would move the specific metric; treat a larger rewrite as a separate, later conversation if the small fix doesn't move the number enough.

Tricky questions

Is "this code is hard to read" alone a strong tech-debt argument?
Most people say

Yes, readability is self-evidently worth fixing on its own.

Actually

It's necessary context, not a complete argument. Pair it with a number, time to onboard someone to this file, bug rate traced to this file, or incidents linked to misreading it, or it will keep losing to feature work that has a visible number attached.

Does a bigger measured cost always justify a bigger fix?
Most people say

Yes, if the cost is high enough, a full rewrite is justified.

Actually

A high measured cost justifies spending more to fix it, but the smallest change that moves the number is still the right first move; a rewrite is a separate decision, made only if the small fix is tried and doesn't move the metric enough.

Scenario question

You're handed a 2-year-old backlog ticket, "Refactor the settlement retry logic," with no description beyond "it's bad." Leadership wants to cut it in planning. What do you do?

Model answer

I'd pull the incident history first and attach real numbers: pages per month traceable to this pattern, the hotfix rate on changes touching it, and time lost to incidents it made worse. If those numbers are small, I'd agree it should get cut, because that's often the correct call. If they're not small, especially once I check whether other services share the same pattern, I'd rescope the ticket to the smallest fix that would move those specific numbers, attach an honest payback estimate like the one in the production example, and re-pitch it as a measured cost comparison instead of a vague cleanliness request.

Interview corner

CommonHow do you make a case for paying down tech debt?
“I baseline the recurring cost with real numbers, pages per month, hotfix rate, lead-time impact, estimate the fix cost and the expected benefit honestly, including the uncertainty, and propose the smallest change that would move that specific number. If I can't name a number that goes down, I don't have an argument yet.”
TrickYour tech-debt ticket has a measured cost of only 2 hours a month. Should you still push for a fix?
“Probably not as its own project. I'd check whether the same pattern is duplicated elsewhere first, since the real argument is often the aggregated cost across every place it was copy-pasted, not any one service's small slice of it.”
Experience checkTell me about a tech-debt argument that actually got funded.
“I aggregated on-call pages from an unbounded-retry pattern copy-pasted across several services, which individually looked minor but totaled real hours a month, and proposed one shared backoff library with a payback under two months. Framing it as the combined cost, not any single service's number, is what got it prioritized.”
DesignHow do you avoid a tech-debt fix ballooning into an unscoped rewrite?
“I define the smallest change that would move the specific metric I baselined, before any implementation starts, and treat a bigger rewrite as a separate conversation to have only if the small fix is tried and doesn't move the number enough.”

Memory hook

Mopping costs something every single time it rains; a roofer costs one number, once. Put the two totals side by side, not a feeling next to a feature's business case.

If you remember only one thing:

If you can't name a number that goes down after the fix, it's a style preference, not a tech-debt argument.

Self-check

  1. What three numbers belong in a tech-debt pitch before anyone proposes a fix?

    A baseline recurring cost (pages/toil hours/failure rate/lead-time impact), the fix cost, and the expected benefit stated with its uncertainty, not a fabricated ROI figure.

  2. Why does a single service's toil number often fail to justify a fix that a shared library would?

    Because the same root cause is often duplicated across many services; the real payback is against the combined cost across all of them, which a single service's slice can't show on its own.

  3. What should always be defined before a tech-debt fix ships?

    How you'll re-measure the baseline afterward, so the pitch is falsifiable instead of a one-time persuasion exercise.

Prove it

Pull your own team's incident log or on-call history for one recurring root cause. Compute the monthly toil cost in hours and the equivalent cost at your team's rate, estimate a bounded fix and its cost, and write the one-page pitch using the structure in this card. Paste the numbers and the recommendation.

ce-11-06Card 6 of 6

Communicate incidents and close ownership loops

MediumP2 · Very likelyTalk about it11 min
incident-communicationstatus-updateshandoverownership

Why you care

During an incident, the engineer actually fixing the problem and the people asking "any update?" are often not the same conversation, and a bad one damages trust faster than the incident itself does. Interviewers ask about incident communication because a wrong answer, going silent until you have the full answer, or stating a guessed root cause as fact, predicts exactly how your next real incident will go: badly, with escalations layered on top of the actual problem. The fix that fixes the system fast is only half the job; the other half is whether anyone trusted the process while it happened.

Explain like I'm 5

Air traffic control gives a steady cadence of updates, "still on approach, no change," even when nothing has changed, because silence from a control tower means something different and worse than "nothing new yet." A pilot getting no response doesn't assume everything's fine; they assume the radio is dead. An incident update works the same way: a committed cadence, even with "no change," is what keeps everyone from assuming the worst and escalating past you.

Core concept

A useful incident update has six fixed parts, every time: impact (concrete: how many users, what percentage of orders, how much money is at risk, not "some things are broken"), known facts versus hypotheses, kept explicitly separate, current mitigation (what's actually being done right now), a named owner (one person, never "the team"), and a committed next-update time, stated even if there's no news yet.

Two disciplines make this work:

  • Avoid premature root-cause claims. Say "we believe X" until there's direct evidence (a log line, a metric, a reproduction), not "it was X." Retracting a stated cause damages trust faster than an honest "still investigating" ever does.
  • Close ownership loops explicitly. A handover (shift change, escalation) isn't complete until the new owner has the current facts, hypotheses, and mitigation state handed to them and confirms they've accepted it; re-discovering the state from scratch wastes time that should go into fixing the problem. Follow-up actions get tracked as owned tickets with a due date, and are closed only once someone verifies the fix actually behaves correctly in production, not merely that the code merged.

Decision rule: if your next update isn't scheduled, silence will be read as "nothing is happening," even if you're making real progress; commit to a cadence and keep it, especially when the update is "no change yet."

Minimal example A structured update rejects an incomplete one

PythonRequired fields, not a free-text status lineruns
REQUIRED_FIELDS = ("impact", "facts", "hypotheses", "mitigation", "owner", "next_update")


def format_update(**fields):
    missing = [f for f in REQUIRED_FIELDS if not fields.get(f)]
    if missing:
        raise ValueError(f"incomplete update, missing: {missing}")
    return "\n".join(f"{k}: {fields[k]}" for k in REQUIRED_FIELDS)


update = format_update(
    impact="~12% of settlement webhooks delayed, no data lost",
    facts="error rate on /webhooks/settle jumped at 02:14 UTC, right after the 02:10 deploy",
    hypotheses="suspect the new retry middleware; not yet confirmed",
    mitigation="rolled back the 02:10 deploy at 02:22 UTC, error rate dropping",
    owner="Priya",
    next_update="02:50 UTC",
)
print(update)

try:
    format_update(impact="x")
except ValueError as e:
    print("ValueError:", e)
Output
impact: ~12% of settlement webhooks delayed, no data lost
facts: error rate on /webhooks/settle jumped at 02:14 UTC, right after the 02:10 deploy
hypotheses: suspect the new retry middleware; not yet confirmed
mitigation: rolled back the 02:10 deploy at 02:22 UTC, error rate dropping
owner: Priya
next_update: 02:50 UTC
ValueError: incomplete update, missing: ['facts', 'hypotheses', 'mitigation', 'owner', 'next_update']

In production A cadence tracker that flags a missed update and a guarded handover

Handing over ownership doesn't pause the cadence requirement; a gap after a handover is still a breach, because silence reads the same way to everyone watching regardless of whose shift it is.

PythonEnforce an update cadence and an explicit handover acceptruns
class Incident:
    def __init__(self, cadence_minutes=30):
        self.cadence = cadence_minutes
        self.updates = []
        self.owner = None

    def update(self, minute, owner, status):
        if self.updates:
            gap = minute - self.updates[-1][0]
            if gap > self.cadence:
                print(f"  [cadence breach: {gap}min since last update, limit {self.cadence}min]")
        self.owner = owner
        self.updates.append((minute, owner, status))
        print(f"t+{minute}m ({owner}): {status}")

    def handover(self, minute, new_owner, accepted):
        if not accepted:
            raise RuntimeError("handover not accepted, previous owner stays on the hook")
        print(f"t+{minute}m: {self.owner} hands over to {new_owner}, accepted")
        self.owner = new_owner


inc = Incident(cadence_minutes=30)
inc.update(0, "Priya", "investigating, error rate spiking right after the 02:10 deploy")
inc.update(25, "Priya", "mitigating, rolling back the 02:10 deploy")
inc.handover(60, "Dev", accepted=True)
inc.update(130, "Dev", "monitoring, error rate back to baseline")
Output
t+0m (Priya): investigating, error rate spiking right after the 02:10 deploy
t+25m (Priya): mitigating, rolling back the 02:10 deploy
t+60m: Priya hands over to Dev, accepted
  [cadence breach: 105min since last update, limit 30min]
t+130m (Dev): monitoring, error rate back to baseline
Scenario A

In your world

The situation

Your loader job fails at 2am. You post once in the incident channel, "db issue, looking into it," and go quiet for two hours while you actually diagnose and fix it.

What actually breaks

By the time it's genuinely fixed, three people have escalated because nobody heard anything for two hours, support is fielding angry pings with nothing to tell customers, and when the day-shift engineer takes over at hour 1 they re-ask every question you already answered, losing 20 minutes re-establishing state you'd already established, even though your actual fix was fast.

What you'd do on Monday

Commit to a cadence up front and keep it even when there's no news, and hand over the actual state, not just a one-line mention:

PythonA committed cadence survives "nothing changed yet"runs
def format_update(**fields):
    required = ("impact", "facts", "hypotheses", "mitigation", "owner", "next_update")
    return "\n".join(f"{k}: {fields.get(k, 'unchanged')}" for k in required)


print(format_update(
    impact="nightly loader did not run; no customer-facing orders affected yet",
    facts="loader failed at 02:00 with a connection timeout to the replica",
    hypotheses="suspect replica failover from the 01:50 maintenance window; not confirmed",
    mitigation="pointing the loader at the primary while investigating the replica",
    owner="You",
    next_update="in 30 minutes, even if there is no change",
))
Output
impact: nightly loader did not run; no customer-facing orders affected yet
facts: loader failed at 02:00 with a connection timeout to the replica
hypotheses: suspect replica failover from the 01:50 maintenance window; not confirmed
mitigation: pointing the loader at the primary while investigating the replica
owner: You
next_update: in 30 minutes, even if there is no change
Scenario B

Imagine you work at Twilio

50 services, roughly 20 incidents a month

The numbers

Illustrative: with 20 incidents/month across 50 services, each touching multiple downstream integration teams, an unscheduled "update when I have news" habit generates on the order of 15 ad hoc status-check pings per incident from stakeholders, about 300 unplanned interruptions a month company-wide, each one pulling the on-call engineer away from actually fixing anything.

What the Scenario-A answer does here

Scenario A's structured update still works for a single incident, but "post whenever you have news" doesn't scale across roughly 20 concurrent-ish incidents a month with multiple stakeholder teams each wanting their own channel updated: the on-call engineer ends up spending more time repeating status than fixing the problem.

What flips at this scale

At this scale the flip is one incident channel or status doc as the single source of truth, updated on a fixed cadence, with stakeholders told to watch that one place instead of pinging individuals; a status field anyone can read (investigating/mitigating/monitoring/resolved) replaces N separate personal updates with one place the owner updates once.

The trade-off you'd defend

I would defend one scheduled-cadence channel of record over ad hoc updates to whoever asks, trading some stakeholders feeling like they weren't personally updated for the on-call engineer actually having time left to fix the incident instead of narrating it to everyone separately.

Illustrative figures to make the arithmetic concrete; not Twilio-published numbers.

Gotchas

A premature root-cause claim gets retracted

Symptom

"It was a bad deploy" gets announced, then retracted an hour later when the real cause turns out to be a database failover.

Cause

The first plausible explanation got stated as confirmed fact before any direct evidence backed it.

Fix

Label it as a hypothesis ("we believe," "suspected") until a log line, metric, or reproduction confirms it, and only then state it as a fact.

Silence between updates

Symptom

Stakeholders repeatedly ping "any update?" during an incident that's actually progressing fine.

Cause

No committed next-update time was given, so silence reads as "nothing is happening" by default.

Fix

Commit to a cadence up front and keep it even when the update is "no change yet."

Handover without a real accept

Symptom

The new on-call engineer re-asks questions that were already answered, wasting diagnostic time that's already been spent once.

Cause

Ownership changed with a one-line mention instead of an explicit handover of the current facts, hypotheses, and mitigation state.

Fix

Require a written handover summary and an explicit acceptance from the new owner before the previous owner steps away.

A follow-up ticket closes on "merged," not "verified"

Symptom

An "add a retry alert" follow-up is marked done, but nobody checks the alert actually fires when it should.

Cause

The ticket's definition of done was the code merging, not the behavior being confirmed in production.

Fix

Close follow-up tickets only after someone verifies the fix behaves as intended in production, not just that it shipped.

Tricky questions

Does staying silent until you've found the root cause look more competent than frequent "still investigating" updates?
Most people say

Yes, waiting for certainty avoids looking confused in front of stakeholders.

Actually

Silence reads as "nothing is happening" and erodes trust faster than an honest, scheduled "still investigating, next update in 20 minutes." Stakeholders need a cadence they can rely on, not certainty they have to wait for.

Is a handover complete once you've told someone "I'm heading off, it's yours"?
Most people say

Yes, saying it out loud transfers responsibility.

Actually

A handover is complete only once the new owner has the current facts, hypotheses, and mitigation state, and has explicitly confirmed they've accepted ownership; otherwise the previous owner is still on the hook even if they've logged off.

Scenario question

You're handed an ongoing incident at hour 2, inheriting it from someone going off-shift with a one-line message: "db stuff, might be the migration." What do you do before touching anything?

Model answer

I'd ask for the actual handover state before I touch anything: the current impact numbers, what's a confirmed fact versus a hypothesis, what mitigation is already in place, and explicitly confirm I now own it, rather than diagnosing blind from a one-line guess. "Might be the migration" is a hypothesis, not a handed-off fact, so I'd want to know what evidence supports it before I either chase it further or rule it out, instead of re-running two hours of diagnosis from scratch.

Interview corner

CommonWhat should every incident update include?
“Impact in concrete terms, facts kept separate from hypotheses, the current mitigation, a named owner, and a committed time for the next update, even if there's no change yet. Missing any of those, I'd treat the update as incomplete.”
TrickYou're not sure of the root cause yet, but people want an answer now. What do you say?
“I'd say what we believe, clearly labeled as a hypothesis, not what caused it as fact. Stating a guessed cause as confirmed and having to retract it later damages trust more than an honest 'still investigating, next update in 20 minutes' ever does.”
Experience checkTell me about an incident handover that went badly.
“A night-shift engineer handed off with one line, 'db stuff,' and I spent the first 20 minutes of my shift re-discovering facts they already had. Now I ask for an explicit handover summary, facts, hypotheses, mitigation state, before I take ownership, and I give the same when I hand off.”
DesignHow do you keep incident communication from eating the time needed to actually fix things?
“I'd push toward one channel or status doc as the source of truth, updated on a fixed cadence, instead of answering separate pings from every stakeholder team individually. The owner updates one place once; that's what actually scales past a handful of concurrent incidents.”

Memory hook

Air traffic control says "still on approach, no change" on a schedule, because silence reads as a dead radio, not as "everything's fine."

If you remember only one thing:

Commit to a next-update time and keep it even with no news, and never state a hypothesis as a confirmed cause.

Self-check

  1. What are the six required parts of an incident update?

    Impact, known facts, hypotheses (kept separate from facts), current mitigation, a named owner, and a committed next-update time.

  2. Why is "we believe X" safer to say mid-incident than "it was X"?

    Because retracting a confirmed cause that turns out wrong damages trust faster than an honest, labeled hypothesis that later turns out to be right or wrong.

  3. When is a handover actually complete?

    Only once the new owner has the current facts, hypotheses, and mitigation state, and has explicitly confirmed they've accepted ownership, not when the previous owner simply announces they're leaving.

Prove it

Write a four-update timeline for a sample incident (real or invented), each update hitting all six required fields, including one handover with an explicit accept and one follow-up ticket that you verify in production rather than just mark merged. Paste the timeline.