Shelf · Core Eng · Chapter 10

Chapter 10: Config, release & supply chain

Your Jenkins job reads config from three places and nobody wrote down which one wins; this chapter makes that precedence explicit, then follows the same artifact from version number to signed, scanned, and on-call-ready. By the end you can design config and feature flags with no surprises, apply 12-factor and semver to a real release, and prove the thing running in prod is the thing you tested and trust.

Difficulty
Priority
ce-10-01Card 1 of 6

Configuration and secrets: precedence and feature flags

MediumP2 · Very likelyKnow cold12 min
configenvironment-variablessecretsfeature-flagsprecedence
Needs: nothing. Start here.

Why you care

Every ETL job you have written already has a config precedence, even if nobody designed it: a default in the code, a value in a checked-in file, an environment variable Jenkins sets, maybe a --batch-size flag someone added for a one-off run. If you cannot say which one wins without running the job, you cannot predict its behaviour in a given environment, and neither can the next person who touches it. Interviewers ask "how does your app get its config" because the answer reveals whether you have a contract or a pile of overrides nobody can trace. Secrets and feature flags are the sharpest edge of this: a secret that leaks through the wrong source, or a flag nobody remembers to turn off, is a production incident with your name on the commit.

Explain like I'm 5

Think of a thermostat with four ways to set the temperature: a factory default inside the unit, a weekly schedule programmed once, a phone app that can override today, and someone's hand turning the physical dial right now. If you walk in and the room is freezing, you need to know which of those four wins, in what order, or you will blame the wrong one. Config precedence is exactly that ladder: default, file, environment, and whatever override touched it last. A feature flag is the same dial, but for behaviour instead of temperature: it decides which code path runs right now, and if nobody notes down who set it and why, you are back to standing in a cold room guessing.

Core concept

  • An explicit precedence contract: defaults < config file < environment variables < CLI/runtime flags (or whatever order your team picks). The exact order matters less than that it is written down and identical in every environment.
  • Secrets are a special config class: never in defaults, never in a checked-in file in plaintext. They arrive via environment or a secret manager at runtime, are logged only in redacted form, and rotate without a code deploy.
  • Startup validation: a service should fail fast at boot if a required key is missing or malformed, not three hours into a batch run.
  • Feature flags are config with an owner, an expiry and a fallback: who owns it, when it gets removed, and what happens if the flag system itself is unreachable — always define the safe default yourself.
  • No universal framework precedence exists. Spring, Django settings, python-dotenv, and flag SDKs each default to a different order; your job is to make your own order explicit and tested, not assume a library's default matches your mental model.

Use file-based config for values that differ by environment but rarely change (table names, bucket names). Use environment variables for anything injected by the deployment platform (secrets, hostnames). Use a flag system only for things you intend to flip without a deploy, and never for permanent configuration — a flag with no removal date is dead code with extra ceremony.

Minimal example Precedence merge exposes a type bug

PythonPrecedence merge exposes a type bugruns
import os

defaults = {"timeout": 30, "batch_size": 500}
file_cfg = {"batch_size": 1000}
os.environ["APP_TIMEOUT"] = "5"
env_cfg = {k[4:].lower(): v for k, v in os.environ.items() if k.startswith("APP_")}

resolved = {**defaults, **file_cfg, **env_cfg}
print(resolved["batch_size"], resolved["timeout"], type(resolved["timeout"]).__name__)
Output
1000 5 str

In production Config loader: precedence, validation, redaction

PythonConfig loader with an explicit precedence contract and redactionruns
import json
import os
from pathlib import Path

Path("config.json").write_text(json.dumps({"batch_size": 2000, "s3_bucket": "orders-raw"}))

REQUIRED = {"db_dsn", "s3_bucket", "batch_size"}
SECRET_KEYS = {"db_dsn"}


def load_config(path, env_prefix="ETL_", cli_overrides=None):
    defaults = {"batch_size": 500, "db_dsn": "postgresql://localhost/dev"}
    file_cfg = json.loads(Path(path).read_text()) if Path(path).exists() else {}
    env_cfg = {
        k[len(env_prefix):].lower(): v
        for k, v in os.environ.items()
        if k.startswith(env_prefix)
    }
    cli_cfg = cli_overrides or {}

    resolved = {**defaults, **file_cfg, **env_cfg, **cli_cfg}
    if "batch_size" in resolved:
        resolved["batch_size"] = int(resolved["batch_size"])

    missing = REQUIRED - resolved.keys()
    if missing:
        raise RuntimeError(f"missing required config: {sorted(missing)}")
    return resolved


def redacted(cfg):
    return {k: ("***" if k in SECRET_KEYS else v) for k, v in cfg.items()}


os.environ["ETL_DB_DSN"] = "postgresql://prod-host/orders"

cfg = load_config("config.json", cli_overrides={"batch_size": "5000"})
print(redacted(cfg))
print(cfg["batch_size"], type(cfg["batch_size"]).__name__)
Output
{'batch_size': 5000, 'db_dsn': '***', 's3_bucket': 'orders-raw'}
5000 int

See it happen Four sources, one resolved value

defaultstimeout=30config.jsonbatch_size=1000env varAPP_TIMEOUT=5CLI flag--batch-size 50resolved config
  1. Start from defaults baked into the code: timeout=30, batch_size=500.
  2. The checked-in file overrides batch_size only; timeout is untouched.
  3. The environment variable overrides timeout, and it arrives as a string, not an int.
  4. The CLI flag wins last: batch_size becomes 50 for this one run only.
  5. Nothing validated the type of timeout, so '5' as a string silently reaches code that expects an int.
Notice: four sources feed one box, and the last one to write wins. Nothing here checked that the winning value was even the right type.
Scenario A

In your world

The situation

Your settlement job's defaults point at a dev database, a checked-in config.json sets the real table names, and Jenkins injects the prod DB host through an environment variable. Someone, debugging a one-off issue last sprint, added a --db-host flag to the scheduled job's launch script "just for that run" and forgot to remove it.

What actually breaks

The CLI flag outranks the environment variable, so every scheduled run has quietly been writing to a decommissioned read-replica for two weeks. The job logs 48,213 rows processed and exits 0, because writing to the wrong, still-reachable database isn't an error. Finance notices the production totals haven't moved before anyone notices the flag.

What you'd do on Monday

Make precedence visible, not just correct. Log the resolved value and its source for every non-secret key at boot, so an unexpected override (a CLI flag in a scheduled job) is loud instead of silent:

PythonLog which source won for every key, so an override can't hideruns
def resolve_with_source(defaults, file_cfg, env_cfg, cli_cfg):
    resolved, source = {}, {}
    for layer, name in [(defaults, "default"), (file_cfg, "file"), (env_cfg, "env"), (cli_cfg, "cli")]:
        for k, v in layer.items():
            resolved[k] = v
            source[k] = name
    return resolved, source


resolved, source = resolve_with_source(
    {"db_host": "dev-db"}, {}, {"db_host": "prod-host"}, {"db_host": "debug-replica"}
)
print(resolved["db_host"], "from", source["db_host"])
if source["db_host"] == "cli":
    print("WARNING: scheduled job running with a CLI override on db_host")
Output
debug-replica from cli
WARNING: scheduled job running with a CLI override on db_host
Scenario B

Imagine you work at CRED

400 internal services, 4 config sources each

The numbers

Illustrative: 400 internal services, each reading config from 4 sources (default, file, env, flag service) is up to 1,600 individual settings. If even 0.5% of env-injected overrides are stale after a migration, that's 8 services quietly pointed at the wrong backing resource at any time, and none of it is visible by reading a repo, because the override lives outside it.

What the Scenario-A answer does here

The Scenario A fix (log the resolved config and source at boot) still works per service, but reading 400 boot logs by hand to find the 8 bad ones does not scale: at 2 minutes a log, that's over 13 hours of manual review to answer one question.

What flips at this scale

At this scale config stops being a per-service concern and becomes a product: a central config/flag service every service pulls from and reports its resolved values to, with an alert when a service's effective config changes outside a deploy. The flip is from "each service logs its own config" to "one system can answer which services are pointed at X right now", because that question is unanswerable from 400 separate log streams.

The trade-off you'd defend

I would defend a central config/flag service with change auditing over per-service files and env vars, trading a new piece of infrastructure (and a new single point of failure to design a safe fallback for) for the ability to answer "who is on the old database" in one query instead of 400 greps.

Illustrative figures chosen for the arithmetic; not CRED-published numbers.

Gotchas

An undocumented override targets the wrong database

Symptom

The job logs success and writes rows, but finance says the numbers are wrong — the rows landed in a decommissioned read-replica, not the primary.

Cause

A --db-host CLI flag left over from a one-off debugging run outranks the environment variable Jenkins sets, and nobody printed which source won at startup.

Fix

Log the resolved value and its source for every non-secret key at boot, and treat an unexpected source (a CLI override in a scheduled job) as a startup warning, not silence.

Secrets land in logs

Symptom

A print(config) or an unhandled exception traceback shows the DB password in Jenkins console output, visible to every logged-in Jenkins user.

Cause

Config and secrets are stored in the same dict with no distinction, and generic logging dumps the whole thing.

Fix

Mark secret keys explicitly, redact them in any representation that could be logged, and never let str(config) or default exception formatting include them.

A flag never gets removed

Symptom

Six months later the code still branches on if flag_enabled("new_calc"), both branches have drifted, and nobody is sure which one is actually safe to delete.

Cause

The flag was created to de-risk a release, but no removal date or owner was recorded, so cleanup had no trigger.

Fix

Record an owner and a remove-by date at creation, and fail a check (or page the owner) when a flag passes its removal date, as shown in Version releases and expire risky flags's production example.

Tricky questions

Does using environment variables for config count as "secure"?
Most people say

Yes, env vars are safer than hardcoding, so that's enough.

Actually

Env vars beat hardcoding, but they are still visible to anything that can read /proc/<pid>/environ, a crash dump, or container-inspect output. For real secrets, a secret manager with access-controlled retrieval and rotation is the actual answer; env vars are just a delivery mechanism, not a vault.

If a config value isn't set anywhere, is the default always safe to fall back to?
Most people say

Yes, that's what defaults are for.

Actually

A default is safe only if it is a safe failure value, not a safe-looking placeholder. A default DB host of localhost looks fine in dev and silently wrong if a prod override is missing, which is why startup validation should require certain keys explicitly rather than let them fall through to a default.

Scenario question

You're handed a service where DEBUG=True is hardcoded as the default and prod sets DEBUG=False through an environment variable, but a recent prod incident showed stack traces with internal paths in an API error response. What's going on and what do you change?

Model answer

I'd check whether the environment variable actually reaches the process: a typo in the variable name, a missing export in the entrypoint script, or bool("False") evaluating to True in Python would all leave DEBUG effectively on in prod despite the override looking correct. I'd fix the immediate leak by making the default False and parsing boolean env vars explicitly instead of relying on truthiness, so a misconfigured or missing override fails safe instead of failing open. The trade-off is a stricter default makes local dev slightly less convenient, so I'd also log the resolved value at boot so this is visible, not discovered through an incident.

Interview corner

CommonHow do you decide what goes in a config file versus an environment variable versus a secret manager?
“I'd put things that differ by environment but aren't sensitive, like table names, in a tracked file. Env vars are my seam for whatever the deployment platform injects, including which file to even read. Actual secrets go through a secret manager with access control and rotation, never a plaintext file, because env vars are visible to more things than a secret store API.”
TrickIf an env var and a config file both set the same key, which wins, and why does it matter?
“I'd say it depends on the precedence contract the team wrote down, but the question that matters more is: can you tell me without reading code? If not, the specific order matters less than it being documented, consistent, and loggable at startup.”
Experience checkTell me about a config bug that reached production.
“I had a flag's default flip when we upgraded its SDK, and because our app read the flag's library default directly instead of pinning our own fallback, a feature that was supposed to be always-on went dark for a day. Now I always define the fallback explicitly in our code, never trust the SDK's.”
DesignHow would you design feature flag ownership for a 50-engineer org?
“I'd require every flag to be created with an owner, a removal date, and a documented fallback for when the flag service is unreachable. I'd add a scheduled check that pages the owner, not just logs a warning, once a flag passes its removal date, because warnings get ignored and leftover flags become permanent, undocumented branches.”

Memory hook

A thermostat has four hands on the dial; config precedence is just naming them in order.

If you remember only one thing:

Write the precedence order down, and log which source won for every key at boot.

Self-check

  1. Why did resolved["timeout"] print as the string '5' instead of the int 5 in the minimal example?

    Environment variables are always strings; the merge never coerced the type, so the env override's string value overwrote the int default unchanged.

  2. What's the operational difference between a config value and a secret?

    A secret needs access control and rotation and must never appear unredacted in logs; a config value just needs a documented source and precedence.

  3. What should happen when a feature flag passes its removal date?

    Something loud: a failing check or a page to the owner, not a silent no-op, so the flag actually gets cleaned up.

  4. Why is localhost a risky default for a database host?

    It looks like it works everywhere, so a missing prod override can go unnoticed instead of failing loudly.

Prove it

Write a resolve_config(defaults, file_cfg, env_prefix, cli_overrides) function that returns both the merged config and a per-key dict of which source won. Feed it a case where an env var and a CLI flag both set batch_size, and print the per-key source map proving the CLI value won and why.

ce-10-02Card 2 of 6

Apply 12-factor principles to a real release

MediumP3 · Role-dependentTalk about it12 min
twelve-factorstatelessbacking-servicesbuild-release-rundisposability

Why you care

Interviewers reach for "12-factor" because it is a checklist for the single biggest category of production incident in young backend teams: an app that works on a laptop and breaks, or cannot scale, the moment it is deployed to more than one box. Most of the twelve factors are not independent trivia; they are one idea — the running process should be disposable and stateless, and every piece of state that must survive should live outside it — expressed twelve times from different angles. If you can only state three of them, you can usually reconstruct the rest, because they fall out of that one idea. Knowing which one your current Jenkins job violates is more useful than reciting the list.

Explain like I'm 5

Think of a hotel. The blueprint is the codebase: one blueprint, many identical rooms built from it. The key card is config: it unlocks a specific room for a specific stay, without the blueprint itself changing. The room's phone line is a backing service: you plug into the wall socket, and you don't care which provider is behind it, as long as the jack is the same shape. Construction is the build, check-in is the release, and the actual stay is run — three different stages, never confused. A room holds no memory of the last guest; that's a stateless process. Housekeeping can clear a room fast and a guest can check out instantly: that's disposability. And the hotel's central ledger, not notes scribbled on a nightstand, is where you look for what actually happened: that's logs as an event stream.

Core concept

  • Codebase: one codebase tracked in version control, many deploys from it.
  • Dependencies: declared and isolated explicitly, not "whatever happens to already be on this box."
  • Config: stored in the environment, per Configuration and secrets: precedence and feature flags's precedence contract, not baked into the code.
  • Backing services: a database, cache, or queue is an attached resource, swappable by changing config, never by changing code.
  • Build, release, run: strictly separated. A build produces one immutable artifact. A release pairs that artifact with environment-specific config. Run starts processes from a release and never recompiles.
  • Processes: stateless. Anything that must survive a restart lives in a backing service, not in process memory.
  • Port binding: the app exports its own service via a port, rather than depending on being injected into an external webserver.
  • Concurrency: scale out by adding identical processes, not by making one process heavier.
  • Disposability: fast startup, graceful shutdown on SIGTERM, and a design that tolerates a SIGKILL without corrupting data.
  • Dev/prod parity: the same kind of backing services in dev as prod — the same database engine, not SQLite locally and Postgres in prod.
  • Logs: written to stdout as an event stream; the platform routes and aggregates them, the app doesn't manage log files.
  • Admin processes: one-off tasks (migrations, backfills) run with the same code, dependencies and config as the app, not a hand-copied script.

Treat the factors as one rule with eleven corollaries: nothing the process needs to survive should live inside the process. Apply it strictly to anything handling traffic or running on a schedule you don't personally babysit; relax it only for a true one-off admin script, and even then run it with the same config and dependencies as the real app.

Minimal example In-memory state does not survive a restart

PythonIn-memory state does not survive a restartruns
class App:
    def __init__(self):
        self.sessions = {}

    def login(self, user):
        self.sessions[user] = "active"

    def is_logged_in(self, user):
        return self.sessions.get(user) == "active"


app = App()
app.login("alice")
print(app.is_logged_in("alice"))

app = App()  # simulates a fresh process after a restart/redeploy
print(app.is_logged_in("alice"))
Output
True
False

In production Build once, release many times, run from the release record

PythonBuild once, release many times, run from the release recordruns
import hashlib
import json
from pathlib import Path


def build(src_files):
    digest = hashlib.sha256("".join(src_files).encode()).hexdigest()[:12]
    return {"artifact_id": digest, "files": src_files}


def release(artifact, config, release_no):
    return {"release": release_no, "artifact_id": artifact["artifact_id"], "config": config}


def run(release_record):
    print(
        f"running release {release_record['release']} "
        f"(artifact {release_record['artifact_id']}) "
        f"against {release_record['config']['db_host']}"
    )


artifact = build(["etl.py", "requirements.txt"])

staging = release(artifact, {"db_host": "staging-db"}, release_no=41)
prod = release(artifact, {"db_host": "prod-db"}, release_no=42)

run(staging)
run(prod)

Path("releases.json").write_text(json.dumps([staging, prod], indent=2))
print(staging["artifact_id"] == prod["artifact_id"])
Output
running release 41 (artifact b77beb71f050) against staging-db
running release 42 (artifact b77beb71f050) against prod-db
True

See it happen Build, release and run stay separate stages

one codebasebuildone artifactsha:b77be...release+ configrun(stateless processes)backing service(Postgres)
  1. One codebase feeds the build stage. Nothing has run yet.
  2. Build produces exactly one immutable artifact. No config is in it yet.
  3. Release binds that same artifact to environment config, twice: staging and prod are two releases from one build.
  4. Run starts disposable processes from a release record. It never recompiles.
  5. Any state that must survive a restart lives in the backing service, not in the running process.
  6. The naive version keeps that state in the process instead, where a restart silently erases it.
Notice: the artifact is built once and never changes between frames; only config and the backing service relationship change around it.
Scenario A

In your world

The situation

Your nightly settlement job used to run on one long-lived VM. It kept the "last processed id" in a module-level variable, written once at startup from a hardcoded number and updated in a loop, because the box never restarted mid-run anyway.

What actually breaks

The job gets moved to run on an ephemeral container platform that recycles the container between runs. The module-level variable resets to the hardcoded number every single run, so each run reprocesses rows the previous run already loaded, duplicating settlement totals. No exception fires, because reading an int and looping over rows is not an error.

What you'd do on Monday

Move the checkpoint outside the process, so a fresh container reads where the last one left off instead of starting from a hardcoded number:

PythonA checkpoint stored outside the process survives a restartruns
import sqlite3


def ensure_checkpoint_table(db):
    db.execute("CREATE TABLE IF NOT EXISTS checkpoints (job TEXT PRIMARY KEY, last_id INTEGER)")


def get_checkpoint(db, job):
    row = db.execute("SELECT last_id FROM checkpoints WHERE job = ?", (job,)).fetchone()
    return row[0] if row else 0


def set_checkpoint(db, job, last_id):
    db.execute(
        "INSERT INTO checkpoints (job, last_id) VALUES (?, ?) "
        "ON CONFLICT(job) DO UPDATE SET last_id = excluded.last_id",
        (job, last_id),
    )
    db.commit()


db = sqlite3.connect("state.db")  # stands in for Postgres
ensure_checkpoint_table(db)

print("resume from:", get_checkpoint(db, "settlements"))
set_checkpoint(db, "settlements", 48213)

db.close()
db = sqlite3.connect("state.db")  # simulates a fresh process on a new container
print("resume from:", get_checkpoint(db, "settlements"))
Output
resume from: 0
resume from: 48213
Scenario B

Imagine you work at Paytm

50k requests/min across stateless workers

The numbers

Illustrative: 50k requests/min is about 833 req/s. One worker process handling 20 req/s needs roughly 42 concurrent processes to keep up. At that count, any in-memory per-request state — a session dict, a module-level cache keyed by user — is split 42 ways with no guarantee the same user ever hits the same process twice.

What the Scenario-A answer does here

The Scenario A fix (checkpoint in Postgres instead of a module global) still works, but "just add more processes" for concurrency breaks any code that assumed one process holds the full picture, like a module-level counter used for a crude rate limit, or a per-process cache that now disagrees across 42 copies.

What flips at this scale

At this concurrency, the backing service isn't optional infrastructure, it's the only consistent view of state across every process: a shared cache or the database itself. Scaling out means "start more identical, interchangeable processes," which only works if none of them keeps a private truth.

The trade-off you'd defend

I would defend scaling out identical stateless processes over scaling up one bigger process, trading a dependency on a shared backing service (another thing that can be slow or down) for horizontal scaling that doesn't require a rewrite every time load doubles.

Illustrative figures chosen for the arithmetic; not Paytm-published numbers.

Gotchas

A module-level cache disagrees across processes

Symptom

Two requests to the "same" service return different answers depending on which worker process handled them.

Cause

Concurrency via multiple processes means each has its own memory; a cache in a module global is really N separate caches, not one.

Fix

Move shared state to a backing service (Redis, Postgres) every process reads the same way, per the processes and backing-services factors.

An admin script bypasses the app's config

Symptom

A one-off backfill run from someone's laptop writes to the wrong environment, or with a different library version than the deployed app.

Cause

The admin task was a hand-copied script instead of the same codebase, dependencies and config as the app.

Fix

Run admin/management tasks via the same artifact and release config, for example the same container image with a different entrypoint command.

SIGTERM is ignored, so shutdown is never graceful

Symptom

Deploys or scale-downs show truncated writes or connection errors in logs, as if the process was yanked mid-batch.

Cause

The process doesn't handle SIGTERM, so the platform waits out a grace period and then sends SIGKILL, with no chance to finish or checkpoint.

Fix

Install a signal handler that stops accepting new work and finishes or checkpoints the current unit before exiting; see Make ENTRYPOINT, CMD and shutdown predictable for the container-level half of this.

Tricky questions

Does "store config in the environment" mean put a .env file in the repo?
Most people say

Yes, as long as it's not hardcoded in the source file.

Actually

A .env file checked into the repo is still code, just one file removed; it ships the same value to every environment that deploys that commit. The factor means the deploy target supplies the value — real process environment variables, a secret manager, platform-injected config — not that it lives in a different file of the same repo.

If my app only ever runs as one process, do stateless processes even matter?
Most people say

No, that's only relevant once you scale horizontally.

Actually

It matters the moment that one process restarts, which every process eventually does, whether from a deploy, a crash, or an autoscaler recycling it. Disposability and statelessness are about surviving a restart without losing or duplicating work; concurrency just makes the consequence show up sooner.

Scenario question

You're handed a Flask app that writes uploaded files to /tmp/uploads on the box it's running on, and a background thread in the same process reads them an hour later for processing. It's moving from one VM to a container platform that can reschedule the container to a different host at any time. What breaks, and what do you change?

Model answer

The write and the later read assume the same filesystem is still there an hour on, which a disposable, rescheduled container doesn't guarantee; files written to local disk vanish or become unreadable if the container moves or restarts. I'd treat the uploads as a backing-service problem: write directly to object storage at upload time, and have the background job pull from there by a persisted job id, not a local path. The trade-off is a bit more latency and an extra network hop per file versus local disk, but it's the only way the design survives the process actually being disposable, which on this platform it now has to be.

Interview corner

CommonName the one idea behind the twelve factors.
“I'd say it's statelessness: nothing the running process needs should live only inside it, because the process can be killed, restarted or scaled at any time. Most of the twelve factors are that idea applied to config, dependencies, logs, or admin tasks.”
TrickDoes using Docker automatically make my app 12-factor?
“No. Docker gives me build/release/run separation for free if I use it right, but I can still write a container that keeps a cache in memory, writes to its own writable layer, or needs a specific startup order. The factors are about app design, not the packaging.”
Experience checkWhere has violating dev/prod parity bitten you?
“I've seen a job tested against SQLite locally pass every test, then fail in prod on a Postgres-specific constraint the SQLite schema didn't enforce. Now I push for the same database engine locally, even in a disposable container, specifically to catch that class of bug before prod does.”
DesignHow would you make a nightly batch job disposable?
“I'd have it checkpoint progress externally per batch, catch SIGTERM to finish or safely abandon the current unit of work, and make reruns idempotent on a natural key, so killing it and restarting it anywhere produces the same end state as letting it finish uninterrupted.”

Memory hook

A hotel room holds no memory of the last guest; neither should your process.

If you remember only one thing:

Nothing the process needs should live only in the process.

Self-check

  1. Why did the second App() in the minimal example return False for is_logged_in?

    A fresh instance starts with an empty sessions dict; nothing about the login was stored outside the object, so a "restart" loses it.

  2. What's the difference between a "release" and a "build" in the production example?

    A build produces one immutable artifact; a release pairs that same artifact with environment-specific config, so two releases can share one build.

  3. Why are logs treated as an event stream instead of files the app manages?

    So the app doesn't own log rotation, retention or routing — the platform aggregates stdout, and that works the same in every environment.

  4. Why should an admin/backfill script use the same artifact as the app?

    So it runs the same code, dependencies and config as production, instead of drifting from whatever happens to be on someone's laptop.

Prove it

Take any script you've written that keeps an in-memory counter or cache across loop iterations. Rewrite it so that state lives in a SQLite table, kill the process mid-run with Ctrl+C, restart it, and paste output showing it resumes from the persisted value instead of zero.

ce-10-03Card 3 of 6

Version releases and expire risky flags

MediumP3 · Role-dependentTalk about it12 min
semverversioningchangelogfeature-flagsrelease-policy

Why you care

A "patch" release that quietly changes a response field's type breaks every consumer who trusted the version number instead of reading the diff, and that's exactly the failure semantic versioning exists to prevent: a version number is a promise about compatibility, not a changelog entry. The same discipline applies to feature flags from Configuration and secrets: precedence and feature flags: a flag with no planned removal date becomes an untested second code path that outlives the person who understood it. Interviewers ask about semver because misreading a dependency's version bump is one of the most common causes of "it worked yesterday" on a team that pins loosely; they ask about flag lifecycle because "how do you turn this off" is the question nobody asks until an incident forces it.

Explain like I'm 5

A version number is a label on a parcel, not a guess about what's inside. MAJOR says "this parcel is a different shape, your old shelf might not fit it." MINOR says "same shape, we added a pocket, nothing you own has to change." PATCH says "we fixed a tear, same shape, same pockets." A prerelease sticker (-beta.1) says "still being packed, don't put this on the shelf yet." If the sticker on the parcel doesn't match what's actually inside, everyone downstream who sorted their shelves by the sticker gets it wrong — which is exactly what happens when a "patch" silently changes shape.

Core concept

  • MAJOR.MINOR.PATCH: MAJOR for incompatible changes, MINOR for backward-compatible additions, PATCH for backward-compatible fixes (Semantic Versioning 2.0.0, semver.org).
  • A prerelease tag (-alpha, -beta.1) sorts lower than the same numeric release and signals "not yet the real thing"; build metadata (+buildid) is informational and ignored in comparisons.
  • The version number is a promise about the public interface, not about internal implementation. "We rewrote the internals" is a patch only if nothing observable to a consumer changed.
  • A changelog is the version number's evidence: every release should be able to point at what changed and why the bump is the one it is.
  • Artifact identity ties a version to one exact build (see CI/CD and promotion: immutable artifacts to rollback's build-once-promote-many): "v2.3.1" should mean one specific set of bytes, not "whatever was tagged v2.3.1 this time we rebuilt it."
  • Feature flags are versioned behaviour inside a release: staged exposure (percentage or allowlist rollout), a safe fallback if the flag system is unreachable, a test matrix covering flag-on and flag-off, and a removal date treated as part of the release plan, not an afterthought.

Bump by the most severe change in the release, not the smallest: one breaking field in an otherwise tiny release still needs MAJOR. Use a prerelease tag for anything you want feedback on before it's a promise. Treat a feature flag like a loan with a due date: if it's still there past the removal date, that's debt the same as an unpatched library.

Minimal example String sort breaks version order

PythonString sort breaks version orderruns
versions = ["1.9.0", "1.10.0", "1.2.0"]
print(sorted(versions))


def version_key(v):
    return tuple(int(p) for p in v.split("."))


print(sorted(versions, key=version_key))
Output
['1.10.0', '1.2.0', '1.9.0']
['1.2.0', '1.9.0', '1.10.0']

In production Flags expire; staged rollout is deterministic per user

PythonFlags expire; staged rollout is deterministic per userruns
import hashlib
from datetime import date

FLAGS = {
    "new_settlement_calc": {
        "enabled_pct": 25,
        "owner": "payments-team",
        "created": date(2026, 6, 1),
        "remove_by": date(2026, 9, 1),
        "fallback": False,
    },
}


def bucket(user_id, salt):
    digest = hashlib.sha256(f"{salt}:{user_id}".encode()).hexdigest()
    return int(digest[:8], 16) % 100


def flag_enabled(name, user_id, today=date(2026, 9, 15)):
    flag = FLAGS.get(name)
    if flag is None:
        return False
    if today > flag["remove_by"]:
        raise RuntimeError(
            f"flag '{name}' is {(today - flag['remove_by']).days} days past its "
            f"removal date; owner {flag['owner']} must clean it up"
        )
    return bucket(user_id, name) < flag["enabled_pct"]


for uid in ["user-1", "user-2", "user-3", "user-4"]:
    print(uid, flag_enabled("new_settlement_calc", uid, today=date(2026, 7, 1)))

try:
    flag_enabled("new_settlement_calc", "user-1", today=date(2026, 9, 15))
except RuntimeError as e:
    print("RuntimeError:", e)
Output
user-1 False
user-2 False
user-3 False
user-4 True
RuntimeError: flag 'new_settlement_calc' is 14 days past its removal date; owner payments-team must clean it up
Scenario A

In your world

The situation

Your platform team ships a "patch" that renames a JSON field in an internal API response your ETL job consumes. It's called a patch because "it's a one-line change" on their side.

What actually breaks

Your job does row.get("amount_cents"), which now returns None since the field was renamed to amount. No exception fires, nulls land in a finance table, and the dashboard shows zero totals for a day before anyone notices they don't add up.

What you'd do on Monday

Treat a field rename or removal as a breaking, MAJOR-worthy change regardless of line count, and make your own consumer fail loudly on a missing required field instead of silently defaulting it:

PythonA required field should fail loudly, not default to Noneruns
def parse_response(payload, required_fields):
    missing = [f for f in required_fields if f not in payload]
    if missing:
        raise KeyError(f"response missing required fields {missing}; check the API's changelog/version")
    return {f: payload[f] for f in required_fields}


old_shape = {"order_id": "O1", "amount_cents": 4200}
new_shape = {"order_id": "O1", "amount": 4200}  # "amount_cents" renamed to "amount"

print(parse_response(old_shape, ["order_id", "amount_cents"]))
try:
    print(parse_response(new_shape, ["order_id", "amount_cents"]))
except KeyError as e:
    print("KeyError:", e)
Output
{'order_id': 'O1', 'amount_cents': 4200}
KeyError: "response missing required fields ['amount_cents']; check the API's changelog/version"
Scenario B

Imagine you work at Groww

staged rollout to 10M users, 1% canary

The numbers

Illustrative: 10M users, a 1% canary is 100,000 users on the new code path — large enough that a 0.1% error rate is 100 affected users, not zero, which is big enough to detect but small enough to contain.

What the Scenario-A answer does here

The Scenario A fix (fail loudly on a missing field, bump MAJOR for breaking changes) still protects consumers, but "just enable the flag for everyone at midnight" does not: at 10M users, a bucketing bug skewing toward one cohort means the canary population isn't representative, and you find out from a complaint spike, not from your healthy-looking 1% metric.

What flips at this scale

At this scale, flag rollout needs its own observability: compare error rate and key business metrics between the flagged and unflagged cohorts in real time, not just "percent enabled." The removal-date enforcement from the production example becomes a release-blocking check, not a reminder, because at 10M users a forgotten flag means 10M users quietly running undocumented behaviour.

The trade-off you'd defend

I would defend cohort-compared metrics over a flat enablement percentage, accepting the extra dashboard and bucketing work, because a percentage with no comparison can hide a localized failure inside a healthy-looking average.

Illustrative figures chosen for the arithmetic; not Groww-published numbers.

Version bump vs what the consumer must do

Change typeBumpConsumer actionExampleRisk if misclassified
Backward-compatible bug fixPATCHNoneFix a null-handling edge caseLow — consumers already expected the old behaviour was a bug
New, optional capabilityMINOROptionalAdd a new optional query parameterLow, unless the "optional" addition quietly changes a default
Removed, renamed or redefined behaviourMAJORRequired, often before upgradingDrop a deprecated field, change a status code's meaningHigh — consumers who trusted "same major" break silently
Not yet a promisePRERELEASE (-alpha, -beta.1)Opt-in only, expect change3.0.0-beta.1Low in prod, as long as nobody depends on it there

Gotchas

A "patch" changes behaviour a consumer depended on

Symptom

A downstream job starts erroring or silently nulling fields right after a dependency bump labelled a patch release.

Cause

The change broke an implicit contract (a renamed field, a changed default, a stricter validation) the publisher didn't consider part of the public interface.

Fix

Treat any observable behaviour change as the compatibility surface, bump MAJOR, and read changelogs, not just version numbers, before upgrading a pinned dependency.

String-sorting versions puts 1.10.0 before 1.9.0

Symptom

A "latest version" picker or a sort in a script selects an older release because "1.10.0" < "1.9.0" as plain text.

Cause

Lexicographic string comparison compares digit by digit, not numeric value, so a two-digit minor version sorts before a single-digit one.

Fix

Parse into a tuple of integers (or use a real semver library) before comparing or sorting, as in the minimal example.

A flag's fallback is undefined when the flag service is down

Symptom

During a flag-service outage, every flagged code path either all turns on or all throws, instead of falling back to the documented safe default.

Cause

The integration never set an explicit default for "flag service unreachable", so it inherited whatever the client library happens to do.

Fix

Define and test the fallback value for every flag explicitly, and treat "flag service down" as a test case, not an edge case nobody wrote.

Tricky questions

Does bumping MINOR mean "small change"?
Most people say

Yes, MINOR is for small changes and PATCH is for tiny ones.

Actually

MINOR means backward-compatible additions, no matter how large; adding a big new optional subsystem is still MINOR if nothing existing breaks. Size is irrelevant to the bump; compatibility is the only axis semver measures.

Is 2.0.0-alpha lower precedence than 1.9.0 because it's "just a prerelease"?
Most people say

Yes — it's unreleased, so it ranks below any real release, including 1.9.0.

Actually

Semver compares the core MAJOR.MINOR.PATCH numbers first, regardless of prerelease status: 2.0.0-alpha's core version (2.0.0) is still greater than 1.9.0's, so the real order is 1.9.0 < 2.0.0-alpha < 2.0.0. A prerelease tag only ranks below its own full release, not below every earlier release.

Scenario question

You're handed a client library at 3.4.2 and told "upgrading to 3.5.0 is safe, it's just a minor." After upgrading, a method you call starts raising a DeprecationWarning that your test suite treats as an error, and CI goes red. Is this a semver violation, and what do you do?

Model answer

No — a DeprecationWarning on a method that still works is exactly what MINOR allows: a backward-compatible addition, or an announcement of a future removal, not a break. The method still runs; nothing about the public interface's actual behaviour changed. I'd fix my own test configuration, since treating warnings as errors is my team's choice, not the library's broken promise, and I'd note the deprecation so we plan for the eventual MAJOR removal instead of being surprised by it later.

Interview corner

CommonWhat's the difference between a MINOR and a PATCH release?
“I'd say MINOR adds backward-compatible functionality, so existing callers are unaffected but new capability is available; PATCH is a backward-compatible bug fix with no new functionality. Neither should break an existing caller — only MAJOR is allowed to.”
TrickIs v2.0.0-rc.1 safe to run in production?
“I'd treat the -rc.1 tag as the publisher explicitly telling me not yet. It has lower precedence than the final 2.0.0 by design, and running a prerelease in prod means accepting a promise that hasn't been finalized, which I wouldn't do without a documented exception.”
Experience checkHow have you handled a feature flag that outlived its purpose?
“I've found flags still in the codebase a year after their rollout finished, both branches drifted, and nobody wanted to touch either. Now I push for a removal date and owner recorded at creation, and I treat a flag's expiry like a dependency's end-of-life — something that fails a check, not something that just sits in a wiki page.”
DesignHow would you roll out a risky, schema-affecting change safely?
“I'd gate it behind a flag with staged, bucketed rollout, compare error rate and business metrics between the flagged and control cohorts, and keep a tested fallback for when the flag can't be read. I'd also bump the version MAJOR if the change touches anything consumers can observe, so the version number itself doesn't lie about the risk.”

Memory hook

A version number is a label on the parcel, not a guess about what's inside.

If you remember only one thing:

Bump by the most severe observable change, not the smallest diff.

Self-check

  1. Why does sorted(["1.9.0", "1.10.0"]) return ['1.10.0', '1.9.0']?

    String comparison is character by character; '1' < '9' at the second character makes '1.10.0' sort first, which is wrong numerically.

  2. In the production example, why did user-4 get the flag but not user-1 through user-3?

    The rollout bucket is a deterministic hash of the user id against the enabled percentage, not a coin flip, so the same user always lands in the same bucket.

  3. What does build metadata (+buildid) do to version precedence?

    Nothing — it's informational and ignored when comparing versions, unlike a prerelease tag.

  4. Why should a feature flag have a removal date even if it's working fine?

    Because "working fine" isn't the same as "tested as the only code path"; past the removal date, the untested alternate branch is risk with no benefit.

Prove it

Write semver_key(v) that correctly sorts ['1.9.0', '1.10.0', '2.0.0-alpha', '2.0.0', '1.2.0'] into true semver precedence order (a prerelease ranks below its own release, otherwise numeric). Print the sorted list and assert it matches the expected order.

ce-10-04Card 4 of 6

Supply chain: SBOM, lockfiles, provenance and hygiene

HardP3 · Role-dependentTalk about it13 min
sbomlockfilesprovenancesupply-chainslsasigning

Why you care

Log4Shell and incidents like SolarWinds and the xz-utils backdoor made "what's actually in your running artifact, and who built it" a question every security review now asks, and it's a different question from "did we write secure code." An SBOM answers "what's inside" (inventory). A lockfile answers "did we get the bytes we reviewed" (pinning). Provenance answers "who built this and how" (origin). A signature answers "has this specific artifact been tampered with since." None of the four answers "is this safe" on its own, and conflating them is exactly how a perfectly signed, provenance-complete artifact ships a known-vulnerable package. Interviewers in platform or security-adjacent roles ask about this because confusing inventory with safety is the single most common gap.

Explain like I'm 5

Think of a shipping container crossing a border. The SBOM is the packing list: exactly what's inside, box by box. The lockfile is the sealed manifest from the warehouse saying "these are the exact boxes we inspected, not substitutes." Provenance is the shipping company's signed record of which port it left from and which crew loaded it. A signature is the tamper-evident seal on the container door. Customs can check all four and wave the container through — and still find out later that one of the boxes, correctly listed, correctly sealed, correctly tracked, legitimately contains something dangerous. The paperwork proves the chain of custody, not that the contents are safe.

Core concept

  • SBOM (Software Bill of Materials): a structured inventory of every component in a build — name, version, usually a hash — generated automatically as part of the build, not hand-maintained.
  • Lockfiles: pin exact resolved versions, and ideally hashes, of direct and transitive dependencies, so two installs from the same lockfile get the same bytes. A lockfile is reproducibility, not safety: it pins a vulnerable version just as faithfully as a safe one.
  • Provenance: a record of where, when and how an artifact was produced, generated by the build platform itself, not asserted by a developer's laptop, so a consumer can check "was this actually built by our CI from our source" rather than trusting a label. SLSA (currently v1.2) formalizes provenance attestation and defines graduated build-integrity levels; it doesn't certify the code is bug-free, only that the build process is trustworthy and reproducible.
  • Signature verification: cryptographic proof that the specific bytes you're about to run are the specific bytes that were built and attested, not something swapped in transit or in a registry.
  • Dependency hygiene: vetting new dependencies before adopting them, monitoring existing ones for disclosed vulnerabilities, and keeping an accurate, current inventory — the ongoing practice the SBOM feeds.

Separate three different questions and check all three, because each only answers its own: identity (is this the artifact that was built and tested — digest match), origin (do I trust who built it and how — provenance/builder id), and safety (does it contain a known-vulnerable component — SBOM against a vulnerability feed). A green check on one says nothing about the other two.

Under the hood

Modern provenance and signing increasingly skip long-lived private keys in favour of "keyless" signing: the build platform authenticates to a certificate authority with a short-lived OIDC identity token (proving "I am this CI workflow, running for this repo"), gets an ephemeral signing certificate bound to that identity, signs the artifact, and the signature plus certificate are recorded in a public, append-only transparency log, so a forged or backdated signature would also have to forge a log entry anyone can audit. This is why builder.id in an SLSA provenance statement is called a trust base: verifying it means checking that the identity in the certificate matches an allow-listed CI identity, not just that some valid-looking signature exists. A stolen long-lived signing key is a single point of failure that keyless signing is specifically designed to remove, at the cost of depending on the certificate authority and transparency log being available and honest.

Minimal example A lockfile hash catches a swapped artifact

PythonA lockfile hash catches a swapped artifactruns
import hashlib


def digest(content):
    return hashlib.sha256(content).hexdigest()[:16]


trusted_content = b"package-v1.4.2-wheel-bytes"
lockfile = {"orders-wrapper": digest(trusted_content)}

tampered_content = b"package-v1.4.2-wheel-bytes-with-a-backdoor"

print(digest(trusted_content) == lockfile["orders-wrapper"])
print(digest(tampered_content) == lockfile["orders-wrapper"])
Output
True
False

In production SBOM and provenance are not the same question as safety

PythonIdentity, origin and safety are three separate checksruns
import hashlib

TRUSTED_BUILDERS = {"github-actions://athene-org/etl-pipeline"}
KNOWN_VULNERABLE = {("left-pad", "1.0.0")}  # denylist by (name, version)


def build_sbom(components):
    return [
        {"name": n, "version": v, "hash": hashlib.sha256(f"{n}{v}".encode()).hexdigest()[:12]}
        for n, v in components
    ]


def build_provenance(artifact_digest, builder_id, commit):
    return {"artifact_digest": artifact_digest, "builder_id": builder_id, "commit": commit}


def verify_artifact(artifact_digest, provenance, sbom):
    checks = {}
    checks["identity"] = artifact_digest == provenance["artifact_digest"]
    checks["origin_trusted"] = provenance["builder_id"] in TRUSTED_BUILDERS
    checks["no_known_vulns"] = not any(
        (c["name"], c["version"]) in KNOWN_VULNERABLE for c in sbom
    )
    return checks


sbom = build_sbom([("orders-wrapper", "2.3.0"), ("left-pad", "1.0.0")])
provenance = build_provenance("sha256:abc123", "github-actions://athene-org/etl-pipeline", "a1b2c3d")

result = verify_artifact("sha256:abc123", provenance, sbom)
print(result)
print("safe to deploy:", all(result.values()))
Output
{'identity': True, 'origin_trusted': True, 'no_known_vulns': False}
safe to deploy: False

See it happen Identity, origin and safety are three separate gates

source repo@ commit a1b2c3dCI builder(trusted identity)artifactsha256:abc123SBOM(component list)provenance +signatureverify gateblocked
  1. CI, not a laptop, builds the artifact and generates its SBOM in the same run.
  2. The artifact is signed; the signature and provenance record the builder identity and source commit.
  3. The verify gate checks identity: does the digest match what CI tested? Yes.
  4. It checks origin: is the builder identity on the trusted allow-list? Yes.
  5. It checks safety against the SBOM: one listed component matches a known vulnerability.
  6. Identity and origin alone don't clear the gate. A fully signed, fully attested artifact still gets blocked.
Notice: the artifact never changes across frames; only how much of it has been checked does. Two green checks and one red one still means blocked.
Scenario A

In your world

The situation

Your Jenkins job runs pip install -r requirements.txt fresh on every build, straight from PyPI, with no lockfile.

What actually breaks

A transitive dependency gets a new release mid-afternoon. The exact same commit, built again six hours later, resolves a different version than the morning's build that was "tested" in staging. Nobody rebuilt staging, so what ships to prod was never actually the thing anyone verified.

What you'd do on Monday

Pin the whole resolved tree with hashes, and build the artifact once, test that exact artifact, and promote the same bytes rather than rebuilding per environment, per CI/CD and promotion: immutable artifacts to rollback:

Shell
pip-compile --generate-hashes requirements.in -o requirements.lock
pip install --require-hashes -r requirements.lock
Scenario B

Imagine you work at Nykaa

1,200 direct packages across 80 services

The numbers

Illustrative: 80 services x ~15 direct dependencies each is about 1,200 direct packages. With a typical ratio of 5-8 transitive dependencies per direct one, the real component count lands around 6,000-9,000. Reviewing that by hand at 30 seconds per component is 50-75 hours, once.

What the Scenario-A answer does here

The Scenario A fix (a lockfile per service, built once and promoted) pins what each service uses, but it doesn't answer "which of our 80 services currently ship the one component with a newly disclosed CVE" — that needs an aggregated inventory across all 80 lockfiles, which doesn't exist if each team's lockfile only lives in that team's own repo.

What flips at this scale

At this scale, the SBOM has to be generated automatically for every build and collected centrally, so "which services are exposed to CVE-2026-XXXX" becomes one query against a database of SBOMs, not 80 separate greps through repos owned by teams who may not reply to a Slack message the same day.

The trade-off you'd defend

I would defend automated, centrally-collected SBOMs per build over per-team manual tracking, trading the cost of scanning every CI build for the ability to answer a disclosed-CVE exposure question in minutes instead of days during an actual incident.

Illustrative figures chosen for the arithmetic; not Nykaa-published numbers.

Gotchas

A valid signature does not mean a safe dependency

Symptom

An artifact passes signature verification and provenance checks cleanly, and still ships a component with a publicly disclosed critical CVE.

Cause

Signing proves the bytes weren't tampered with after the build and who built them; it says nothing about whether the components inside were safe when they were built in.

Fix

Run identity, origin and safety as three separate checks, as in the production example, and block promotion if any one fails, not just if the signature is invalid.

A lockfile pins the version, not safety

Symptom

A dependency scan flags a CVE in a package your lockfile has pinned for eight months, with a fix already released upstream.

Cause

A lockfile guarantees reproducibility — the same bytes every install — which is the opposite of automatic updates; pinning a vulnerable version is just as reproducible as pinning a safe one.

Fix

Scan pinned versions against a vulnerability feed on a schedule, independent of whether anyone is actively editing the lockfile.

The SBOM was generated from source, not from what actually shipped

Symptom

The SBOM lists a dependency version that doesn't match what's actually inside the deployed container.

Cause

Build-time substitutions, a different base image, or a cached layer with an older version can diverge from the declared source manifest.

Fix

Generate the SBOM from the built artifact itself, not from source files, so it reflects what will actually run.

A rolling tag breaks the identity check

Symptom

"The same image tag" behaves differently between two deploys with no code change on your side.

Cause

A mutable tag (like :latest or an unpinned base image tag) can point at different bytes over time; identity verification against a tag, not a digest, isn't verifying anything stable.

Fix

Pin and verify by digest (sha256:...), the same discipline Build cache-friendly, repeatable Dockerfiles applies to base images.

Tricky questions

If an SBOM lists a component, does that mean it was scanned for vulnerabilities?
Most people say

Yes, having it in the SBOM means it's been checked.

Actually

An SBOM is inventory, not a scan result. It says the component is present; a separate step has to compare that inventory against a vulnerability database (or a denylist, as in the production example) to say anything about safety.

Does pip install -r requirements.txt with pinned == versions give the same guarantees as a lockfile?
Most people say

Yes, pinning the version is the same as a lockfile.

Actually

Pinning direct dependencies with == doesn't pin their transitive dependencies unless every one of those also happens to be pinned somewhere. A real lockfile resolves and pins the entire dependency tree, usually with hashes, so the install is reproducible all the way down, not just at the top level.

Scenario question

You're handed an artifact that passed every gate: digest matches what CI tested, provenance shows it was built by your trusted pipeline from the right commit, and it's signed. Security still blocks the release. What are they most likely checking that the pipeline didn't, and how do you find out?

Model answer

I'd assume they ran a vulnerability scan against the SBOM, which the identity and provenance checks don't cover at all — those only prove the bytes are the ones that were built, by whom, and from where, not what's safe inside them. I'd pull the SBOM for this artifact and diff it against the vulnerability database security used, which should name the exact component and CVE. If it's a transitive dependency with no fix yet available, I'd look for a documented, time-boxed exception process rather than either shipping it silently or blocking indefinitely with no path forward.

Interview corner

CommonWhat's the difference between an SBOM and a lockfile?
“I'd say a lockfile pins the exact versions your build resolves to, so installs are reproducible; an SBOM is the resulting inventory of what's actually in the artifact, generated from the build, which is what you scan for vulnerabilities and quote in a security review.”
TrickIf an artifact is signed, is it safe to deploy?
“No — signed tells me the bytes weren't tampered with and I know who built them. It says nothing about whether the components inside have known vulnerabilities. I check identity, origin and SBOM-based safety as three separate gates.”
Experience checkHave you dealt with a supply-chain scare, even a near-miss?
“A transitive dependency got yanked from PyPI mid-day over a security issue, and because we weren't using a lockfile, two builds six hours apart on the same commit resolved different versions. We moved to pinned, hashed lockfiles and build-once-promote-many specifically so that can't happen silently again.”
DesignHow would you add supply-chain checks to an existing pipeline without blocking every release on day one?
“I'd start by generating an SBOM on every build and just recording it, no blocking, to build a baseline of what we actually ship. Then I'd turn on blocking for new critical CVEs with no existing exception, and only after that add provenance and signature verification as a release gate, so I'm not shipping three new blocking checks the same day with no rollback plan.”

Memory hook

A sealed, signed, perfectly tracked container can still legally carry something dangerous inside.

If you remember only one thing:

Identity, origin and safety are three separate checks — a pass on one proves nothing about the other two.

Self-check

  1. In the production example, why does verify_artifact return no_known_vulns: False even though identity and origin_trusted are both True?

    Identity and origin check the digest and the builder, not the SBOM contents; left-pad 1.0.0 is on the vulnerability denylist regardless of how trustworthy the build was.

  2. What does a lockfile guarantee that an SBOM doesn't?

    Reproducibility: the same resolved versions, and often hashes, on every install. An SBOM only reports what ended up in a given build.

  3. What does SLSA provenance's builder.id field represent?

    The trust base — the build platform and process trusted to have faithfully run the build and recorded what happened, which a consumer checks against an allow-list.

  4. Why does keyless signing use a transparency log?

    So that even a short-lived, identity-bound certificate can't produce an undetectable forged signature — any signing event is recorded in a public, append-only log anyone can audit.

Prove it

Write a function that takes a list of (component, version) tuples and a small vulnerability-feed dict, and returns which components are flagged. Then write verify(artifact) that checks digest match, builder allow-list membership, and the vulnerability flag as three independent booleans, and print which specific check(s) failed when a fully signed, fully provenance-attested artifact still contains a flagged component.

ce-10-05Card 5 of 6

Maintain dependencies and verify the released artifact

MediumP3 · Role-dependentTalk about it12 min
dependency-managementvulnerability-triageartifact-verificationpatching

Why you care

Most teams either never update dependencies until a security scanner screams, or update everything constantly and spend a sprint fixing whatever an upgrade broke; neither is dependency maintenance, both are reactions. The actual skill is triage: telling a direct dependency bump apart from a transitive one that rode along uninvited, judging which CVE actually matters for how you use a package, and verifying the one artifact you're about to ship, not the source tree you built it from three steps ago. Interviewers in any role touching production ask "how do you know what you're about to deploy is safe" because "we ran a scanner once" is a tool, not an answer.

Explain like I'm 5

A direct dependency is a tenant you signed a lease with. A transitive dependency is whoever that tenant sublets a room to, without necessarily telling you. You pinned the tenant's name and unit number — the direct version — but the sublet changes on its own schedule unless your lease explicitly locks that down too. Verifying the final artifact is like inspecting the actual building before anyone moves in, not re-reading the lease on your desk and assuming the building still matches it.

Core concept

  • Direct dependencies are what you declare; transitive dependencies are what those pull in. Pinning a direct version does not pin the transitive versions it resolves to, unless a lockfile locks the whole tree.
  • Patch ownership: someone — a person or a rotation — is responsible for triaging a disclosed vulnerability, judging severity-in-context (does your usage even reach the vulnerable code path), and either upgrading, patching, or filing a documented, time-boxed exception.
  • Scan the artifact that will run, not just the manifest: a vulnerability in a transitive dependency only shows up if the scanner looks at what's actually bundled into the built thing, which is why Supply chain: SBOM, lockfiles, provenance and hygiene's SBOM is generated from the build.
  • Verify by digest: compare the digest of what's about to deploy against the digest of what CI actually tested. If they don't match, something rebuilt between test and deploy, and you are no longer deploying the tested artifact.
  • Isolated build/cache trust: a build should pull from a trusted, access-controlled cache or registry, not an open mirror anyone could have poisoned. Documented exceptions (an accepted CVE, a pinned-but-vulnerable version with no fix yet) need an owner, a reason and an expiry, same as a feature flag.
  • No scanner guarantees safety: a scanner only catches what's in its database, so passing a scan is evidence, not proof; zero-days and misconfigurations it doesn't check for still get through.

Separate "can I safely upgrade this" (direct, usually lower-risk, test it) from "did my dependency tree drift under me" (transitive, needs a lockfile and periodic diffing) from "is the thing I'm about to deploy actually clean" (artifact scan at the promotion gate, every time, not just when someone remembers).

Minimal example A pinned direct dependency still drifts through its own deps

PythonA pinned direct dependency still drifts through its own depsruns
pip_install_log_monday = {"etl-toolkit": "2.1.0", "urllib-core": "1.9.0"}
pip_install_log_friday = {"etl-toolkit": "2.1.0", "urllib-core": "1.9.4"}

print(pip_install_log_monday == pip_install_log_friday)
changed = {
    k for k in pip_install_log_monday if pip_install_log_monday[k] != pip_install_log_friday[k]
}
print("direct dependency pinned:", True, "| drifted packages:", changed)
Output
False
direct dependency pinned: True | drifted packages: {'urllib-core'}

In production Scan the artifact that will actually run

PythonScan the artifact that will actually runruns
from datetime import date

VULN_DB = {
    "orders-wrapper==2.3.0": [{"id": "CVE-2026-1111", "severity": "high"}],
}

ACCEPTED_EXCEPTIONS = {
    "CVE-2026-1111": {
        "ticket": "SEC-482",
        "expires": date(2026, 10, 1),
        "reason": "no exploitable path in our usage",
    },
}


def scan_artifact(components, today):
    findings = []
    for component in components:
        for vuln in VULN_DB.get(component, []):
            exception = ACCEPTED_EXCEPTIONS.get(vuln["id"])
            if exception and today <= exception["expires"]:
                continue
            findings.append((component, vuln["id"], vuln["severity"]))
    return findings


def promote(tested_digest, artifact_digest, components, today):
    if tested_digest != artifact_digest:
        raise RuntimeError("artifact digest does not match the digest CI tested; refusing to promote")
    findings = scan_artifact(components, today)
    if findings:
        raise RuntimeError(f"blocking findings without an active exception: {findings}")
    return "promoted"


components = ["orders-wrapper==2.3.0"]
print(promote("sha256:abc", "sha256:abc", components, today=date(2026, 9, 15)))
try:
    print(promote("sha256:abc", "sha256:abc", components, today=date(2026, 10, 15)))
except RuntimeError as e:
    print("RuntimeError:", e)
Output
promoted
RuntimeError: blocking findings without an active exception: [('orders-wrapper==2.3.0', 'CVE-2026-1111', 'high')]
Scenario A

In your world

The situation

Before every deploy, your Jenkins job runs pip install -U -r requirements.txt "to pick up security fixes", then builds and deploys straight from that install, with no separate test run against that newly resolved set.

What actually breaks

An upgraded transitive dependency flips a default (strict mode in a date-parsing library). A job that has quietly tolerated slightly malformed vendor timestamps for two years starts raising on every row. The deploy that broke it shows zero diff in your own repo, because nothing in your repo changed.

What you'd do on Monday

Separate "update dependencies" from "deploy." Run upgrades as their own change with the full test suite against the new lockfile, merge only when green, then build once from the merged lockfile and promote that exact artifact — never upgrade implicitly as part of a deploy:

PythonDiff two resolved dependency sets before trusting "no code changed"runs
tested_lock = {"etl-toolkit": "2.1.0", "date-parser": "3.4.0", "urllib-core": "1.9.0"}
deploy_lock = {"etl-toolkit": "2.1.0", "date-parser": "3.5.0", "urllib-core": "1.9.4"}

DIRECT = {"etl-toolkit", "date-parser"}


def diff_lockfiles(tested, deploy, direct):
    changed = {k: (tested[k], deploy[k]) for k in tested if tested.get(k) != deploy.get(k)}
    return {k: ("direct" if k in direct else "transitive", old, new) for k, (old, new) in changed.items()}


for pkg, (kind, old, new) in diff_lockfiles(tested_lock, deploy_lock, DIRECT).items():
    print(f"{pkg} ({kind}): {old} -> {new}")
Output
date-parser (direct): 3.4.0 -> 3.5.0
urllib-core (transitive): 1.9.0 -> 1.9.4
Scenario B

Imagine you work at Chargebee

nightly scans across 300 repos

The numbers

Illustrative: 300 repos x ~50 resolved dependencies each (direct plus transitive) is about 15,000 dependency edges. If just 5% of repos depend on one widely-used package that gets a disclosed CVE, that's 15 repos needing a coordinated patch, findable only by searching resolved lockfiles, not by reading 300 requirements.txt files for a name that might not even appear there directly.

What the Scenario-A answer does here

The Scenario A fix (separate upgrade PRs, test before promote) protects each repo individually, but triaging repo by repo at this count means someone manually checks 300 lockfiles for one package name whenever a new CVE drops, which doesn't finish before the exposure window matters.

What flips at this scale

At this scale, triage flips from "check each repo when something comes up" to "query a central index of every repo's resolved dependencies for a given package and version," built from the same per-build SBOMs as Supply chain: SBOM, lockfiles, provenance and hygiene. "Are we exposed" becomes answerable in minutes across all 300 repos, with an owner auto-notified per repo instead of a company-wide message asking people to go check.

The trade-off you'd defend

I would defend investing in a central dependency index over per-repo manual triage, trading the cost of standardizing SBOM generation across every repo for the ability to answer an exposure question across the whole fleet before an exploit window closes.

Illustrative figures chosen for the arithmetic; not Chargebee-published numbers.

Where a check runs vs what it actually proves

CheckWhere it runsWhat it provesWhat it misses
Static scan of declared manifestSource tree, before buildKnown CVEs in versions you declaredTransitive versions actually resolved at install time
Lockfile hash verificationInstall stepInstalled bytes match what was reviewedWhether those reviewed bytes are themselves safe
Scan of the built artifactAfter build, before promotionWhat will actually run, including bundled transitive depsZero-days, anything not yet in the scanner's database
Digest comparison at the promotion gatePromotion, immediately before deployYou're deploying the exact bytes CI tested, not a rebuildWhether that one artifact is itself free of vulnerabilities

Gotchas

Pinning the direct dependency isn't enough

Symptom

requirements.txt hasn't changed in months, but a build from last week behaves differently from a build run today.

Cause

An unpinned transitive dependency's version range allows a new release to be resolved on each fresh install, as shown in the minimal example.

Fix

Use a full lockfile that pins the entire resolved tree, direct and transitive, and only update it deliberately through its own reviewed diff.

Scanning only the source misses what actually ships

Symptom

A scanner run against requirements.txt reports clean, but a scan of the deployed container finds a vulnerable package.

Cause

The declared manifest and the actually-resolved, actually-bundled set of packages can diverge, through build-time substitution, a stale cached layer, or an unpinned transitive range.

Fix

Scan the built artifact, not the source tree, as the authoritative check, per Supply chain: SBOM, lockfiles, provenance and hygiene.

A clean scan today doesn't mean clean tomorrow

Symptom

An artifact passes its scan at build time, sits in a registry for three weeks, then a CVE is disclosed for a component it contains, and nobody re-checks an artifact already "approved."

Cause

Vulnerability databases update continuously; a point-in-time scan result has a shelf life, and gates that only scan once at build time miss newly disclosed CVEs in artifacts awaiting deploy.

Fix

Re-scan artifacts sitting in a registry on a schedule, or re-scan at the promotion gate immediately before deploy, not only at build time.

Tricky questions

Does "we scan our dependencies" mean the same thing regardless of when and what you scan?
Most people say

Yes, a scan is a scan.

Actually

Scanning source manifests, scanning a lockfile, and scanning the built artifact can all report differently, because each sees a different, sometimes stale, view of what's actually resolved and bundled. "We scan" only means something once you can say which of those three, and how often.

If a scanner reports zero findings, is the artifact safe?
Most people say

Yes, a clean scan means it's safe to ship.

Actually

A clean scan means nothing in the scanner's current database matches what it looked at. It says nothing about zero-days, misconfigurations the scanner doesn't check, or vulnerabilities disclosed an hour after the scan ran.

Scenario question

You're handed a policy that says "critical and high CVEs block a release." A release is blocked on a critical CVE in a transitive dependency, but your team has no realistic path to reach the vulnerable code — it's in a code path your usage never calls. Ship on time, or block the release?

Model answer

I would not override the block silently. I'd document a time-boxed exception: who reviewed it, why the vulnerable code path is unreachable in your usage, a ticket, and an expiry date forcing a re-check or upgrade later — the same pattern as the production example's ACCEPTED_EXCEPTIONS. The trade-off is this adds process overhead to what might be a genuinely safe release, but a severity-based policy with no exception path either blocks every release forever on noise or trains people to bypass it quietly, which is worse than a documented, expiring exception.

Interview corner

CommonWhat's the difference between updating a direct dependency and a transitive one?
“A direct dependency is one I declared and chose a version for; a transitive one is pulled in by something I depend on, and pinning my direct version doesn't pin that unless a lockfile locks the whole resolved tree. I treat transitive drift as something to diff and review, not something that just happens.”
TrickYour scanner is green. Are you done?
“No — green means nothing in its database matched what it scanned, as of when it ran, against what it scanned. I still check that it scanned the actual artifact, not just the manifest, and that the scan is recent relative to deploy, not three weeks stale.”
Experience checkTell me about a dependency upgrade that broke something unexpected.
“A transitive date-parsing library we never declared directly changed a default from lenient to strict parsing, and a job that had quietly tolerated slightly malformed vendor timestamps for years started raising on every row the day it rode along with an unrelated upgrade. We now diff the full resolved lockfile on every dependency PR, not just the package someone meant to change.”
DesignHow would you set up ongoing dependency maintenance for a team that currently updates reactively?
“I'd separate update PRs from release PRs so upgrades get their own full test run, add a scheduled scan of the actual built artifact, not just source, with an owner for triage, and require any accepted vulnerability to have a ticket and an expiry instead of a silent ignore. That turns 'we'll deal with it when a scanner screams' into a routine with a known cadence.”

Memory hook

A lease with the tenant's name on it doesn't control who they sublet to.

If you remember only one thing:

Pin the whole resolved tree, and scan what will actually run, not what you declared.

Self-check

  1. In the minimal example, why do the two install logs differ even though the direct dependency version is identical?

    The transitive dependency urllib-core wasn't pinned by a lockfile, so it resolved to a newer version on the second install.

  2. What three things does the production example's promote function check before shipping?

    That the artifact digest matches what was tested, that it has no blocking vulnerability findings, and that any finding without an active, unexpired exception blocks the promotion.

  3. Why can a scan of requirements.txt miss something a scan of the built container finds?

    The declared manifest isn't necessarily what got resolved and bundled; build-time substitutions or unpinned ranges can diverge from it.

  4. Why does an accepted vulnerability exception need an expiry?

    So it gets re-evaluated instead of becoming a permanent, unreviewed gap — the same reason a feature flag needs a removal date.

Prove it

Take two lockfile-style dicts representing "tested" and "about to deploy" dependency sets with at least one transitive drift. Write and run diff_lockfiles to flag it, then extend it to raise if any drifted package is on a small denylist, and show the exception path (an entry with an expiry date) letting a documented drift through.

ce-10-06Card 6 of 6

Prepare runbooks and prove on-call readiness

MediumP3 · Role-dependentTalk about it12 min
runbookson-callincident-readinessescalationhandover

Why you care

A runbook that says "if the pipeline fails, investigate" is not a runbook, it's a wish. The actual job of a runbook is to let someone who is not the original author, at 2 a.m., with partial information, perform a safe check and a known mitigation without guessing or waking up the one engineer who "just knows." On-call readiness is the proof that this actually works: a rotation with no verified pager access, no rehearsed rollback, and a six-month-stale handover note is not covered, it's a liability wearing a schedule. Interviewers ask about this because "how does your team handle incidents" separates people who have genuinely held the pager from people who have only ever been upstream of one.

Explain like I'm 5

A fire extinguisher's instruction label doesn't say "figure out the fire." It says pull the pin, aim low, squeeze, sweep — in that order, by anyone, under stress, without prior training. A runbook is that label for a specific alarm: here's the symptom, here's the one safe thing to check first, here's what to do if the check confirms it, here's who to call if it doesn't. On-call readiness is making sure the extinguisher is actually where the label says, charged, and that the person on shift has actually used one before, not just read the label for the first time while the room fills with smoke.

Core concept

  • A runbook entry has five parts: symptom (what you'll actually observe — an alert name, a metric, a log pattern), a safe check (read-only, approved, doesn't make things worse while you figure out what's wrong), escalation (who and when, before you're out of ideas, not after), mitigation/rollback (the action, with its own risk noted), and verification (how you confirm it actually worked, not just that you did something).
  • Runbooks are living operational artifacts tied to a specific alert or symptom, not a generic study guide or a wiki page nobody has opened since it was written; the production example models one as checkable logic, not prose, precisely because prose can't be tested.
  • On-call readiness, concretely: the on-call person has working access to the systems and dashboards the runbook names (verified, not assumed), is actually in the alert's paging rotation, has a current handover from the previous shift, and has rehearsed the mitigation at least once, not just read about it.
  • Handover is a deliberate step: what's currently degraded, what's being watched, what changed recently, and anything suppressed or acknowledged. Stale or missing handover is itself an incident waiting to compound.
  • Rehearsal — a game day, a drill, a documented dry run — is what turns "there's a runbook" into "this works," the same way a disaster-recovery plan (SLOs and incident signals: RED/USE and error budgets, Disaster recovery) is unproven until someone has actually failed over using it.

Write a runbook entry the moment you've debugged something for the second time, not the first — the first time is still discovery. Treat "nobody has rehearsed this" as equivalent to "this runbook doesn't exist" when assessing real readiness, because an untested procedure under incident stress fails the same way untested code does.

Minimal example A runbook is a checkable decision tree, not prose

PythonA runbook is a checkable decision tree, not proseruns
RUNBOOK = [
    {
        "symptom": "queue depth rising, workers idle",
        "check": lambda m: m["queue_depth"] > 1000 and m["worker_cpu"] < 10,
        "mitigation": "restart worker pool; check for a deadlocked consumer",
    },
    {
        "symptom": "queue depth rising, workers busy",
        "check": lambda m: m["queue_depth"] > 1000 and m["worker_cpu"] >= 10,
        "mitigation": "scale out workers; this is a load problem, not a hang",
    },
]


def diagnose(metrics):
    for step in RUNBOOK:
        if step["check"](metrics):
            return step["mitigation"]
    return "no matching symptom; escalate to on-call lead"


print(diagnose({"queue_depth": 4000, "worker_cpu": 3}))
print(diagnose({"queue_depth": 4000, "worker_cpu": 70}))
print(diagnose({"queue_depth": 50, "worker_cpu": 3}))
Output
restart worker pool; check for a deadlocked consumer
scale out workers; this is a load problem, not a hang
no matching symptom; escalate to on-call lead

In production On-call readiness before a handover, not during an incident

PythonOn-call readiness check before a handoverruns
from datetime import date

HANDOVER_MAX_AGE_DAYS = 2


def readiness_check(engineer, today):
    issues = []
    if not engineer.get("has_runbook_access"):
        issues.append("missing runbook access")
    if not engineer.get("has_alert_routing"):
        issues.append("not in the pager rotation for this service")
    last_handover = engineer.get("handover_date")
    if last_handover is None or (today - last_handover).days > HANDOVER_MAX_AGE_DAYS:
        issues.append("handover notes missing or stale")
    if not engineer.get("rehearsed_rollback"):
        issues.append("has not rehearsed the rollback runbook")
    return issues


today = date(2026, 9, 30)

ready = {
    "has_runbook_access": True,
    "has_alert_routing": True,
    "handover_date": date(2026, 9, 29),
    "rehearsed_rollback": True,
}
not_ready = {
    "has_runbook_access": True,
    "has_alert_routing": False,
    "handover_date": date(2026, 9, 20),
    "rehearsed_rollback": False,
}

for name, eng in [("priya", ready), ("arjun", not_ready)]:
    issues = readiness_check(eng, today)
    print(name, "READY" if not issues else issues)
Output
priya READY
arjun ['not in the pager rotation for this service', 'handover notes missing or stale', 'has not rehearsed the rollback runbook']

See it happen A page becomes a verified mitigation, step by step

pageron-call engineerrunbookserviceP1 page: settlement job failinglook up: queue depth risingsafe check: queue depth, worker CPUrun safe checkqueue_depth=4000, cpu=3%apply mitigation: restart worker poolupdate: mitigated, verifying recoverylog outcome + handover note
  1. The page fires from a concrete SLO breach, not a guess.
  2. The engineer finds the matching symptom entry instead of improvising.
  3. The runbook names a read-only check first, no mitigation yet.
  4. Only the documented, approved query is executed.
  5. This confirms a stuck consumer, not a load spike.
  6. The mitigation matches the branch the runbook specifies for this exact symptom.
  7. A status update goes out before root cause is known, naming facts versus hypothesis.
  8. The loop closes with a timestamped handover entry for the next on-call.
Notice: every arrow into the service is a documented step from the runbook; nothing here is the engineer improvising from memory.
Scenario A

In your world

The situation

Your team's only "runbook" for the nightly settlement job is a Slack thread from eight months ago where a senior engineer fixed a similar-looking failure, and the on-call rotation is informally "whoever's awake."

What actually breaks

The job fails at 2 a.m. on a weekend that engineer is on leave and unreachable. The on-call engineer, two weeks into the rotation, finds the old Slack thread, isn't sure if it's the same symptom, and spends 90 minutes making an uninformed guess before escalating — by which time the job has retried itself into a worse state with duplicate partial writes.

What you'd do on Monday

Write the runbook entry now, as data a person two weeks in can execute, with an explicit, time-boxed escalation trigger instead of leaving escalation to a stressed individual's judgment:

PythonA runbook entry a stranger can execute, not a Slack threadruns
RUNBOOK_ENTRY = {
    "symptom": "nightly settlement job: alert 'rows_loaded == 0', job marked success",
    "safe_check": "query audit.settlement_runs WHERE run_date = today(); compare row_count to source manifest",
    "escalation": {"after_minutes": 20, "contact": "data-platform-oncall-secondary"},
    "mitigation": "re-run the job from the last good checkpoint; do NOT re-run from scratch (duplicates settled rows)",
    "verify": "row_count in audit table matches source manifest count, and matches target table count",
}


def follow_runbook(entry, minutes_elapsed, check_result):
    if check_result == "mismatch":
        if minutes_elapsed >= entry["escalation"]["after_minutes"]:
            return f"escalate to {entry['escalation']['contact']}"
        return entry["mitigation"]
    return "no action needed"


print(follow_runbook(RUNBOOK_ENTRY, minutes_elapsed=5, check_result="mismatch"))
print(follow_runbook(RUNBOOK_ENTRY, minutes_elapsed=25, check_result="mismatch"))
print(follow_runbook(RUNBOOK_ENTRY, minutes_elapsed=5, check_result="match"))
Output
re-run the job from the last good checkpoint; do NOT re-run from scratch (duplicates settled rows)
escalate to data-platform-oncall-secondary
no action needed
Scenario B

Imagine you work at Twilio

follow-the-sun on-call across 3 regions

The numbers

Illustrative: 400 pages/month across a follow-the-sun rotation of 3 regions is about 13 pages/day, split roughly evenly, meaning any one on-call shift of 8 hours can expect 4-5 pages even on a calm day — not rare enough to treat each one as a fresh investigation.

What the Scenario-A answer does here

The Scenario A fix (write the runbook entry, structure it as checkable logic) works for one engineer on one shift, but with 3 regions handing off every 8 hours, "ask the person who fixed it last time" fails differently: that person is now asleep in a different timezone for two of every three shifts, and a handover note that's just "all good, nothing to report" doesn't tell the incoming region what's actually been flapping.

What flips at this scale

At this scale, handover becomes a structured artifact, not a courtesy message: open issues, suppressed or acknowledged alerts, anything watched but not yet actioned, carried forward region to region. Readiness, as in the production example, gets checked automatically before a shift starts, not assumed because someone's name is on the calendar.

The trade-off you'd defend

I would defend an automated, structured handover and readiness check over a trust-based "ask the previous shift" handoff, trading the overhead of maintaining that structure for incoming engineers who are never starting from zero context, which matters most exactly when the person who could explain it is asleep.

Illustrative figures chosen for the arithmetic; not Twilio-published numbers.

Gotchas

A runbook that only the author can execute

Symptom

The documented "fix" references internal knowledge ("restart it the way we usually do") that isn't actually written down anywhere in the entry.

Cause

The runbook was written by, and implicitly for, the person who already knows the system, so unstated steps got skipped.

Fix

Have someone outside the original incident try to execute the runbook from the document alone, and treat their questions as the missing steps to add.

Escalation happens only after someone gives up

Symptom

An incident ran for over an hour before anyone outside the original on-call engineer heard about it.

Cause

The runbook, or the engineer's habit, has no explicit "escalate by this time regardless of progress" rule, so escalation depends on when someone feels stuck.

Fix

Put a time-boxed escalation trigger in the runbook itself, as in the production example's after_minutes, so it isn't a judgment call made alone under stress.

Readiness assumed from a calendar entry

Symptom

The person whose name is on the on-call calendar can't actually access the dashboard the runbook tells them to check.

Cause

Being scheduled and being equipped are treated as the same thing; access provisioning wasn't verified before the shift started.

Fix

Run, and require passing, a readiness check — access, pager routing, current handover, rehearsed rollback — before a shift starts, not after the first page of it.

Tricky questions

Is a wiki page describing the architecture a runbook?
Most people say

Yes, it explains the system, which is what you need during an incident.

Actually

An architecture doc explains how the system is supposed to work; a runbook tells you what to do when it isn't. During an incident you need the second one, fast, and conflating them means someone reads architecture prose while the clock runs instead of following a symptom-to-action path.

Does "we have a runbook for this" mean the team is on-call ready?
Most people say

Yes, if the document exists, the team can handle it.

Actually

An unrehearsed runbook is unproven; it can be stale, wrong about current system behaviour, or technically correct but unusable under stress. Readiness means someone has actually executed it — a drill, a real incident, a game day — not just that it exists in a wiki.

Scenario question

You're handed an on-call rotation where the previous person's handover note says "all good, nothing to report," but you get paged twice in your first two hours for an alert that, it turns out, has been firing and auto-resolving for the past three days. What does this tell you about the handover process, and what do you change?

Model answer

It tells me "nothing to report" hid real information: a flapping alert that was known but never escalated or fixed. I'd change the handover format to require listing anything that fired in the last shift, even if it auto-resolved, with a one-line note on whether it's being tracked, rather than letting "nothing to report" be a valid default. The trade-off is a slightly heavier handover for a quiet shift, which is worth it the moment a quiet-looking shift was actually masking a pattern the next person had no way to know about.

Interview corner

CommonWhat are the required parts of a good runbook entry?
“I'd want a specific, observable symptom, a safe read-only check to confirm it, a mitigation or rollback with its own risk noted, an explicit escalation trigger with a contact, and a way to verify the mitigation actually worked, not just that an action was taken.”
TrickYour team has runbooks for every known alert. Are you on-call ready?
“Not necessarily. I'd ask whether anyone other than the author has actually executed each one, whether access and paging are verified for the current rotation, and whether handover is structured, not just a 'nothing to report' message. A document existing and a team being ready are different claims.”
Experience checkDescribe an incident where the lack of a runbook made things worse.
“I've seen an on-call engineer, new to the rotation, spend most of an outage figuring out what was even normal before they could tell what was wrong, because there was no documented safe check. By the time they escalated, a retry loop had made the data problem worse. We wrote the runbook the next day, which was a day late.”
DesignHow would you build on-call readiness for a new rotation before it goes live?
“I'd require a documented runbook for the top known failure modes, verify each new on-call engineer's access and paging routing explicitly, run at least one rehearsal or game day against a real, non-prod failure, and only then let that person carry the pager solo, rather than assuming a training doc and good intentions are enough.”

Memory hook

A fire extinguisher's label doesn't say "figure out the fire" — it says pull, aim, squeeze, sweep.

If you remember only one thing:

An unrehearsed runbook is unproven; readiness means someone has actually executed it.

Self-check

  1. Why does diagnose escalate instead of returning a mitigation for the third input in the minimal example?

    Its metrics (queue_depth=50) don't match either defined symptom check, and a runbook that can't match a symptom should hand off to a human, not guess.

  2. What's the difference between a mitigation and a root-cause fix in a runbook?

    A mitigation restores the symptom, like restarting the pool; it's meant to be fast and known-safe, not necessarily the full explanation, which can come later in a postmortem.

  3. Why does follow_runbook check minutes_elapsed before returning the mitigation?

    To separate "try the known mitigation first" from "escalate because it's taking too long," so escalation isn't left to a stressed individual's judgment alone.

  4. What makes a handover note "structured" instead of a courtesy message?

    It lists specific items — ongoing issues, suppressed alerts, recent changes — rather than a free-text summary that can hide information behind "all good."

Prove it

Write a runbook entry for a real failure you've debugged before, or a plausible one from your current job, structured as symptom/safe_check/escalation/mitigation/verify like the production example. Have a teammate, or a future version of yourself after a week, execute only from the written entry with no extra context, and note every question they had to ask — those questions are the missing steps.