Shelf · Core Eng · Chapter 8

Chapter 8: CI/CD for real

Your Jenkins job that rebuilds separately in every environment and calls that "the pipeline" is the single biggest gap between a TCS ETL setup and a product-company release process. This chapter replaces it with one immutable artifact promoted by reference, gates that fail closed instead of fail open, migrations and rollouts that survive a schema change, and a CI job that earns exactly the privilege it needs and no more.

Difficulty
Priority
ce-08-01Card 1 of 6

CI/CD and promotion: immutable artifacts to rollback

MediumP2 · Very likelyKnow cold13 min
ci-cdpipelineartifactspromotionrollback

Why you care

"Walk me through your pipeline" is a standard interview prompt precisely because the honest answer reveals whether you understand promotion or just understand "the build turned green." A Jenkins job that runs docker build separately for dev, staging and prod is not three identical releases — it's three different builds that happen to share a Dockerfile, and nothing stops one of them from quietly resolving a different dependency version than the one that was tested. The job you want tests one artifact once, then moves that exact artifact forward, and can move it backward just as fast when something's wrong. Get this wrong on the job and you ship an incident that passed every test — just not the test of the bytes that actually reached prod.

Explain like I'm 5

Bake one cake, then carry that same cake past every tasting station — staging, then prod. Nobody re-bakes a fresh cake at each station and assumes it tastes the same as the first one, even from the identical recipe: the oven's a little hotter today, the butter's a different brand, the sugar settled differently in the bag. A pipeline that rebuilds per environment is re-baking at every station and hoping nobody notices the difference. A pipeline that promotes is carrying the one cake that was actually tasted.

Core concept

  • A full pipeline has a shape: checkout (get the exact commit), validate (lint, type-check — fast, cheap failures first), test (the suite from Testing strategy: unit, integration, contract and e2e), build (produce one artifact — a container image, a wheel, a binary), publish (push that artifact to a registry under an immutable identity), promote (move that same artifact's reference into an environment), verify (confirm it's healthy there), and rollback (move the reference back if it isn't).
  • Build-once-promote-many is the rule that makes all of this trustworthy: the artifact is built exactly once, identified by something that can't lie about what it contains — a commit SHA plus a content digest, not a mutable tag like latest or v2. Every environment runs that same identity. If staging and prod can't show matching digests, they are not running "the same release," no matter what the Jenkinsfile says.
  • Promotion is a pointer move, not a rebuild. "Promote to prod" means: change what the prod environment's deployment config points at, to an artifact that already exists in the registry and already passed every earlier gate. It should take seconds, because nothing is being compiled, installed or linked — it already was, once.
  • Rollback is the same pointer move, backward. It re-points the environment at the previous artifact that was itself promoted and verified — not a git revert followed by a brand-new build. A rollback that triggers a fresh build defeats the entire point: you'd be deploying a fourth, never-tested artifact during an incident, under time pressure, which is the worst possible moment to introduce a new unknown.
  • Delivery is not deployment. Continuous delivery means the pipeline can always produce a release-ready, verified artifact; continuous deployment means that artifact reaches production automatically, with no human in the loop. A team can have excellent continuous delivery and still require a person to click "promote to prod" — that's a deliberate choice about risk, not a failure to "finish" CI/CD.
  • Decision rule: if two environments can't both answer "what digest is running here?" with the same value, you don't have a pipeline — you have two builds that happen to share a Dockerfile. Fix that before adding gates, strategies or anything else in this chapter; everything else assumes it.

Minimal example Promote by reference; rebuilding per environment can drift

PythonPromote by reference; rebuilding per environment can driftruns
import hashlib


def digest(content):
    return hashlib.sha256(content.encode()).hexdigest()[:12]


app_source = "orders_etl_v3"
built_once = digest(app_source)

registry = {}


def publish(name, artifact_digest):
    registry[name] = artifact_digest


publish("orders-etl", built_once)

env_pointers = {}


def promote(env, name):
    env_pointers[env] = registry[name]


for env in ("dev", "staging", "prod"):
    promote(env, "orders-etl")

same_digest_everywhere = len(set(env_pointers.values())) == 1
print("promote-by-reference, one digest for all envs:", same_digest_everywhere)


def rebuild(app_source, dependency_version):
    return digest(app_source + dependency_version)


dev_build = rebuild(app_source, "requests==2.31")
staging_build = rebuild(app_source, "requests==2.31")
prod_build = rebuild(app_source, "requests==2.32")  # resolved later; a newer pin was available

print("rebuild-per-env, staging matches prod:", staging_build == prod_build)
print("rebuild-per-env, dev matches staging:", dev_build == staging_build)
Output
promote-by-reference, one digest for all envs: True
rebuild-per-env, staging matches prod: False
rebuild-per-env, dev matches staging: True

In production A promotion pipeline where rollback never rebuilds

PythonA promotion pipeline where rollback never rebuildsruns
import hashlib
from dataclasses import dataclass, field


def digest(payload: str) -> str:
    return hashlib.sha256(payload.encode()).hexdigest()[:10]


@dataclass
class Artifact:
    name: str
    source_commit: str
    locked_deps: tuple
    tests_passed: bool

    @property
    def artifact_id(self) -> str:
        payload = self.source_commit + "|" + "|".join(self.locked_deps)
        return digest(payload)


@dataclass
class Pipeline:
    registry: dict = field(default_factory=dict)
    env_pointer: dict = field(default_factory=dict)
    promotion_log: list = field(default_factory=list)

    def build_once(self, artifact: Artifact) -> str:
        if not artifact.tests_passed:
            raise RuntimeError(f"refusing to publish {artifact.name}: tests did not pass")
        self.registry[artifact.artifact_id] = artifact
        return artifact.artifact_id

    def promote(self, env: str, artifact_id: str):
        if artifact_id not in self.registry:
            raise KeyError(f"{artifact_id} was never published; cannot promote an unbuilt artifact")
        self.env_pointer[env] = artifact_id
        self.promotion_log.append((env, artifact_id, "promote"))

    def rollback(self, env: str):
        history = [eid for e, eid, action in self.promotion_log if e == env and action == "promote"]
        if len(history) < 2:
            raise RuntimeError(f"no earlier verified artifact to roll {env} back to")
        previous = history[-2]
        self.env_pointer[env] = previous
        self.promotion_log.append((env, previous, "rollback"))
        return previous


pipeline = Pipeline()

release_1 = Artifact("orders-etl", "a1b2c3d", ("psycopg2==2.9", "boto3==1.34"), tests_passed=True)
r1_id = pipeline.build_once(release_1)
pipeline.promote("staging", r1_id)
pipeline.promote("prod", r1_id)

release_2 = Artifact("orders-etl", "f9e8d7c", ("psycopg2==2.9", "boto3==1.35"), tests_passed=True)
r2_id = pipeline.build_once(release_2)
pipeline.promote("staging", r2_id)
pipeline.promote("prod", r2_id)   # looked fine in staging, promoted to prod too

print("prod currently running release 2:", pipeline.env_pointer["prod"] == r2_id)

rolled_back_to = pipeline.rollback("prod")
print("prod rolled back to the previous verified artifact, not rebuilt:", rolled_back_to == r1_id)
print("rollback took zero new builds; registry still has only 2 artifacts:", len(pipeline.registry) == 2)

try:
    pipeline.promote("prod", digest("never-built"))
except KeyError as e:
    print(f"KeyError: {e}")
Output
prod currently running release 2: True
prod rolled back to the previous verified artifact, not rebuilt: True
rollback took zero new builds; registry still has only 2 artifacts: True
KeyError: 'dfcbc0cb11 was never published; cannot promote an unbuilt artifact'

See it happen One artifact, promoted forward, then promoted back

build oncepromotepromoterollback: re-pointcheckoutvalidatetestbuildregistry(publish)stagingprod(rolled back)
  1. Code moves through checkout, validate and test before anything is built.
  2. Build produces exactly one immutable artifact and publishes it once, by digest.
  3. That same artifact, not a rebuild, is promoted to staging.
  4. The identical artifact is promoted again, unchanged, to prod.
  5. A regression surfaces in prod a few hours later.
  6. Rollback re-points prod at the previous verified digest already sitting in the registry; nothing rebuilds.
Notice: promotion moves a reference to one built artifact forward; rollback moves that same reference backward. Neither step ever rebuilds.
Scenario A

In your world

The situation

Your Jenkinsfile has a docker build step inside each environment's own stage — Build & Deploy Staging and Build & Deploy Prod — because that's how it was written two years ago, and it's "worked" every time staging passed.

What actually breaks

Between the staging run on Friday and the prod run on Monday, a transitive dependency with no pinned version resolves to a newer release that changes a currency-rounding default. Staging's build (Friday) still has the old resolution cached; prod's build (Monday) pulls fresh and gets the new one. The nightly settlement job processes its usual 50,000 rows; 12,406 of them get rounded differently, caught by a broad except Exception that was already in the loader, logged as a warning, and written anyway. The build is green. Finance's end-of-day reconciliation shows a ₹3.2 lakh variance three days later, and nobody can say which run introduced it because "the same code" ran in both places.

What you'd do on Monday

Build the image exactly once, right after tests pass, and push it under a digest. Every later stage promotes that digest; it never calls docker build again:

TextJenkinsfile stage order: build once, promote twice
stage('Build') {
    steps {
        sh 'docker build -t orders-etl:${GIT_COMMIT} .'
        sh 'docker push orders-etl:${GIT_COMMIT}'
    }
}
stage('Promote to staging') {
    steps {
        sh 'kubectl set image deploy/orders-etl orders-etl=orders-etl:${GIT_COMMIT} -n staging'
    }
}
stage('Promote to prod') {
    when { expression { params.APPROVED_FOR_PROD } }
    steps {
        sh 'kubectl set image deploy/orders-etl orders-etl=orders-etl:${GIT_COMMIT} -n prod'
    }
}

${GIT_COMMIT} plus the registry's own digest is the artifact's identity. docker inspect --format '{{.Id}}' against staging and prod now returns the same value, which is the one fact last Monday's incident report didn't have.

Scenario B

Imagine you work at Etsy

50 promotions a day across one pipeline

The numbers

Illustrative: a team doing roughly 50 promotions a day, each one touching three environments (dev, staging, prod). Rebuilding per environment at ~10 minutes a build costs 50 × 3 × 10 = 1,500 build-minutes a day, or 25 CI-hours. Build-once-promote-many costs 50 × 10 = 500 build-minutes a day — a saving of 1,000 minutes (16.7 hours) of CI compute daily, on top of removing the byte-mismatch risk entirely.

What the Scenario-A answer does here

Scenario A's fix (one build, promoted by digest) still holds at this volume, but "just add more CI runners to absorb the 3x rebuilds" doesn't fix the actual problem — more parallel capacity makes three divergent builds finish faster, it doesn't make them identical. Throwing hardware at nondeterminism just hides it behind a shorter wait.

What flips at this scale

At 50 promotions a day, "which environment is this artifact allowed into" stops being inline branching logic inside the pipeline script and becomes configuration owned by each environment itself — a protected-environment setting, not a Jenkinsfile if. Rollback becomes "move the pointer back," fully decoupled from whatever retry logic the build job has, so an incident response never waits on CI capacity.

The trade-off you'd defend

I would defend build-once-promote-many with a thin, environment-owned promotion config over rebuilding per environment, trading a marginally bigger artifact-registry retention bill for guaranteed byte-identical releases and a lot less CI compute.

Illustrative figures chosen to make the arithmetic concrete; not Etsy-published numbers.

Gotchas

Rebuilding per environment promotes different bytes than were tested

Symptom

Staging passes; the "same" build fails or behaves differently in prod, with no code difference anyone can find in the diff.

Cause

Each environment triggered its own build, and an unpinned dependency, cache state or base-image tag resolved differently between the two builds.

Fix

Build once, publish by digest, and promote that single artifact's reference into every environment; never run a second docker build for the same release.

Rollback that triggers a rebuild

Symptom

An incident rollback takes as long as a normal deploy, and occasionally introduces a brand-new bug nobody has seen before.

Cause

"Rollback" was implemented as git revert plus a fresh build from the reverted commit, not a pointer move to a previously verified artifact.

Fix

Keep the last few promoted artifacts retained in the registry; rollback changes the environment pointer back to one of them, with no build step involved.

Calling it "CI/CD" when it's only CI

Symptom

An audit or incident review assumes production releases are automatic, and is surprised to learn a person clicks "promote" every time.

Cause

The team has continuous delivery (always a release-ready artifact) but not continuous deployment (automatic promotion to prod), and the two got used interchangeably.

Fix

Name the actual capability in runbooks and SLAs — "delivery" if a human gates production, "deployment" only if nothing does.

Tricky questions

Does having a Jenkins pipeline mean we're doing continuous deployment?
Most people say

Yes — it's a CI/CD pipeline, so every merge eventually reaches production automatically.

Actually

Only if nothing between "artifact published" and "running in prod" requires a human. A pipeline that stops for a manual promote button is continuous delivery, not continuous deployment — a real and often deliberate distinction, not a lesser version of the same thing.

If staging and prod build from the identical commit, is the artifact identical?
Most people say

Yes — same commit, same code, same artifact.

Actually

Not guaranteed. Unless dependencies are pinned and the build is reproducible, two separate builds of the same commit can resolve different transitive versions or base-image bytes. "Same commit" guarantees the same source; it does not guarantee the same artifact.

Scenario question

You're handed a pipeline where staging and prod each run their own docker build step off the same Jenkinsfile, and a bug just shipped to prod that staging never caught. What do you check first, and what do you change?

Model answer

I'd check whether the two environments actually ran byte-identical artifacts — docker inspect --format '{{.Id}}' on both — before assuming the code itself is at fault. If the digests differ, that's the bug: two separate builds, not one promoted artifact. I'd change the pipeline to build once right after tests pass, publish that single artifact by digest, and have every environment's stage promote the same reference instead of calling docker build again. The trade-off I'd name: this makes the very first environment slightly slower to go live, since nothing can start until that one build finishes, in exchange for every later environment being guaranteed to run exactly what was tested.

Interview corner

CommonWhat's the difference between continuous delivery and continuous deployment?
“Delivery means the pipeline can always produce a release-ready, tested artifact. Deployment means that artifact reaches production automatically, with no human gate. I'd never assume a team has the second just because they clearly have the first.”
TrickIf a Jenkins job rebuilds the image separately for every environment, is that still CI/CD done right?
“No — that's the one thing I'd fix before anything else. Rebuilding per environment means you're not actually testing what you ship; you're testing something that merely shares a Dockerfile with what ships. Build once, promote the same artifact by reference.”
Experience checkTell me about a time a rollback actually mattered.
“A release that rebuilt on rollback picked up a dependency bump that had landed between the original deploy and the incident, so the 'rollback' wasn't actually the previous working version — it was a fourth, untested one. After that I made sure rollback always re-points to an artifact that was already promoted and verified, never a fresh build.”
DesignHow would you design a promotion pipeline for a payments service that needs an approval gate before prod?
“I'd build once and publish by digest right after tests pass, let it promote to staging automatically, then gate the prod promotion on a required reviewer against that same digest — never a separate build triggered by the approval. The approval authorizes a specific, already-tested artifact, not a new build.”

Memory hook

Bake one cake and carry it past every tasting station — never bake a fresh one at each station and hope it tastes the same.

If you remember only one thing:

Build the artifact exactly once; every environment gets the same bytes by promoting a reference, never by rebuilding.

Self-check

  1. Why can "the same commit" still produce two different artifacts?

    Unless dependencies and the build environment are pinned, two separate builds of the same source can resolve different transitive versions or base-image bytes, producing different artifact digests.

  2. What should a rollback actually do, mechanically?

    Re-point the environment's pointer to a previously promoted, verified artifact already sitting in the registry — not trigger a new build from a reverted commit.

  3. What's the practical difference between continuous delivery and continuous deployment?

    Delivery means a release-ready artifact always exists; deployment means that artifact reaches production with no human gate. A team can have one without the other.

Prove it

Write a short script that models a two-environment promotion pipeline: build one artifact with a content-based identifier, publish it to a dict-based "registry," promote it to two environments by writing the identifier into each environment's "pointer," then roll one environment back to a previous identifier without calling the build function again. Paste the pointer values before and after the rollback, and the one line proving the build function was called exactly once.

ce-08-02Card 2 of 6

Compare declarative Jenkins and GitHub Actions side by side

MediumP3 · Role-dependentTalk about it12 min
jenkinsgithub-actionspipeline-syntaxrunnersmigration

Why you care

Most product-company backend teams run GitHub Actions, GitLab CI or something Actions-shaped; "I only know Jenkins" is a real gap on a resume built entirely on Jenkins button-clicking. Interviewers ask "how would this look in Actions?" not to test trivia, but because translating a mental model — not just syntax — is the actual skill: knowing that agent becomes runs-on, that credentials work differently, and that one Jenkins directive (post) has no clean Actions equivalent at all. Porting a pipeline by syntax lookup alone is exactly how a cleanup step quietly disappears.

Explain like I'm 5

Two companies can run the same org chart with different job titles. "VP of Engineering" in one company does roughly what "Director of Platform" does in the other — but not every title lines up, and some responsibilities (say, "the person who turns off the lights at the end of the day") simply don't exist as a named role everywhere. Translating a Jenkinsfile to a GitHub Actions workflow is exactly that: most roles map cleanly, a few don't map at all, and assuming they all do is how something quietly stops happening.

Core concept

  • The core structural mapping, confirmed against GitHub's own migration guide: Jenkins pipeline → Actions workflow; agent → runs-on (or container); stages → jobs; stage → a named job; steps → steps; environment → env; when → if; triggers → on; Jenkins cron syntax → on.schedule; an upstream-job dependency → needs; Jenkins parallel/matrix axes → strategy.matrix.
  • post has no direct equivalent. Jenkins' post { always/success/failure/cleanup } block is a named feature of the pipeline DSL; Actions has nothing with the same name. The pattern it's modeled as is explicit if: always() or if: failure() conditions on individual steps, or a separate job gated with needs and if: failure() on the upstream job.
  • Credentials are scoped differently. Jenkins binds a stored credential into an environment variable with credentials('id'), masked by Jenkins' own credential store. Actions exposes secrets.<NAME> in workflow syntax, and secrets can additionally be scoped to a named Environment so only jobs that explicitly target that environment can read them — a scoping mechanism Jenkins credentials don't have in the same form.
  • Manual approval looks different too. Jenkins' input step pauses a running pipeline for a human click. Actions achieves the same outcome through required reviewers configured on a deployment Environment: a job referencing that environment pauses, waiting for approval, before it can run or read the environment's secrets.
  • Runner ownership and statefulness differ. Jenkins agent commonly points at a long-lived, team-owned node that persists between jobs. Actions' hosted runners are a fresh virtual machine per job by default; self-hosted runners exist too, but inherit the same statefulness Jenkins agents have — which matters for Integrate often without granting CI excessive trust's trust boundary.
  • Decision rule: translate by directive, not by vibe — and explicitly check for anything with no listed equivalent (post is the big one) before trusting a ported pipeline to behave the same way the old one did.

Minimal example Translating Jenkins directives to Actions keys — and what has no match

PythonTranslating Jenkins directives to Actions keys, and what has no matchruns
JENKINS_TO_ACTIONS = {
    "pipeline": "workflow",
    "agent": "runs-on",
    "stages": "jobs",
    "stage": "job",
    "steps": "steps",
    "environment": "env",
    "when": "if",
    "triggers": "on",
    "post": None,  # no 1:1 directive; model as if: always()/failure() step(s)
}

jenkinsfile_directives = ["pipeline", "agent", "stages", "stage", "steps", "environment", "when", "post"]

for directive in jenkinsfile_directives:
    actions_equivalent = JENKINS_TO_ACTIONS[directive]
    if actions_equivalent is None:
        print(f"{directive:12s} -> NO DIRECT EQUIVALENT: model as if: always()/failure() step(s)")
    else:
        print(f"{directive:12s} -> {actions_equivalent}")
Output
pipeline     -> workflow
agent        -> runs-on
stages       -> jobs
stage        -> job
steps        -> steps
environment  -> env
when         -> if
post         -> NO DIRECT EQUIVALENT: model as if: always()/failure() step(s)

In production Porting a Jenkinsfile's stages, with the gaps flagged

PythonPorting a Jenkinsfile's stages, with the gaps flaggedruns
from dataclasses import dataclass, field


@dataclass
class JenkinsStage:
    name: str
    steps: list
    post_always: list = field(default_factory=list)


@dataclass
class JenkinsPipeline:
    stages: list


def port_to_actions(pipeline: JenkinsPipeline) -> dict:
    """Translate a toy Jenkinsfile model into a toy Actions workflow model,
    and flag anything with no direct equivalent."""
    jobs = {}
    warnings = []
    for stage in pipeline.stages:
        job_id = stage.name.lower().replace(" ", "-")
        jobs[job_id] = {"runs-on": "ubuntu-latest", "steps": list(stage.steps)}
        if stage.post_always:
            warnings.append(
                f"stage '{stage.name}' has post{{always}} steps {stage.post_always}; "
                "'post' has no direct Actions directive, add them as steps with if: always()"
            )
    return {"jobs": jobs, "warnings": warnings}


nightly_etl = JenkinsPipeline(stages=[
    JenkinsStage("Checkout", ["git checkout"]),
    JenkinsStage("Build", ["docker build -t orders-etl ."]),
    JenkinsStage("Load", ["python load.py"], post_always=["cleanup_tmp_s3_prefix()"]),
])

ported = port_to_actions(nightly_etl)

for job_id, job in ported["jobs"].items():
    print(f"job {job_id}: runs-on={job['runs-on']}, steps={job['steps']}")

print("warnings:")
for w in ported["warnings"]:
    print(" -", w)
Output
job checkout: runs-on=ubuntu-latest, steps=['git checkout']
job build: runs-on=ubuntu-latest, steps=['docker build -t orders-etl .']
job load: runs-on=ubuntu-latest, steps=['python load.py']
warnings:
 - stage 'Load' has post{always} steps ['cleanup_tmp_s3_prefix()']; 'post' has no direct Actions directive, add them as steps with if: always()
Scenario A

In your world

The situation

Your team is told to "port this one Jenkinsfile to GitHub Actions" for a nightly ETL job. You translate agent any to runs-on: ubuntu-latest, stages/stage to jobs, and when { branch 'main' } to if: github.ref == 'refs/heads/main' — a clean, confident mapping for everything the migration guide actually lists.

What actually breaks

The Jenkinsfile's post { always { cleanupWorkspace(); cleanup_tmp_s3_prefix() } } block has nothing to map to, so it's dropped during the port rather than deliberately reimplemented. The job itself still goes green every night. After three weeks (21 nightly runs), 21 orphaned tmp/etl-run-* prefixes sit in the S3 bucket, one per run, until the bucket's lifecycle-monitoring alert fires on an object-count threshold nobody connected to "we migrated this job last month."

What you'd do on Monday

Add the cleanup back explicitly, since Actions has no single directive for it — either as a step with if: always() in the same job, or as a separate job that depends on the others and runs regardless of their outcome:

YAML.github/workflows/nightly-etl.yml: explicit cleanup where `post` used to be
jobs:
  load:
    runs-on: ubuntu-latest
    steps:
      - run: python load.py
  cleanup:
    needs: load
    if: always()
    runs-on: ubuntu-latest
    steps:
      - run: python cleanup_tmp_s3_prefix.py

Nothing about the actual ETL logic changes — the fix is remembering that one Jenkins directive had no Actions equivalent, and writing the equivalent behaviour back in by hand.

Scenario B

Imagine you work at LinkedIn

400 repos migrating off self-hosted Jenkins

The numbers

Illustrative: 400 repositories migrating their Jenkinsfiles to Actions. Even a well-intentioned 2% per-repo miss rate on translating directives with no direct equivalent (post blocks, input approvals) means roughly 400 × 2% ≈ 8 repos silently lose their cleanup or approval behaviour during the port, with the migrated workflow still reporting green.

What the Scenario-A answer does here

Scenario A's fix (explicit if: always() cleanup) still works per repository, but "have each of the 400 teams hand-translate their own Jenkinsfile" doesn't scale — the error compounds once per repo, and nobody notices a silently dropped cleanup step until its absence causes a cost or capacity problem weeks later.

What flips at this scale

At fleet scale, the fix moves from "translate carefully" to "translate once, centrally." A shared, tested reusable workflow (or composite action) encodes the cleanup-after-failure pattern a single time; every repo's workflow calls it instead of 400 teams independently reconstructing the same post { always } behaviour from a migration-guide table.

The trade-off you'd defend

I would defend one centrally owned, reusable workflow for cross-cutting patterns like cleanup and approval over 400 independent hand ports, trading a little team autonomy for consistent behaviour across the fleet.

Illustrative figures for the arithmetic; not LinkedIn-published numbers.

Jenkins (declarative) vs GitHub Actions

AxisJenkins (declarative)GitHub Actions
Pipeline definitionJenkinsfile, interpreted by a central Jenkins controllerworkflow YAML in .github/workflows/, dispatched by GitHub itself
Where it runsagent picks a label, often a long-lived, team-owned noderuns-on picks a hosted runner (fresh VM per job) or a self-hosted one
StatefulnessAgents are commonly reused across many jobs; leftover files can persistGitHub-hosted runners are ephemeral and torn down after each job
Credentialsenvironment { X = credentials('id') }, masked by Jenkins' credential storesecrets.X, optionally scoped to a named Environment
Manual approvalinput step pauses the pipeline for a humanrequired reviewers on a deployment Environment pause the job before secrets/run
Cleanup / notifypost { always/success/failure/cleanup }no single directive — model with if: always()/if: failure() steps or a dependent job

Gotchas

A dropped post block during a port

Symptom

A migrated workflow reports green every run, but something that used to happen after every build (cleanup, a notification) quietly stops happening.

Cause

Jenkins' post directive has no 1:1 Actions equivalent, and a syntax-by-syntax port has nothing to translate it into unless someone adds the behaviour back explicitly.

Fix

Reimplement the behaviour as if: always()/if: failure() steps, or a separate dependent job, and verify it actually runs by checking its own log output, not just the overall workflow status.

Assuming self-hosted-agent statefulness carries over to hosted runners

Symptom

A job that depended on a file left over from a previous Jenkins agent run behaves differently or fails outright after porting to Actions' default hosted runners.

Cause

GitHub-hosted runners are a fresh VM per job; nothing from a previous run persists unless it's explicitly cached or restored.

Fix

Make the job's dependencies explicit (cache steps, artifact downloads) instead of relying on leftover state from a prior run.

Treating Jenkins credential masking and Actions secret masking as identical

Symptom

A value that was reliably redacted from Jenkins console logs shows up in plaintext in an Actions log.

Cause

Only values registered through secrets.<NAME> are automatically masked in Actions; a value copied into a plain env var from some other source isn't covered by that masking.

Fix

Pull sensitive values through secrets.<NAME> directly in the step that needs them, rather than relaying them through an intermediate plain environment variable.

Tricky questions

Is porting a Jenkinsfile to Actions basically a 1:1 syntax swap?
Most people say

Yes — map each directive to its Actions equivalent and the behaviour carries over.

Actually

Close for stages/steps/env/when, but there's no direct equivalent for post, and runner statefulness differs by default (reused Jenkins agents vs ephemeral hosted runners). A port that only maps syntax can silently drop cleanup logic or assume state that no longer exists.

Does an environment variable in an Actions workflow get masked in logs the same way a Jenkins credentials() binding does?
Most people say

Yes — any sensitive-looking env var is automatically redacted either way.

Actually

No — Actions only masks values pulled through secrets.<NAME>. A value assigned into a plain env var from somewhere else isn't automatically covered, even if the equivalent Jenkins binding would have masked it.

Scenario question

You're handed a Jenkinsfile with a post { always { notifySlack(); archiveLogs() } } block and told to "port it to Actions, same behaviour." What's your approach?

Model answer

I'd map the structural pieces first — pipeline/stages/steps/when have clean Actions equivalents — then stop at post specifically, since I know it has no direct match. I'd reimplement notifySlack() and archiveLogs() as explicit steps with if: always() in the same job (or a dependent job with if: failure() if they should only fire on failure), and then actually trigger a failing run once to confirm both still execute, rather than trusting the YAML to behave the same way the old block did. The trade-off I'd name: this takes longer than a pure syntax port, but a port that silently drops post behaviour looks identical to a correct one until the thing it was doing is missed.

Interview corner

CommonHow would you map a Jenkins declarative pipeline to a GitHub Actions workflow?
“Pipeline becomes workflow, stages become jobs, steps stay steps, agent becomes runs-on, when becomes if, and environment becomes env. The one gap I always flag is post — Actions has no single directive for it, so I reimplement it with if: always() or if: failure() steps.”
TrickDoes GitHub Actions mask any environment variable that looks like a secret?
“I'd say no — only values pulled through the secrets context get automatic masking. If I copy a value into a plain env var from somewhere else, it isn't covered just because it holds something sensitive.”
Experience checkHave you migrated a Jenkins pipeline to another CI platform?
“Yes — the structural mapping was the easy part. The thing that actually bit us was the post block having no direct equivalent, so a cleanup step silently stopped running until we noticed orphaned temp files piling up weeks later.”
DesignHow would you port 400 repositories off Jenkins without 400 teams each making the same mistake?
“I'd build one shared, tested reusable workflow for the cross-cutting patterns — cleanup, notifications, approval gates — and have every repo's workflow call that, instead of trusting 400 independent hand translations to all remember the directives with no 1:1 equivalent.”

Memory hook

Different org chart, same org — most titles map cleanly, but check for the one role nobody renamed because it simply doesn't exist on the other chart.

If you remember only one thing:

Translate by directive mapping, but explicitly check what has no equivalent (post is the one that bites) before trusting a ported pipeline.

Self-check

  1. What's the Actions equivalent of a Jenkins agent directive?

    runs-on (or container), which selects where the job executes.

  2. Why doesn't a straight syntax port of post { always {...} } work in Actions?

    There's no directive with the same meaning; it has to be modeled explicitly as if: always()/if: failure() steps or a separate dependent job, or the behaviour silently disappears.

  3. What's the practical difference in statefulness between a typical Jenkins agent and a GitHub-hosted runner?

    A Jenkins agent is often a long-lived node reused across many jobs, so leftover files can persist; a GitHub-hosted runner is a fresh VM per job by default, with nothing carried over unless explicitly cached.

Prove it

Take any small Jenkinsfile (yours or a sample one) with at least one post { always {...} } block. Write the GitHub Actions workflow YAML you believe is equivalent, including an explicit if: always() step or dependent job standing in for the post block. Then write one sentence identifying every directive in the original Jenkinsfile that had no 1:1 entry in the mapping table, and how you modeled each one.

ce-08-03Card 3 of 6

Turn test and security results into quality gates

MediumP2 · Very likelyKnow cold12 min
quality-gatestestingsecurityfail-closedevidence

Why you care

A pipeline that marks a build green because the security scanner timed out and produced nothing is worse than one with no scanner at all — the first one looks safe. Interviewers ask about quality gates because "what happens when a required check doesn't run" separates people who've only ever seen a gate pass from people who've had to explain to a security team why thirty releases shipped with zero scan evidence and a green dashboard the whole time. The fix isn't a smarter scanner; it's a gate that treats "no evidence" as a failure, not a pass.

Explain like I'm 5

Think of airport security: each gate needs a stamped boarding pass as evidence you cleared the check before it. A traveler with no stamp doesn't get waved through just because the scanner behind them happened to be broken that hour — no stamp means held, every time, regardless of why the stamp is missing. A quality gate that treats a missing report as a pass is the equivalent of waving through anyone whose scanner glitched.

Core concept

  • Order gates cheap-to-expensive. Lint and type-checks run in seconds; unit tests from Testing strategy: unit, integration, contract and e2e next; integration tests after that; the slowest and most expensive checks — a security or supply-chain scan — last. Failing fast on a cheap check saves the pipeline from ever paying for an expensive one that was going to fail anyway.
  • Fail closed on missing evidence. A gate's decision has exactly three honest outcomes: pass (the report exists and is clean), fail (the report exists and isn't clean), or fail (no report exists at all). "No report" must map to the same outcome as "report says no" — never to "pass." A scanner timeout, a crashed job, or a misconfigured path are all failures of the same kind, not special cases that deserve a pass.
  • Exceptions are signed off, not silent. Sometimes a gate genuinely needs to be bypassed — a known false positive, a vendor fix not yet available. That's a named, dated waiver: who approved it, which gate, and when it expires. A waiver with no expiry is a permanent hole with a signature on it.
  • Evidence is tied to the artifact, not the run. A test report or scan report is only meaningful if it's stored against the specific artifact identity from CI/CD and promotion: immutable artifacts to rollback — its commit/digest — so a gate decision made today can be audited six months later against the exact bytes it was about.
  • Decision rule: order gates by cost, fail closed on any missing required report, and treat a waiver as an exception with an owner and an expiry date — never as a quiet workaround nobody has to re-justify.

Minimal example A missing report is not a clean report

PythonA missing report is not a clean reportruns
def naive_gate(report_path, reports):
    """Wrong: treats a missing report as if nothing failed."""
    result = reports.get(report_path)
    if result is None:
        return "PASS (no report found, assumed clean)"
    return "PASS" if result["vulnerabilities"] == 0 else "FAIL"


def fail_closed_gate(report_path, reports):
    """Right: a missing report is itself a failure."""
    result = reports.get(report_path)
    if result is None:
        return "FAIL (required report missing)"
    return "PASS" if result["vulnerabilities"] == 0 else "FAIL"


reports = {
    "unit-tests.json": {"vulnerabilities": 0},
    # "security-scan.json" was never produced this run; the scanner timed out
}

print("naive gate on missing security report:", naive_gate("security-scan.json", reports))
print("fail-closed gate on missing security report:", fail_closed_gate("security-scan.json", reports))
print("fail-closed gate on present, clean report:", fail_closed_gate("unit-tests.json", reports))
Output
naive gate on missing security report: PASS (no report found, assumed clean)
fail-closed gate on missing security report: FAIL (required report missing)
fail-closed gate on present, clean report: PASS

In production Fast-first gates with fail-closed and a dated waiver

PythonFast-first gates with fail-closed and a dated waiverruns
from dataclasses import dataclass
from datetime import date


@dataclass
class GateResult:
    name: str
    report_present: bool
    passed: bool


@dataclass
class Waiver:
    gate_name: str
    signed_off_by: str
    expires: date


def run_gate(name, results_by_name):
    result = results_by_name.get(name)
    if result is None:
        return GateResult(name, report_present=False, passed=False)
    return GateResult(name, report_present=True, passed=result)


def evaluate_pipeline(gate_order, results_by_name, waivers, today):
    active_waivers = {w.gate_name: w for w in waivers if w.expires >= today}
    outcomes = []
    for name in gate_order:
        gate = run_gate(name, results_by_name)
        if not gate.passed and name in active_waivers:
            w = active_waivers[name]
            outcomes.append((name, "WAIVED", f"signed off by {w.signed_off_by}, expires {w.expires}"))
            continue
        outcomes.append((name, "PASS" if gate.passed else "FAIL", "report missing" if not gate.report_present else ""))
        if not gate.passed:
            break  # fail fast: stop at the first unwaived failure
    return outcomes


gate_order = ["lint", "unit-tests", "integration-tests", "security-scan"]
results_by_name = {
    "lint": True,
    "unit-tests": True,
    "integration-tests": True,
    # "security-scan" missing: scanner timed out against the internal mirror
}

print("no waiver:")
for row in evaluate_pipeline(gate_order, results_by_name, [], today=date(2026, 9, 1)):
    print(" ", row)

expired_waiver = [Waiver("security-scan", "security-team@co", date(2026, 1, 1))]
print("expired waiver, same outcome as no waiver:")
for row in evaluate_pipeline(gate_order, results_by_name, expired_waiver, today=date(2026, 9, 1)):
    print(" ", row)

active_waiver = [Waiver("security-scan", "security-team@co", date(2026, 12, 31))]
print("active, named, dated waiver:")
for row in evaluate_pipeline(gate_order, results_by_name, active_waiver, today=date(2026, 9, 1)):
    print(" ", row)
Output
no waiver:
  ('lint', 'PASS', '')
  ('unit-tests', 'PASS', '')
  ('integration-tests', 'PASS', '')
  ('security-scan', 'FAIL', 'report missing')
expired waiver, same outcome as no waiver:
  ('lint', 'PASS', '')
  ('unit-tests', 'PASS', '')
  ('integration-tests', 'PASS', '')
  ('security-scan', 'FAIL', 'report missing')
active, named, dated waiver:
  ('lint', 'PASS', '')
  ('unit-tests', 'PASS', '')
  ('integration-tests', 'PASS', '')
  ('security-scan', 'WAIVED', 'signed off by security-team@co, expires 2026-12-31')

See it happen A missing report that looks the same as a clean one, until the gate checks

lintunit testsintegrationsecurity scangateFAIL (closed)
  1. Lint and unit tests run first, since they are fast and cheap to fail on.
  2. Integration tests run next, slower, but still ahead of the expensive security scan.
  3. The security scanner times out against the internal mirror and produces no report at all.
  4. A naive gate treats a missing report as nothing to worry about and waves the build through.
  5. A fail-closed gate treats that same missing report as a failure instead, and blocks promotion.
Notice: a missing report and a clean report look identical to a gate that doesn't check which one it got — fail closed is what tells them apart.
Scenario A

In your world

The situation

Your Jenkins pipeline's security-scan stage occasionally times out reaching an internal Artifactory mirror. The stage is wrapped in a post { always { ... } } block that was written to keep the pipeline from "getting stuck," and it quietly lets the overall build report SUCCESS even when the scan step itself never produced a report.

What actually breaks

Over the last month, 14 of 60 nightly builds shipped with zero vulnerability-scan evidence — all 14 marked green, indistinguishable in the Jenkins dashboard from the 46 that scanned cleanly. Nobody notices until a security audit asks for scan evidence on a specific release and the pipeline has none to show for it.

What you'd do on Monday

Check for the report file explicitly, and fail the stage — not just log a warning — when it's missing:

Pythonsecurity_gate.py: fail closed when the report file is missing
import sys
from pathlib import Path

report_path = Path("security-scan.json")
if not report_path.exists():
    print("FAIL: required security-scan.json report is missing", file=sys.stderr)
    sys.exit(1)

Wiring this as its own pipeline step, checked before the build is allowed to promote, turns "the scanner glitched" into a visible, actionable failure instead of an invisible gap in the audit trail.

Scenario B

Imagine you work at Myntra

60 service repos scanning nightly

The numbers

Illustrative: 60 microservice repos each running a nightly security gate. If 5% of scans silently no-op due to a flaky connection to the scanner, that's 3 repos a night shipping with zero scan evidence; over a 30-night month that compounds to roughly 90 ungated releases across the fleet, each indistinguishable from a clean one on the dashboard.

What the Scenario-A answer does here

Scenario A's fix (fail closed on a missing report) still works per repo, but "just retry the scan until it produces a report" at fleet scale only moves the flakiness into a longer queue — it doesn't define what happens if a retry never succeeds before a release deadline, and a long retry loop blocks every other repo waiting behind it on shared scanner capacity.

What flips at this scale

At this scale, evidence storage separates from the pass/fail decision entirely: results are retained in a store keyed by artifact digest, retries and backfills happen against that store independently of any single pipeline run, and the gate's job becomes "does a non-expired, matching report exist for this exact artifact" — plus an explicit, owned, time-boxed waiver path for the genuine cases where a scan can't finish in time.

The trade-off you'd defend

I would defend decoupled evidence storage with named, time-boxed waivers over either blocking every release on a flaky scanner or silently treating a missing result as a pass, trading some process overhead for an audit trail that survives a review six months later.

Illustrative figures for the arithmetic; not Myntra-published numbers.

Gotchas

Missing report treated as a pass

Symptom

A dashboard shows a run as fully green, but no security-scan artifact exists anywhere for that run when someone goes looking for it later.

Cause

The gate's logic checked "did the report say it was clean," which returns a falsy "no result found" the same way a genuinely clean report would if the check isn't explicit.

Fix

Check for the report's existence as its own condition, and fail closed when it's absent, regardless of what any downstream check says.

Expensive gates running before cheap ones

Symptom

A trivial lint error is only discovered after a 20-minute integration-test and scan run has already completed.

Cause

Gates were added to the pipeline in whatever order someone wrote them, not ordered by cost.

Fix

Reorder so lint and unit tests run first; let the pipeline fail on a one-line typo in seconds, not twenty minutes later.

A "temporary" waiver with no expiry

Symptom

A signed-off exception from eight months ago is still silently bypassing a gate, and nobody remembers why it was ever approved.

Cause

The waiver was granted without a stated expiry date, so it never came back up for re-justification.

Fix

Require every waiver to carry an expiry; treat an expired waiver exactly like no waiver at all.

Tricky questions

If the security scanner errors out, should the pipeline just skip that stage and keep going?
Most people say

Yes — skip it gracefully so a tooling outage doesn't block delivery.

Actually

No. Skipping converts a tooling outage into an invisible policy bypass. Fail closed instead: a missing result is a failure, and if the team genuinely needs to proceed anyway, that's a named, dated waiver — not a silent skip.

Does running the security scan at all mean the release has real evidence behind it?
Most people say

Yes — if the scan step is in the pipeline, the release is covered.

Actually

Only if the scan actually produced a report for that specific run and the gate checked for it. A scan step that's present in the YAML but silently times out every third run gives the appearance of coverage without the evidence.

Scenario question

You're handed a report that 14 of the last 60 nightly builds shipped with no security-scan evidence at all, all marked green. What do you fix first, and how do you make sure it's actually fixed?

Model answer

I'd fix the gate's logic before touching the scanner itself: a missing report has to fail the stage, full stop, regardless of why it's missing. Then I'd verify the fix by deliberately breaking the scanner's connectivity in a test run and confirming the pipeline now fails instead of going green — not just reading the code and assuming it's correct. The trade-off I'd name: fail-closed will occasionally block a release over a transient scanner blip rather than a real vulnerability, and that's the right trade against the alternative, which is 14 releases a month with no evidence at all.

Interview corner

CommonWhat does it mean for a quality gate to "fail closed"?
“I'd say it means a missing or unavailable result is treated as a failure, the same as a result that explicitly failed. I never let the gate default to pass just because it couldn't get an answer.”
TrickIf a scanner is flaky and keeps blocking releases, is the right fix to make the gate more lenient?
“No, I'd fix the flakiness or add a named, expiring waiver — not make the gate pass on missing evidence. Leniency there just turns a tooling problem into a silent coverage gap.”
Experience checkHave you found a gap where a required check wasn't actually running?
“Yes — a security scan step that silently no-opped on timeout, and the overall pipeline still reported success. I added an explicit check for the report file's existence, which turned that into a visible failure instead of an invisible gap.”
DesignHow would you order the gates in a new pipeline from scratch?
“I'd order them cheapest and fastest first — lint, type-checks — then unit tests, then integration tests, then the most expensive checks like a full security or supply-chain scan last. I want to fail fast on the cheap ones so the pipeline never pays for an expensive check that was going to fail anyway.”

Memory hook

A missing boarding-pass stamp doesn't get you through security just because the scanner behind you broke — no stamp means held, every time.

If you remember only one thing:

A missing required report is a failure, not a pass — fail closed, and let any real exception be a named, dated waiver instead of a silent skip.

Self-check

  1. Why is "fail closed on a missing report" different from "fail if the report says there's a problem"?

    A gate that only checks the report's contents treats "no report" the same as "nothing to report," which is a silent pass; failing closed means absence of evidence is itself treated as a failure.

  2. What two things must a valid waiver record?

    Who signed off on it and when it expires — an unexpiring waiver with no named approver is effectively a permanent, unaccountable bypass.

  3. Why does gate order matter even if every gate eventually runs?

    Cheap, fast checks (lint, unit tests) catch trivial errors in seconds; running them last wastes the time already spent on slower, more expensive checks that didn't need to run at all.

Prove it

Write a small gate-evaluation function that takes an ordered list of gate names and a dict of results keyed by gate name. Run it once with a result missing for one gate and confirm the pipeline reports FAIL for that gate specifically (not PASS), then add a dated waiver for that same gate and confirm the outcome changes to WAIVED with the approver's name shown. Paste both outcome lists.

ce-08-04Card 4 of 6

Put database migrations in a safe pipeline

HardP2 · Very likelyTalk about it14 min
database-migrationsexpand-contractpipelinelocksdeployment-safety

Why you care

A deploy that goes green and still corrupts data is almost always a schema change, not application logic — rollback can undo code, but it cannot un-delete a column's data or un-apply a backfill that already overwrote values. Interviewers probe migration placement in a pipeline because it's the fastest way to tell whether someone has actually caused — or cleaned up after — a locked table in production, versus someone who's only run alembic upgrade head on a laptop with ten rows in it.

Explain like I'm 5

Renovating a bridge while traffic still crosses it means you build the new lanes beside the old ones first — expand — and let cars use whichever lane they're already used to. Only once every car has moved onto the new lanes do you close and remove the old ones — contract. Pour both halves of the bridge on the same night, in one step, and some car is going to be driving on concrete that isn't cured yet.

Core concept

  • Compatibility before promotion. A migration must leave the schema in a state both the old app version (still running somewhere) and the new app version (about to be promoted) can read and write safely. The migration runs, is verified, and only then does app promotion from CI/CD and promotion: immutable artifacts to rollback begin — never the other way around, and never simultaneously.
  • Serialized execution. Exactly one migration runs pipeline-wide at a time. Two pipelines racing the same table — a hotfix and a scheduled release both touching the same migration lock — must be rejected, not interleaved; the second one waits or fails loudly, it never runs halfway alongside the first.
  • Lock and time budgets. A schema change that takes a long-held table lock must have a stated time budget before it's allowed to run against production traffic. If it can't fit the budget, it needs an online-migration pattern instead of a blind retry at a quieter hour.
  • Backfill is visible and resumable. Filling in a new column for millions of existing rows is a real, long-running job — not a side effect of the ALTER TABLE statement. It needs observable progress and the ability to resume after a partial failure, not a silent background UPDATE nobody can check on.
  • Forward repair, not rollback, fixes a bad migration. If a migration fails partway through, the fix is a new forward migration that completes or corrects the change — rolling the schema itself backward can destroy rows already written under the new shape. App code can roll back; a destructive schema change usually can't.
  • Expand, backfill, contract is the whole pattern in three words — the full mechanics belong to Safe database migrations: expand-contract; this card only owns where that pattern sits inside the pipeline, as a distinct, serialized, pre-promotion step.
  • Decision rule: never let a migration and an app promotion race each other, and never run the contract step until the backfill it depends on is independently confirmed complete.

Under the hood

PostgreSQL's own documentation is specific about one case that trips people up: adding a column with a constant DEFAULT only writes that default into the table's metadata — it does not rewrite existing rows, "making the ALTER TABLE very fast even on large tables." But that fast path has a real exception: adding a column with a volatile default (something like clock_timestamp()), a generated column, an identity column, or certain constrained domain types forces a full rewrite of the table and all its indexes — under the same ACCESS EXCLUSIVE lock the statement always acquires. The statement looks identical in your migration file either way; whether it's instant or minutes-long under a lock depends entirely on which of those two cases it falls into, which is exactly why "it worked fine in staging on a small table" tells you nothing about lock duration on a production-sized one.

Minimal example A constant default is metadata-only; a volatile one rewrites every row

PythonA constant default is metadata-only; a volatile one rewrites every rowruns
def alter_table_lock_seconds(row_count, has_volatile_default_or_rewrite):
    """Model: a constant-default ADD COLUMN only touches table metadata
    (near-instant); a volatile default/constraint rewrites every row
    under the same ACCESS EXCLUSIVE lock, so lock time scales with rows."""
    if has_volatile_default_or_rewrite:
        rewritten_rows_per_second = 50_000
        return round(row_count / rewritten_rows_per_second, 2)
    return 0.01  # metadata-only change: effectively constant


rows_small = 50_000
rows_large = 25_000_000

print("constant default, 50k rows   -> lock seconds:", alter_table_lock_seconds(rows_small, False))
print("constant default, 25M rows   -> lock seconds:", alter_table_lock_seconds(rows_large, False))
print("volatile default, 25M rows   -> lock seconds:", alter_table_lock_seconds(rows_large, True))
print("rewrite case locks longer than metadata-only case:",
      alter_table_lock_seconds(rows_large, True) > alter_table_lock_seconds(rows_large, False))
Output
constant default, 50k rows   -> lock seconds: 0.01
constant default, 25M rows   -> lock seconds: 0.01
volatile default, 25M rows   -> lock seconds: 500.0
rewrite case locks longer than metadata-only case: True

In production Expand, backfill, contract — and a migration that refuses to run twice at once

sqlite3 stands in for Postgres below; the lock-contention and expand/backfill/contract logic are the point, not the specific engine.

PythonExpand, backfill, contract, and a migration that refuses to run twice at onceruns
import sqlite3  # stands in for Postgres in this example


def run_serialized_migration(conn, migration_sql, lock_table="migration_lock"):
    """Only one migration runs at a time pipeline-wide; a second caller
    trying to run while one is in flight is rejected, not interleaved."""
    cur = conn.cursor()
    cur.execute(f"CREATE TABLE IF NOT EXISTS {lock_table} (locked INTEGER)")
    cur.execute(f"SELECT locked FROM {lock_table}")
    row = cur.fetchone()
    if row and row[0] == 1:
        raise RuntimeError("migration already in progress; refusing to run concurrently")
    cur.execute(f"DELETE FROM {lock_table}")
    cur.execute(f"INSERT INTO {lock_table} VALUES (1)")
    conn.commit()
    try:
        cur.executescript(migration_sql)
        conn.commit()
    finally:
        cur.execute(f"UPDATE {lock_table} SET locked = 0")
        conn.commit()


conn = sqlite3.connect(":memory:")
conn.execute("CREATE TABLE settlements (id INTEGER PRIMARY KEY, amount REAL)")
conn.execute("INSERT INTO settlements (amount) VALUES (100.0), (250.5)")
conn.commit()

# Step 1 (expand): add the new column nullable; no NOT NULL yet.
# Old app code (unaware of settled_at) and new app code can both run safely.
run_serialized_migration(conn, "ALTER TABLE settlements ADD COLUMN settled_at TEXT")


def old_app_write(conn, amount):
    conn.execute("INSERT INTO settlements (amount) VALUES (?)", (amount,))
    conn.commit()


def new_app_write(conn, amount, settled_at):
    conn.execute("INSERT INTO settlements (amount, settled_at) VALUES (?, ?)", (amount, settled_at))
    conn.commit()


old_app_write(conn, 75.0)                 # old app, during the dual-version overlap
new_app_write(conn, 310.0, "2026-09-30")  # new app, during the same overlap

rows = conn.execute("SELECT amount, settled_at FROM settlements ORDER BY id").fetchall()
print("rows during expand (nullable column, both app versions write fine):")
for r in rows:
    print(" ", r)

# Backfill: fill settled_at for the rows that predate the new column.
conn.execute("UPDATE settlements SET settled_at = 'backfilled' WHERE settled_at IS NULL")
conn.commit()

backfilled_count = conn.execute(
    "SELECT COUNT(*) FROM settlements WHERE settled_at = 'backfilled'"
).fetchone()[0]
print("rows backfilled before contract step:", backfilled_count)

remaining_nulls = conn.execute(
    "SELECT COUNT(*) FROM settlements WHERE settled_at IS NULL"
).fetchone()[0]
print("safe to add the NOT NULL constraint now (0 remaining NULLs):", remaining_nulls == 0)

# A second pipeline run tries to start a migration while this one is "in flight".
cur = conn.cursor()
cur.execute("UPDATE migration_lock SET locked = 1")
conn.commit()
try:
    run_serialized_migration(conn, "ALTER TABLE settlements ADD COLUMN region TEXT")
except RuntimeError as e:
    print(f"RuntimeError: {e}")
Output
rows during expand (nullable column, both app versions write fine):
  (100.0, None)
  (250.5, None)
  (75.0, None)
  (310.0, '2026-09-30')
rows backfilled before contract step: 3
safe to add the NOT NULL constraint now (0 remaining NULLs): True
RuntimeError: migration already in progress; refusing to run concurrently

See it happen Contract is the one step a rollback can't safely undo

expand(add nullable col)promote app v2backfill(visible, resumable)contract(safe)
  1. Expand: add the new column nullable; both the old and new app versions can read and write safely.
  2. Promote: the new app version ships while an old instance may still be running elsewhere.
  3. Backfill: existing rows get the new column filled in, visibly and resumably, not silently.
  4. A naive pipeline runs the contract step here, before the backfill is confirmed complete.
  5. Only once backfill is verified complete does contract run: add NOT NULL, then drop the old column.
Notice: contract is the one step that can lose data if run too early or rolled back — never run it until expand and backfill are both independently verified complete.
Scenario A

In your world

The situation

Your Jenkins pipeline runs alembic upgrade head as a build step in the same stage that deploys the new app version, with no separate lock around it. A hotfix pipeline and the regular nightly release both happen to trigger within the same few minutes.

What actually breaks

Both pipelines reach the migration step close together. One adds a NOT NULL column with no default directly, the other is mid-deploy with new app code that already expects the column to exist, while old rows haven't been backfilled yet. 1,842 of that night's 50,000 settlement inserts fail with null value in column "settled_at" violates not-null constraint — caught by a broad except Exception already in the loader, logged as a warning, not raised, and the job finishes green.

What you'd do on Monday

Pull migrations out into their own serialized, pre-promotion step, and apply them as expand-then-contract instead of a direct NOT NULL add:

Pythonmigrate.py: a serialized step that runs before any app promotion
import fcntl  # advisory file lock standing in for a DB-level advisory lock

with open("/tmp/migration.lock", "w") as lockfile:
    fcntl.flock(lockfile, fcntl.LOCK_EX)   # blocks until any other migration finishes
    run_migration("ALTER TABLE settlements ADD COLUMN settled_at TEXT")  # expand: nullable
    # backfill runs separately, as its own observable, resumable job

The NOT NULL constraint only gets added in a later contract step, run only after a backfill job confirms zero remaining nulls — exactly the sequence from the production example above.

Scenario B

Imagine you work at Instacart

40 sharded Postgres databases, nightly migrations

The numbers

Illustrative: 40 shards, each averaging 25M rows, needing the same schema change rolled out nightly. A full-rewrite ALTER (volatile default, or an engine without the metadata-only optimization) at roughly 50,000 rows/second costs ~500 seconds (8.3 minutes) per shard; run one shard at a time, that's 40 × 8.3 ≈ 333 minutes (5.5 hours) of serialized lock time across the fleet.

What the Scenario-A answer does here

Scenario A's fix (serialize, expand-contract) still holds, but "run the ALTER TABLE on all 40 shards in parallel to save time" breaks: it spikes write-lock contention cluster-wide at the same instant, and a single slow shard (bigger table, replication lag) now blocks the whole fleet's cutover simultaneously instead of just blocking its own deploy.

What flips at this scale

At fleet scale, migrations get a per-shard time budget and roll out shard-by-shard — canary the schema change on one shard first, verify it, then fan out — decoupling one slow shard's migration from the whole fleet's deploy readiness, with backfills running as their own throttled, resumable jobs separate from the schema-only DDL step.

The trade-off you'd defend

I would defend a staged, time-budgeted per-shard rollout over an all-at-once parallel migration, trading total migration wall-clock time for not turning one slow shard into a fleet-wide incident.

Illustrative figures for the arithmetic; not Instacart-published numbers.

Gotchas

ALTER TABLE ADD COLUMN silently rewrites the whole table

Symptom

An ALTER TABLE that was instant in staging holds a lock for minutes in production and blocks reads and writes the whole time.

Cause

The column has a volatile default, a generated/identity definition, or a constrained domain type — any of which forces a full table-and-index rewrite under the same lock a metadata-only change would otherwise take instantly.

Fix

Prefer a constant default (or no default) for the fast metadata-only path; if a rewrite is unavoidable, give it an explicit time budget and schedule it outside peak traffic, or use an online-migration tool.

Two pipelines racing the same migration

Symptom

A hotfix and a scheduled release both trigger near-simultaneously, and the resulting errors don't match either change cleanly on its own.

Cause

Nothing serialized migration execution pipeline-wide, so both runs attempted schema changes against the same table close together.

Fix

Put a single, pipeline-wide lock (advisory lock, lock table, or equivalent) around the migration step so a second attempt waits or fails loudly instead of interleaving.

Assuming an app rollback also rolls back the schema

Symptom

After rolling back a bad release, errors continue, now from the old app code encountering a schema it was never written to expect.

Cause

The migration already ran forward and may have dropped a column or changed a type; rolling back only the app code doesn't undo a destructive schema change, and sometimes can't.

Fix

Fix forward with a new migration instead of expecting the schema to roll back; keep expand-contract's old-and-new compatibility window wide enough that a code rollback never needs the old schema back.

Tricky questions

If the app deploy fails and gets rolled back, does the database schema roll back with it?
Most people say

Yes — rolling back the release reverts the schema change too.

Actually

No. Migrations in a pipeline are typically forward-only; rolling back app code while the schema has moved forward only works because expand-contract kept the old code compatible with the new schema in the first place. A rollback doesn't undo the migration — the compatibility was arranged in advance.

Is adding a NOT NULL column directly safe as long as the pipeline's tests pass?
Most people say

Yes — if tests are green, the migration is safe.

Actually

Not necessarily. Tests running against a small, fresh test database won't reveal a lock held for minutes against a 25-million-row production table, or rows mid-flight failing a constraint that didn't exist when they were written. Lock duration and backfill completeness are production-scale concerns tests at a small scale can't surface.

Scenario question

You're handed a hotfix pipeline that runs a schema migration in the same stage as the app deploy, with no locking, and told "it's always worked." What do you check, and what do you change before the next release?

Model answer

I'd check whether anything actually prevents two pipeline runs from touching the same migration concurrently — most likely nothing does, and "always worked" just means it never happened to race before. I'd pull the migration into its own serialized step with an explicit lock, ahead of any app promotion, and verify the change is expand-compatible (both old and new app versions can read/write) before promotion starts. The trade-off I'd name: this adds a small amount of pipeline time for the lock and the compatibility check, in exchange for migrations that can't silently interleave with a release under time pressure.

Interview corner

CommonWhy can't a database migration just be rolled back the same way application code is?
“Because a destructive schema change can delete data on the way forward, and there's no code-level rollback that brings deleted data back. I fix a bad migration forward with a new migration, not by trying to reverse it.”
TrickIf staging shows an ALTER TABLE finishing instantly, does that prove it's safe in production?
“No. Whether an ALTER TABLE is instant or rewrites the whole table under a lock depends on whether the default or constraint is volatile, and that effect only shows up at real row counts. Staging having ten rows tells me nothing about lock duration on a 25-million-row production table.”
Experience checkHave you seen two pipelines race a migration against each other?
“Yes — a hotfix and a scheduled release both triggered close together with no serialization around the migration step, and the resulting constraint violations didn't cleanly match either change on its own. After that I put a single lock around migration execution, pipeline-wide.”
DesignHow would you roll out the same schema change across 40 sharded databases safely?
“I'd canary it on one shard first, verify lock duration and app behavior there, then fan out shard-by-shard with an explicit time budget per shard — never all 40 in parallel, since one slow shard would otherwise block the whole fleet's cutover at once.”

Memory hook

Pour both new lanes of the bridge first, let traffic choose either one, and only pull up the old lanes once everyone's already moved over.

If you remember only one thing:

Migrations are a distinct, serialized, pre-promotion step — expand and backfill before you ever contract, and fix forward, never backward, once a schema change has landed.

Self-check

  1. Why does a constant-default ADD COLUMN behave so differently from a volatile-default one?

    A constant default only updates the table's metadata, so it's near-instant even on a huge table; a volatile default (or certain constraints) forces a full rewrite of the table and its indexes under the same lock.

  2. Why is "fix forward" the right response to a failed migration, instead of rolling it back?

    A destructive schema change (a dropped column, an overwritten backfill) can lose data on the way forward that a rollback cannot restore; a new corrective migration is the safer path.

  3. What does "serialized" mean for migrations running inside a pipeline?

    Only one migration executes pipeline-wide at any moment — a second pipeline run attempting one concurrently is rejected or made to wait, never interleaved with the first.

Prove it

Using sqlite3 (standing in for Postgres), write a migration script that adds a nullable column, backfills existing rows, verifies zero remaining nulls, and only then adds a NOT NULL-equivalent check before "contracting." Then simulate two concurrent attempts at the same migration using a simple lock (a lock table row, or fcntl.flock on a lock file) and paste the error the second attempt raises.

ce-08-05Card 5 of 6

Choose blue-green, canary and rollback signals

HardP3 · Role-dependentTalk about it13 min
deployment-strategyblue-greencanaryrollbackkubernetes

Why you care

Naming blue-green, canary and rolling deploys is interview trivia; explaining what actually decides a rollback — and why a schema change can take rollback off the table entirely — is the senior answer. A team that eyeballs a dashboard for two minutes after a deploy and calls it "canaried" is one slow-building regression away from a 9 AM incident, because the thing that catches a leak isn't a strategy's name, it's a defined, automatic health gate watching real traffic for long enough to matter.

Explain like I'm 5

Blue-green is two complete stages and a spotlight that swings instantly from one to the other — fast to switch back, but you're paying to light both stages at once during the switch. Canary is sending one trusted bird into the mine first and watching it closely before anyone else goes in. Rolling is replacing band members one at a time mid-song, so for a while half the band on stage is old and half is new, playing together.

Core concept

  • Blue-green: two full, independent environments; traffic switches at the router/load-balancer level, all at once. Rollback is just switching the spotlight back — close to instant — but it costs double the running infrastructure during the switch window, and anything with shared state (a database, a cache) has to tolerate both colors touching it simultaneously if the cutover isn't instantaneous for every component.
  • Canary: a small, defined slice of real traffic goes to the new version first, watched against a health/SLO gate for a defined bake time, then ramped up in stages. It catches a regression before full exposure, at the cost of a longer total rollout and the need for an actual traffic-splitting mechanism — not just "run fewer replicas and assume the split matches."
  • Rolling: the default for a Kubernetes Deployment — pods replaced a few at a time, no second environment needed. For the whole rollout window, old and new versions serve traffic simultaneously, which is the same compatibility requirement blue-green has during its switch, just stretched out longer.
  • Rollback authority: who or what actually decides to roll back. Manual means an on-call engineer's judgment call; automatic means a burn-rate or error-rate gate halts and reverts the rollout itself, without waiting for a human to notice. A strategy without a defined trigger has no real rollback plan — it has a strategy name and a hope.
  • External effects and schema changes constrain rollback. If the new version wrote data in a shape the old version can't read — the direct link to Put database migrations in a safe pipeline's expand-contract requirement — "rolling back the app" is no longer a clean revert. Any strategy that lets old and new versions run concurrently (canary, rolling) requires that compatibility; only blue-green's instant, no-overlap cutover sidesteps it, and only if nothing else shared between the two colors breaks that assumption too.
  • Decision rule: pick blue-green when you need an instant, overlap-free cutover and can afford double infrastructure briefly; pick canary when you need to catch a regression on a slice of real traffic before full exposure; pick rolling as the low-overhead default when neither extreme is required — but in every case, define the automatic gate that triggers rollback before the first pod switches, not after.

Under the hood

A plain Kubernetes Service load-balances connections across every pod matching its selector — it has no built-in concept of weighting traffic by percentage; the official docs are explicit that percentage-based traffic splitting needs something beyond a plain Service, such as an Ingress controller, the Gateway API, or a service mesh. That means a "10% canary" implemented as "run 1 new-version pod alongside 9 old-version pods behind the same Service" is really a 1-in-10 replica ratio, not a guaranteed 10% traffic ratio — the two only match if load happens to be even across every pod, which is a real assumption, not a given. Anyone claiming a precise canary percentage without a weighted router in front of it is really describing a replica count.

Minimal example A plain Service's "10% canary" is really 1 of 10 replicas

PythonA plain Service's "10% canary" is really 1 of 10 replicasruns
def route_requests(total_requests, canary_replicas, baseline_replicas):
    """Models a plain Kubernetes Service: it load-balances roughly evenly
    across every pod matching its selector, with no percentage weighting.
    'canary at 10%' on a plain Service really means '1 of 10 replicas'."""
    total_replicas = canary_replicas + baseline_replicas
    counts = {"canary": 0, "baseline": 0}
    for i in range(total_requests):
        target = "canary" if i % total_replicas < canary_replicas else "baseline"
        counts[target] += 1
    return counts


counts = route_requests(total_requests=1000, canary_replicas=1, baseline_replicas=9)
canary_share = counts["canary"] / sum(counts.values())
print("requests routed to canary:", counts["canary"], "of", sum(counts.values()))
print("canary share equals replica share (1/10):", canary_share == 0.1)

small_fleet = route_requests(total_requests=1000, canary_replicas=1, baseline_replicas=2)
print("canary share with only 3 total replicas:", small_fleet["canary"] / sum(small_fleet.values()))
Output
requests routed to canary: 100 of 1000
canary share equals replica share (1/10): True
canary share with only 3 total replicas: 0.334

In production An automatic rollback gate driven by relative error rate, not a guess

PythonAn automatic rollback gate driven by relative error rate, not a guessruns
from dataclasses import dataclass


@dataclass
class BakeWindowResult:
    requests: int
    errors: int

    @property
    def error_rate(self) -> float:
        return self.errors / self.requests if self.requests else 0.0


def canary_gate(baseline, canary, max_relative_increase, min_sample):
    """Automatic rollback decision: require enough samples to mean anything,
    then compare the canary's error rate against the baseline's, not an
    absolute threshold copied from an unrelated service's template."""
    if canary.requests < min_sample:
        return "HOLD", f"only {canary.requests} canary requests seen, below the {min_sample} needed to decide"
    baseline_error_rate = baseline.error_rate if baseline.error_rate > 0 else 0.0001
    relative_increase = (canary.error_rate - baseline_error_rate) / baseline_error_rate
    if relative_increase > max_relative_increase:
        return "ROLLBACK", f"canary error rate {canary.error_rate:.4f} is {relative_increase:.1f}x baseline {baseline_error_rate:.4f}"
    return "PROMOTE", f"canary error rate {canary.error_rate:.4f} within {max_relative_increase}x of baseline"


baseline = BakeWindowResult(requests=18000, errors=18)         # 0.1% baseline error rate
healthy_canary = BakeWindowResult(requests=2000, errors=3)      # 0.15%, within tolerance
regressed_canary = BakeWindowResult(requests=2000, errors=40)   # 2%, a real regression
sparse_canary = BakeWindowResult(requests=40, errors=0)         # not enough samples yet

for label, canary in [("healthy", healthy_canary), ("regressed", regressed_canary), ("sparse", sparse_canary)]:
    decision, reason = canary_gate(baseline, canary, max_relative_increase=3.0, min_sample=500)
    print(f"{label:9s} -> {decision}: {reason}")
Output
healthy   -> PROMOTE: canary error rate 0.0015 within 3.0x of baseline
regressed -> ROLLBACK: canary error rate 0.0200 is 19.0x baseline 0.0010
sparse    -> HOLD: only 40 canary requests seen, below the 500 needed to decide

See it happen Two gates, not a glance — each one earns the next ramp or reverses it

promotepromoterollbackcanary~10% replicasgate 1canary~50%gate 2100%(old retired)previous version(rollback target)
  1. The new version starts at roughly 10% of replicas; the rest still run the previous version.
  2. Gate 1 checks the canary's error rate and latency against the baseline over a defined bake window.
  3. Gate 1 passes: traffic ramps up to roughly 50%.
  4. Gate 2 checks the same signals again, now at a larger sample size.
  5. If gate 2 fails, rollback re-points traffic back to the previous version, not a rebuild.
  6. If gate 2 passes instead, the rollout finishes at 100% and the previous version is retired.
Notice: two gates, not one glance at a dashboard — each one decides, on real traffic and a real sample size, whether the rollout earns the next percentage or gets reversed.
Scenario A

In your world

The situation

It's 2 AM. On-call flips the router from blue to green, watches the dashboard for two minutes, sees nothing alarming, and goes back to sleep. The old (blue) environment gets torn down an hour later to free up the budget.

What actually breaks

The new version has a slow memory leak that only shows up under sustained load — load that simply wasn't there during a quiet overnight window. By 9 AM, p95 latency has crept from 220ms to 4.1 seconds, and 1 in 12 checkout requests is timing out. There's no "blue" left to switch back to, and nobody re-checked after the two-minute glance that declared it fine.

What you'd do on Monday

Replace the human glance with an explicit, automatic health/SLO gate — error rate and p95 latency under defined thresholds, over a defined bake window — and keep the old environment running through that whole window instead of tearing it down the moment the switch happens:

Pythondeploy_gate.py: a bake window gate, not a two-minute glance
BAKE_WINDOW_MINUTES = 30
MAX_P95_MS = 400
MAX_ERROR_RATE = 0.01

metrics = poll_metrics(window_minutes=BAKE_WINDOW_MINUTES)  # needs real metrics backend
if metrics.p95_ms > MAX_P95_MS or metrics.error_rate > MAX_ERROR_RATE:
    switch_traffic_back_to_blue()
    raise SystemExit("gate failed; rolled back to blue, old environment stays up")

The old environment doesn't get torn down until that gate has actually passed, not until a person has looked away satisfied.

Scenario B

Imagine you work at Dropbox

500 pods behind one service, canary at 10% replicas

The numbers

Illustrative: 500 pods behind a service, canary at a 10% replica ratio = 50 canary pods. If the service handles roughly 200,000 requests/minute, those 50 pods absorb about 20,000 requests/minute — enough for a statistically meaningful error-rate signal within a few minutes, not hours.

What the Scenario-A answer does here

Scenario A's fix (an automatic bake-window gate) still works, but "always canary at a flat 10% for 10 minutes, no matter the service" breaks for a low-traffic internal tool: 10% of a 50 requests/minute service is 5 requests/minute, nowhere near enough samples in 10 minutes to tell a real regression from noise.

What flips at this scale

At this scale, canary sample size and bake duration get computed from the service's own traffic — enough requests to detect the smallest regression that actually matters, per the gate's own math — instead of a flat percentage and duration copied from a template built for a much higher-traffic service.

The trade-off you'd defend

I would defend a traffic-aware canary sizing and bake-time policy over one fixed percentage and duration applied to every service, trading a bit of per-service configuration effort for canaries that are statistically meaningful whether the service is high- or low-traffic.

Illustrative figures for the arithmetic; not Dropbox-published numbers.

Gotchas

"10% canary" on a plain Service is really 1 of 10 replicas

Symptom

A "10% canary" catches a regression far later than expected, or the measured traffic split doesn't match the configured replica ratio at all.

Cause

A plain Kubernetes Service load-balances across matching pods with no percentage weighting; the "percentage" is only as accurate as replica-count ratio, which assumes perfectly even load per pod.

Fix

Use an Ingress controller, Gateway API, or service mesh for a true weighted split if the exact percentage matters, or treat the replica-ratio approximation explicitly as an approximation when sizing the canary.

Schema changes make rollback non-trivial

Symptom

Rolling back after a bad canary doesn't actually fix anything, because data written in the new format during the overlap is unreadable by the old version.

Cause

The release included a schema change that wasn't expand-compatible, so old and new versions couldn't safely coexist during the canary/rolling overlap window in the first place.

Fix

Confirm expand-contract compatibility (Put database migrations in a safe pipeline) before choosing any strategy that lets old and new versions run concurrently; use blue-green's instant, no-overlap cutover instead if that compatibility can't be guaranteed.

No defined trigger for rollback

Symptom

Rollback decisions vary run to run, based on whoever is on call and how closely they happened to look.

Cause

The team has a strategy name (canary, blue-green) but no stated metric and threshold that actually triggers a rollback.

Fix

Define the exact signal (error rate, latency, a specific SLO burn rate) and the exact threshold before the first deploy using that strategy, not during the next incident.

Tricky questions

Is a canary configured at 10% guaranteed to receive exactly 10% of production traffic?
Most people say

Yes, by definition — 10% canary means 10% of traffic.

Actually

Only with a weighted router in front of it. On a plain Kubernetes Service, "10%" is really a replica-count ratio, which only equals a traffic ratio if load happens to be even across every pod — an assumption, not a guarantee.

If a two-minute post-deploy check looks fine, is it safe to tear down the old environment right away?
Most people say

Yes — free up the resources once the switch looks clean.

Actually

No. Some regressions (leaks, slow degradation under sustained load) don't show up in a two-minute glance; keep the old environment live through a defined bake window with automatic gates before retiring it.

Scenario question

You're handed the job of picking a rollout strategy for a payments authorization service, and this release also changes the format of a column it writes to. What do you choose, and why?

Model answer

The schema change decides this before the strategy preference does — if old and new versions can't both safely read and write that column during an overlap, canary and rolling are unsafe until the change is made expand-compatible, because both let old and new versions serve traffic concurrently. I'd choose blue-green specifically for its instant, no-overlap cutover, accepting the cost of running double infrastructure briefly, over a canary that would quietly mix old and new writes against an incompatible schema. The trade-off I'd name: blue-green gives up canary's "catch it on a slice of traffic first" benefit, which I'd only accept because the schema constraint removed canary as a safe option in the first place.

Interview corner

CommonWhat's the practical difference between blue-green, canary and rolling deploys?
“I'd say blue-green switches all traffic at once between two full environments, with fast rollback but double infrastructure during the switch. Canary ramps a small, watched slice of real traffic up gradually. Rolling replaces pods a few at a time with no second environment, but old and new run concurrently for the whole window, same as canary's concurrency requirement.”
TrickDoes setting a canary to 10% in Kubernetes guarantee 10% of traffic?
“I'd say not with a plain Service — that's really a 1-in-10 replica ratio, and it only matches 10% of traffic if load happens to be even across pods. I'd want a weighted router in front, like an Ingress controller or a mesh, before I'd trust a precise percentage.”
Experience checkHave you seen a deploy look fine right after switching, then fail hours later?
“Yes — a memory leak that only showed up under daytime load, well after a quick overnight check had already declared the deploy fine and the old environment was torn down. I replaced the manual glance with an automatic bake-window gate after that.”
DesignHow would you decide which rollout strategy to use for a release that includes a schema change?
“I'd check expand-contract compatibility first. If old and new app versions can both safely use the changed schema, canary or rolling both work. If they can't, I'd use blue-green specifically because its instant cutover avoids the concurrent old-and-new window the other two strategies require.”

Memory hook

Send one trusted bird into the mine first, and watch it closely before anyone else goes in — not a two-minute glance after the whole crew's already down there.

If you remember only one thing:

Define the exact metric and threshold that triggers rollback before the first pod switches — a strategy's name is not a rollback plan.

Self-check

  1. Why doesn't a plain Kubernetes Service guarantee an exact traffic percentage for a canary?

    It load-balances across matching pods with no built-in percentage weighting, so a "10% canary" is really a 1-in-10 replica ratio that only matches 10% of traffic if load is even across pods.

  2. Why can a schema change rule out canary or rolling deploys specifically?

    Both let old and new app versions serve traffic concurrently for a window; if the schema change isn't compatible with both versions reading and writing it, that concurrent window becomes unsafe.

  3. What's the difference between manual and automatic rollback authority?

    Manual relies on a person's judgment call to decide and act; automatic means a defined metric and threshold trigger the halt and revert themselves, without waiting for someone to notice.

Prove it

Write a small canary-gate function that takes a baseline error rate, a canary error rate, a sample size, and thresholds for both relative error increase and minimum sample size. Run it against three scenarios — a healthy canary, a regressed canary, and a canary with too few samples — and paste the three decisions (PROMOTE/ROLLBACK/HOLD) along with the specific numbers that produced each one.

ce-08-06Card 6 of 6

Integrate often without granting CI excessive trust

MediumP2 · Very likelyTalk about it13 min
trunk-based-developmentci-securitysecretsgithub-actionssupply-chain

Why you care

"Integrate often" is good advice that gets dangerous the moment it's paired with "and let CI do whatever it needs to." A pipeline that runs with more privilege or persistence than the specific job in front of it requires is exactly how an external contributor's pull request ends up with a production credential — not through some exotic exploit, but through a documented, well-known trigger that was configured for convenience. Interviewers who ask "walk me through what your CI job can actually do" are checking whether you've only ever pressed Build, or whether you can explain the full blast radius of the job you clicked.

Explain like I'm 5

Integrating often is like saving a document every few minutes instead of once a month — small diffs, easy to see what broke, easy to undo. CI trust is a separate question: it's handing out keys. Give the office cleaner a master key to the building because they need to vacuum the lobby, and now anyone who can get a vacuum cord taped to that key has the whole building, not just the lobby.

Core concept

  • Trunk-based development with short-lived branches keeps integration diffs small and conflicts cheap to resolve — the branching half of "integrate often," covered as a strategy in Choose a team branching strategy. Feature flags are what let a merge to trunk stay dark in production until it's actually ready, decoupling "merged" from "released."
  • Secrets should be scoped, not blanket. A secret available to every job in a repo is available to every job, including ones that don't need it. Environment-scoped secrets, read-only tokens by default, and short-lived credentials over long-lived ones all shrink what a compromised job can actually reach.
  • The default GITHUB_TOKEN is broader than it needs to be unless you say otherwise. GitHub's own security guidance recommends explicitly setting the default permission for GITHUB_TOKEN to read-only, specifically because leaving it at whatever the repository or org default happens to be can grant write access a given job never needed.
  • Pin third-party actions to a full commit SHA, not a mutable tag. uses: some/action@v1 can point at different code tomorrow than it does today, with zero diff in your own workflow file — the same digest-pinning lesson from Build cache-friendly, repeatable Dockerfiles, applied to a dependency you don't control at all. GitHub's own hardening guidance states pinning to a full-length commit SHA is currently the only way to use an action as an immutable release.
  • The trust boundary that actually matters: pull_request vs pull_request_target. A pull_request workflow runs in an unprivileged context by default. A pull_request_target workflow runs in the base repository's privileged context — with its secrets and write access — even when triggered by a fork's PR. Checking out and executing that fork's own code under pull_request_target is a documented way to hand an attacker your secrets; GitHub's guidance is explicit that such workflows "must not explicitly check out untrusted code, including from pull request forks."
  • Self-hosted runners are often persistent. A job that compromises one can leave something behind for whatever runs there next — which is exactly why official guidance says self-hosted runners "should almost never be used for public repositories." Ephemeral, hosted runners default to a fresh VM per job for a reason.
  • Decision rule: default every job to least privilege and ephemeral compute; grant broader privilege or persistent infrastructure only to specific, reviewed jobs that need it — never to "whatever happens to trigger on a pull request."

Minimal example Deny by default — a permission only exists if it was granted

PythonDeny by default: a permission only exists if it was grantedruns
DEFAULT_PERMISSIONS = {"contents": "read"}  # the recommended, explicit safe default


def can_perform(action, granted_permissions):
    """Deny-by-default: an action is only allowed if its exact permission
    was explicitly granted, never because nothing said no."""
    required = {
        "checkout": "contents:read",
        "push_tag": "contents:write",
        "deploy": "deployments:write",
    }[action]
    scope, level = required.split(":")
    granted_level = granted_permissions.get(scope)
    return granted_level == level or (level == "read" and granted_level == "write")


untrusted_pr_job_permissions = DEFAULT_PERMISSIONS
reviewed_release_job_permissions = {"contents": "write", "deployments": "write"}

for job_name, perms in [("untrusted PR job", untrusted_pr_job_permissions), ("reviewed release job", reviewed_release_job_permissions)]:
    print(f"{job_name}:")
    for action in ("checkout", "push_tag", "deploy"):
        print(f"  can {action}:", can_perform(action, perms))
Output
untrusted PR job:
  can checkout: True
  can push_tag: False
  can deploy: False
reviewed release job:
  can checkout: True
  can push_tag: True
  can deploy: True

In production A CI trust checker — fork checkout, self-hosted runners, mutable action pins

PythonA CI trust checker: fork checkout, self-hosted runners, mutable action pinsruns
from dataclasses import dataclass, field


@dataclass
class WorkflowJob:
    name: str
    trigger: str
    checks_out_pr_head: bool
    runner: str
    repo_visibility: str
    uses_actions: list = field(default_factory=list)
    requests_secrets: bool = False


def check_ci_trust(job: WorkflowJob) -> list:
    warnings = []
    if job.trigger == "pull_request_target" and job.checks_out_pr_head:
        warnings.append(
            "pull_request_target checks out the PR head: fork-controlled code runs with "
            "this job's secrets and write access"
        )
    if job.runner == "self-hosted" and job.repo_visibility == "public":
        warnings.append(
            "self-hosted runner used for a public repo trigger: a malicious job can "
            "persist on the runner for whatever job runs next"
        )
    for action in job.uses_actions:
        ref = action.split("@")[1] if "@" in action else ""
        if ref and not ref.strip().startswith(tuple("0123456789abcdef")):
            warnings.append(f"'{action}' is pinned to a mutable ref, not a commit SHA; it can change later")
    return warnings


risky_preview_job = WorkflowJob(
    name="preview-build",
    trigger="pull_request_target",
    checks_out_pr_head=True,
    runner="self-hosted",
    repo_visibility="public",
    uses_actions=["some/deploy-action@v1"],
    requests_secrets=True,
)

safe_label_job = WorkflowJob(
    name="label-pr",
    trigger="pull_request_target",
    checks_out_pr_head=False,
    runner="github-hosted",
    repo_visibility="public",
    uses_actions=["actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683"],
    requests_secrets=True,
)

for job in (risky_preview_job, safe_label_job):
    warnings = check_ci_trust(job)
    print(f"{job.name}: {len(warnings)} warning(s)")
    for w in warnings:
        print(" -", w)
Output
preview-build: 3 warning(s)
 - pull_request_target checks out the PR head: fork-controlled code runs with this job's secrets and write access
 - self-hosted runner used for a public repo trigger: a malicious job can persist on the runner for whatever job runs next
 - 'some/deploy-action@v1' is pinned to a mutable ref, not a commit SHA; it can change later
label-pr: 0 warning(s)

See it happen Same pull request, two different trust boundaries

fork PRpull_request jobpull_request_target jobDEPLOY_TOKENopens PRruns unprivileged, no secretssame PR, different triggerchecks out PR headattacker-controlled script reads ittoken value
  1. A fork opens a pull request against the repository.
  2. Under plain pull_request, the job runs in an unprivileged context with no repository secrets by default.
  3. The same pull request also matches a separate pull_request_target workflow.
  4. That workflow explicitly checks out the fork's own code into a privileged job.
  5. Because this job runs in the base repository's privileged context, the fork-controlled code can read DEPLOY_TOKEN.
  6. The token leaves the pipeline entirely, inside a run that still shows up as a normal green check.
Notice: pull_request and pull_request_target look like a one-word trigger difference; they are actually two different trust boundaries, and only one of them is safe to pair with a fork checkout.
Scenario A

In your world

The situation

Your team open-sourced an internal tool and wants faster CI feedback on external contributions. Someone adds pull_request_target to the preview-build workflow so it can access the DEPLOY_TOKEN secret needed to post a preview, and checks out github.event.pull_request.head.sha for convenience, same as the old internal-only workflow did.

What actually breaks

An external contributor's PR modifies the workflow's own build script to send the DEPLOY_TOKEN value to their own server as a query parameter. Because the job runs under pull_request_target, it executes with the base repository's privileged context and secret available, and the exfiltration happens inside what looks like just another green "preview build" check on the PR. The token has push access to 14 downstream repos; the leak isn't discovered until three days later, when one of those repos shows an unexpected tag nobody on the team pushed.

What you'd do on Monday

Never check out untrusted PR code under pull_request_target. Split the job: an unprivileged build runs under plain pull_request with no secrets, and anything that genuinely needs DEPLOY_TOKEN runs only after a human reviews the diff:

YAML.github/workflows/preview.yml: unprivileged build, privileged step gated by review
on:
  pull_request:
jobs:
  build:
    runs-on: ubuntu-latest
    permissions:
      contents: read
    steps:
      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
      - run: ./build.sh
  preview:
    needs: build
    environment: preview-with-review   # required reviewer gates this job
    runs-on: ubuntu-latest
    steps:
      - run: ./post_preview.sh
        env:
          DEPLOY_TOKEN: ${{ secrets.DEPLOY_TOKEN }}

The build job never sees DEPLOY_TOKEN at all; the job that does only runs after a named reviewer approves it against the environment it's scoped to.

Scenario B

Imagine you work at CRED

120 repos, one dangerous workflow pattern

The numbers

Illustrative: 120 repositories, each with its own CI workflows. If even 5% (6 repos) have a pull_request_target plus secrets plus fork-checkout misconfiguration, that's 6 live secret-exfiltration paths open simultaneously across the org — and one exploited path is exactly as damaging as a single leaked production credential, regardless of how many of the other 114 repos are configured correctly.

What the Scenario-A answer does here

Scenario A's fix (never checkout untrusted code under a privileged trigger) still holds, but "ban pull_request_target entirely, org-wide" over-corrects: some jobs genuinely need it safely — labeling a PR, posting a comment — without ever touching the fork's own code, and a blanket ban just pushes teams toward worse ad hoc workarounds.

What flips at this scale

At org scale, this becomes centrally enforced policy rather than 120 teams each independently remembering one dangerous combination — a required workflow template, or an org-level Actions policy restricting which repos may use self-hosted runners or pull_request_target, with narrow, reviewed exceptions.

The trade-off you'd defend

I would defend a centrally enforced policy with reviewed exceptions over trusting every team to independently get this right, trading some central-team overhead for not needing all 120 teams to be CI-security experts.

Illustrative figures for the arithmetic; not CRED-published numbers.

Gotchas

Untrusted PR code on a persistent, credentialed runner reaches release privileges

Symptom

A self-hosted runner used for public-repo PR checks behaves oddly on its next job, or a later audit finds an unexplained process or cached credential on it.

Cause

Self-hosted runners are commonly persistent and reused; a malicious one-time job can leave a lasting compromise behind for whatever runs there next, which is why official guidance says they should almost never be used for public repositories.

Fix

Use ephemeral, hosted runners for anything that executes untrusted PR code; reserve self-hosted, persistent runners for jobs with no untrusted-input path.

pull_request_target plus a fork checkout

Symptom

A "harmless" preview-build check turns out to have had access to a production secret the whole time, discoverable only by reading the workflow trigger carefully.

Cause

pull_request_target runs in the base repository's privileged context even for fork PRs; explicitly checking out the fork's head inside that job hands attacker-controlled code the ability to read whatever that job can read.

Fix

Never check out untrusted PR code under pull_request_target; use plain pull_request for anything that builds or runs fork code, and keep privileged steps separate and reviewer-gated.

A mutable tag on a third-party action

Symptom

A workflow's behavior changes with no corresponding diff in the repository's own history.

Cause

uses: some/action@v1 references a mutable tag the action's maintainer (or an attacker who compromises their account) can repoint to different code at any time.

Fix

Pin third-party actions to a full-length commit SHA, the same discipline as pinning a base image digest.

Tricky questions

Does the default GITHUB_TOKEN only ever get read access unless I explicitly ask for more?
Most people say

Yes, it's always minimal by default, so there's nothing to configure.

Actually

No — official guidance specifically recommends setting the default permission to read-only, because the out-of-the-box default can be broader than that. Minimal access isn't automatic; it's a setting worth checking and setting explicitly.

Is pull_request_target just a newer, better version of pull_request?
Most people say

Yes, it's the newer trigger, so it's generally preferred.

Actually

No — they represent fundamentally different trust levels. pull_request_target runs in the base repository's privileged context (secrets, write access) even for a fork's PR, which is exactly why checking out untrusted code there is dangerous in a way pull_request isn't.

Scenario question

You're handed a workflow that uses pull_request_target to run tests on fork PRs "for speed," and now it also needs a deploy secret for a preview build. What's your first question, and what would you change?

Model answer

My first question is whether this job checks out the fork's own code — if it does, pairing that with pull_request_target is already a documented secret-exfiltration pattern before the deploy secret even enters the picture. I'd split the work into an unprivileged job (plain pull_request, no secrets, builds the fork's code) and a separate, reviewer-gated job scoped to an Environment that holds the deploy secret, so the secret is only reachable after a human looks at the diff. The trade-off I'd name: this adds a manual approval step to the preview flow, which is slower than letting it run fully automatically — a cost I'd accept over a secret being reachable by anyone who can open a pull request.

Interview corner

CommonWhat's the risk of using pull_request_target with a fork's pull request?
“I'd say it runs in the base repository's privileged context — secrets, write access — even though the triggering code came from a fork. If that job also checks out the fork's head, I'd expect attacker-controlled code to run with privileges it should never have.”
TrickIs a self-hosted runner just a cheaper version of a GitHub-hosted one?
“Not when untrusted code is involved. Self-hosted runners are often persistent, so a compromise can outlive the job that caused it. I'd only use one for jobs with no untrusted-input path, and lean on ephemeral hosted runners for anything triggered by a public PR.”
Experience checkHave you reviewed a CI workflow and found it trusted more than it should have?
“Yes — a preview-build workflow used pull_request_target with secrets and an explicit fork checkout, which meant any external contributor's PR could potentially read that secret. I split it into an unprivileged build job and a separate, reviewer-approved job that holds the secret.”
DesignHow would you prevent 120 teams from each independently making the same CI trust mistake?
“I'd enforce it centrally rather than rely on each team's judgment — a required workflow template, or an org-level policy restricting pull_request_target and self-hosted runners to reviewed exceptions, so the dangerous combination simply isn't available by default.”

Memory hook

Don't hand the person vacuuming the lobby a master key to the whole building just because they need to clean one room.

If you remember only one thing:

Give a job only the privilege and persistence its own work needs — never whatever privilege happens to be available to the trigger that started it.

Self-check

  1. Why is pull_request_target riskier than pull_request for a fork's PR?

    It runs in the base repository's privileged context — with secrets and write access — even though the triggering code came from the fork, unlike pull_request, which runs unprivileged by default.

  2. Why are self-hosted runners discouraged for public repositories?

    They're often persistent and reused across jobs; a malicious job triggered by an untrusted PR can leave a compromise behind for whatever job runs there next.

  3. What's the practical risk of uses: some/action@v1 compared to pinning a commit SHA?

    A tag is mutable — the maintainer, or an attacker who compromises their account, can repoint it to different code later with no diff visible in your own repository's history; a full commit SHA can't be silently repointed the same way.

Prove it

Find (or write) a small GitHub Actions workflow that triggers on pull_request_target. Identify whether it checks out the PR's head ref, whether it references any secrets, and whether every third-party action it uses is pinned to a commit SHA or a mutable tag. Paste the specific lines that would need to change to make it safe for a public repository, and why each one matters.