CI/CD and promotion: immutable artifacts to rollback
Why you care
"Walk me through your pipeline" is a standard interview prompt precisely because the honest answer reveals whether you understand promotion or just understand "the build turned green." A Jenkins job that runs docker build separately for dev, staging and prod is not three identical releases — it's three different builds that happen to share a Dockerfile, and nothing stops one of them from quietly resolving a different dependency version than the one that was tested. The job you want tests one artifact once, then moves that exact artifact forward, and can move it backward just as fast when something's wrong. Get this wrong on the job and you ship an incident that passed every test — just not the test of the bytes that actually reached prod.
Explain like I'm 5
Bake one cake, then carry that same cake past every tasting station — staging, then prod. Nobody re-bakes a fresh cake at each station and assumes it tastes the same as the first one, even from the identical recipe: the oven's a little hotter today, the butter's a different brand, the sugar settled differently in the bag. A pipeline that rebuilds per environment is re-baking at every station and hoping nobody notices the difference. A pipeline that promotes is carrying the one cake that was actually tasted.
Core concept
- A full pipeline has a shape: checkout (get the exact commit), validate (lint, type-check — fast, cheap failures first), test (the suite from Testing strategy: unit, integration, contract and e2e), build (produce one artifact — a container image, a wheel, a binary), publish (push that artifact to a registry under an immutable identity), promote (move that same artifact's reference into an environment), verify (confirm it's healthy there), and rollback (move the reference back if it isn't).
- Build-once-promote-many is the rule that makes all of this trustworthy: the artifact is built exactly once, identified by something that can't lie about what it contains — a commit SHA plus a content digest, not a mutable tag like
latestorv2. Every environment runs that same identity. If staging and prod can't show matching digests, they are not running "the same release," no matter what the Jenkinsfile says. - Promotion is a pointer move, not a rebuild. "Promote to prod" means: change what the prod environment's deployment config points at, to an artifact that already exists in the registry and already passed every earlier gate. It should take seconds, because nothing is being compiled, installed or linked — it already was, once.
- Rollback is the same pointer move, backward. It re-points the environment at the previous artifact that was itself promoted and verified — not a
git revertfollowed by a brand-new build. A rollback that triggers a fresh build defeats the entire point: you'd be deploying a fourth, never-tested artifact during an incident, under time pressure, which is the worst possible moment to introduce a new unknown. - Delivery is not deployment. Continuous delivery means the pipeline can always produce a release-ready, verified artifact; continuous deployment means that artifact reaches production automatically, with no human in the loop. A team can have excellent continuous delivery and still require a person to click "promote to prod" — that's a deliberate choice about risk, not a failure to "finish" CI/CD.
- Decision rule: if two environments can't both answer "what digest is running here?" with the same value, you don't have a pipeline — you have two builds that happen to share a Dockerfile. Fix that before adding gates, strategies or anything else in this chapter; everything else assumes it.
Minimal example Promote by reference; rebuilding per environment can drift
import hashlib
def digest(content):
return hashlib.sha256(content.encode()).hexdigest()[:12]
app_source = "orders_etl_v3"
built_once = digest(app_source)
registry = {}
def publish(name, artifact_digest):
registry[name] = artifact_digest
publish("orders-etl", built_once)
env_pointers = {}
def promote(env, name):
env_pointers[env] = registry[name]
for env in ("dev", "staging", "prod"):
promote(env, "orders-etl")
same_digest_everywhere = len(set(env_pointers.values())) == 1
print("promote-by-reference, one digest for all envs:", same_digest_everywhere)
def rebuild(app_source, dependency_version):
return digest(app_source + dependency_version)
dev_build = rebuild(app_source, "requests==2.31")
staging_build = rebuild(app_source, "requests==2.31")
prod_build = rebuild(app_source, "requests==2.32") # resolved later; a newer pin was available
print("rebuild-per-env, staging matches prod:", staging_build == prod_build)
print("rebuild-per-env, dev matches staging:", dev_build == staging_build)promote-by-reference, one digest for all envs: True
rebuild-per-env, staging matches prod: False
rebuild-per-env, dev matches staging: TrueIn production A promotion pipeline where rollback never rebuilds
import hashlib
from dataclasses import dataclass, field
def digest(payload: str) -> str:
return hashlib.sha256(payload.encode()).hexdigest()[:10]
@dataclass
class Artifact:
name: str
source_commit: str
locked_deps: tuple
tests_passed: bool
@property
def artifact_id(self) -> str:
payload = self.source_commit + "|" + "|".join(self.locked_deps)
return digest(payload)
@dataclass
class Pipeline:
registry: dict = field(default_factory=dict)
env_pointer: dict = field(default_factory=dict)
promotion_log: list = field(default_factory=list)
def build_once(self, artifact: Artifact) -> str:
if not artifact.tests_passed:
raise RuntimeError(f"refusing to publish {artifact.name}: tests did not pass")
self.registry[artifact.artifact_id] = artifact
return artifact.artifact_id
def promote(self, env: str, artifact_id: str):
if artifact_id not in self.registry:
raise KeyError(f"{artifact_id} was never published; cannot promote an unbuilt artifact")
self.env_pointer[env] = artifact_id
self.promotion_log.append((env, artifact_id, "promote"))
def rollback(self, env: str):
history = [eid for e, eid, action in self.promotion_log if e == env and action == "promote"]
if len(history) < 2:
raise RuntimeError(f"no earlier verified artifact to roll {env} back to")
previous = history[-2]
self.env_pointer[env] = previous
self.promotion_log.append((env, previous, "rollback"))
return previous
pipeline = Pipeline()
release_1 = Artifact("orders-etl", "a1b2c3d", ("psycopg2==2.9", "boto3==1.34"), tests_passed=True)
r1_id = pipeline.build_once(release_1)
pipeline.promote("staging", r1_id)
pipeline.promote("prod", r1_id)
release_2 = Artifact("orders-etl", "f9e8d7c", ("psycopg2==2.9", "boto3==1.35"), tests_passed=True)
r2_id = pipeline.build_once(release_2)
pipeline.promote("staging", r2_id)
pipeline.promote("prod", r2_id) # looked fine in staging, promoted to prod too
print("prod currently running release 2:", pipeline.env_pointer["prod"] == r2_id)
rolled_back_to = pipeline.rollback("prod")
print("prod rolled back to the previous verified artifact, not rebuilt:", rolled_back_to == r1_id)
print("rollback took zero new builds; registry still has only 2 artifacts:", len(pipeline.registry) == 2)
try:
pipeline.promote("prod", digest("never-built"))
except KeyError as e:
print(f"KeyError: {e}")prod currently running release 2: True
prod rolled back to the previous verified artifact, not rebuilt: True
rollback took zero new builds; registry still has only 2 artifacts: True
KeyError: 'dfcbc0cb11 was never published; cannot promote an unbuilt artifact'See it happen One artifact, promoted forward, then promoted back
- Code moves through checkout, validate and test before anything is built.
- Build produces exactly one immutable artifact and publishes it once, by digest.
- That same artifact, not a rebuild, is promoted to staging.
- The identical artifact is promoted again, unchanged, to prod.
- A regression surfaces in prod a few hours later.
- Rollback re-points prod at the previous verified digest already sitting in the registry; nothing rebuilds.
In your world
The situation
Your Jenkinsfile has a docker build step inside each environment's own stage — Build & Deploy Staging and Build & Deploy Prod — because that's how it was written two years ago, and it's "worked" every time staging passed.
What actually breaks
Between the staging run on Friday and the prod run on Monday, a transitive dependency with no pinned version resolves to a newer release that changes a currency-rounding default. Staging's build (Friday) still has the old resolution cached; prod's build (Monday) pulls fresh and gets the new one. The nightly settlement job processes its usual 50,000 rows; 12,406 of them get rounded differently, caught by a broad except Exception that was already in the loader, logged as a warning, and written anyway. The build is green. Finance's end-of-day reconciliation shows a ₹3.2 lakh variance three days later, and nobody can say which run introduced it because "the same code" ran in both places.
What you'd do on Monday
Build the image exactly once, right after tests pass, and push it under a digest. Every later stage promotes that digest; it never calls docker build again:
stage('Build') {
steps {
sh 'docker build -t orders-etl:${GIT_COMMIT} .'
sh 'docker push orders-etl:${GIT_COMMIT}'
}
}
stage('Promote to staging') {
steps {
sh 'kubectl set image deploy/orders-etl orders-etl=orders-etl:${GIT_COMMIT} -n staging'
}
}
stage('Promote to prod') {
when { expression { params.APPROVED_FOR_PROD } }
steps {
sh 'kubectl set image deploy/orders-etl orders-etl=orders-etl:${GIT_COMMIT} -n prod'
}
}${GIT_COMMIT} plus the registry's own digest is the artifact's identity. docker inspect --format '{{.Id}}' against staging and prod now returns the same value, which is the one fact last Monday's incident report didn't have.
Imagine you work at Etsy
The numbers
Illustrative: a team doing roughly 50 promotions a day, each one touching three environments (dev, staging, prod). Rebuilding per environment at ~10 minutes a build costs 50 × 3 × 10 = 1,500 build-minutes a day, or 25 CI-hours. Build-once-promote-many costs 50 × 10 = 500 build-minutes a day — a saving of 1,000 minutes (16.7 hours) of CI compute daily, on top of removing the byte-mismatch risk entirely.
What the Scenario-A answer does here
Scenario A's fix (one build, promoted by digest) still holds at this volume, but "just add more CI runners to absorb the 3x rebuilds" doesn't fix the actual problem — more parallel capacity makes three divergent builds finish faster, it doesn't make them identical. Throwing hardware at nondeterminism just hides it behind a shorter wait.
What flips at this scale
At 50 promotions a day, "which environment is this artifact allowed into" stops being inline branching logic inside the pipeline script and becomes configuration owned by each environment itself — a protected-environment setting, not a Jenkinsfile if. Rollback becomes "move the pointer back," fully decoupled from whatever retry logic the build job has, so an incident response never waits on CI capacity.
The trade-off you'd defend
I would defend build-once-promote-many with a thin, environment-owned promotion config over rebuilding per environment, trading a marginally bigger artifact-registry retention bill for guaranteed byte-identical releases and a lot less CI compute.
Illustrative figures chosen to make the arithmetic concrete; not Etsy-published numbers.
Gotchas
Rebuilding per environment promotes different bytes than were tested
Staging passes; the "same" build fails or behaves differently in prod, with no code difference anyone can find in the diff.
Each environment triggered its own build, and an unpinned dependency, cache state or base-image tag resolved differently between the two builds.
Build once, publish by digest, and promote that single artifact's reference into every environment; never run a second docker build for the same release.
Rollback that triggers a rebuild
An incident rollback takes as long as a normal deploy, and occasionally introduces a brand-new bug nobody has seen before.
"Rollback" was implemented as git revert plus a fresh build from the reverted commit, not a pointer move to a previously verified artifact.
Keep the last few promoted artifacts retained in the registry; rollback changes the environment pointer back to one of them, with no build step involved.
Calling it "CI/CD" when it's only CI
An audit or incident review assumes production releases are automatic, and is surprised to learn a person clicks "promote" every time.
The team has continuous delivery (always a release-ready artifact) but not continuous deployment (automatic promotion to prod), and the two got used interchangeably.
Name the actual capability in runbooks and SLAs — "delivery" if a human gates production, "deployment" only if nothing does.
Tricky questions
Yes — it's a CI/CD pipeline, so every merge eventually reaches production automatically.
Only if nothing between "artifact published" and "running in prod" requires a human. A pipeline that stops for a manual promote button is continuous delivery, not continuous deployment — a real and often deliberate distinction, not a lesser version of the same thing.
Yes — same commit, same code, same artifact.
Not guaranteed. Unless dependencies are pinned and the build is reproducible, two separate builds of the same commit can resolve different transitive versions or base-image bytes. "Same commit" guarantees the same source; it does not guarantee the same artifact.
Scenario question
You're handed a pipeline where staging and prod each run their own docker build step off the same Jenkinsfile, and a bug just shipped to prod that staging never caught. What do you check first, and what do you change?
Model answer
I'd check whether the two environments actually ran byte-identical artifacts — docker inspect --format '{{.Id}}' on both — before assuming the code itself is at fault. If the digests differ, that's the bug: two separate builds, not one promoted artifact. I'd change the pipeline to build once right after tests pass, publish that single artifact by digest, and have every environment's stage promote the same reference instead of calling docker build again. The trade-off I'd name: this makes the very first environment slightly slower to go live, since nothing can start until that one build finishes, in exchange for every later environment being guaranteed to run exactly what was tested.
Interview corner
CommonWhat's the difference between continuous delivery and continuous deployment?
TrickIf a Jenkins job rebuilds the image separately for every environment, is that still CI/CD done right?
Experience checkTell me about a time a rollback actually mattered.
DesignHow would you design a promotion pipeline for a payments service that needs an approval gate before prod?
Memory hook
Bake one cake and carry it past every tasting station — never bake a fresh one at each station and hope it tastes the same.
Build the artifact exactly once; every environment gets the same bytes by promoting a reference, never by rebuilding.
Self-check
Why can "the same commit" still produce two different artifacts?
Unless dependencies and the build environment are pinned, two separate builds of the same source can resolve different transitive versions or base-image bytes, producing different artifact digests.
What should a rollback actually do, mechanically?
Re-point the environment's pointer to a previously promoted, verified artifact already sitting in the registry — not trigger a new build from a reverted commit.
What's the practical difference between continuous delivery and continuous deployment?
Delivery means a release-ready artifact always exists; deployment means that artifact reaches production with no human gate. A team can have one without the other.
Write a short script that models a two-environment promotion pipeline: build one artifact with a content-based identifier, publish it to a dict-based "registry," promote it to two environments by writing the identifier into each environment's "pointer," then roll one environment back to a previous identifier without calling the build function again. Paste the pointer values before and after the rollback, and the one line proving the build function was called exactly once.