Shelf · Core Eng · Chapter 6

Chapter 6: Docker & containers

Your Jenkins job already runs somewhere; this chapter makes that "somewhere" a first-class thing you design instead of a black box you copy from a template. By the end you can explain why a running container is disposable, write a Dockerfile that rebuilds fast and reproducibly, and diagnose a crashing container without guessing.

Difficulty
Priority
ce-06-01Card 1 of 6

Container fundamentals: images, layers and runtime

MediumP2 · Very likelyKnow cold14 min
dockerimageslayerscontainersisolationlifecycle
Needs: nothing. Start here.

Why you care

"What's the difference between an image and a container?" is a common Docker interview question precisely because a wrong answer predicts a real mistake: treating a running container as somewhere to keep a fix. Anyone who has only ever run docker build from a copied Dockerfile eventually has to explain why their live patch "disappeared." Knowing that an image is an immutable stack of layers, and a container is that stack plus one disposable writable layer, is the fact that separates people who have shipped containers from people who have only run docker run hello-world.

Explain like I'm 5

An image is a sealed cookbook: every page (layer) is fixed, and the same cookbook can sit on many kitchen shelves at once, byte for byte identical. A container is a chef working from that cookbook with exactly one sheet of scratch paper clipped to the front. The chef can scribble anything on the scratch paper, but the cookbook itself never changes, and when the chef clocks out, the scratch paper goes in the bin. The next chef who opens the same cookbook starts from a blank sheet, not from your scribbles.

Core concept

  • An image is an ordered stack of read-only layers, one per filesystem-changing instruction in the Dockerfile that built it. Layers are content-addressed, so two images that share a base literally share the same bytes on disk; nothing is duplicated just because two teams both wrote FROM python:3.12-slim.
  • A container is an image plus exactly one thin, writable layer on top, plus an isolated process view: its own PID namespace, network namespace, mount namespace and (usually) a resource budget enforced by cgroups. Nothing about the image changes when a container writes a file; the write lands only in that container's writable layer.
  • Lifecycle: created, running, paused or stopped, then removed. Stopping a container keeps its writable layer and metadata around, so docker start resumes it as it was. Removing it (docker rm) throws that writable layer away for good. This is the whole story behind "my fix disappeared."
  • Networks and volumes exist precisely because the writable layer is not a place to keep anything you care about: a volume is Docker-managed storage that outlives the container that mounts it, and a network is how containers find each other without depending on any one container's private state.
  • Decision rule: never treat a running container as the source of truth for anything that must survive a restart. If it needs to survive, it belongs in the image (change the Dockerfile or the source, then rebuild) or outside the container entirely (a volume, a bind mount, or an external service). If you can't say where a piece of state lives when the container is removed, you don't actually know if it's safe to remove.
  • Containers are not lightweight VMs with the same boundary. There is no hypervisor and no separate kernel: every container on a host shares that kernel, and isolation comes from namespaces (what a process can see) and cgroups (what it can use), not virtualized hardware. That is why containers start in milliseconds, and why a kernel exploit's blast radius across containers differs from a VM escape.

Under the hood

Docker's default storage driver (overlay2 on Linux) implements this with a real union filesystem: each image layer is a "lower" directory, mounted read-only, and the container's writable layer is the "upper" directory. Reading a file walks the stack top-down and returns the first hit. The first time a container writes to a file that only exists in a lower, read-only layer, the driver does a copy-up: it copies that file into the upper directory first, then applies the write there — which is why the very first write to a large file that came from the image can be noticeably slower than every write after it, and why editing a file inside a running container never touches the image layer it came from.

Minimal example Layers are read-only; the writable layer is not

PythonCopy-up semantics, modeled as a tiny union filesystemruns
class Layer(dict):
    """One immutable image layer: a set of files it adds or changes."""


class Container:
    """An image (ordered read-only layers) plus one writable layer on top."""

    def __init__(self, image_layers):
        self.image_layers = image_layers   # read-only, shared
        self.writable = {}                 # this container's own scratch layer

    def read(self, path):
        if path in self.writable:
            return self.writable[path]
        for layer in reversed(self.image_layers):
            if path in layer:
                return layer[path]
        raise KeyError(path)

    def write(self, path, content):
        self.writable[path] = content       # copy-up: never touches image_layers


base = Layer({"os-release": "debian-13"})
deps = Layer({"site-packages/requests": "2.32"})
image_layers = [base, deps]

job1 = Container(image_layers)
job2 = Container(image_layers)

job1.write("os-release", "PATCHED-debian-13")   # a live hotfix inside job1 only

print(job1.read("os-release"))
print(job2.read("os-release"))
print(base["os-release"])
Output
PATCHED-debian-13
debian-13
debian-13

In production Removing a container discards its writable layer, not the image

Two ETL job runs, job-run-1 and job-run-2, both start from the same built image, orders-etl:v1. One of them gets hand-patched while running, the other doesn't, and only one of them gets removed.

PythonRemoving a container discards its writable layer, not the imageruns
class Container:
    def __init__(self, name, image_layers):
        self.name = name
        self.image_layers = image_layers
        self.writable = {}

    def read(self, path):
        if path in self.writable:
            return self.writable[path]
        for layer in reversed(self.image_layers):
            if path in layer:
                return layer[path]
        raise KeyError(f"{path} not found in {self.name}")

    def write(self, path, content):
        self.writable[path] = content

    def remove(self):
        self.writable = None   # docker rm: the writable layer is gone


base = {"python": "3.12-slim"}
deps = {"site-packages/psycopg2": "2.9"}
app = {"etl/load.py": "v1"}
orders_etl_v1 = [base, deps, app]   # this is the built image, orders-etl:v1

job_run_1 = Container("job-run-1", orders_etl_v1)
job_run_2 = Container("job-run-2", orders_etl_v1)

job_run_1.write("etl/load.py", "v1-with-hotfix")   # exec'd in and patched live

print("job_run_1 sees:", job_run_1.read("etl/load.py"))
print("job_run_2 sees:", job_run_2.read("etl/load.py"))

job_run_1.remove()
try:
    job_run_1.read("etl/load.py")
except TypeError:
    print("job_run_1's writable layer is gone, only the image remains")

print("image layer untouched:", app["etl/load.py"])
print("job_run_2 still fine:", job_run_2.read("etl/load.py"))
Output
job_run_1 sees: v1-with-hotfix
job_run_2 sees: v1
job_run_1's writable layer is gone, only the image remains
image layer untouched: v1
job_run_2 still fine: v1

See it happen One image, two containers, one hotfix that never leaves its own container

docker rundocker runbasepython:3.12-slimdepspsycopg2appetl/load.py v1job-run-1writable layerjob-run-2writable layer
  1. The image is three fixed, read-only layers: base, deps, app.
  2. job-run-1 starts: Docker adds one empty writable layer on top of the same image.
  3. job-run-1 is patched live; only its own writable layer changes.
  4. job-run-2 starts from the exact same image and sees the original app layer, untouched by job-run-1's patch.
  5. job-run-1 is removed: its writable layer is deleted with it.
  6. The image layers, and job-run-2, are exactly as they were before job-run-1 ever existed.
Notice: the image never moves. Only the thin writable layer on top of it is created, changed and thrown away, once per container.
Scenario A

In your world

The situation

Your nightly ETL container fails at 1 AM with ModuleNotFoundError: No module named 'ujson'. You SSH into the Jenkins agent, docker exec -it into the running container, pip install ujson, and manually re-trigger the job. It goes green. You go back to bed.

What actually breaks

The container is recreated fresh from orders-etl:2.3 on every run — that's the whole point of disposability. The next night, Jenkins pulls the same image, starts a brand-new container with an empty writable layer, and hits the exact same ModuleNotFoundError. This time it pages on-call, who has no idea a "fix" was ever applied to a container that no longer exists.

What you'd do on Monday

The rule from Scenario A is the fix: state that must survive belongs in the image, not in a running container. Add the dependency to the actual source of truth and rebuild:

TextDockerfile: the fix lives here, not in a shell inside a container
FROM python:3.12-slim
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
CMD ["python", "load.py"]
Textrequirements.txt: add the missing dependency and rebuild
ujson==5.10.0

Rebuild and re-tag deliberately (docker build -t orders-etl:2.4 .), so the next run pulls a version that actually has the fix baked into its layers, not into a container that will be gone by morning.

Scenario B

Imagine you work at Docker Inc

one 45 MB base layer shared across 50,000 images

The numbers

Illustrative: a single python:3.12-slim base layer is roughly 45 MB. Sharing it once across 50,000 images built FROM it costs ~45 MB total; storing it as a separate copy per image would cost 50,000 × 45 MB ≈ 2.25 TB. A pull that already has that layer cached also skips re-downloading those 45 MB, at a typical registry speed of ~100–150 MB/s.

What the Scenario-A answer does here

The Scenario A rule (never patch a running container and expect it to persist) still holds at any scale, but "let every team pick whatever base image they like" breaks the storage and pull-time economics: teams choosing python:3.12 versus -slim versus -bookworm get zero layer sharing even though their code is nearly identical, multiplying both storage and the bytes every CI runner pulls cold.

What flips at this scale

At registry scale, base-image choice stops being a per-repo preference and becomes a shared contract: a small number of pinned, team-owned base images that most services build FROM, so that the layer most likely to already be cached on a given CI runner or node is the one everybody actually uses. The flip is governance over the FROM line, not just Dockerfile hygiene inside any one repo.

The trade-off you'd defend

I would defend a small, versioned set of shared base images over letting every team choose independently, trading a little central coordination (someone owns bumping and deprecating those base images) for real savings in registry storage and CI pull time across the whole fleet.

Illustrative figures chosen to make the arithmetic concrete; not Docker Inc.-published numbers.

Gotchas

A live fix inside a container disappears

Symptom

A patch applied with docker exec works immediately, then the exact same bug reappears after a restart, a redeploy or an autoscaling event, with no error explaining why.

Cause

The patch lives only in that container's writable layer; recreating the container starts from a fresh, empty writable layer over the same unfixed image.

Fix

Make the change in the Dockerfile or application source and rebuild the image; treat docker exec as read-only investigation, not a place to leave fixes.

Assuming containers isolate like VMs

Symptom

A container process observes host-level resource pressure it was never given a limit for, or a security review flags that no per-container kernel boundary exists.

Cause

Containers share the host kernel; namespaces and cgroups isolate what a process can see and use, but there is no hypervisor and no separate kernel underneath.

Fix

Set explicit resource limits, don't run untrusted code expecting VM-grade isolation, and reach for gVisor, Kata or a real VM when that boundary is actually required.

Two containers from the same image "disagree"

Symptom

Container A's change is invisible from container B, even though docker inspect shows both were started from the identical image ID.

Cause

Each container gets its own private writable layer; the shared image layers are read-only, so nothing written in one container's layer is visible from another's.

Fix

If containers genuinely need to share writes, put that data in a shared volume, not either writable layer.

Tricky questions

Is a container basically a lightweight VM?
Most people say

Yes, it's a VM with less overhead — same isolation, just cheaper.

Actually

No. There's no hypervisor and no separate kernel; every container on the host shares that one kernel. Isolation comes from namespaces and cgroups, a software boundary inside one kernel, which is why containers start in milliseconds but also why a kernel exploit's blast radius is different from a VM escape.

Does removing a container delete the image it came from?
Most people say

Yes, the container "used up" the image, so removing the container frees it.

Actually

No. The image is independent, read-only, and can be referenced by other containers or other images built FROM it; it survives every container that used it being removed, and is only deleted with docker rmi once nothing still references it.

Scenario question

You're handed a support ticket: "I fixed the bug by editing a file inside the running container and tested it live — it worked. Two hours later the same bug is back." What happened, and what do you tell them to do differently?

Model answer

I'd explain that the fix lived only in that container's writable layer, and something recreated the container between the test and the ticket — a restart, a health-check recreation, or a routine redeploy — which throws that layer away and starts clean from the same, still-unfixed image. I'd tell them to change the Dockerfile or the source, rebuild and re-tag, then deploy that. The trade-off I'd flag: editing a live container is fine to test a hypothesis, but never a fix, since nothing about it survives the very thing containers are meant to do, which is get recreated.

Interview corner

CommonWhat's the difference between a Docker image and a Docker container?
“I'd say an image is a fixed, read-only stack of layers, and a container is that image plus one thin writable layer and an isolated process view. Multiple containers can run from one image at once, each with its own private writable layer, and none of them can change the image underneath.”
TrickIf I run three containers from one 5 GB image, how much disk does that actually cost?
“I'd say roughly 5 GB total, not 15, because the image's layers are read-only and shared on disk; each container only adds its own writable layer, which starts near zero and grows only with what that specific container writes.”
Experience checkHave you ever hotfixed a running container? What happened afterward?
“Yes, early on, patching a missing dependency inside a running ETL container to unblock a job. It worked that run and disappeared the next night when the scheduler recreated the container from the same unfixed image, which is exactly why I now only ever treat exec as a way to test a hypothesis before changing the Dockerfile.”
DesignWhy aren't containers just cheaper virtual machines?
“I'd point out there's no hypervisor and no separate kernel — every container on a host shares that host's kernel, and isolation is namespaces plus cgroups, not virtualized hardware. That's the trade-off I'd name: much faster startup, but a different, generally weaker isolation boundary than a VM.”

Memory hook

An image is a sealed cookbook; a container is a chef with one sheet of scratch paper, thrown out when the shift ends.

If you remember only one thing:

Anything you want to survive a restart lives in the image or outside the container — never in a container's writable layer.

Self-check

  1. Why does patching a file inside a running container not survive a restart?

    The patch only exists in that container's private writable layer; a restart or recreation starts from a fresh writable layer over the same, unchanged image.

  2. If two containers run from the same 5 GB image, how much extra disk does the second one cost?

    Only the size of its own writable layer (near zero until it writes something); the 5 GB of image layers is shared, not duplicated.

  3. Why is a container not "a lightweight VM" in terms of isolation?

    It shares the host's kernel; there is no hypervisor. Isolation is namespaces (visibility) and cgroups (resource limits), a weaker boundary than virtualized hardware.

Prove it

On a machine with Docker installed: build a small image, run two containers from it, write a different file into each container's writable layer, and show with docker diff <container> that each container's diff is independent. Then docker rm one of them and show the image (docker images) and the other container are both unaffected. Paste the three docker diff outputs and the final docker images line.

ce-06-02Card 2 of 6

Build cache-friendly, repeatable Dockerfiles

MediumP2 · Very likelyKnow cold12 min
dockerdockerfilebuild-cachedockerignorereproducibility

Why you care

A Dockerfile that reinstalls every dependency on every single commit, even when only a comment changed, is one of the most common and most fixable CI complaints on a Python team. It's also a favourite interview question — "why do you COPY requirements.txt before COPY . .?" — because the answer proves whether you understand cache invalidation or just copy-pasted a template. Get the order wrong on the job and your builds get slower every sprint while everyone blames "Docker" instead of the four lines that actually caused it.

Explain like I'm 5

Think of each Dockerfile instruction as a transparent stencil stacked on the one before it. Changing any stencil means every stencil stacked on top of it has to be redrawn too, even if their own artwork didn't change. So you put the stencil that changes least — the base image, the OS packages — at the bottom, and the one that changes on every commit — your application code — right at the top, so redrawing it never forces you to redraw the ones underneath.

Core concept

  • Docker builds a Dockerfile top to bottom, and (with the classic builder) caches one layer per filesystem-changing instruction. A build reuses a cached layer only if its parent layer was also reused and its own instruction content matches exactly — for COPY/ADD, that includes a checksum of the files being copied, not just the instruction text.
  • Cache invalidation cascades downstream, never upstream. The moment one layer misses, every instruction after it in the file must rebuild too, regardless of whether those later instructions' own content changed.
  • Order by real change frequency, cheapest-to-stay-stable first: pinned base image, then rarely-changing OS packages, then the dependency manifest (requirements.txt) plus the install step, then the application code, then CMD/ENTRYPOINT. COPY requirements.txt . followed by RUN pip install -r requirements.txt, only then COPY . ., is the whole trick.
  • .dockerignore keeps the build context small by excluding things like .git, virtual environments and data dumps before the daemon even sees them — a bloated context gets uploaded on every single build, cache hit or not, so it slows every build regardless of caching.
  • Digest pins. A tag like python:3.12-slim is mutable: the publisher can repoint it at new bytes at any time, so an identical Dockerfile can produce different image contents on two different days with no diff in your repo. Pinning FROM python:3.12-slim@sha256:<digest> fixes the exact bytes, so an update becomes a deliberate, reviewable diff instead of a silent surprise on rebuild day.
  • Decision rule: order instructions by how rarely their content actually changes, not by how naturally the app reads top to bottom, and pin anything whose silent movement would surprise you.

Minimal example A Dockerfile's cache breaks downstream from the first miss

PythonReordering a Dockerfile changes which layers survive a rebuildruns
def cache_hits(instructions, cached_keys):
    """Walk instructions in order; a layer only hits if every layer before
    it also hit, because each key is chained to its parent's key."""
    parent = "FROM python:3.12-slim"
    hits = []
    still_ok = True
    for content in instructions:
        key = (parent, content)
        hit = still_ok and key in cached_keys
        hits.append(hit)
        still_ok = hit
        parent = key
    return hits


def keys_for(instructions):
    parent = "FROM python:3.12-slim"
    keys = set()
    for content in instructions:
        key = (parent, content)
        keys.add(key)
        parent = key
    return keys


good_v1 = ["COPY requirements.txt .", "RUN pip install -r requirements.txt", "COPY app/ ."]
good_v2 = ["COPY requirements.txt .", "RUN pip install -r requirements.txt", "COPY app/ . (v2)"]

bad_v1 = ["COPY . .", "RUN pip install -r requirements.txt"]
bad_v2 = ["COPY . . (v2)", "RUN pip install -r requirements.txt"]

print("good order, code-only change:", cache_hits(good_v2, keys_for(good_v1)))
print("bad  order, code-only change:", cache_hits(bad_v2, keys_for(bad_v1)))
Output
good order, code-only change: [True, True, False]
bad  order, code-only change: [False, False]

In production A tiny Dockerfile linter, plus a rebuild-cost estimate

This models what a linter would flag (a mutable base tag, code copied before install) and estimates the arithmetic behind "a commit that only touches application code."

PythonA tiny Dockerfile linter: ordering, pins, and rebuild-cost estimateruns
from dataclasses import dataclass


@dataclass
class Layer:
    instruction: str
    seconds: int          # time this layer takes to (re)build
    changed_this_commit: bool


def lint(order):
    """order: list of instruction strings, FROM first."""
    warnings = []
    if "@sha256:" not in order[0]:
        warnings.append(
            f"unpinned base image: {order[0].removeprefix('FROM ')!r} can select different bytes on rebuild"
        )
    copy_all = next((i for i, s in enumerate(order) if s.startswith("COPY .")), None)
    install = next((i for i, s in enumerate(order) if s.startswith("RUN pip install")), None)
    if copy_all is not None and install is not None and copy_all < install:
        warnings.append("COPY . . before the install step: every code change reinstalls dependencies")
    return warnings


def rebuild_seconds(layers):
    """Once one layer's content changed, every layer after it must rebuild
    too, whether or not that later layer's own content changed."""
    total, cache_broken = 0, False
    for layer in layers:
        cache_broken = cache_broken or layer.changed_this_commit
        if cache_broken:
            total += layer.seconds
    return total


good_order = [
    "FROM python:3.12-slim@sha256:abc123",
    "COPY requirements.txt .",
    "RUN pip install -r requirements.txt",
    "COPY . .",
]
bad_order = [
    "FROM python:3.12-slim",
    "COPY . .",
    "RUN pip install -r requirements.txt",
]

print("good order warnings:", lint(good_order))
print("bad order warnings: ", lint(bad_order))

# A commit that only touches application code, not requirements.txt:
good_layers = [
    Layer(good_order[0], 3, False),
    Layer(good_order[1], 2, False),
    Layer(good_order[2], 35, False),
    Layer(good_order[3], 4, True),
]
bad_layers = [
    Layer(bad_order[0], 3, False),
    Layer(bad_order[1], 4, True),
    Layer(bad_order[2], 35, False),
]

print("good order rebuild seconds:", rebuild_seconds(good_layers))
print("bad  order rebuild seconds:", rebuild_seconds(bad_layers))
Output
good order warnings: []
bad order warnings:  ["unpinned base image: 'python:3.12-slim' can select different bytes on rebuild", 'COPY . . before the install step: every code change reinstalls dependencies']
good order rebuild seconds: 4
bad  order rebuild seconds: 39

See it happen The same one-line code change, two very different rebuilds

good orderFROMCOPY req.txtRUN installCOPY app/bad orderFROMCOPY . .RUN install
  1. Build 1, cold cache: every layer in both orders builds from scratch.
  2. Only application code changes. In the good order, FROM and the install step still hit the cache.
  3. In the bad order, the code copy sits before install, so the code change busts the install layer too.
Notice: the good order rebuilds 4 seconds of work for a code-only commit; the bad order rebuilds 39, for the exact same one-line change.
Scenario A

In your world

The situation

Your team's Jenkins pipeline runs docker build on every pull request. The Dockerfile does COPY . . first, then RUN pip install -r requirements.txt, because that's the order someone wrote it in two years ago.

What actually breaks

Every single commit, including ones that touch only a docstring, reinstalls every dependency from scratch, because the code copy comes before the install step and busts its cache every time. The build has crept from 40 seconds to 6 minutes as the dependency list grew, and reviewers have started batching several small PRs together "to save CI time," which just makes each review bigger and slower.

What you'd do on Monday

Reorder so the manifest is copied and installed before the rest of the source arrives:

TextDockerfile: copy the manifest, install, then copy the rest
FROM python:3.12-slim@sha256:abc123
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
CMD ["python", "load.py"]

Nothing else about the job changes — the fix is purely instruction order, and it's the same 4-second-versus-39-second arithmetic from the minimal example.

Scenario B

Imagine you work at Uber

3,000 CI builds a day across microservices

The numbers

Illustrative: 3,000 merge-triggered builds/day across a microservice fleet, each reinstalling roughly 400 MB of dependencies at ~20 MB/s when the cache is busted needlessly, costing about 20 seconds of dead CI time per build. 3,000 builds × 20 s ≈ 16.7 CI-hours wasted per day, purely from cache misses that a correctly ordered Dockerfile would have avoided.

What the Scenario-A answer does here

Scenario A's fix (copy the manifest before the code) still helps, but "put the manifest first everywhere" isn't the whole story in a shared codebase: if a central, shared internal package gets version-bumped in a common requirements file daily, every service that copies that file busts its own install layer every day too, even services nobody touched.

What flips at this scale

At fleet scale the unit of caching moves up a level: a small number of team-owned base images already carry the common, slow-changing dependencies as pre-built layers, published on their own release cadence, and each service's own Dockerfile only adds its thin, genuinely volatile layer on top. The flip is treating the base image itself as a build artifact with a lifecycle, not something every service quietly re-derives from a bare language tag.

The trade-off you'd defend

I would defend a small set of versioned, centrally owned base images over every service managing its full dependency chain independently, trading a coordination cost — someone has to own base-image bumps — for build-time savings that scale with the size of the fleet instead of against it.

Illustrative arithmetic; not Uber-published figures.

Gotchas

COPY . . before the install step

Symptom

Every rebuild reinstalls all dependencies, even for commits that never touched requirements.txt.

Cause

Cache invalidation cascades downstream from the first changed layer; if the code copy sits above the install step, every code change busts the install layer too.

Fix

Copy the dependency manifest and install first, then copy the rest of the source.

A mutable base-image tag

Symptom

The exact same Dockerfile, rebuilt weeks apart with no repo changes, produces an image with different contents or a different vulnerability count.

Cause

A tag like python:3.12-slim can be repointed to a new digest by the publisher; a mutable tag selects different bytes on a rebuild.

Fix

Pin FROM <image>@sha256:<digest> and bump the digest as a deliberate, reviewed change.

A bloated build context slows every build

Symptom

Sending build context to Docker daemon: 1.2GB prints before anything else runs, even on builds that hit the cache for every layer.

Cause

No .dockerignore, so .git, virtual environments and data files get uploaded to the daemon regardless of what's actually COPYed.

Fix

Add a .dockerignore that excludes anything not needed inside the image.

Tricky questions

Does copying the manifest separately from the code always help?
Most people say

Yes, splitting COPY requirements.txt from COPY . . always improves caching.

Actually

Only if the manifest is genuinely more stable than the code. If it changes just as often as the code, the split buys nothing — ordering follows real change frequency, not habit.

Is RUN apt-get update && apt-get install -y X the same as two separate RUN lines?
Most people say

Yes, splitting them is cleaner and has the same effect.

Actually

No. Splitting means a later rebuild can reuse a stale, cached apt-get update layer while running a fresh apt-get install, which can fail to find the package or install an outdated version. Keep update and install in one RUN so both happen together on any cache miss.

Scenario question

You're handed a Dockerfile whose builds have crept from 30 seconds to 5 minutes over six months, even though the app is barely 2,000 lines and nobody touched the Dockerfile. What do you check, in order?

Model answer

I'd first check what's actually being rebuilt, with docker history on the image, to see which layers are large and whether the same ones rebuild every time. Then I'd look at instruction order for exactly this pattern — code copied before install — since dependency lists tend to grow quietly over six months even if the Dockerfile itself didn't change. I'd also check the build context size for creeping bloat (a .dockerignore that never got updated as the repo grew). The trade-off I'd name: reordering is a five-minute fix with no behaviour change, so I'd do that before touching base images or caching infrastructure.

Interview corner

CommonWhy do you copy requirements.txt before the rest of the source?
“Because cache invalidation cascades downstream from the first changed layer. If I copy the manifest and install dependencies before copying application code, a code-only commit still hits the cache for the install step, instead of reinstalling everything on every change.”
TrickDoes putting the dependency file first always speed up builds?
“I'd say not automatically — it only helps if the manifest genuinely changes less often than the code. If a shared internal package bumps that file daily, the split doesn't save anything, because the layer that was supposed to be stable isn't.”
Experience checkHave you fixed a slow Docker build before? What was actually wrong?
“Yes — I had a Dockerfile that copied the whole repo before installing dependencies, so every commit reinstalled everything. Reordering it dropped the build from minutes back to seconds with zero change to what actually ran.”
DesignHow would you keep build times flat as a fleet of services grows?
“I'd push the slow-changing, shared dependencies into a small number of versioned base images built on their own schedule, so each service's Dockerfile only owns its thin, genuinely volatile layer on top, instead of every service re-deriving the same heavy install step independently.”

Memory hook

Cache breaks downstream from the first miss, never upstream — put what changes least at the top of the file.

If you remember only one thing:

Order Dockerfile instructions by how rarely their content actually changes, not by how naturally the app reads.

Self-check

  1. Why does a code-only commit still reinstall dependencies in a Dockerfile that does COPY . . before RUN pip install?

    Cache invalidation cascades downstream from the first changed layer; the code copy sits above the install step, so any code change busts the install layer's cache too.

  2. What's the practical difference between FROM python:3.12-slim and FROM python:3.12-slim@sha256:<digest>?

    The tag is mutable and can point to different bytes over time; the digest pins the exact image contents, so a rebuild only changes when you deliberately bump it.

  3. Why does a missing .dockerignore slow down builds even when every layer hits the cache?

    The build context is uploaded to the daemon before caching is even checked; a bloated context (like an included .git folder) costs time on every build regardless of cache hits.

Prove it

Write two versions of a small Dockerfile for any tiny Python script with a requirements.txt — one with COPY . . before the install step, one with it after. Build each twice: once cold, once after changing only a comment in the app code. On a machine with Docker installed, time both second builds with docker build --no-cache=false and paste the real wall-clock difference.

ce-06-03Card 3 of 6

Choose multi-stage, slim or distroless images

MediumP3 · Role-dependentTalk about it12 min
dockermulti-stage-buildsslimdistrolessimage-size

Why you care

"Why is our 200-line Flask app's image 1.4 GB?" is a real ticket, and "just switch to distroless" is a real way to turn that ticket into an incident, because a minimal base can silently break a C-extension wheel that needs glibc, or leave a container with no shell exactly when on-call needs one. Interviewers who ask "slim, alpine or distroless?" are checking whether you've actually shipped one of these and hit its specific failure mode, not whether you can name all three.

Explain like I'm 5

A multi-stage build is a workshop and a shipping crate. You build the furniture with a full workshop — saws, glue, sawdust everywhere, a compiler if you're building something from raw materials — and then you pack only the finished chair into the crate that actually ships. The workshop, sawdust and all, never leaves the building.

Core concept

  • Multi-stage builds use more than one FROM in a single Dockerfile, each starting an independent build stage. COPY --from=<stage> pulls named artifacts — a compiled binary, an installed virtual environment, static files — from an earlier stage into a later one. Only the final stage's base image and contents decide what actually ships; everything in earlier stages that isn't explicitly copied forward is discarded.
  • slim (e.g. python:3.12-slim) is Debian-based with most docs and extra packages stripped out, but it still has a real shell, apt, and glibc. Most published Python wheels on PyPI are glibc-linked (manylinux), which makes slim the safe default for Python: smaller than the full image, without changing which wheels install cleanly.
  • alpine uses musl libc instead of glibc and is much smaller, but many C-extension packages have no musl wheel on PyPI, so pip silently falls back to compiling from source — which needs a compiler and headers you may not have budgeted for, and turns a "small image" into a slow, fragile build stage.
  • distroless (Google's gcr.io/distroless, maintained outside Docker Inc.) ships no shell and no package manager at all, only the application and its runtime dependencies; the smallest variant is roughly 2 MiB. The trade is debugging: there's nothing to docker exec into, so the maintainers publish a separate :debug tag with a busybox shell for local reproduction, and production debugging needs an external tool instead.
  • Decision rule: default to slim for Python services. Move to multi-stage before you move to a smaller base — it usually removes more bloat with zero compatibility risk. Only adopt alpine or distroless after checking that every dependency has what it needs (wheels, certs, any shell scripts you rely on), and always measure the resulting image with docker images / docker history rather than assuming smaller is automatically safe.

Minimal example Multi-stage arithmetic — what never ships

PythonSingle-stage vs multi-stage image size, in MBruns
# Illustrative layer sizes in MB, loosely modeled on a real Python build stage.
build_stage_layers = {
    "base-debian": 80,
    "gcc-and-headers": 250,
    "pip-wheel-cache": 180,
    "venv-with-deps": 140,
    "app-code": 5,
}
runtime_stage_layers = {
    "distroless-python-base": 45,
    "venv-with-deps": 140,   # copied in with COPY --from, nothing else from the builder
    "app-code": 5,
}

single_stage_total = sum(build_stage_layers.values())
multi_stage_total = sum(runtime_stage_layers.values())

print("single-stage image size MB:", single_stage_total)
print("multi-stage image size MB: ", multi_stage_total)
print("bytes never shipped, MB:", single_stage_total - multi_stage_total)
print("shipped image is smaller:", multi_stage_total < single_stage_total)
Output
single-stage image size MB: 655
multi-stage image size MB:  190
bytes never shipped, MB: 465
shipped image is smaller: True

In production Does this base actually have the wheels it needs?

Measuring beats assuming: check each candidate base against the packages you actually install before picking the smallest one.

PythonBase-image fit check: wheel availability, size and shell accessruns
from dataclasses import dataclass


@dataclass
class Package:
    name: str
    manylinux_wheel: bool   # prebuilt wheel for glibc-based bases (slim, distroless)
    musl_wheel: bool        # prebuilt wheel for alpine


@dataclass
class Base:
    name: str
    libc: str                 # "glibc" or "musl"
    has_shell: bool
    base_size_mb: int


def check_base(base, packages):
    """Return (estimated_size_mb, warnings) for shipping `packages` on `base`."""
    warnings = []
    size = base.base_size_mb
    for pkg in packages:
        has_wheel = pkg.manylinux_wheel if base.libc == "glibc" else pkg.musl_wheel
        if not has_wheel:
            warnings.append(
                f"{pkg.name}: no prebuilt wheel for {base.libc} on {base.name}, "
                "will compile from source (needs build tools in this stage)"
            )
            size += 40  # rough cost of pulling in a compiler toolchain layer
        else:
            size += 6   # a normal installed wheel
    if not base.has_shell:
        warnings.append(f"{base.name} has no shell: docker exec debugging needs an external tool")
    return size, warnings


psycopg2 = Package("psycopg2-binary", manylinux_wheel=True, musl_wheel=False)
requests = Package("requests", manylinux_wheel=True, musl_wheel=True)

slim = Base("python:3.12-slim", libc="glibc", has_shell=True, base_size_mb=45)
alpine = Base("python:3.12-alpine", libc="musl", has_shell=True, base_size_mb=12)
distroless = Base("gcr.io/distroless/python3-debian13", libc="glibc", has_shell=False, base_size_mb=20)

for base in (slim, alpine, distroless):
    size, warnings = check_base(base, [psycopg2, requests])
    print(f"{base.name}: ~{size} MB, {len(warnings)} warning(s)")
    for w in warnings:
        print("  -", w)
Output
python:3.12-slim: ~57 MB, 0 warning(s)
python:3.12-alpine: ~58 MB, 1 warning(s)
  - psycopg2-binary: no prebuilt wheel for musl on python:3.12-alpine, will compile from source (needs build tools in this stage)
gcr.io/distroless/python3-debian13: ~32 MB, 1 warning(s)
  - gcr.io/distroless/python3-debian13 has no shell: docker exec debugging needs an external tool

See it happen What actually crosses the stage boundary

COPY --from=buildCOPY --from=buildgcc + headerspip wheel cachevenv (deps)app sourceruntime stagedistroless-python
  1. The builder stage installs gcc, fills the pip wheel cache, and builds the venv and app; nothing has shipped yet.
  2. The final stage copies only the finished venv across the stage boundary.
  3. It copies the app source the same way.
  4. gcc and the wheel cache never cross into the runtime stage; they stay in the discarded builder.
  5. The shipped image is the runtime stage alone: a small base, the venv, and the app, nothing else.
Notice: only what's named in COPY --from ever leaves the builder stage. Every build tool stays behind with it.
Scenario A

In your world

The situation

Your ETL container image is 1.3 GB, built with a single FROM python:3.12 and a plain RUN pip install -r requirements.txt, and it takes 90 seconds to pull onto every fresh Jenkins agent.

What actually breaks

Roughly half that image is gcc, build tooling and pip's wheel cache, none of which the running job ever uses — they were only needed to install a couple of packages with C extensions, and a single-stage Dockerfile has no way to leave them behind.

What you'd do on Monday

Split the build into a builder stage that compiles what it needs, and a runtime stage that only copies the finished virtual environment across:

TextDockerfile: multi-stage, only the venv and app cross the boundary
FROM python:3.12-slim AS build
RUN apt-get update && apt-get install -y --no-install-recommends gcc
COPY requirements.txt .
RUN python -m venv /venv && /venv/bin/pip install -r requirements.txt

FROM python:3.12-slim
COPY --from=build /venv /venv
COPY . .
CMD ["/venv/bin/python", "load.py"]

gcc and the wheel cache never reach the final image; the runtime stage is just the slim base, the venv, and the app.

Scenario B

Imagine you work at Airbnb

200 services, 50 cold starts an hour at peak

The numbers

Illustrative: 200 microservices, autoscaling adding roughly 50 new containers per hour at peak. At a registry pull throughput of ~150 MB/s per node, a 1.2 GB image costs about 8 seconds to pull cold versus 1.7 seconds for a 250 MB slimmed image — a difference of about 6.3 seconds per cold start. Fifty cold starts an hour × 6.3 seconds saved ≈ 5.25 minutes of scale-up latency shaved off every hour, purely from image size, while an autoscaler is racing a traffic spike.

What the Scenario-A answer does here

Scenario A's fix (multi-stage) still helps at this scale, but "switch every service to distroless because it's smallest" breaks: some services use a package with a native extension that needs a libc the new base doesn't ship, or a startup healthcheck script that calls curl or bash, neither of which exist in a shell-less image. Swapping the base without checking dependency compatibility trades a smaller image for a build failure, or worse, a healthcheck failure discovered mid-incident.

What flips at this scale

At fleet scale, base-image choice becomes a measured, per-service decision instead of a blanket policy: each service's actual runtime dependencies — wheels needed, shell scripts used, certificate bundles required — get checked before it moves off slim, and anything that does go distroless keeps a :debug-tagged sibling image available so on-call isn't stuck without a shell mid-incident.

The trade-off you'd defend

I would defend measuring each service before minimizing its base image over a fleet-wide "everything goes distroless" mandate, trading a slower rollout for not trading away incident-response time in exchange for image size.

Illustrative figures for the arithmetic; not Airbnb-published numbers.

Gotchas

Alpine plus a C-extension package with no musl wheel

Symptom

A build that "always just installed" suddenly takes five times longer, or fails outright with a compiler error, right after switching the base image to alpine.

Cause

Most PyPI wheels are glibc-linked (manylinux); alpine's musl libc has no matching wheel for many packages, so pip compiles from source and needs build tools that may not be installed.

Fix

Default to a glibc-based image (slim) for Python unless you've confirmed musl wheels exist for every dependency, or deliberately add build tools in a dedicated builder stage.

Distroless breaks a shell-based healthcheck

Symptom

A container is marked unhealthy immediately after switching base images, with nothing unusual in the application's own logs.

Cause

The healthcheck or a startup script called sh -c "curl ...", and the new base has neither a shell nor curl.

Fix

Implement health checks as HTTP checks performed by the orchestrator from outside the container, not as shell scripts run inside it; or use a distroless variant that includes the specific static binary you need.

Assuming a smaller image always means a faster deploy

Symptom

Swapping to a smaller base doesn't noticeably change deploy time.

Cause

Pull time may already be dominated by a base layer shared and cached across services; the actual bottleneck was somewhere else, like app startup or a migration step.

Fix

Measure the real bottleneck first (docker history, actual pull timing) before optimizing image size.

Tricky questions

Is alpine always the smallest, safest default for a Python image?
Most people say

Yes, alpine is the standard minimal base, so it's the safe default.

Actually

Not for Python specifically — musl-versus-glibc wheel mismatches make alpine builds slower and less reliable for many common packages. Slim or distroless are usually the safer minimal choices for Python workloads.

Does using multi-stage automatically make the final image small?
Most people say

Yes, using FROM more than once shrinks the image by itself.

Actually

Only if the final stage's base and its COPY --from lines are chosen to exclude the build tools. Multi-stage is a mechanism, not a guarantee — you can still COPY --from an entire environment, dev dependencies included, and ship a bloated image anyway.

Scenario question

You're handed an image that's 900 MB, told to get it under 300 MB without breaking anything, and you don't know yet whether switching base images is safe. What do you do first?

Model answer

I'd measure before changing anything that could break compatibility — docker history to see which layers are actually the biggest, and a list of what the app needs at runtime versus what was only needed to build it. Then I'd introduce a multi-stage build against the same base image first, which isolates one variable: is the bloat from leftover build tools, or from the base image itself being fat. Only after confirming the base is still the remaining cost would I test a smaller base against the app's real dependency list. The trade-off I'd name: I'd rather ship two smaller, separately verifiable changes than one that changes both variables at once and makes any resulting incident hard to diagnose.

Interview corner

CommonSlim, alpine or distroless: how do you choose?
“I default to slim for Python because most wheels are glibc-linked. I move to distroless once I've verified every dependency's wheels, certs and any shell scripts still work without a shell. I mostly avoid alpine for Python unless I've specifically checked musl wheel availability.”
TrickDoes multi-stage guarantee a smaller image?
“I'd say no — it's a mechanism, not a guarantee. If the final stage still copies a whole build environment or unnecessary dev dependencies across the boundary, the image can stay just as bloated as a single-stage build.”
Experience checkHave you shipped a distroless or alpine image that broke something unexpected?
“Yes — a healthcheck script that called curl inside the container stopped working the moment we moved to a shell-less base, because there was nothing to run it. We moved the check to the orchestrator instead of adding a shell back.”
DesignHow would you shrink a fleet's average image size without breaking services?
“I'd start with multi-stage everywhere, since it has no compatibility risk, then only move individual services to a smaller base after checking their actual runtime dependencies — never as a blanket policy applied to the whole fleet at once.”

Memory hook

A workshop and a shipping crate — the sawdust never leaves the building.

If you remember only one thing:

Multi-stage decides what leaves the workshop; base-image choice decides what the crate is made of. Measure both before you ship.

Self-check

  1. Why does a multi-stage build usually produce a smaller image than a single-stage one?

    Only the final stage's base and whatever is explicitly copied with COPY --from end up in the shipped image; build tools and intermediate artifacts from earlier stages are discarded.

  2. Why might switching from slim to alpine make a Python build slower, not just smaller?

    Many PyPI wheels only ship glibc (manylinux) builds; alpine's musl libc often has no matching wheel, so pip compiles the package from source instead of installing a prebuilt one.

  3. What's the debugging trade-off of a distroless image?

    It has no shell or package manager, so docker exec into an interactive shell fails; debugging needs the vendor's separate :debug tag or an external tool attached to the same namespaces.

Prove it

Take any small Python service with at least one C-extension dependency (e.g. psycopg2 or cryptography). Build it three ways — single-stage on python:3.12, multi-stage on python:3.12-slim, and (if you have Docker available) on python:3.12-alpine — and record the three image sizes with docker images, plus whether the alpine build succeeded without extra build dependencies. Paste all three sizes and the alpine result.

ce-06-04Card 4 of 6

Make ENTRYPOINT, CMD and shutdown predictable

MediumP2 · Very likelyKnow cold12 min
dockerentrypointcmdsignalsshutdownpid1

Why you care

A container that always takes exactly ten seconds to stop, no matter what it's doing, is not a coincidence — it's docker stop's default grace period expiring because nothing inside ever received the signal it sent. Interviewers ask about ENTRYPOINT shell form versus exec form because it's the fastest way to tell whether you've actually debugged a container that wouldn't die cleanly, or just copied a Dockerfile template without reading what it does.

Explain like I'm 5

Exec-form ENTRYPOINT puts your program directly on stage as the lead actor, PID 1. Shell-form puts a stagehand in front who relays your program's lines through a script. When the director shouts "cut!" — a SIGTERM — they're shouting at the stagehand, who might just walk off without telling the actor anything, so the actor keeps performing right up until the theatre physically cuts the power (SIGKILL, once the grace period runs out).

Core concept

  • CMD and ENTRYPOINT each support two forms: exec form (["cmd", "arg"]), which runs the executable directly with no shell involved, and shell form (cmd arg), which is silently wrapped as /bin/sh -c "cmd arg".
  • With exec-form ENTRYPOINT plus CMD, ENTRYPOINT is the fixed program and CMD supplies its default arguments, overridable at docker run time — the standard pattern for "fixed binary, configurable flags." Shell-form ENTRYPOINT ignores CMD and any docker run arguments entirely, per Docker's own reference.
  • Shell form starts your program as a child of /bin/sh -c, not as PID 1, and, again per Docker's reference, that shell "does not pass signals." docker stop sends SIGTERM to PID 1; if PID 1 is a shell that doesn't forward it, your actual process never sees the signal and keeps running until the grace period expires and Docker sends SIGKILL — 10 seconds by default on Linux, configurable per call with docker stop -t/--time.
  • Decision rule: use exec form for both ENTRYPOINT and CMD unless you genuinely need shell features (globbing, env-var expansion) in the entrypoint itself — and if you do, exec your real command as the shell script's last line so it replaces the shell process instead of staying its child.
  • Startup validation belongs at the top of the entrypoint, failing fast and exiting non-zero before serving any traffic, rather than starting the app and letting the orchestrator discover the problem later through failed health checks.
  • Non-root: add a USER instruction. Without one, a container runs as root by default, and because containers share the host kernel, a compromise inside the app gets root-equivalent filesystem access inside that container's namespace — a bigger blast radius than the same bug in a process that was never root to begin with.

Under the hood

PID 1 does not automatically get the kernel's usual default signal handling that every other process gets; nothing installs a default SIGTERM handler for it unless the process itself registers one. This is ordinary Linux init behaviour, not a Docker quirk, and it's exactly why containers need either an application that installs its own signal handler, or docker run --init, which — per Docker's own documentation — "inserts a tiny init-process into the container as the main process, and handles reaping of all processes when the container exits," including forwarding signals down to the real workload.

Minimal example Exec form receives SIGTERM directly; a shell wrapper doesn't forward it

PythonPID 1 identity decides whether SIGTERM ever reaches the appruns
class ShellWrapperEntrypoint:
    """Models `CMD myscript.sh` (shell form): the shell is PID 1, and it
    does not forward signals to the child it started."""

    def __init__(self):
        self.child_got_sigterm = False

    def receive_sigterm(self):
        pass   # the shell has no handler and does not relay it downward


class ExecFormEntrypoint:
    """Models `CMD ["python", "load.py"]` (exec form): the app itself is
    PID 1 and receives the signal directly."""

    def __init__(self):
        self.child_got_sigterm = False

    def receive_sigterm(self):
        self.child_got_sigterm = True


def docker_stop(entrypoint, grace_period_s=10):
    entrypoint.receive_sigterm()
    if entrypoint.child_got_sigterm:
        return "exited cleanly before the grace period"
    return f"no response; SIGKILL sent after {grace_period_s}s"


shell_form = ShellWrapperEntrypoint()
exec_form = ExecFormEntrypoint()

print("shell form CMD:", docker_stop(shell_form))
print("exec  form CMD:", docker_stop(exec_form))
Output
shell form CMD: no response; SIGKILL sent after 10s
exec  form CMD: exited cleanly before the grace period

In production A graceful drain finishes more of the in-flight batch

A worker that reacts to a stop signal by finishing its current batch instead of dying mid-row.

PythonGraceful drain vs an immediate kill, mid-batchruns
from collections import deque


class NaiveWorker:
    """No SIGTERM handler: the process dies wherever it happens to be."""

    def __init__(self, queue):
        self.queue = deque(queue)
        self.committed = []

    def run_until_stopped(self, stop_after=None):
        for i, row in enumerate(list(self.queue)):
            if stop_after is not None and i == stop_after:
                break   # SIGKILL lands here, mid-batch
            self.committed.append(row)
        return self.committed


class GracefulWorker:
    """Exec-form entrypoint with a real handler: stop pulling new rows,
    finish committing the current batch, then exit."""

    def __init__(self, queue, batch_size=5):
        self.queue = deque(queue)
        self.batch_size = batch_size
        self.committed = []
        self.stopping = False

    def sigterm(self):
        self.stopping = True

    def run_until_stopped(self, stop_after=None):
        i = 0
        while self.queue:
            if stop_after is not None and i == stop_after:
                self.sigterm()
            if self.stopping and i % self.batch_size == 0 and self.committed:
                break   # finished the current batch, now drain and exit
            self.committed.append(self.queue.popleft())
            i += 1
        return self.committed


rows = list(range(1, 23))   # 22 settlement rows still in flight

naive = NaiveWorker(rows)
naive_result = naive.run_until_stopped(stop_after=13)
print("naive worker committed:", len(naive_result), "of", len(rows), "-> rows lost:", len(rows) - len(naive_result))

graceful = GracefulWorker(rows, batch_size=5)
graceful_result = graceful.run_until_stopped(stop_after=13)
print("graceful worker committed:", len(graceful_result), "of", len(rows), "-> rows lost:", len(rows) - len(graceful_result))
Output
naive worker committed: 13 of 22 -> rows lost: 9
graceful worker committed: 15 of 22 -> rows lost: 7

See it happen A clean shutdown, because PID 1 is the app itself

docker stopload.py (PID 1, exec form)in-flight batchSIGTERMhandler: stop pulling new rowsfinish committingbatch committedexit 0(shell-form contrast)
  1. docker stop sends SIGTERM straight to PID 1, because ENTRYPOINT/CMD are in exec form.
  2. The registered SIGTERM handler flips a flag; no new work is pulled from the queue.
  3. The current batch keeps committing so it isn't left half-written.
  4. The batch finishes well inside the 10-second default grace period.
  5. The process exits cleanly on its own; SIGKILL is never needed.
  6. If ENTRYPOINT/CMD were shell form instead, PID 1 would be /bin/sh, which drops SIGTERM silently, and this sequence would end in a SIGKILL at the 10-second mark instead.
Notice: exec form makes your code PID 1, which is the only way it can actually see the SIGTERM docker stop sends.
Scenario A

In your world

The situation

Your nightly loader's Dockerfile has CMD python load.py, written as a plain string, which Docker treats as shell form. A deploy restarts the container mid-run.

What actually breaks

python load.py is a child of /bin/sh -c, not PID 1, so it never receives the SIGTERM the deploy sends. After the 10-second grace period, Docker sends SIGKILL, which tears down the container outright, mid-transaction. The staging table is left with a partially committed batch of 2,000 rows, and the next run either duplicates them or trips a unique-key error, either way costing a manual cleanup Monday morning.

What you'd do on Monday

Switch to exec form and let the app react to the signal on its own terms:

TextDockerfile: exec form puts the app itself at PID 1
FROM python:3.12-slim
COPY . .
CMD ["python", "load.py"]
Pythonload.py: install a SIGTERM handler (needs a real container/process, not the sandbox)
import signal
import sys

stopping = False


def handle_sigterm(signum, frame):
    global stopping
    stopping = True


signal.signal(signal.SIGTERM, handle_sigterm)

for batch in read_batches():           # existing batch loop
    if stopping:
        break                          # stop pulling new batches; current one already committed
    load_batch(batch)

sys.exit(0)
Scenario B

Imagine you work at Grab

5,000 in-flight ride requests per deploy

The numbers

Illustrative: a driver-matching service holds roughly 5,000 in-flight ride requests at any moment during peak, deployed via a rolling update touching about 30 pods every 20 minutes. If shutdown isn't graceful, each pod recreation risks whatever requests were mid-flight on it: 30 pods × ~150 in-flight requests per pod ≈ 4,500 requests at risk per deploy if SIGKILL lands before a drain, versus near zero if SIGTERM triggers a drain inside the grace window.

What the Scenario-A answer does here

Scenario A's fix (exec form plus a handler that drains current work) still applies, but "just raise the grace period to 60 seconds so nothing gets killed" breaks at fleet scale: a slow drain multiplied across dozens of pods stretches every rolling deploy by minutes, and a process that's genuinely stuck — the exact bug graceful shutdown was meant to avoid masking — now blocks the whole deploy for a full minute per pod instead of failing fast.

What flips at this scale

At this scale, graceful shutdown needs two more pieces beyond the app's own handler: the orchestrator stops routing new traffic to a pod (readiness flips false) before SIGTERM is even sent, so "stop accepting new work" happens at the load balancer, not only inside the process; and the app's own drain timeout must be shorter than the platform's kill grace period, with margin, so the app always finishes on its own terms before the platform's hard kill fires.

The trade-off you'd defend

I would defend a short, enforced drain timeout inside the app plus readiness-based traffic draining at the platform level, over one long global grace period, trading a small chance of dropping a slow outlier request for deploys that stay fast and predictable across the whole fleet.

Illustrative figures for the arithmetic; not Grab-published numbers.

Gotchas

Shell-form CMD/ENTRYPOINT swallows shutdown signals

Symptom

docker stop always takes the full grace period — exactly 10 seconds, every time — and the app never logs a clean shutdown message.

Cause

The shell became PID 1 and doesn't forward SIGTERM to the real process; Docker eventually sends SIGKILL once the timeout expires.

Fix

Use exec form, or exec the real command as the shell script's last line so it replaces the shell instead of staying its child.

Shell-form ENTRYPOINT ignores CMD and run-time args entirely

Symptom

docker run myimage --flag never actually applies --flag, no matter how CMD or the run command is written.

Cause

Shell-form ENTRYPOINT ignores both CMD and any command-line arguments, per Docker's own reference.

Fix

Write ENTRYPOINT in exec form so CMD and run-time arguments append to it correctly.

Running as root by default

Symptom

A security review flags every service container as running as root, or an RCE inside the app gets root-equivalent filesystem access inside the container.

Cause

No USER instruction; the default user is root.

Fix

Add a dedicated user (RUN useradd -m appuser then USER appuser) and verify the app still has the file access it actually needs.

Tricky questions

Does docker stop immediately kill the container?
Most people say

Yes, stop means kill right away.

Actually

No — it sends SIGTERM first and waits a grace period (10 seconds by default on Linux, configurable) before sending SIGKILL. A shell-form entrypoint just makes that wait pointless, because nothing inside ever acted on the SIGTERM.

If a Dockerfile only sets CMD with no ENTRYPOINT, is signal handling automatically fine?
Most people say

Yes, the shell-form problem is specific to ENTRYPOINT only.

Actually

No — the same /bin/sh -c wrapping applies to shell-form CMD when there's no ENTRYPOINT either. The fix is identical: write it in exec form.

Scenario question

You're handed a service that always takes exactly 10 seconds to stop, no matter what it's doing, and a teammate says "that's just how Docker works." True or false, and what do you check?

Model answer

Not the full story — 10 seconds is the default maximum grace period before SIGKILL, not a mandatory wait; a process that actually handles SIGTERM and finishes draining should exit well before that, often in under a second. I'd check whether ENTRYPOINT/CMD is shell or exec form first, with docker inspect --format '{{.Config.Entrypoint}} {{.Config.Cmd}}', since shell form is the single most common reason nothing exits early. If it's already exec form, I'd check whether the app installs a SIGTERM handler at all — most languages don't shut down gracefully by default without you writing one, and that's the trade-off: a handler is a few lines of code in exchange for deploys that don't cost a full grace period every time.

Interview corner

CommonWhat's the practical difference between exec form and shell form for CMD/ENTRYPOINT?
“I'd explain that exec form runs the executable directly, so it becomes PID 1 and receives signals like SIGTERM straight from docker stop. Shell form wraps it in /bin/sh -c, which becomes PID 1 instead and doesn't forward signals, so the real process never learns it's supposed to stop.”
TrickA container always takes the full grace period to stop. What's your first guess?
“I'd check whether CMD or ENTRYPOINT is written in shell form first, since that's the most common reason SIGTERM never reaches the app. If it's already exec form, I'd check whether the app installs a signal handler at all.”
Experience checkHave you seen a deploy lose in-flight work because of this?
“Yes, I had a nightly loader written with shell-form CMD that got SIGKILLed mid-batch during a deploy, leaving a half-committed table. Switching to exec form and adding a SIGTERM handler that finished the current batch before exiting fixed it.”
DesignHow would you make a rolling deploy of a stateful worker safe?
“I'd make the process handle SIGTERM by stopping new work and draining what's in flight, keep that drain timeout shorter than the platform's kill grace period, and have the orchestrator stop routing new traffic before it even sends the signal.”

Memory hook

Exec form puts your app on stage; shell form puts a stagehand in front who might never relay the cue to leave.

If you remember only one thing:

If docker stop always takes the full grace period, suspect shell-form CMD/ENTRYPOINT first.

Self-check

  1. Why does shell-form ENTRYPOINT prevent a process from receiving SIGTERM cleanly?

    The shell (/bin/sh -c), not the application, becomes PID 1, and that shell doesn't forward signals to the child process it started.

  2. What's the default grace period between SIGTERM and SIGKILL on Linux, and how do you change it?

    10 seconds by default; configurable per call with docker stop -t/--time, or set as the container's own default with --stop-timeout at docker run/create time, which docker stop then uses if you pass no -t.

  3. Why does running as root inside a container matter if the container is already isolated?

    Containers share the host kernel; without a user namespace remapping, root inside the container has root-equivalent filesystem access within that container's view, widening the blast radius of any compromise.

Prove it

On a machine with Docker installed: build two tiny images running the same sleep-based script, one with CMD sleep 300 (shell form) and one with CMD ["sleep", "300"] (exec form). Run both, then time docker stop on each with the default timeout. Paste both real elapsed times and confirm the shell-form container takes the full grace period while the exec-form one exits sooner.

ce-06-05Card 5 of 6

Persist data and connect local services

MediumP2 · Very likelyKnow cold12 min
dockervolumesbind-mountsnetworkingcomposedepends-on

Why you care

"Works on my machine" Compose setups that break the moment a teammate clones the repo trace almost every time to one of two things: a bind mount pointing at a host path that only exists on your laptop, or a service that starts before the thing it depends on is actually ready to accept connections. Interviewers ask about volumes versus bind mounts, and about what depends_on really guarantees, because getting either wrong means shipping a docker-compose.yml that "usually works," which is worse than one that fails loudly every time.

Explain like I'm 5

A volume is a locker Docker manages for you — you don't need to know exactly where it physically lives, only that it survives after the container using it is gone. A bind mount is more like pointing at a specific drawer in your own house — it only works if that exact drawer exists on whichever machine you're standing in. Compose is the seating chart that tells every container which table (network) to sit at, and by what name to call the others at the same table.

Core concept

  • Volumes are storage Docker itself creates and tracks (docker volume create, or implicitly via -v myvol:/data), living outside any single container's writable layer. They survive docker rm, are portable across container recreations, and are the default choice for anything a container needs to keep — a database's data directory, an upload folder.
  • Bind mounts attach a specific host path into the container (-v /host/path:/container/path). They're tied to that host's filesystem layout and permissions — genuinely useful for local development (live-editing source without a rebuild), but the wrong tool for anything that has to work identically on someone else's machine.
  • Networks and DNS: containers on the same user-defined Docker network resolve each other by name through Docker's built-in DNS, which is why a Compose service can call db:5432 instead of an IP address. Containers on different networks, or one on the default bridge and one attached via Compose, simply cannot resolve each other by name.
  • Published vs internal ports: EXPOSE in a Dockerfile is documentation read by tooling, not a mapping — it does not make a port reachable from the host. Only -p host:container (or Compose's ports:) actually publishes a port; without it, a port is reachable only from other containers on the same network, which is exactly what you want for a database that should never be reachable from outside the stack.
  • Compose's depends_on in its short form (or condition: service_started) only waits for the dependency's container process to start — not for it to finish initializing or accept connections; this is documented Compose behaviour, not a gap in your config. Actually waiting for readiness needs condition: service_healthy on the dependent, paired with a healthcheck: block on the dependency.
  • Decision rule: use volumes for anything that must persist and be portable across machines, bind mounts only for local-dev conveniences you're prepared to have differ machine to machine, publish only the ports that genuinely need to be reachable from outside the network, and never assume a dependency is ready just because Compose says it started.

Minimal example DNS resolves by network membership; publishing decides host access

PythonSame-network name resolution vs an unpublished port, modeledruns
class DockerNetwork:
    """A user-defined network with built-in DNS: containers on it can
    resolve each other by service name."""

    def __init__(self, name):
        self.name = name
        self.services = {}          # service_name -> "address"
        self.published_ports = {}   # service_name -> host_port

    def join(self, service_name, address):
        self.services[service_name] = address

    def publish(self, service_name, host_port):
        self.published_ports[service_name] = host_port

    def resolve(self, from_service, target_service):
        if from_service not in self.services:
            raise LookupError(f"{from_service} is not on network {self.name}")
        if target_service not in self.services:
            raise LookupError(f"{target_service}: Name or service not known")
        return self.services[target_service]

    def reach_from_host(self, service_name):
        if service_name not in self.published_ports:
            raise LookupError(f"{service_name}: connection refused, port not published to host")
        return f"localhost:{self.published_ports[service_name]}"


appnet = DockerNetwork("appnet")
appnet.join("db", "10.0.1.2")
appnet.join("etl", "10.0.1.3")
appnet.publish("etl", 8080)   # only the app's health port is published, not the db

print("etl resolves db by name:", appnet.resolve("etl", "db"))

other_net_container = DockerNetwork("othernet")
other_net_container.join("reporting", "10.0.2.5")
try:
    appnet.resolve("reporting", "db")
except LookupError as e:
    print("cross-network lookup fails:", e)

print("host can reach etl:", appnet.reach_from_host("etl"))
try:
    appnet.reach_from_host("db")
except LookupError as e:
    print("host cannot reach db directly:", e)
Output
etl resolves db by name: 10.0.1.2
cross-network lookup fails: reporting is not on network appnet
host can reach etl: localhost:8080
host cannot reach db directly: db: connection refused, port not published to host

In production `service_started` versus `service_healthy`

Pythondepends_on started vs healthy, modeled against a slow-initializing dependencyruns
class Dependency:
    """Models a Postgres container's startup: the process starts
    immediately, but it only becomes healthy after a few init polls."""

    def __init__(self, polls_until_healthy=3):
        self.started = True             # true the instant the process launches
        self._polls_until_healthy = polls_until_healthy
        self._polls_done = 0

    @property
    def healthy(self):
        return self._polls_done >= self._polls_until_healthy

    def healthcheck_poll(self):
        self._polls_done += 1


def connect_when(condition_met, db):
    """Simulate an ETL container's first connection attempt, made the
    instant `condition_met(db)` is true."""
    while not condition_met(db):
        db.healthcheck_poll()
    return "connected" if db.healthy else "connection refused (db not ready)"


db_a = Dependency(polls_until_healthy=3)
db_b = Dependency(polls_until_healthy=3)

# depends_on with the short form / condition: service_started
db_a.healthcheck_poll()   # one poll has run by the time "started" is already true
result_started = "connected" if db_a.healthy else "connection refused (db not ready)"
print("depends_on: service_started ->", result_started)

# depends_on with condition: service_healthy
result_healthy = connect_when(lambda d: d.healthy, db_b)
print("depends_on: service_healthy ->", result_healthy)
Output
depends_on: service_started -> connection refused (db not ready)
depends_on: service_healthy -> connected

See it happen Two separate walls — name resolution, and host access

db:5432 (DNS)-p 8080:8080(no -p)db(healthy)etl containerhost machine
  1. etl and db share a Docker network; only etl's port is published to the host.
  2. etl resolves 'db' by name over Docker's built-in DNS, no IP address needed.
  3. The host can reach etl on localhost:8080 because that port was explicitly published.
  4. The host cannot reach db directly at all; it was never published, which is the point for a database.
  5. etl's first connection attempt fires the instant db's process starts, if depends_on only waits for service_started.
  6. With condition: service_healthy, etl waits for db's own healthcheck to pass before connecting at all.
Notice: name resolution and port publishing are two separate walls — same-network containers always see each other by name, and the host only sees whatever was explicitly published.
Scenario A

In your world

The situation

Your local docker-compose.yml has depends_on: - db (short form) for the ETL container, and db: image: postgres:16 with a bind mount, ./pgdata:/var/lib/postgresql/data, that only exists because you created it once by hand months ago.

What actually breaks

A new teammate clones the repo and runs docker compose up. Postgres either creates an empty ./pgdata with permissions that confuse it on their OS, or the ETL container's very first connection attempt fails with connection refused, because depends_on only waited for the Postgres container process to start, not for Postgres to finish initializing. Compose reports the ETL container as exited/crashed, so they assume the application code is broken, not the compose file.

What you'd do on Monday

Swap the bind mount for a named volume so any machine gets a clean, Docker-managed data directory, and add a health-gated dependency so the ETL container actually waits for readiness, not just a started process:

YAMLdocker-compose.yml: named volume + health-gated dependency
services:
  db:
    image: postgres:16
    volumes:
      - pgdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 2s
      timeout: 2s
      retries: 10
  etl:
    build: .
    depends_on:
      db:
        condition: service_healthy
volumes:
  pgdata:
Scenario B

Imagine you work at Zerodha

50 compose runs a day across a 12-person team

The numbers

Illustrative: a 12-person team runs a 3-service local stack (order-gateway, positions-db, cache) roughly 50 times a day across their laptops while testing order-matching logic. When a schema change makes the old bind-mounted data directory stale, about 1 in 5 runs needs a manual rm -rf pgdata and restart, costing roughly 10 minutes each time. 50 runs/day × 20% × 10 minutes ≈ 100 minutes/day, or roughly 8 dev-hours a week across the team, spent on a problem a named volume with a fresh-init lifecycle would avoid entirely.

What the Scenario-A answer does here

Scenario A's fix (named volume plus healthcheck) still holds locally, but "just add depends_on and a healthcheck to every service" doesn't scale to production — Compose's dependency ordering and health gating are a local-dev, single-host convenience. They say nothing about cross-host failure, retries under a network partition, or what happens when a dependency starts healthy and becomes unhealthy five minutes after the dependent already connected; Compose doesn't restart a dependent when a previously healthy dependency later fails.

What flips at this scale

Past local dev, "wait until the dependency is actually ready" moves from Compose's depends_on into two other places: the application's own connection retry with backoff, and the orchestrator's readiness/liveness probes in production. Compose's health-gated startup is training wheels for a good local loop, not a substitute for the app defending itself against a dependency that's healthy now and unhealthy later.

The trade-off you'd defend

I would defend using Compose health-gating for a reliable local dev loop while still writing the app's own connection retries, over relying on either alone, trading a little duplicated logic for a stack that fails the same predictable way both locally and in production.

Illustrative figures for the arithmetic; not Zerodha-published numbers.

Gotchas

depends_on (short form) doesn't wait for readiness

Symptom

The dependent container's first connection attempt fails with "connection refused" right after docker compose up, more often on a slower machine.

Cause

Short-form depends_on only waits for the dependency's container process to start, not for it to finish its own startup work.

Fix

Add a healthcheck: to the dependency and condition: service_healthy on the dependent.

A bind-mounted data directory has the wrong permissions, or doesn't exist, for a new clone

Symptom

permission denied or initdb: directory ... exists but is not empty on a teammate's machine, while it works fine on yours.

Cause

Bind mounts point at a specific host path with that host's ownership and permission model; nothing guarantees the same path or permissions exist elsewhere.

Fix

Use a named volume for anything Docker itself should own and initialize.

Two containers can't reach each other by name

Symptom

Name or service not known between two containers that are both clearly running.

Cause

They're on different Docker networks — commonly, one was started outside Compose on the default bridge network. Per-network DNS only resolves names within the same network.

Fix

Put both containers on the same user-defined network, or attach one to the other's network explicitly.

Tricky questions

If docker compose ps shows Postgres as "running," is it safe to connect?
Most people say

Yes, running means ready.

Actually

Not necessarily — "running" (service_started) only means the container process started; Postgres may still be initializing its data directory. A real readiness check (pg_isready, an app-level ping) via service_healthy is what actually confirms it can accept connections.

Does EXPOSE in a Dockerfile make a port reachable from the host?
Most people say

Yes, EXPOSE opens the port to the host machine.

Actually

No — EXPOSE is documentation read by tooling like Compose; only -p host:container (or Compose's ports:) actually publishes a port to the host.

Scenario question

You're handed a docker-compose.yml where the app container intermittently fails to connect to Postgres on docker compose up, but always succeeds on a second docker compose restart app. Diagnose it without just adding a sleep.

Model answer

I'd name the mechanism first: depends_on in its default form waited for Postgres's process to start, not for Postgres to finish initializing, so the app's first connection attempt is racing Postgres's own startup. The restart "fixes" it by accident, because by then Postgres is already ready. I'd add a healthcheck: on the Postgres service and switch the app's dependency to condition: service_healthy, so the app never even attempts a connection until Postgres proves it's ready. The trade-off against a sleep: a healthcheck costs a little extra compose config and a small polling interval, but it stays correct even if Postgres's startup gets slower later, where a fixed sleep would eventually start failing again.

Interview corner

CommonVolumes vs bind mounts: how do you choose?
“I use volumes for anything that needs to persist and be portable — a database's data directory — because Docker manages them consistently across machines. I use bind mounts only for local-dev conveniences, like live-editing source code, where I'm fine with it depending on the host's exact filesystem.”
TrickDoes EXPOSE in a Dockerfile make a service reachable from my laptop?
“I'd say no — EXPOSE is just documentation for tooling. Nothing is actually reachable from the host until you publish it with -p or Compose's ports:, which is a separate, deliberate step.”
Experience checkWhat's a local Compose bug you've actually hit?
“I had an app container intermittently failing to connect to Postgres right after docker compose up, because depends_on only waited for the container to start, not for Postgres to be ready. A healthcheck plus condition: service_healthy fixed it for good.”
DesignHow would you make a local Compose stack resilient to dependency startup order?
“I'd add a real healthcheck to every stateful dependency and gate dependents on service_healthy, not service_started, and I'd still have the app itself retry its first connection with backoff, since that's the same defense it needs in production anyway.”

Memory hook

A volume is a locker Docker keeps for you; a bind mount is a drawer at a specific address — it only works if that address exists.

If you remember only one thing:

"Started" is not "ready." Gate on a healthcheck, not on depends_on alone.

Self-check

  1. Why does a bind-mounted Postgres data directory sometimes fail on a teammate's machine but not yours?

    Bind mounts point at a specific host path with that host's permissions and layout; nothing guarantees that path exists or has compatible permissions on a different machine.

  2. What does depends_on: condition: service_started actually guarantee?

    Only that the dependency's container process has started — not that it has finished initializing or can accept connections.

  3. Why can't two containers on different Docker networks resolve each other by name?

    Docker's built-in DNS only resolves service names within the same user-defined network; containers on different networks (or the default bridge) have no shared name resolution.

Prove it

On a machine with Docker installed: write a two-service Compose file (an app and a Postgres dependency) with depends_on: condition: service_healthy and a real pg_isready healthcheck. Run docker compose up several times, then remove the healthcheck and switch to plain depends_on: - db, and run it several more times. Paste any failure you observe in the second version that didn't happen in the first.

ce-06-06Card 6 of 6

Diagnose and scan the shipped container

MediumP2 · Very likelyKnow cold11 min
dockerdebugginglogsinspectexecscanning

Why you care

Getting paged for a container stuck in a restart loop, at whatever hour, is exactly when you need to know which of four commands actually answers your question instead of guessing with docker exec -it sh and finding there's no shell to exec into. Interviewers ask "how would you debug a crashing container" specifically to separate people who've only run docker run hello-world from people who've actually been on the other end of that page.

Explain like I'm 5

Think of the container as a sealed shipping container on a ship. docker logs is reading the manifest paperwork taped to the outside — no need to open anything. docker inspect is the customs record: full metadata, also with the container sealed. docker stats is a live gauge welded to the outside, showing fuel and cargo weight in real time. docker exec is actually cutting a hole and climbing in — only possible if there's something inside to stand on, and it changes nothing about the sealed cargo itself, which is the image.

Core concept

  • docker logs [-f] [--since] shows stdout/stderr the container's main process has written. It's the first thing to check, it works even after the container has exited (as long as it wasn't removed), and it needs nothing installed inside the container.
  • docker inspect returns full JSON metadata: config (entrypoint, cmd, env, user), state (exit code, OOMKilled true/false, start/finish timestamps), mounts and networks. --format pulls one field instead of the whole blob, e.g. docker inspect --format '{{.State.OOMKilled}}', which alone answers "did this crash from an application error or get killed for memory."
  • docker exec runs a new process inside a container's existing namespaces, and it requires two things: the container must still be running, and the executable you're asking for must exist inside that container's filesystem. A shell-less (distroless) image fails docker exec -it <c> sh outright, with an error like exec: "sh": executable file not found in $PATH.
  • docker stats shows live CPU%, memory usage against its limit, network I/O and block I/O per container — the fastest way to confirm whether a container is actually near a resource limit before assuming the bug is in application logic.
  • Image size and scanning: docker history <image> shows per-layer size, useful for "why is this 900 MB." A vulnerability scanner such as docker scout cves <image> analyzes an image's contents against vulnerability databases and reports counts by severity with fixed-version guidance where available. A scan result is an inventory of known vulnerabilities in identifiable packages as of that scan — not a guarantee the image is safe, and not a substitute for knowing whether the build itself was trustworthy (that's provenance, Supply chain: SBOM, lockfiles, provenance and hygiene).
  • Decision rule: read before you touch. docker logs and docker inspect answer most questions with zero risk of changing container state. Reach for docker exec only once you need to interactively test something, and only after confirming there's a shell to exec into. Treat a clean scan as "no known CVEs found today," not "verified safe."

Minimal example Classify a crash from `docker inspect`-style state, no exec required

PythonOOMKilled vs exit code: telling the two apart from state aloneruns
def classify_failure(state):
    """state mirrors the fields docker inspect reports under .State."""
    if state["Running"]:
        return "healthy"
    if state["OOMKilled"]:
        return "oom-killed"
    if state["ExitCode"] == 137:
        return "killed-by-signal (not OOM; check who sent SIGKILL)"
    if state["ExitCode"] == 0:
        return "exited cleanly"
    return f"app-error (exit code {state['ExitCode']}, {state['Error'] or 'no message'})"


states = [
    {"Running": True, "OOMKilled": False, "ExitCode": 0, "Error": ""},
    {"Running": False, "OOMKilled": True, "ExitCode": 137, "Error": ""},
    {"Running": False, "OOMKilled": False, "ExitCode": 137, "Error": ""},
    {"Running": False, "OOMKilled": False, "ExitCode": 1, "Error": "ValueError: missing config"},
    {"Running": False, "OOMKilled": False, "ExitCode": 0, "Error": ""},
]

for s in states:
    print(classify_failure(s))
Output
healthy
oom-killed
killed-by-signal (not OOM; check who sent SIGKILL)
app-error (exit code 1, ValueError: missing config)
exited cleanly

In production An OOM-trend alert and a scan-severity gate

Two small pieces of logic that mirror what docker stats and a scanner report: a resource trend check, and a pass/fail policy over severity counts.

Pythondocker stats-style OOM trend, and a CI scan-severity gateruns
def oom_risk(usage_mb_by_minute, limit_mb, warn_ratio=0.85):
    """docker stats-style trend check: rising usage close to the limit."""
    rising = all(b > a for a, b in zip(usage_mb_by_minute, usage_mb_by_minute[1:]))
    near_limit = usage_mb_by_minute[-1] / limit_mb >= warn_ratio
    if rising and near_limit:
        return f"OOM risk: {usage_mb_by_minute[-1]}MB of {limit_mb}MB and still climbing"
    return f"stable: {usage_mb_by_minute[-1]}MB of {limit_mb}MB"


nightly_etl = [420, 610, 780, 890]      # sampled once a minute during the load
steady_service = [300, 305, 298, 302]

print(oom_risk(nightly_etl, limit_mb=1024))
print(oom_risk(steady_service, limit_mb=1024))


def scan_gate(cves, max_high=2):
    """Pass/fail policy for a CI scan step, same shape as `docker scout cves`
    output grouped by severity."""
    counts = {"CRITICAL": 0, "HIGH": 0, "MEDIUM": 0, "LOW": 0}
    for cve in cves:
        counts[cve["severity"]] += 1
    if counts["CRITICAL"] > 0:
        return "FAIL", counts
    if counts["HIGH"] > max_high:
        return "FAIL", counts
    return "PASS", counts


base_image_bump = [
    {"id": "CVE-2024-1111", "severity": "HIGH"},
    {"id": "CVE-2024-2222", "severity": "MEDIUM"},
    {"id": "CVE-2024-3333", "severity": "LOW"},
]
verdict, counts = scan_gate(base_image_bump)
print(f"scan gate: {verdict}, {counts}")
Output
OOM risk: 890MB of 1024MB and still climbing
stable: 302MB of 1024MB
scan gate: PASS, {'CRITICAL': 0, 'HIGH': 1, 'MEDIUM': 1, 'LOW': 1}
Scenario A

In your world

The situation

A nightly ETL container in staging has started restarting every few minutes. Jenkins shows the job "failed," and someone's first instinct is docker exec -it <container> bash to poke around inside.

What actually breaks

By the time anyone execs in, either the container has already restarted with a fresh ID and no memory of what happened, or the image is a shell-less variant chosen for this job specifically, so docker exec fails outright. Twenty minutes get burned chasing a shell instead of the ten seconds it takes to read what Docker already recorded.

What you'd do on Monday

Check the evidence that already exists before touching anything:

ShellRead before you touch: logs and inspected state, no shell required
docker logs --since 10m orders-etl-nightly
docker inspect --format '{{.State.OOMKilled}} {{.State.ExitCode}}' orders-etl-nightly

For a restart-looping memory issue, that prints something like true 137 — OOMKilled: true, exit code 137 (SIGKILL) — which points straight at a memory limit, not a code bug, before a single shell is opened.

Scenario B

Imagine you work at Datadog

200 of 5,000 containers OOM-looping at once

The numbers

Illustrative: a fleet of 5,000 containers, where a bad deploy causes 200 of them to start OOM-killing in a loop. If each one requires a human to docker exec into a live instance to diagnose, at roughly 5 minutes per manual check that's 200 × 5 minutes ≈ 16.7 engineer-hours to triage a single bad deploy. Reading the OOMKilled field and memory metrics already collected by an agent, instead, triages the same 200 containers with one dashboard query in under a minute.

What the Scenario-A answer does here

Scenario A's fix (read logs and inspected state before exec) still works one container at a time, but "have on-call manually inspect every restarting container" doesn't scale past a handful of incidents. At fleet scale you need exit reason and resource usage exported continuously, so a dashboard already answers "how many containers are OOMKilled right now" before anyone runs a single docker command.

What flips at this scale

At this scale, diagnosis moves from "inspect a specific container after the fact" to "query pre-aggregated signals across the fleet," and docker inspect/docker exec become the tools you reach for on the one container you've already identified as interesting, not the way you find it in the first place. That's the same shift from cold container CLI commands to an observability stack.

The trade-off you'd defend

I would defend investing in fleet-wide exit-reason and resource metrics over relying on manual per-container inspection, trading upfront instrumentation effort for triage time that stays roughly flat as the fleet grows instead of scaling with it.

Illustrative figures for the arithmetic; not Datadog-published numbers.

Gotchas

docker exec into a shell-less container fails

Symptom

OCI runtime exec failed: exec failed: unable to start container process: exec: "sh": executable file not found in $PATH.

Cause

A distroless (or otherwise minimal) image ships no shell or package manager by design.

Fix

Reproduce locally with the image's :debug tag if one exists, or attach an ephemeral debug container to the same namespaces at the platform level; don't add a shell to the production image just to make exec work.

Reading exit code 137 without checking OOMKilled

Symptom

Assuming exit code 137 always means "the app crashed" and spending time reading application code for a bug that isn't there.

Cause

137 is 128 + 9 (SIGKILL) — the same exit code whether the kernel's OOM killer sent it or something else did.

Fix

Always check .State.OOMKilled before assuming a code bug, and compare against the container's configured memory limit.

Trusting a clean vulnerability scan as "safe to ship"

Symptom

An image passes a scan on Monday, ships, and a critical CVE for one of its packages is published Wednesday, with no re-scan scheduled.

Cause

A scan reports known vulnerabilities against the databases available at scan time; it's a snapshot, not a standing guarantee, and it says nothing about whether the build itself was trustworthy.

Fix

Re-scan on a schedule, not only at build time, and pair scanning with provenance checks (Supply chain: SBOM, lockfiles, provenance and hygiene) rather than treating "0 vulnerabilities found" as "verified safe."

Tricky questions

Is docker exec -it <c> bash the right first move for a crash-looping container?
Most people say

Yes, get in and look around first.

Actually

Usually not — if the container keeps crash-looping, by the time you exec in you may be looking at a fresh container with no history of the crash. docker logs and docker inspect on the container that already failed carry the actual evidence, and both keep working even after it stops.

Does a passing vulnerability scan mean the image has no security problems?
Most people say

Yes, a clean scan means it's safe.

Actually

It means no known CVEs were found in the packages the scanner could identify, as of that scan. It says nothing about misconfigurations, the application's own logic, secrets baked into a layer, or whether the build itself was trustworthy.

Scenario question

You're handed a container that's been restart-looping in staging for the last hour, and your only access is the Docker CLI on the host, with no extra logging deployed yet. Walk through your first three commands and what each rules in or out.

Model answer

First, docker logs --since 1h <container>, because it costs nothing and shows the last thing the app itself said before dying. Second, docker inspect --format '{{.State.OOMKilled}} {{.State.ExitCode}} {{.State.Error}}' <container>, because it tells me whether the kernel killed it for memory versus the process exiting on its own. Third, docker stats on a currently running instance, if I can catch one alive, to see whether memory is climbing toward the limit right now. Only after those three, and only if there's a shell to exec into, would I reach for docker exec — trading a slightly slower first pass for not chasing a hypothesis the logs had already ruled out.

Interview corner

CommonHow would you debug a container that keeps restarting?
“I'd start with docker logs and docker inspect, since both work without touching the container and often answer the question outright, especially the OOMKilled field. I'd only reach for docker exec once I need to interactively test something, and only if there's a shell available.”
TrickYour first move is docker exec -it bash. What could go wrong?
“Two things — the image might have no shell at all if it's distroless, and if the container keeps crash-looping, I might be execing into a fresh instance that has no memory of the actual failure. Logs and inspected state on the container that already failed are more reliable.”
Experience checkWhat's a real incident where OOMKilled vs exit code mattered?
“A nightly job kept exiting 137 and we assumed a code bug for a while, until inspect showed OOMKilled: true — the fix was raising the memory limit and fixing an unbounded batch size, not touching the application logic at all.”
DesignHow would you make container diagnosis scale across a large fleet?
“I'd export exit reason and resource trend metrics continuously from every container, so a dashboard answers 'how many are OOMKilled right now' before anyone runs a single docker command on an individual instance.”

Memory hook

Read the manifest and the customs record before you cut a hole in the container.

If you remember only one thing:

Check docker logs and docker inspect before you reach for docker exec — most answers are already recorded.

Self-check

  1. Why can docker logs still help even after a container has already exited?

    It reads stdout/stderr that Docker retained from the container's main process, independent of whether the container is currently running, as long as it wasn't removed.

  2. What single field in docker inspect tells you whether a container was killed for memory?

    .State.OOMKilled, a boolean separate from the exit code.

  3. Why does a clean vulnerability scan not mean an image is fully safe?

    It only reports known vulnerabilities in identifiable packages as of that scan; it says nothing about misconfiguration, application logic, embedded secrets, or the trustworthiness of the build itself.

Prove it

On a machine with Docker installed: run a small script in a container with --memory 50m that deliberately allocates more memory than that limit. Confirm with docker inspect --format '{{.State.OOMKilled}}' that it reports true, and paste that output alongside the exit code.