Container fundamentals: images, layers and runtime
Why you care
"What's the difference between an image and a container?" is a common Docker interview question precisely because a wrong answer predicts a real mistake: treating a running container as somewhere to keep a fix. Anyone who has only ever run docker build from a copied Dockerfile eventually has to explain why their live patch "disappeared." Knowing that an image is an immutable stack of layers, and a container is that stack plus one disposable writable layer, is the fact that separates people who have shipped containers from people who have only run docker run hello-world.
Explain like I'm 5
An image is a sealed cookbook: every page (layer) is fixed, and the same cookbook can sit on many kitchen shelves at once, byte for byte identical. A container is a chef working from that cookbook with exactly one sheet of scratch paper clipped to the front. The chef can scribble anything on the scratch paper, but the cookbook itself never changes, and when the chef clocks out, the scratch paper goes in the bin. The next chef who opens the same cookbook starts from a blank sheet, not from your scribbles.
Core concept
- An image is an ordered stack of read-only layers, one per filesystem-changing instruction in the Dockerfile that built it. Layers are content-addressed, so two images that share a base literally share the same bytes on disk; nothing is duplicated just because two teams both wrote
FROM python:3.12-slim. - A container is an image plus exactly one thin, writable layer on top, plus an isolated process view: its own PID namespace, network namespace, mount namespace and (usually) a resource budget enforced by cgroups. Nothing about the image changes when a container writes a file; the write lands only in that container's writable layer.
- Lifecycle: created, running, paused or stopped, then removed. Stopping a container keeps its writable layer and metadata around, so
docker startresumes it as it was. Removing it (docker rm) throws that writable layer away for good. This is the whole story behind "my fix disappeared." - Networks and volumes exist precisely because the writable layer is not a place to keep anything you care about: a volume is Docker-managed storage that outlives the container that mounts it, and a network is how containers find each other without depending on any one container's private state.
- Decision rule: never treat a running container as the source of truth for anything that must survive a restart. If it needs to survive, it belongs in the image (change the Dockerfile or the source, then rebuild) or outside the container entirely (a volume, a bind mount, or an external service). If you can't say where a piece of state lives when the container is removed, you don't actually know if it's safe to remove.
- Containers are not lightweight VMs with the same boundary. There is no hypervisor and no separate kernel: every container on a host shares that kernel, and isolation comes from namespaces (what a process can see) and cgroups (what it can use), not virtualized hardware. That is why containers start in milliseconds, and why a kernel exploit's blast radius across containers differs from a VM escape.
Under the hood
Docker's default storage driver (overlay2 on Linux) implements this with a real union filesystem: each image layer is a "lower" directory, mounted read-only, and the container's writable layer is the "upper" directory. Reading a file walks the stack top-down and returns the first hit. The first time a container writes to a file that only exists in a lower, read-only layer, the driver does a copy-up: it copies that file into the upper directory first, then applies the write there — which is why the very first write to a large file that came from the image can be noticeably slower than every write after it, and why editing a file inside a running container never touches the image layer it came from.
Minimal example Layers are read-only; the writable layer is not
class Layer(dict):
"""One immutable image layer: a set of files it adds or changes."""
class Container:
"""An image (ordered read-only layers) plus one writable layer on top."""
def __init__(self, image_layers):
self.image_layers = image_layers # read-only, shared
self.writable = {} # this container's own scratch layer
def read(self, path):
if path in self.writable:
return self.writable[path]
for layer in reversed(self.image_layers):
if path in layer:
return layer[path]
raise KeyError(path)
def write(self, path, content):
self.writable[path] = content # copy-up: never touches image_layers
base = Layer({"os-release": "debian-13"})
deps = Layer({"site-packages/requests": "2.32"})
image_layers = [base, deps]
job1 = Container(image_layers)
job2 = Container(image_layers)
job1.write("os-release", "PATCHED-debian-13") # a live hotfix inside job1 only
print(job1.read("os-release"))
print(job2.read("os-release"))
print(base["os-release"])PATCHED-debian-13
debian-13
debian-13In production Removing a container discards its writable layer, not the image
Two ETL job runs, job-run-1 and job-run-2, both start from the same built image, orders-etl:v1. One of them gets hand-patched while running, the other doesn't, and only one of them gets removed.
class Container:
def __init__(self, name, image_layers):
self.name = name
self.image_layers = image_layers
self.writable = {}
def read(self, path):
if path in self.writable:
return self.writable[path]
for layer in reversed(self.image_layers):
if path in layer:
return layer[path]
raise KeyError(f"{path} not found in {self.name}")
def write(self, path, content):
self.writable[path] = content
def remove(self):
self.writable = None # docker rm: the writable layer is gone
base = {"python": "3.12-slim"}
deps = {"site-packages/psycopg2": "2.9"}
app = {"etl/load.py": "v1"}
orders_etl_v1 = [base, deps, app] # this is the built image, orders-etl:v1
job_run_1 = Container("job-run-1", orders_etl_v1)
job_run_2 = Container("job-run-2", orders_etl_v1)
job_run_1.write("etl/load.py", "v1-with-hotfix") # exec'd in and patched live
print("job_run_1 sees:", job_run_1.read("etl/load.py"))
print("job_run_2 sees:", job_run_2.read("etl/load.py"))
job_run_1.remove()
try:
job_run_1.read("etl/load.py")
except TypeError:
print("job_run_1's writable layer is gone, only the image remains")
print("image layer untouched:", app["etl/load.py"])
print("job_run_2 still fine:", job_run_2.read("etl/load.py"))job_run_1 sees: v1-with-hotfix
job_run_2 sees: v1
job_run_1's writable layer is gone, only the image remains
image layer untouched: v1
job_run_2 still fine: v1See it happen One image, two containers, one hotfix that never leaves its own container
- The image is three fixed, read-only layers: base, deps, app.
- job-run-1 starts: Docker adds one empty writable layer on top of the same image.
- job-run-1 is patched live; only its own writable layer changes.
- job-run-2 starts from the exact same image and sees the original app layer, untouched by job-run-1's patch.
- job-run-1 is removed: its writable layer is deleted with it.
- The image layers, and job-run-2, are exactly as they were before job-run-1 ever existed.
In your world
The situation
Your nightly ETL container fails at 1 AM with ModuleNotFoundError: No module named 'ujson'. You SSH into the Jenkins agent, docker exec -it into the running container, pip install ujson, and manually re-trigger the job. It goes green. You go back to bed.
What actually breaks
The container is recreated fresh from orders-etl:2.3 on every run — that's the whole point of disposability. The next night, Jenkins pulls the same image, starts a brand-new container with an empty writable layer, and hits the exact same ModuleNotFoundError. This time it pages on-call, who has no idea a "fix" was ever applied to a container that no longer exists.
What you'd do on Monday
The rule from Scenario A is the fix: state that must survive belongs in the image, not in a running container. Add the dependency to the actual source of truth and rebuild:
FROM python:3.12-slim
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
CMD ["python", "load.py"]ujson==5.10.0Rebuild and re-tag deliberately (docker build -t orders-etl:2.4 .), so the next run pulls a version that actually has the fix baked into its layers, not into a container that will be gone by morning.
Imagine you work at Docker Inc
The numbers
Illustrative: a single python:3.12-slim base layer is roughly 45 MB. Sharing it once across 50,000 images built FROM it costs ~45 MB total; storing it as a separate copy per image would cost 50,000 × 45 MB ≈ 2.25 TB. A pull that already has that layer cached also skips re-downloading those 45 MB, at a typical registry speed of ~100–150 MB/s.
What the Scenario-A answer does here
The Scenario A rule (never patch a running container and expect it to persist) still holds at any scale, but "let every team pick whatever base image they like" breaks the storage and pull-time economics: teams choosing python:3.12 versus -slim versus -bookworm get zero layer sharing even though their code is nearly identical, multiplying both storage and the bytes every CI runner pulls cold.
What flips at this scale
At registry scale, base-image choice stops being a per-repo preference and becomes a shared contract: a small number of pinned, team-owned base images that most services build FROM, so that the layer most likely to already be cached on a given CI runner or node is the one everybody actually uses. The flip is governance over the FROM line, not just Dockerfile hygiene inside any one repo.
The trade-off you'd defend
I would defend a small, versioned set of shared base images over letting every team choose independently, trading a little central coordination (someone owns bumping and deprecating those base images) for real savings in registry storage and CI pull time across the whole fleet.
Illustrative figures chosen to make the arithmetic concrete; not Docker Inc.-published numbers.
Gotchas
A live fix inside a container disappears
A patch applied with docker exec works immediately, then the exact same bug reappears after a restart, a redeploy or an autoscaling event, with no error explaining why.
The patch lives only in that container's writable layer; recreating the container starts from a fresh, empty writable layer over the same unfixed image.
Make the change in the Dockerfile or application source and rebuild the image; treat docker exec as read-only investigation, not a place to leave fixes.
Assuming containers isolate like VMs
A container process observes host-level resource pressure it was never given a limit for, or a security review flags that no per-container kernel boundary exists.
Containers share the host kernel; namespaces and cgroups isolate what a process can see and use, but there is no hypervisor and no separate kernel underneath.
Set explicit resource limits, don't run untrusted code expecting VM-grade isolation, and reach for gVisor, Kata or a real VM when that boundary is actually required.
Two containers from the same image "disagree"
Container A's change is invisible from container B, even though docker inspect shows both were started from the identical image ID.
Each container gets its own private writable layer; the shared image layers are read-only, so nothing written in one container's layer is visible from another's.
If containers genuinely need to share writes, put that data in a shared volume, not either writable layer.
Tricky questions
Yes, it's a VM with less overhead — same isolation, just cheaper.
No. There's no hypervisor and no separate kernel; every container on the host shares that one kernel. Isolation comes from namespaces and cgroups, a software boundary inside one kernel, which is why containers start in milliseconds but also why a kernel exploit's blast radius is different from a VM escape.
Yes, the container "used up" the image, so removing the container frees it.
No. The image is independent, read-only, and can be referenced by other containers or other images built FROM it; it survives every container that used it being removed, and is only deleted with docker rmi once nothing still references it.
Scenario question
You're handed a support ticket: "I fixed the bug by editing a file inside the running container and tested it live — it worked. Two hours later the same bug is back." What happened, and what do you tell them to do differently?
Model answer
I'd explain that the fix lived only in that container's writable layer, and something recreated the container between the test and the ticket — a restart, a health-check recreation, or a routine redeploy — which throws that layer away and starts clean from the same, still-unfixed image. I'd tell them to change the Dockerfile or the source, rebuild and re-tag, then deploy that. The trade-off I'd flag: editing a live container is fine to test a hypothesis, but never a fix, since nothing about it survives the very thing containers are meant to do, which is get recreated.
Interview corner
CommonWhat's the difference between a Docker image and a Docker container?
TrickIf I run three containers from one 5 GB image, how much disk does that actually cost?
Experience checkHave you ever hotfixed a running container? What happened afterward?
DesignWhy aren't containers just cheaper virtual machines?
Memory hook
An image is a sealed cookbook; a container is a chef with one sheet of scratch paper, thrown out when the shift ends.
Anything you want to survive a restart lives in the image or outside the container — never in a container's writable layer.
Self-check
Why does patching a file inside a running container not survive a restart?
The patch only exists in that container's private writable layer; a restart or recreation starts from a fresh writable layer over the same, unchanged image.
If two containers run from the same 5 GB image, how much extra disk does the second one cost?
Only the size of its own writable layer (near zero until it writes something); the 5 GB of image layers is shared, not duplicated.
Why is a container not "a lightweight VM" in terms of isolation?
It shares the host's kernel; there is no hypervisor. Isolation is namespaces (visibility) and cgroups (resource limits), a weaker boundary than virtualized hardware.
On a machine with Docker installed: build a small image, run two containers from it, write a different file into each container's writable layer, and show with docker diff <container> that each container's diff is independent. Then docker rm one of them and show the image (docker images) and the other container are both unaffected. Paste the three docker diff outputs and the final docker images line.