Images, layers and isolation

Why it matters

Image design decides how fast you build, how fast nodes pull and start your service, how much a security scanner finds, and whether production runs exactly what you tested. A Dockerfile that copies the source before installing dependencies rebuilds everything on every commit; an image with a full OS and compilers ships hundreds of vulnerabilities you never use; a container that writes logs into its own file system slows down and loses them on restart. All of it follows from how layers work.

The idea

An image (in the OCI format that Docker and Kubernetes use) is an ordered list of layers, each a tarball of file system changes, plus a configuration (entrypoint, environment, user, exposed ports). Layers are content-addressed by their hash, so identical layers are stored and transferred once. An image is identified immutably by its digest, a hash of its manifest; tags like :latest are movable names pointing at a digest.

Assembling the file system

At run time, the container runtime stacks the image's read-only layers with an overlay file system (overlayfs on Linux) and adds one thin writable layer on top. Reads look down through the stack and return the first version of a file found. A write to an existing file triggers copy-up: the file is copied from its read-only layer into the writable layer and modified there. A deletion records a whiteout marker hiding the lower file.

Consequences: starting a container copies nothing, so it is fast and many containers share one copy of the image on disk and in the page cache. Heavy writes inside the container file system are slow (copy-up of large files, overlay overhead) and vanish when the container is replaced, which is why data, logs and caches belong in volumes or go to stdout, and why the root file system can be mounted read-only.

Build caching

Each Dockerfile instruction that changes files creates a layer, and the builder reuses a cached layer when the instruction and everything before it are unchanged. Order therefore matters: copy the dependency manifest and install dependencies first, then copy the frequently changing source. Then a code change rebuilds only the last layers, and nodes pull only those.

Practice	Why
Order instructions from least to most frequently changing	Maximises cache hits in builds and pulls
Multi-stage builds: compile in a builder image, copy only artefacts into a small runtime image	Compilers, headers and caches never ship
Minimal bases: distroless, alpine, or scratch for static binaries	Fewer files means fewer vulnerabilities and faster pulls
Pin base images by digest and rebuild regularly	Reproducible builds that still pick up security patches
Run as a non-root user, with a read-only root file system	Limits what a compromise can change
Keep secrets out of layers; use build secrets and runtime injection	Layers are permanent; deleting a file in a later layer does not remove it from history
Deploy by digest, not by tag	Production runs exactly the image that was tested and scanned

Image size and startup

Pull time scales with the bytes that are not already on the node. A 1.5 GB image built on a full OS takes tens of seconds to pull on a fresh node, which matters for autoscaling and for spot capacity. A Go or Rust static binary on a scratch or distroless base can be a few tens of megabytes. Lazy-loading formats (eStargz, SOCI) and registry caches near the cluster reduce pull time further for large images.

Isolation, revisited

Layers make the file system per container, but the isolation boundary remains namespaces, cgroups and the security settings from the previous lesson, on a shared kernel. An image that runs as root with every capability, with the host's Docker socket mounted, is effectively root on the host. Image scanning catches known-vulnerable packages; minimal images, non-root users, dropped capabilities and admission policies that reject unsigned or unscanned images close the rest of the gap.

Trade-offs

Layering gives fast starts, shared storage and cached builds at the cost of slow writes inside the container and of history that keeps everything ever added. Minimal images are small and safe and are harder to debug (no shell); ephemeral debug containers fill that gap. Pinning by digest gives reproducibility and requires a process to rebuild for patches.

In an interview

Show the build and the runtime discipline: "A multi-stage build compiles the service and copies the binary onto a distroless base, about 30 MB; dependencies install before the source is copied so code changes rebuild in seconds. Images are deployed by digest after scanning, run as non-root with a read-only root file system, and anything stateful goes to a volume or an external store."