Process lifecycle and signals

Why it matters

Every deploy stops processes, and how they stop decides whether in-flight requests complete or fail. Kubernetes sends SIGTERM, waits, then sends SIGKILL; a service that ignores the first gets killed mid-request by the second. Containers whose main process does not reap children fill up with zombies. Exit codes drive restarts and alerts. All of it rests on the process lifecycle and the signal mechanism.

The idea

A process moves through a small set of states, and the kernel's scheduler and the process's own actions move it between them.

Exit, zombies and orphans

When a process exits, the kernel frees almost everything but keeps a tiny record, its PID and exit status, until the parent collects it with wait. In that window the process is a zombie: it uses no CPU or memory beyond the record, but it holds a PID. A parent that never calls wait accumulates zombies, and enough of them exhausts the PID table. If a parent exits first, its children become orphans and are adopted by PID 1 (the init process), which is expected to reap them.

Exit status is a small integer: 0 for success, non-zero for failure, and 128 plus the signal number when a signal killed the process (137 means killed by SIGKILL, signal 9, which in a container usually means the out-of-memory killer). Supervisors, shells and orchestrators decide whether to restart based on it.

Signals

A signal is an asynchronous notification delivered to a process by the kernel, by another process (kill), or by the terminal. The process can handle it with a function, ignore it, or accept the default action, which is often to terminate.

Graceful shutdown

1. 1

   Catch SIGTERM

   Install a handler that sets a flag or triggers shutdown rather than exiting immediately.
2. 2

   Stop taking new work

   Fail the readiness check so the load balancer stops sending requests, stop accepting connections, stop pulling from queues.
3. 3

   Finish in-flight work

   Let current requests complete and acknowledge or return in-progress messages, up to a deadline shorter than the orchestrator's grace period.
4. 4

   Release resources and exit 0

   Flush logs and metrics, close connections cleanly, exit with status 0.

In Kubernetes, the pod is removed from service endpoints and sent SIGTERM at roughly the same time, then SIGKILL after the grace period (30 seconds by default), so a short pre-stop delay helps the load balancer catch up before the server stops accepting connections.

The PID 1 problem in containers

The first process in a container is PID 1 inside its namespace, and PID 1 is special: the kernel does not apply default signal actions to it, so a SIGTERM with no handler does nothing, and it inherits orphans that it must reap. A shell script or an application that was never designed as an init will ignore SIGTERM (waiting out the grace period until SIGKILL) and leak zombies. Fixes: run the application with exec so it replaces the shell, handle SIGTERM explicitly, and use a tiny init (tini, dumb-init, or the runtime's built-in init option) to forward signals and reap children.

Trade-offs

Signals are simple and universal but coarse: they carry no payload, can interrupt code at awkward moments, and handlers must do very little safely. Graceful shutdown costs deploy time (waiting for in-flight work) in exchange for zero failed requests. SIGKILL guarantees termination at the cost of losing whatever was in progress.

In an interview

Bring it up for deploys and reliability: "Instances handle SIGTERM by failing readiness, draining in-flight requests for up to 20 seconds, then exiting, inside a 30-second grace period, so rolling deploys drop no requests. The container uses a minimal init so signals reach the app and children are reaped."