System calls

Why it matters

Everything a program does beyond pure computation, reading a file, sending a packet, allocating memory, starting a thread, getting the time, is a system call or relies on one. Their cost shapes high-performance code (why buffering matters, why sendfile exists, why io_uring was invented), strace output is often the fastest way to see what a misbehaving process is actually doing, and container security policies are written as lists of allowed system calls.

The idea

A system call is a request from a program to the kernel. Programs run in user mode, where the CPU forbids access to devices, kernel memory and privileged instructions. To make a call, the program puts a call number and arguments in registers and executes a special instruction (syscall on x86-64) that traps into the kernel: the CPU switches to kernel mode, jumps to the kernel's entry point, which validates the arguments, checks permissions, does the work and returns a result (or an error code, surfaced as errno).

The calls every server makes

What a call costs

The trap and return themselves cost on the order of a hundred nanoseconds to a few hundred, depending on the CPU and on security mitigations added after Spectre and Meltdown (which made the boundary crossing more expensive). The real cost is usually the work inside plus side effects: copying data between user and kernel buffers, cache and TLB disruption, and blocking if the call must wait. A program making millions of tiny calls per second spends a large share of its time crossing the boundary.

Avoiding the crossing

1. 1

   Batch

   Buffer writes and read in large chunks; use writev and readv to move several buffers in one call; recvmmsg and sendmmsg for many datagrams.
2. 2

   vDSO

   Calls such as clock_gettime and gettimeofday are served by code the kernel maps into every process, so reading the time does not trap at all.
3. 3

   Zero-copy

   sendfile and splice move file data to a socket inside the kernel without copying it through user space; nginx and Kafka use this to serve files and log segments.
4. 4

   io_uring

   Shared submission and completion rings let a program queue many operations and collect results without a system call per operation.
5. 5

   Kernel bypass

   For extreme cases, DPDK and SPDK drive network cards and NVMe devices directly from user space, avoiding the kernel entirely.

Watching them

strace -f -p PID prints every system call a running process makes, with arguments, results and timing (-T, -c for a summary). It answers "what is it doing?" when logs say nothing: stuck in futex (waiting on a lock), looping on epoll_wait with timeouts, failing connect calls with ECONNREFUSED, open calls failing with ENOENT on a config path. strace slows the process noticeably; eBPF tools (bpftrace, the BCC suite) observe system calls with much lower overhead in production.

System calls as a security boundary

Because every interaction with the outside world goes through system calls, restricting them confines a process. seccomp filters let a process (or a container runtime on its behalf) allow only a list of calls; Docker and Kubernetes apply a default profile blocking dozens of rarely needed, risky ones. Sandboxes such as gVisor intercept system calls and implement them in a user-space kernel, trading performance for a much smaller attack surface.

Trade-offs

The user/kernel boundary is what makes processes safe and the kernel trustworthy, and every crossing costs time. Batching, vDSO, zero-copy and io_uring reduce crossings at the cost of more complex code; kernel bypass removes them at the cost of the kernel's protections and tooling. Filtering system calls hardens a process at the cost of breaking code that needs an unexpected call.

In an interview

Use strace in debugging stories and the boundary in performance ones: "strace showed the worker blocked in futex, so it was waiting on a lock rather than I/O, which led us to the contended connection pool." Or: "Serving large files uses sendfile, so data never crosses into user space; that halved CPU per gigabyte served."