SyscallGuard: Giving Every Linux Job a Budget for Expensive System Calls

November 18, 2026  11 min read  c · linux · seccomp · systems

How I used seccomp user notification to charge each job for its costly syscalls, so one noisy process slows itself down instead of the whole machine, without a kernel module or root.

Run a big build next to a chatty logger on the same machine and the build gets slower. That isn’t because the logger uses much CPU. It’s because it calls fsync a few thousand times a second, and every one of those calls makes the kernel and the disk do real work that the build then has to queue behind.

CPU and memory limits don’t catch this. cgroups can cap CPU time, memory and block I/O bandwidth, but they don’t ask how many expensive system calls a process makes. A job can stay well inside its CPU quota and still hurt everything around it by creating files, starting processes or flushing to disk in a tight loop.

SyscallGuard is my attempt at that missing limit. Each job gets a budget measured in system-call cost. When a job runs out, its expensive calls wait, so the busy job slows down and everyone else keeps their speed.

Charging per call, not per CPU second

Not all system calls cost the same. A read from the page cache is almost free, and an fsync can take milliseconds. SyscallGuard gives each costed syscall a price in cost units, and each job a token bucket that refills at its rate. A call that fits in the bucket goes through. A call that doesn’t either waits until enough budget comes back or fails with an errno.

There are three ways to set a job’s rate:

./build/sgd -r 3000 -l events.jsonl &               # global budget: 3000 units/s
./build/sg-launch -p 8 -n build -- make -j4         # priority share: high
./build/sg-launch -p 2 -n logger -- ./noisy-logger  # priority share: low
./build/sg-launch -r 500 -- ./batch                 # fixed 500 units/s

Priority jobs split the global budget in proportion to their priority, so the build at 8 gets four times the logger’s share. A fixed-rate job gets exactly what you asked for, whatever else is running. When a job ends, sgd prints what happened to it:

[sgd] job 2 (logger) ended after 1.778s: calls=8067 allowed=8067 waited=168 rejected=0 dropped=0 charged=8315 max_wait_ms=9.4

168 of the logger’s calls had to wait, the longest for 9.4 ms, and the build next to it never noticed.

Why seccomp user notification

To charge a call you have to see it before it runs. The obvious ways to do that each had a problem:

  • A kernel module sees everything but needs root to load, and it ties you to one kernel version.
  • ptrace stops the traced process on every syscall, cheap ones included, and a process can only have one tracer.
  • eBPF is good at measuring syscalls, but it can’t park a call for a few milliseconds and then let it continue.

Seccomp user notification (Linux 5.5+) does exactly what I needed. A process installs a seccomp filter, and for any syscall the filter returns SECCOMP_RET_USER_NOTIF on, the kernel suspends the calling thread and sends a notification to a supervisor through a file descriptor. The supervisor decides, replies, and the thread either continues or gets an error back. Any user can install a seccomp filter on their own processes, so none of this needs root.

SyscallGuard is two programs built around that:

  • sg-launch sets no_new_privs, installs the filter with SECCOMP_FILTER_FLAG_NEW_LISTENER, sends the listener fd to the supervisor over a Unix socket, and then execs your command.
  • sgd is the supervisor. It runs one epoll loop over every job’s listener, and for each notification it allows, holds or rejects the call.

Every sg-launch installs its own filter, so every job has its own listener. The fd a notification arrives on already tells sgd which job made the call, with no pid-to-job lookup and no race against pid reuse.

Keeping cheap calls in the kernel

A round trip to a userspace supervisor costs a context switch or two. Paying that on every read would make SyscallGuard the noisiest job on the machine. The filter that sg-launch installs is a small classic BPF program generated from the cost table, and it handles most calls without ever leaving the kernel:

/* Costed calls jump past the remaining checks and RET_ALLOW to RET_USER_NOTIF. */
for (i = 0; i < NTABLE; i++)
    *p++ = (struct sock_filter)BPF_JUMP(BPF_JMP | BPF_JEQ | BPF_K, (unsigned)table[i].nr,
                                        (unsigned char)(NTABLE - i), 0);
*p++ = (struct sock_filter)BPF_STMT(BPF_RET | BPF_K, SECCOMP_RET_ALLOW);
*p++ = (struct sock_filter)BPF_STMT(BPF_RET | BPF_K, SECCOMP_RET_USER_NOTIF);

Before that table, the filter does a few checks that are easy to forget:

  • Architecture. A process on the wrong architecture is killed, because syscall numbers mean different things on different ABIs.
  • x32. On x86-64, x32 syscall numbers (bit 30 set) get ENOSYS. Otherwise a job could reach fsync under a number the table doesn’t list and skip the bill.
  • io_uring. io_uring_setup returns EPERM, because one io_uring_enter can submit hundreds of operations and per-call accounting would see only one.
  • clone3. clone3 returns ENOSYS. Its flags live in a struct in memory, out of the filter’s reach. glibc falls back to clone(), whose flags are in a register, so sgd can tell a new thread from a new process and price them differently.

What a call costs

The seed costs in the source only rank the classes: a write is cheap, an fsync is expensive. To get real numbers, tools/sg_calibrate.py runs a workload under bpftrace, records a latency histogram per syscall on the raw_syscalls tracepoints, and uses each call’s median latency as its cost, at 1 unit per microsecond. On my WSL2 machine (kernel 6.6), two runs agreed within 3%:

Syscall Seed Calibrated
fsync 20 1504
vfork 30 181
clone (process) 30 167
execve 40 112
mkdir 3 71
clone (thread) 10 45
openat 3 4
write 1 3

The seeds got the order right and the scale badly wrong. An fsync turned out to cost 75 times what I’d guessed. Calibration needs root for bpftrace. Enforcing the calibrated costs afterwards doesn’t.

Holding calls without starving anyone

Letting a call through or failing it is the easy part. Making a call wait without causing new problems took most of the design.

Priority with aging. Waiting calls sit in a heap ordered by priority. A low-priority call that has waited long enough has to go eventually, so every -a milliseconds of waiting (500 by default) raises a call’s priority by one. Done naively, that means re-keying the whole heap all the time. It doesn’t have to: a call’s effective priority is prio + wait/q, where wait = now − arrival. At any given moment now is the same for every waiting call, so sorting by effective priority gives the same order as sorting by a fixed key:

return (int64_t)j->acct.prio * cfg.aging_ms * 1000 - arrive_us;

The key is computed once, when the call arrives, and the heap never needs rebuilding.

Per-job FIFO. A job’s later call never overtakes its own earlier waiting call, even if it’s cheaper. Programs assume their syscalls run in the order they made them.

Strict priority on the global bucket. If a high-priority call is waiting for the shared global bucket, new low-priority calls have to wait behind it. Otherwise a stream of small cheap calls would drain the global bucket first, and the expensive call would never fit.

Paced rejections. A program that gets EAGAIN from fork often just tries again straight away. If sgd answered each rejection at once, that retry loop would keep the supervisor busy. Rejections are held for 10 ms by default, and the rejected thread sleeps instead of spinning.

Fork policies. Starting a process is the most expensive thing a job does, and the right response to running out of budget depends on the job. The default, brief, makes a short-of-budget fork wait up to 500 ms and then fail. A throttled build slows down, and a fork bomb still hits a wall quickly. wait treats fork like any other call, and reject fails it right away for scripts you don’t trust.

Borrowing

Strict buckets waste budget. If the build is idle for a moment, its unused budget is wasted while the logger waits. So when a call costs more than a job holds, sgd tries to borrow before making the call wait. The rules keep that from turning into a free-for-all:

  • A lender has no waiting calls, owes nothing itself, and keeps 30% of its capacity in reserve.
  • A borrower can owe at most 50% of its capacity.
  • Lenders must have the same or lower priority than the borrower, unless they’ve been idle for a second.
  • Each loan is at least 10% of the borrower’s capacity, so a throttled job takes a few bigger loans instead of one tiny loan per call.

Tokens move from lender to borrower, so borrowing never creates budget. Half of every later refill goes to repaying the borrower’s oldest loan. As a result, borrowing absorbs bursts but doesn’t raise a job’s sustained rate. With the defaults it can shorten a throttled job by at most half a second. Every loan’s opening, repayment and write-off goes into the event log:

{"ev":"loan_open","loan":1,"from":1,"to":2,"amount":264.987,"repaid":0.000,"age_us":3}
{"ev":"loan_repaid","loan":1,"from":1,"to":2,"amount":264.987,"repaid":264.987,"age_us":532769}

Security: a budget, not a sandbox

SyscallGuard decides how fast a job may make expensive calls, not which calls it may make. It’s still worth being precise about what it guarantees, because the mechanism it uses has a well-known trap.

The TOCTOU trap. When a supervisor answers with SECCOMP_USER_NOTIF_FLAG_CONTINUE, the kernel runs the original syscall with whatever arguments are in memory at that moment. A supervisor that reads a path from the process’s memory, checks it, and then says “continue” can be fooled: another thread rewrites the path between the check and the call. This is why the kernel docs warn against using seccomp notification for security decisions. SyscallGuard never reads the job’s memory. A call’s cost comes from the syscall number and register arguments only: clone flags, and the size argument for calls charged by size. These are copied into the notification when the call stops and can’t be changed afterwards.

no_new_privs. The kernel only lets an unprivileged process install a seccomp filter after it has set no_new_privs, and that flag does useful work here too. Nothing inside a job can gain privileges from a setuid binary or a file capability, so a job can’t leave its limits by running something like sudo. The cost is that sudo, ping (which uses a file capability) and tcpdump don’t work inside a job. Ordinary networking does. For ping, ICMP datagram sockets via net.ipv4.ping_group_range need no capability at all.

A hard stop. A budget limits how fast a job can spend, but a fork storm still grows up to the system’s process limit. sg-launch puts every job in its own process group, so sgctl kill <job> signals the whole tree at once.

Small surfaces. The control socket is mode 0600, so only you can change your jobs. The Prometheus endpoint listens on 127.0.0.1 only.

Seeing what happened

A limiter you can’t observe is a limiter you’ll eventually turn off. sgctl status shows every job’s rate, tokens, waiting calls and loans live. sgctl set-prio and sgctl set-rate change a job while it runs, and sgd -m serves Prometheus metrics. The JSON-lines event log records every decision, and tools/sg_analyze.py turns it into per-job wait percentiles, rejection rates per syscall class, budget utilisation, loan statistics, and Jain’s fairness index over charged units per priority weight. A score of 1.0 means each job got service in exact proportion to its priority.

Limitations

A few things I couldn’t design away:

  • Filters outlive the supervisor. A seccomp filter can’t be removed from a running process. If sgd exits, still-running jobs see ENOSYS on every costed call.
  • Calls are charged before they run. A failed execve during a PATH search still costs a job its budget.
  • Synchronous wake-up is partial. Linux 6.6’s synchronous wake-up speeds up the reply to a job, but sgd sleeps in epoll, which doesn’t pass the same-CPU hint on. Under very heavy notification load, sgd can get dragged across CPUs.
  • sendmsg and recvmsg are free. sg-launch needs sendmsg to hand over the listener before any costed call can be charged, so both stay on the fast path.

Try it

You need Linux 5.5 or newer (5.19+ recommended, 6.6+ for synchronous wake-up) on x86-64 or aarch64, plus gcc and make:

git clone https://github.com/swayam5342/SGD
cd SGD
make && make test

About 2,300 lines of C hold the core library, the supervisor, the launcher and the control client, with unit, Python and integration tests alongside. It’s MIT licensed, and the website walks through the design. If you try fork-storm experiments, use a VM or a spare machine, not a shared host.


← all posts