Follow-up to #910/#912 with hard measurements. Generated seccomp profiles are missing syscalls the kernel demonstrably recorded, which is the root of the "generated profile kills the workload" class (#912).
Method
A probe container runs a single python process (no fork/exec) issuing distinct, rare syscalls at known times: chdir+mkdir at t=0, rename t+10s, link t+25s, symlink t+45s, rmdir t+70s, socket t+100s, then pure sleep. In parallel, a privileged pod dumps the syscalls_per_mntns map (bpftool, 1s interval) for the probe mntns, so kernel ground truth is captured before and between the 30s BPF_MAP_LOOKUP_AND_DELETE_BATCH drains. Learning period 5m. node-agent vendored at v0.3.179.
Kernel ground truth (measured)
The eBPF side is flawless. The map entry for the probe mntns contained, drain window by drain window:
| window |
contents |
| 0 (container start) |
62 syscalls: full startup incl. execve, brk, read, mmap, chdir, rename + footer byte |
| 1 |
chdir, mkdir, rename (markers t0/t10) |
| 2 |
link, symlink (t25/t45) |
| 3 |
rmdir (t70) |
| 4 |
socket (t100) |
| 5-6 |
clock_nanosleep (steady state) |
Union: 67 distinct syscalls. Every marker present. The deliberate no-container-filtering design in program.bpf.c works exactly as intended.
What the profile contained
9 syscalls: clock_nanosleep, close, fstat, getdents64, lseek, openat, prctl, tgkill, write.
58 of 67 lost (87%). All 6 markers lost. execve, read, mmap, brk lost. The losses are not a time window (they span every drain), and not whole-packet (survivors are a strict subset of window 0 plus repeats). node-agent logged zero errors or warnings during the run: no queue-full drops, no enqueue failures, no splits.
Where it is NOT
Ruled out with evidence on the same cluster:
Remaining scope: the userspace path from the mapiter datasource emission through decodeSyscalls/eventCallback/ReportSyscall into containerData. Note ReportSyscall errors with ErrContainerNotFound are swallowed silently (logEventError), so any attribution failure there is invisible.
Why this is severe
The drain deletes the map entry, so each observation exists exactly once; whatever this path drops is unrecoverable. A workload that starts once and idles (most servers) gets a profile missing its startup and one-shot syscalls, and a seccomp profile generated from it prevents the container from starting (#912) or breaks it at runtime.
Suggested directions
- Make the syscalls pipeline idempotent instead of once-only: stop draining (plain lookup instead of lookup-and-delete, entries removed on container removal), so every fetch re-emits the cumulative bitmap and any single loss self-heals on the next fetch. The bitmap is 501 bytes per mntns; re-emission cost is negligible and downstream is already a dedup/union.
- Make every drop in the userspace path loud:
ReportSyscall attribution failures must at minimum increment a metric, not vanish.
- Keep a completeness guard in generators regardless (armosec side: runtime baseline union), because a delta pipeline can never prove completeness.
Full repro manifests and the bpftool watcher script available on request.
Follow-up to #910/#912 with hard measurements. Generated seccomp profiles are missing syscalls the kernel demonstrably recorded, which is the root of the "generated profile kills the workload" class (#912).
Method
A probe container runs a single python process (no fork/exec) issuing distinct, rare syscalls at known times: chdir+mkdir at t=0, rename t+10s, link t+25s, symlink t+45s, rmdir t+70s, socket t+100s, then pure sleep. In parallel, a privileged pod dumps the
syscalls_per_mntnsmap (bpftool, 1s interval) for the probe mntns, so kernel ground truth is captured before and between the 30sBPF_MAP_LOOKUP_AND_DELETE_BATCHdrains. Learning period 5m. node-agent vendored at v0.3.179.Kernel ground truth (measured)
The eBPF side is flawless. The map entry for the probe mntns contained, drain window by drain window:
Union: 67 distinct syscalls. Every marker present. The deliberate no-container-filtering design in
program.bpf.cworks exactly as intended.What the profile contained
9 syscalls: clock_nanosleep, close, fstat, getdents64, lseek, openat, prctl, tgkill, write.
58 of 67 lost (87%). All 6 markers lost. execve, read, mmap, brk lost. The losses are not a time window (they span every drain), and not whole-packet (survivors are a strict subset of window 0 plus repeats). node-agent logged zero errors or warnings during the run: no queue-full drops, no enqueue failures, no splits.
Where it is NOT
Ruled out with evidence on the same cluster:
Remaining scope: the userspace path from the mapiter datasource emission through
decodeSyscalls/eventCallback/ReportSyscallinto containerData. NoteReportSyscallerrors withErrContainerNotFoundare swallowed silently (logEventError), so any attribution failure there is invisible.Why this is severe
The drain deletes the map entry, so each observation exists exactly once; whatever this path drops is unrecoverable. A workload that starts once and idles (most servers) gets a profile missing its startup and one-shot syscalls, and a seccomp profile generated from it prevents the container from starting (#912) or breaks it at runtime.
Suggested directions
ReportSyscallattribution failures must at minimum increment a metric, not vanish.Full repro manifests and the bpftool watcher script available on request.