Count dropped execs instead of losing them silently

README.md

@@ -290,12 +290,16 @@ A BPF program that loads on your laptop can be rejected by an older kernel's ver
 
 `make veristat` loads every program through the verifier on your own kernel and reports per-program complexity. [`.github/workflows/kernel-matrix.yml`](.github/workflows/kernel-matrix.yml) builds the object and boots each kernel in its matrix in a VM, failing if any verifier rejects it. Run the same matrix locally on Linux with KVM using `make veristat-matrix`.
 
-The aggregation has its own suite, which needs no kernel:
+Three suites, split by what they can reach:
 
 ```sh
-node test/heuristics.test.mjs
+node test/heuristics.test.mjs   # folding + outlier scoring — no kernel needed
+bash test/capture.sh            # the kernel seam: seeding, fork propagation, threads
+bash test/drops.sh              # ring-buffer drop accounting under a fork storm
 ```
 
+The first replays recorded captures through `lib/model.js`, so it runs anywhere node does. The other two need a Linux box with the probe built, because they exercise the path from kernel to traced set to ring buffer, which is where two whole-subtree bugs lived: a scope that only ever spread forward through `fork`, so anything already running was invisible, and a parent lookup keyed by tid against a map keyed by tgid, which silently dropped every process spawned by a pre-existing thread. Neither was reachable from a fixture replay.
+
 It runs the folding and the outlier scoring against five recorded captures in `test/` and asserts both halves of the claim: three benign builds produce no findings, and the adversarial ones produce the expected findings. The captures are real probe output rather than synthesized fixtures, deliberately: an earlier synthetic version of this suite passed while the heuristics were badly wrong.
 
 ## Try it without real traffic
@@ -325,6 +329,7 @@ Container mode additionally needs Docker reachable from the host running the pro
 > `exectop` is observability, not enforcement. It tells you what was launched; it does not stop, delay, or modify anything. For a kernel-enforced boundary around what a process can touch, [`agent-lock`](https://github.com/yeet-src/agent-lock) is the sibling that blocks rather than reports.
 
 - **Anything that doesn't exec.** A dependency that does its damage inside Node, Python, or the JVM without launching a program is invisible here. Fetching a URL with `fetch()` looks like nothing; fetching it with `curl` is a row. This is the boundary that matters most when reasoning about what the findings panel can and cannot catch.
+- **Execs beyond about 2,700 a second.** Delivery from the kernel is bound by bytes moved rather than events, and a record carries a 1 KiB argument window, so a parallel fork storm outruns the ring buffer. Measured: 96,000 execs fired across 24 workers, 36,000 captured and 60,000 dropped. The count is not lost, it is reported: the verdict line shows `59,431 dropped` and the headless report says the numbers below it are a floor. Normal builds are nowhere near this (a real `npm install` is tens per second), but a `make -j24` can reach it.
 - **Arguments past 1 KiB.** The kernel copies a fixed 1024-byte window of the argument blob and marks the record truncated. A very long compiler invocation is cut off; the program, its flags and the timing stay correct.
 - **Children that already existed** when you attach with `--pid` or `--container`. Membership propagates at `fork`, so a process that forked before you attached is outside the set until it forks again. Launch mode has no such gap, which is the reason to prefer it.
 - **Which files a process touched, or what it sent.** This is process launches only. For file access see [`agent-lock`](https://github.com/yeet-src/agent-lock), for HTTP see [`container-traffic`](https://github.com/yeet-src/container-traffic), for raw packets [`pktscope`](https://github.com/yeet-src/pktscope).

src/bpf/exectop.bpf.c

@@ -46,9 +46,15 @@ struct exec_event {
 	__u8  args[ARGV_BUF];
 };
 
+
 struct {
 	__uint(type, BPF_MAP_TYPE_RINGBUF);
-	__uint(max_entries, 1 << 22); // 4 MiB — exec storms are bursty
+	// 32 MiB. A record is ~1 KiB (the argv window dominates), so 4 MiB held
+	// only ~4000 events and a parallel fork storm outran it in under a second:
+	// measured 7k of 36k execs arriving, the rest dropped silently. This is
+	// the cheapest lever, and the drop counter above reports what still slips
+	// through rather than letting a burst quietly truncate the picture.
+	__uint(max_entries, 1 << 25);
 } events SEC(".maps");
 
 // The traced set: tgid -> depth below the root. Seeded from userspace with the
@@ -82,6 +88,23 @@ struct {
 	__type(value, __u64);
 } fork_ts SEC(".maps");
 
+// Event counters, read by JS. `emitted` and `dropped` are the two numbers that
+// matter: a burst that outruns the ring buffer loses execs, and losing them
+// silently is worse than the loss. Measured on a 12-way fork storm, 36k execs
+// in a few seconds, only ~7k arrived — 80% gone with nothing on screen to say
+// so. The UI now shows a dropped count when it is non-zero.
+struct counters {
+	__u64 emitted;
+	__u64 dropped;
+};
+
+struct {
+	__uint(type, BPF_MAP_TYPE_ARRAY);
+	__uint(max_entries, 1);
+	__type(key, __u32);
+	__type(value, struct counters);
+} stats_map SEC(".maps");
+
 // Set from JS before the subscription opens (probe.bss). When non-zero, only
 // tasks in this cgroup are eligible — the cgroup is the scope and `traced`
 // narrows it to the actual subtree. When zero, `traced` alone decides.
@@ -102,6 +125,9 @@ static __always_inline int in_scope(__u32 tgid, __u32 *depth_out)
 	struct traced_val *d = bpf_map_lookup_elem(&traced, &k);
 	if (!d)
 		return 0;
+	// target_cgid is inert: the traced tgid set is the scope, seeded from the
+	// process graph and grown at fork. Kept as a knob for a future
+	// cgroup-only mode, and it reads as 0 unless something patches it.
 	if (target_cgid && bpf_get_current_cgroup_id() != target_cgid)
 		return 0;
 	*depth_out = d->depth;
@@ -198,6 +224,22 @@ int on_exec(struct trace_event_raw_sched_process_exec *ctx)
 		e->args_len = (n == 0) ? (__u32)len : 0;
 	}
 
-	bpf_ringbuf_output(&events, e, sizeof(*e), 0);
+	// A variable-length emit (HDR_BYTES + args_len) was tried here to raise
+	// throughput, since the ring is bound by bytes moved and the median argv
+	// blob is only 48 bytes against a 1024-byte field. It does not work: the
+	// ring buffer is bound on the JS side with `btf_struct: "exec_event"`, so
+	// the consumer decodes fixed-size records and a short write delivers
+	// nothing at all (measured: 0 events received). Raising the ceiling would
+	// mean a smaller ARGV_BUF, which truncates 12% of real records, or a
+	// second smaller event type. Left as a documented limit instead: the
+	// drop counter reports what is lost rather than hiding it.
+	long sent = bpf_ringbuf_output(&events, e, sizeof(*e), 0);
+	struct counters *c = bpf_map_lookup_elem(&stats_map, &zero);
+	if (c) {
+		if (sent < 0)
+			__sync_fetch_and_add(&c->dropped, 1);
+		else
+			__sync_fetch_and_add(&c->emitted, 1);
+	}
 	return 0;
 }

src/components/verdict.jsx

@@ -2,9 +2,9 @@
 // before you read anything else — the total, the rate, a sparkline, the
 // dominant behavior, and whether anything looks out of place.
 import { Box, Text } from "yeet:tui";
-import { C_BAD, C_DIM, C_FAINT, C_OK, C_TITLE, BUCKET, fmtAge, fmtCount, fmtRate, pad, sparkline } from "@/lib/format.js";
+import { C_BAD, C_DIM, C_FAINT, C_OK, C_TITLE, C_WARN, BUCKET, fmtAge, fmtCount, fmtRate, pad, sparkline } from "@/lib/format.js";
 
-export default ({ tick, stats, buckets, outliers, idle, width: W }) => (
+export default ({ tick, stats, buckets, outliers, idle, dropped, width: W }) => (
   <Box direction="column" height="2">
     <Text height="1" break="none">
       {() => {
@@ -59,6 +59,13 @@ export default ({ tick, stats, buckets, outliers, idle, width: W }) => (
         if (quiet != null && quiet > 10) {
           runs.push(<Text fg={C_DIM}>{`  ·  quiet for ${fmtAge(quiet)}`}</Text>);
         }
+        // A dropped exec means the counts below are a floor, not a total. Say
+        // so where the totals are, rather than letting a burst quietly
+        // truncate the picture.
+        const lost = dropped?.() ?? 0;
+        if (lost > 0) {
+          runs.push(<Text bold fg={C_WARN}>{`  ·  ${fmtCount(lost)} dropped`}</Text>);
+        }
         // Clear the rest of the line (see above — no erase-in-line exists).
         const used = runs.reduce((n, r) => n + String(r?.props?.children ?? "").length, 0);
         runs.push(<Text>{" ".repeat(Math.max(0, W() - used - 2))}</Text>);

src/main.jsx

@@ -19,7 +19,7 @@
  */
 import { Box, Text, computed, mount, signal } from "yeet:tui";
 import {
-  buckets, flashes, folds, idle, outliers, paused, seedExisting, seedRoot, setCgroup, setPaused, stats, status, tick,
+  buckets, dropped, flashes, folds, idle, outliers, paused, seedExisting, seedRoot, setPaused, stats, status, tick,
 } from "@/probes/exec.js";
 import { commOf, containerRoot, descendantsOf, listContainers, procTable } from "@/lib/scope.js";
 import { C_FAINT, C_DIM } from "@/lib/format.js";
@@ -211,7 +211,7 @@ const Root = (size) => {
   return (
     <Box>
       <TitleBar scope={scope} status={status} paused={paused} />
-      <Verdict tick={tick} stats={stats} buckets={buckets} outliers={outliers} idle={idle} width={width} />
+      <Verdict tick={tick} stats={stats} buckets={buckets} outliers={outliers} idle={idle} dropped={dropped} width={width} />
       <Rule label="doing" width={width} />
       <Buckets tick={tick} stats={stats} buckets={buckets} width={width} maxRows={6} />
       <Rule label="doesn't fit" width={width} focused={outFocused} />

src/probes/capture.js

@@ -9,14 +9,26 @@
 //
 // It also writes the raw normalized stream to /tmp/exectop-capture.json so a
 // run can be replayed against changed heuristics without re-running the load.
-import { HashMap, RingBuf } from "yeet:bpf";
+import { ArrayMap, HashMap, RingBuf } from "yeet:bpf";
 import { control } from "./probe.js";
 import { normalize } from "../lib/argv.js";
 import { createModel } from "../lib/model.js";
 import { descendantsOf, procTable } from "../lib/scope.js";
 
 const traced = new HashMap(control, "traced");
 const events = new RingBuf(control, "events");
+const statsMap = new ArrayMap(control, "stats_map");
+
+// Kernel-side drop count. A non-zero value means the ring filled faster than
+// it drained, so every number in the report is a floor rather than a total.
+const dropped = async () => {
+  try {
+    const c = await statsMap.lookup(0);
+    return Number(c?.dropped ?? 0);
+  } catch {
+    return 0;
+  }
+};
 
 const root = Number(yeet.args?._?.[0] ?? 0);
 const secs = Number(yeet.args?._?.[1] ?? 30);
@@ -42,14 +54,16 @@ const sub = await events.subscribe((w) => {
   model.add(e);
 });
 
-setTimeout(() => {
-  report();
+setTimeout(async () => {
+  await report();
   yeet.exit();
 }, secs * 1000);
 
-function report() {
+async function report() {
   const total = model.total;
-  console.log(`\n=== ${total} execs in ${model.elapsed.toFixed(0)}s (${model.rate().toFixed(1)}/s) ===\n`);
+  const lost = await dropped();
+  console.log(`\n=== ${total} execs in ${model.elapsed.toFixed(0)}s (${model.rate().toFixed(1)}/s)${lost ? ` — ${lost} DROPPED` : ""} ===\n`);
+  if (lost) console.log(`[warn] ${lost} execs were dropped: the ring buffer filled faster than it drained, so every count below is a floor.\n`);
 
   console.log("-- doing --");
   for (const b of model.buckets()) {

src/probes/exec.js

@@ -2,14 +2,26 @@
 // ring buffer once, feeds the pure aggregation in lib/model.js, and exposes
 // what the UI reads. The only BPF-aware module besides probe.js.
 import { signal } from "yeet:tui";
-import { DataSec, HashMap, RingBuf } from "yeet:bpf";
+import { ArrayMap, DataSec, HashMap, RingBuf } from "yeet:bpf";
 import { control } from "./probe.js";
 import { normalize } from "../lib/argv.js";
 import { createModel, foldKey } from "../lib/model.js";
 
 const events = new RingBuf(control, "events");
 const traced = new HashMap(control, "traced");
 const bss = new DataSec(control, "probe.bss");
+const statsMap = new ArrayMap(control, "stats_map");
+
+// How many execs the kernel emitted vs dropped. A drop means the ring buffer
+// filled faster than userspace drained it, so the numbers on screen are a
+// floor rather than a count. Polled rather than streamed: it only needs to be
+// right when it is read.
+// A SIGNAL, not a plain variable. The verdict line reads this inside a thunk,
+// and a thunk only re-renders when a signal it read changes — a plain
+// function returning a mutated local never triggers a repaint, so the warning
+// was computed correctly and never drawn.
+export const droppedCount = signal(0);
+export const dropped = () => droppedCount.get();
 
 const model = createModel();
 
@@ -56,10 +68,11 @@ export async function seedExisting(rows) {
   return n;
 }
 
-// Narrow to a cgroup (0 = the pid subtree alone, no cgroup filter).
-export async function setCgroup(cgid) {
-  await bss.patch({ target_cgid: cgid });
-}
+// Note: there is deliberately no cgroup narrowing. An earlier design set a
+// target_cgid in .bss and filtered on it in-kernel, but nothing ever called it,
+// so container mode was always a plain pid subtree. Now that the existing tree
+// is seeded from the process graph, the tgid set IS the scope, and a second
+// overlapping filter would only add a way for the two to disagree.
 
 // One subscription, driving one model.
 //
@@ -105,6 +118,15 @@ await events.subscribe((w) => {
 // per event would spend the budget on work the eye cannot see. The flash decay
 // also needs a heartbeat, so tick advances even when the stream is briefly
 // quiet but flashes are still fading.
+// Poll the kernel counters once a second. Cheap, and only used to warn.
+setInterval(async () => {
+  try {
+    const c = await statsMap.lookup(0);
+    const n = Number(c?.dropped ?? 0);
+    if (n !== droppedCount.get()) droppedCount.set(n);
+  } catch { /* counters are advisory; never break the UI over them */ }
+}, 1000);
+
 setInterval(() => {
   const fading = flashes.size > 0;
   // Tick at least once a second even with nothing arriving, so the idle line

src/probes/probe.js

@@ -18,6 +18,7 @@ async function load() {
         .bind("events", { kind: "ringbuf", btf_struct: "exec_event" })
         .bind("traced", { kind: "hash" })
         .bind("fork_ts", { kind: "hash" })
+        .bind("stats_map", { kind: "array" })
         .bind("probe.bss", { kind: "data" })
         .start();
     } catch (err) { last = err; }

test/drops.sh

@@ -0,0 +1,81 @@
+#!/usr/bin/env bash
+# Ring-buffer drop accounting.
+#
+# Delivery is bound by bytes moved, not events: a 1 KiB record caps the ring at
+# roughly 2,700 execs/s. A parallel fork storm outruns that by an order of
+# magnitude, and the events that do not fit are lost. That is a real limit and
+# it is fine; losing them SILENTLY is not, because every count on screen then
+# reads as a total when it is a floor.
+#
+# This asserts the accounting closes: captured + dropped == what actually ran.
+# Run it in the VM after `make`.
+set -uo pipefail
+cd "$(dirname "$0")/.."
+
+WORKERS=${WORKERS:-24}
+PER=${PER:-4000}
+EXPECT=$((WORKERS * PER))
+
+g="\033[32m"; r="\033[31m"; d="\033[2m"; z="\033[0m"
+fail=0
+ok()   { printf "${g}  ok    %s${z}\n" "$1"; }
+bad()  { printf "${r}  FAIL  %s${z}\n" "$1"; fail=$((fail+1)); }
+
+cat > /tmp/exectop-storm.sh <<STORM
+#!/bin/bash
+sleep 3
+for w in \$(seq 1 $WORKERS); do ( for i in \$(seq 1 $PER); do /bin/true; done ) & done
+wait
+sleep 6
+STORM
+chmod +x /tmp/exectop-storm.sh
+
+printf "${d}firing %d execs across %d workers …${z}\n" "$EXPECT" "$WORKERS"
+setsid /tmp/exectop-storm.sh >/dev/null 2>&1 </dev/null &
+sleep 1
+ROOT=$(pgrep -n -f /tmp/exectop-storm.sh)
+[ -n "$ROOT" ] || { echo "could not start the storm"; exit 1; }
+
+OUT=$(timeout 60 yeet run src/probes/capture.js -- "$ROOT" 25 2>&1)
+CAPTURED=$(printf '%s' "$OUT" | grep -oE '^=== [0-9]+ execs' | grep -oE '[0-9]+' | head -1)
+DROPPED=$(printf '%s' "$OUT" | grep -oE '— [0-9]+ DROPPED' | grep -oE '[0-9]+' | head -1)
+DROPPED=${DROPPED:-0}
+CAPTURED=${CAPTURED:-0}
+TOTAL=$((CAPTURED + DROPPED))
+
+echo
+echo "captured=$CAPTURED  dropped=$DROPPED  sum=$TOTAL  expected>=$EXPECT"
+echo
+
+# The point of the test: nothing vanishes unaccounted for. Shell overhead adds
+# a little (seq, sleep, the subshells), so the sum is a floor, and the slack is
+# for scheduling, not for losses.
+if [ "$TOTAL" -ge "$EXPECT" ]; then
+  ok "accounting closes: captured + dropped >= $EXPECT"
+else
+  bad "accounting leaks: $TOTAL < $EXPECT — $((EXPECT - TOTAL)) execs vanished uncounted"
+fi
+
+# A storm this size must actually overrun the ring. If it does not, either the
+# machine got much faster or the storm is no longer a storm, and the drop path
+# is then untested rather than passing.
+if [ "$DROPPED" -gt 0 ]; then
+  ok "drops are reported, not silent ($DROPPED)"
+else
+  bad "no drops at $EXPECT execs — the drop path went untested; raise WORKERS/PER"
+fi
+
+# And the warning has to reach the reader.
+if printf '%s' "$OUT" | grep -q "were dropped"; then
+  ok "the report says the counts are a floor"
+else
+  bad "drops counted but never explained in the output"
+fi
+
+echo
+if [ "$fail" -eq 0 ]; then
+  printf "${g}PASS — every exec is either captured or counted as dropped${z}\n"
+else
+  printf "${r}FAIL — %d check(s) failed${z}\n" "$fail"
+fi
+exit "$fail"