feat(memtrack): attach allocator probes on demand

[not-matthias]

Attach allocator uprobes only to libraries the tracked process tree actually maps, instead of pre-attaching every allocator discovered on the system at startup.

A BPF fentry on security_mmap_file signals the first mapping of each unknown executable inode. A background worker SIGSTOPs the mapping processes, classifies the file by its allocator symbols, attaches probes, and resumes them. This:

• covers dlopen'd allocators and statically linked allocators in spawned children (previously missed entirely)
• skips libraries the benchmark never loads
• removes the startup discovery scan (system-wide library glob + build-dir walk), which was slow on nix-heavy systems

The eager discovery path and CODSPEED_MEMTRACK_BINARIES are removed since the watcher supersedes them (breaking: the env var is no longer read).

The crate API is now Tracker (owns the attach worker) and Session (owns the spawned child and its event pipeline); ring buffers are polled through a generic RingBufferPoller with caller-supplied parsing, replacing the keepalive forwarding thread.

Review focus: the stop-the-world attach in attach_worker.rs (fixpoint stop + synchronous ring buffer drain) — its correctness argument is that a full consume() after all producers are stopped cannot miss requests.

Fixes COD-1801

[codspeed-hq]

Merging this PR will not alter performance

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

✅ 17 untouched benchmarks

---

_{Comparing cod-1801-only-attach-to-used-libraries-in-memtrack (6794d28) with main (c17b77f)}

[greptile-apps]

Greptile Summary

This PR replaces the eager startup allocator discovery scan (system-wide glob + CODSPEED_MEMTRACK_BINARIES) with an on-demand watcher: a BPF fentry/security_mmap_file probe triggers a background worker that SIGSTOP-freezes the mapping process tree, classifies the new inode, attaches uprobes, updates known_inodes, then SIGCONTs. The Tracker/Session API split and generic RingBufferPoller are clean improvements.

• On-demand attach worker (attach_worker.rs, proc_fs.rs, attach.h): fixpoint stop-the-world loop correctly drains the ring buffer only after confirming all known pids are stopped; ContGuard guarantees SIGCONT on every exit path including panics.
• Spawn stub (spawn.rs): shell-level kill -STOP $$; exec "$@" sidesteps the pre_exec/SIGSTOP deadlock; waitpid(WUNTRACED) blocks until the child stops before tracking is armed.
• Breaking removal of CODSPEED_MEMTRACK_BINARIES: the env var is no longer read; callers that relied on it for static-allocator discovery now get equivalent coverage from the watcher automatically.

Confidence Score: 4/5

Safe to merge; the stop-the-world drain correctness is sound for the intended use case of well-behaved benchmarks that clean up children before the root exits.

The core attach correctness argument holds: ring-buffer entries are submitted before SIGSTOP is sent, the fixpoint loop stops all observed pids before draining, and ContGuard guarantees resume on every exit path. The wait_all_stopped function has a subtle all_stopped=true default that could prematurely return if all thread-stat reads are skipped, potentially violating the drain guarantee for that pid. The shutdown drain loop in Worker::run has no iteration bound, meaning it could block indefinitely if tracked children outlive the root process.

crates/memtrack/src/ebpf/proc_fs.rs (wait_all_stopped all_stopped default) and crates/memtrack/src/ebpf/attach_worker.rs (shutdown drain loop bound)

Important Files Changed

Filename	Overview
crates/memtrack/src/ebpf/attach_worker.rs	New background worker that stop-the-world attaches allocator uprobes on demand; fixpoint stop logic looks correct, ContGuard SIGCONT-on-drop guarantees resume on all exit paths, panic/drain error handling wired up
crates/memtrack/src/ebpf/proc_fs.rs	New /proc helpers; wait_all_stopped has a subtle all_stopped=true default that prematurely declares success when no thread stat is readable; resolve_mapping and dev-encoding logic otherwise correct
crates/memtrack/src/ebpf/c/attach.h	New BPF fentry on security_mmap_file; correctly submits ring-buffer entry before bpf_send_signal(SIGSTOP), drop counter on overflow handled, known_inodes dedup prevents repeated requests
crates/memtrack/src/ebpf/poller.rs	RingBufferPoller refactored to generic parse-fn + mpsc::Sender; drain() protocol via ctl channel is sound; shutdown via ctl drop triggers final consume() before thread exit
crates/memtrack/src/ebpf/tracker.rs	Tracker now owns Arc<Mutex> and AttachWorker; spawn() atomically arms tracking while process is stopped then resumes; API simplified
crates/memtrack/src/ebpf/spawn.rs	stopped_command shell-stub sidesteps the pre_exec SIGSTOP deadlock; spawn_stopped uses waitpid(WUNTRACED) to detect the stop before returning
crates/memtrack/src/ebpf/events.rs	New AttachRequest::parse uses same unsafe pointer-cast pattern as parse_event; relies on BPF ring-buffer 8-byte alignment guarantee for the u64 fields in struct attach_request
crates/memtrack/src/session.rs	New Session owns the child process and event pipeline; dropping it stops the event poller and closes the channel, which unblocks rx.iter() collect in tests
crates/memtrack/src/main.rs	Removed Mutex in favour of Arc; session/tracker shutdown order correct; finish() called after encode pipeline drains
crates/memtrack/src/ebpf/c/utils/process_tracking.h	is_enabled() now fails closed (returns 0) on unexpected map-lookup failure instead of open; correct improvement
crates/memtrack/tests/dlopen_tests.rs	New dlopen tests compile fake allocator shared libs at runtime and verify all 100 alloc/free events are captured after on-demand attach
src/cli/exec/mod.rs	CODSPEED_MEMTRACK_BINARIES wiring removed; execute_config no longer needs to resolve binary paths since the watcher supersedes it

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant CLI as main.rs
    participant T as Tracker
    participant W as AttachWorker (bg thread)
    participant BPF as BPF (kernel)
    participant S as Stopped Process

    CLI->>T: Tracker::new()
    T->>BPF: attach fentry/security_mmap_file + tracepoints
    T->>W: AttachWorker::start()
    Note over W: poll_attach ring buffer → mpsc channel

    CLI->>T: tracker.spawn(cmd)
    T->>S: spawn_stopped (sh stub → SIGSTOP)
    T->>BPF: add_tracked_pid(pid)
    T->>S: resume(SIGCONT)

    S->>BPF: mmap(libfoo.so, PROT_EXEC)
    BPF->>BPF: submit attach_request(pid,dev,ino)
    BPF->>S: bpf_send_signal(SIGSTOP)
    Note over S: process stops

    W->>W: recv attach_request batch
    W->>S: wait_all_stopped(pid, 1s)
    W->>W: poller.drain() → consume ring buffer
    W->>W: resolve_mapping(pid, dev, ino)
    W->>BPF: attach_allocator_probes(libjemalloc)
    W->>BPF: insert_known_inode(dev, ino)
    W->>S: ContGuard drop → SIGCONT

    Note over S: process resumes, allocations now tracked

    CLI->>T: tracker.finish()
    T->>W: "shutdown=true"
    W->>W: final drain + process remaining batch
    W-->>T: "Ok() or Err (dropped count > 0)"

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant CLI as main.rs
    participant T as Tracker
    participant W as AttachWorker (bg thread)
    participant BPF as BPF (kernel)
    participant S as Stopped Process

    CLI->>T: Tracker::new()
    T->>BPF: attach fentry/security_mmap_file + tracepoints
    T->>W: AttachWorker::start()
    Note over W: poll_attach ring buffer → mpsc channel

    CLI->>T: tracker.spawn(cmd)
    T->>S: spawn_stopped (sh stub → SIGSTOP)
    T->>BPF: add_tracked_pid(pid)
    T->>S: resume(SIGCONT)

    S->>BPF: mmap(libfoo.so, PROT_EXEC)
    BPF->>BPF: submit attach_request(pid,dev,ino)
    BPF->>S: bpf_send_signal(SIGSTOP)
    Note over S: process stops

    W->>W: recv attach_request batch
    W->>S: wait_all_stopped(pid, 1s)
    W->>W: poller.drain() → consume ring buffer
    W->>W: resolve_mapping(pid, dev, ino)
    W->>BPF: attach_allocator_probes(libjemalloc)
    W->>BPF: insert_known_inode(dev, ino)
    W->>S: ContGuard drop → SIGCONT

    Note over S: process resumes, allocations now tracked

    CLI->>T: tracker.finish()
    T->>W: "shutdown=true"
    W->>W: final drain + process remaining batch
    W-->>T: "Ok() or Err (dropped count > 0)"

Loading

_{Reviews (9): Last reviewed commit: "fix(memtrack): join attach worker thread..." | Re-trigger Greptile}

[GuillaumeLagrange]

Stopping here becuase I want us to dig the TRACEME thing before leaning in the stop wrapper

[GuillaumeLagrange]

lgtm, I recommend you rebase on top of #455 before merging though 😅

[not-matthias]

Yeah, that was the plan :)