gpu: attach to an already-running process, and keep its modules (#124)

[dpsoft]

gpu-cuda-profile could only exec.Command its own workload. Every deployment that matters
attaches to a process it did not start — a sidecar profiles a container the kubelet started, a node
collector profiles processes that were running before it was scheduled — which is why
examples/kubernetes/ carries a manifest marked not-runnable. No cluster configuration fixes a tool
that has to be the parent of what it measures.

-pid attaches to a running process. -duration bounds the run, -drain-every bounds the
timeline's memory, -wait-for-shim bounds the precondition.

Verified on an RTX 3090 against the same workload profiled both ways.

The main risk was late CFI, and it is retired

	A: launched	B: attached 8s late
SampledLaunches	1,006	13,894
StacksWalkedDWARF	1,006	13,894
StacksWalkedNoTables	0	0
StackWalkReachedRoot	1,006	13,894
UnwindEnrollRequests	1	0
KernelNamesUnresolved	0	0

UnwindEnrollRequests: 0 confirms the startup rendezvous never happened — the target passed
cuInit long before this command existed. StacksWalkedNoTables: 0 across 13,894 sampled launches
says the eager registration path carried all of it: a non-zero Config.PID makes Attach
compile that process's CFI tables synchronously, before the uprobe link exists and therefore before
any probe can fire. The window is not narrowed but absent. Without it a late attach would walk its
first stacks with no tables — the ~38% loss #49 measured.

Refused, not ignored

The producer's whole configuration — sampling period, PC-sampling tier — lives in the target's
environment and is fixed at cuInit. -period accepted in attach mode would change nothing and
would then be read off the command line afterwards as though it had, so the profile would be
interpreted at a rate it was not taken at. The tier is worse than inert: it gates whether an
execution may be claimed gpu_serialized="true", and a consumer reading its own shell's environment
would make that claim on nothing. Attach mode takes the zero value, which claims nothing and marks
executions "unknown" — correct-but-weaker, the right direction to be wrong in.

The classification is checked for totality by a test over the registered flag set. A flag added
later and classified nowhere would fall through the refusal and be silently ignored — the exact
failure the refusal prevents, reintroduced by omission. Mutation-tested both ways.

The long window exposed a real defect (commit 2)

The tool snapshotted once, at the end, holding a whole run inside a ring sized for an interval.
At ~5,500 launches/s the 65,536-entry ring fills in ~12s:

ANOMALY: launch cache evicted 45315 launches at capacity (65536 live)
ANOMALY: 45032 executions evicted from the timeline ring before this snapshot
         — that GPU time is missing from the profile entirely

Launch mode gets away with it because its workload stops; a collector never does, so the loss scales
with run length. Timeline.Snapshot already drains and was built to be called repeatedly — nothing
was calling it that way.

	before	after
samples in a 20s attach	65,536	111,056
executions evicted from the ring	45,032	0
launches whose executions could not join	45,668	88 (attach boundary)

The drain deliberately does not c.Flush(), though the end-of-run path must: mid-run, launches
held for a sampled twin are usually a batch away rather than lost, and flushing on a timer would
release them stackless — a periodic drain that silently degraded attribution while claiming to
protect it.

And a reporting bug the drain introduced (commit 3)

Snapshot drains the rings but not the launch cache, so its counters are cumulative and were
re-reported, larger, every interval — one ongoing condition wearing the costume of four fresh
anomalies, unbounded over a collector's lifetime. Anomalies now report once per kind.

Stated, not hidden

The launch-cache anomaly is over-firing on its own terms. It fires on EvictedCapacity > 0
alone and asserts "their executions cannot join", which this run disproves — 111,056 samples from
~110,851 launches, final snapshot joining every execution exactly. Making it precise means teaching
LaunchCache to tell an evicted launch that already joined from one that never did, and reconciling
a cumulative counter against a per-snapshot one: shared join-health semantics with its own tests,
filed as #137 rather than edited in here.

Modules survive a late attach too (commit 5)

The measurement said the module loss was 100%, not partial — so this went in rather than being
deferred. Two gates caused it and only fixing both helps:

• on_module_loaded refused to copy unless a consumer was already present. The vendor hands a
  module's bytes over exactly once, inside that callback, in a buffer it reuses afterwards, and
  CUPTI offers no way to re-enumerate what is loaded. A module not copied there is gone for the life
  of the process — and a CUDA process loads essentially all of them at startup, which is why the
  loss was total.
• on_tick then drained the queue unconditionally. drain() pops and frees whatever it takes
  regardless of whether the offer landed, so draining into a socket nobody is listening on is
  indistinguishable from deleting the modules. Making capture unconditional without this would have
  copied every module and thrown it away 100ms later.

Now: capture always, drain only when a consumer is attached, replay on the attach edge.
shim/core/drain.h's ReplayLog already did the third and nothing called its module half.

Order is load-bearing twice: the replay must precede the drain, because gpu_module_load_v1 carries
bytes_ptr into the adapter's own copy and the drain frees that copy the moment it has offered it.
And it had to move above the Tier B gate — left where it was, if (!g_pc_enabled) return sits
between it and the top, so it would never run at all in the default configuration.

The bounds are the policy, and they are now 512 modules / 64 MiB — the consumer's own
ModuleStore defaults, taken rather than invented. A process nobody ever profiles holds at most
that. The shim's presence is the operator's opt-in, so this memory is only ever held by a process
somebody asked to be able to profile.

Verified on the 3090, attaching six seconds into a running workload:

modules_captured=1  module_retained_unattached=1  cubin_queue_full=0  cubins_sent=1
consumer: ModulesStored:1  ModulesWithLineInfo:1

byte-for-byte the launched run's result. The adapter's wiring needs CUPTI and is verified by that
run; what core holds is the property it rests on — an undrained queue retains every capture with
its bytes intact — plus the bounds, asserted where they live.