Skip to content

gpu: say whether the injection happened, and document how to install it (#121) - #135

Merged
dpsoft merged 1 commit into
mainfrom
feat/121-injection-selfcheck
Sep 8, 2026
Merged

gpu: say whether the injection happened, and document how to install it (#121)#135
dpsoft merged 1 commit into
mainfrom
feat/121-injection-selfcheck

Conversation

@dpsoft

@dpsoft dpsoft commented Sep 8, 2026

Copy link
Copy Markdown
Owner

CUDA_INJECTION64_PATH fails open and silent. A driver that cannot load the library carries on
as though the variable were never set: nothing logged, workload runs normally, profile comes out
empty. "The shim is broken" and "this workload launched no kernels" are indistinguishable in the
output — which is how a shim requiring glibc 2.42, unloadable in every mainstream PyTorch image,
went unnoticed.

The check

The mapping is the one observable that separates the two, and it is the same fact enrolment already
keys on. gpuprobe.ShimIsMappedIn exports it; the tool watches for the shim appearing in the
workload's maps while the workload is alive, and when a run samples no launches it says which
cause it was:

reported
launches > 0 nothing — a line confirming the obvious on every healthy run trains people to skip the output
none, shim mapped injection worked, the adapter reported nothing → points at the perfagent-cupti lines
none, shim absent names the file and pid, says this is what a failed injection looks like, suggests nvidia-portable

Two limits are stated rather than hidden: it inspects this pid, so a workload launched through a
wrapper script does its CUDA work in a child it cannot see; and it only ever produces a diagnostic,
never a failure. A malformed shim is caught earlier still, at attach, with its own message.

Tested both ways, deliberately

A check that can only answer "no" would report every run as broken; one that can only answer "yes"
would confirm injections that never happened, which is worse than not checking. So the test asserts
both against the same process. A path that is not a file is an error, not a confident
negative — answering "not injected" for a mistyped path sends the reader to the driver instead of
to their own command line.

Docs

docs/gpu-injection.md is the install guide this issue implied but never had: the mechanism and
its two dangerous properties, which shim to build and why that distinction is not cosmetic, that
CUPTI is the operator's to supply and where it usually already is, delivery on bare metal / in a
container / in Kubernetes, how to verify, and the capability set including why CAP_SYS_ADMIN is
absent by design.

It also states what is still missing rather than implying completeness: no published image for
the init-container pattern, and no attach-to-running-process mode — which is what stops the
Kubernetes sidecar shape from working at all (#124).

…it (#121)

CUDA_INJECTION64_PATH fails OPEN and SILENT. A driver that cannot load the
library carries on as though the variable were never set: nothing is logged, the
workload runs normally, and the profile comes out empty. "The shim is broken"
and "this workload launched no kernels" are indistinguishable in the output,
which is how a shim requiring glibc 2.42 -- unloadable in every mainstream
PyTorch image -- went unnoticed.

The mapping is the one observable that separates them, and it is the same fact
enrolment already keys on. gpuprobe.ShimIsMappedIn exports it; the GPU tool
watches for the shim appearing in the workload's maps while the workload is
still alive, and when a run samples no launches it names which cause it was:

  launches > 0        say nothing. A line confirming the obvious on every
                      healthy run trains people to skip the output.
  none, shim mapped   injection worked, the adapter reported nothing -- points
                      at the perfagent-cupti lines, where a missing CUPTI now
                      announces itself.
  none, shim absent   names the file and the pid, says this is what a failed
                      injection looks like, and suggests nvidia-portable.

Two limits are stated rather than hidden. The check inspects THIS pid, so a
workload started through a wrapper script does its CUDA work in a child it does
not see; and it only ever produces a diagnostic, never a failure. A malformed
shim is caught earlier still, at attach, with its own message.

Tested both ways against one process, because a check that can only answer "no"
would report every run as broken, and one that can only answer "yes" would
confirm injections that never happened -- worse than not checking. A path that
is not a file is an error rather than a confident negative: answering "not
injected" for a mistyped path sends the reader to the driver instead of to
their own command line.

docs/gpu-injection.md is the install guide this issue implied but never had:
the mechanism and its two dangerous properties, which shim to build and why the
distinction is not cosmetic, that CUPTI is the operator's to supply and where it
usually already is, delivery on bare metal, in a container and in Kubernetes,
how to verify, and the capability set with the reason CAP_SYS_ADMIN is absent.

It also states what is still missing rather than implying completeness: no
published image for the init-container pattern, and no attach-to-running-process
mode, which is what stops the Kubernetes sidecar shape from working at all.
@dpsoft
dpsoft merged commit c969f0a into main Sep 8, 2026
11 checks passed
@dpsoft
dpsoft deleted the feat/121-injection-selfcheck branch September 8, 2026 14:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant