gpu: say whether the injection happened, and document how to install it (#121) - #135
Merged
Conversation
…it (#121) CUDA_INJECTION64_PATH fails OPEN and SILENT. A driver that cannot load the library carries on as though the variable were never set: nothing is logged, the workload runs normally, and the profile comes out empty. "The shim is broken" and "this workload launched no kernels" are indistinguishable in the output, which is how a shim requiring glibc 2.42 -- unloadable in every mainstream PyTorch image -- went unnoticed. The mapping is the one observable that separates them, and it is the same fact enrolment already keys on. gpuprobe.ShimIsMappedIn exports it; the GPU tool watches for the shim appearing in the workload's maps while the workload is still alive, and when a run samples no launches it names which cause it was: launches > 0 say nothing. A line confirming the obvious on every healthy run trains people to skip the output. none, shim mapped injection worked, the adapter reported nothing -- points at the perfagent-cupti lines, where a missing CUPTI now announces itself. none, shim absent names the file and the pid, says this is what a failed injection looks like, and suggests nvidia-portable. Two limits are stated rather than hidden. The check inspects THIS pid, so a workload started through a wrapper script does its CUDA work in a child it does not see; and it only ever produces a diagnostic, never a failure. A malformed shim is caught earlier still, at attach, with its own message. Tested both ways against one process, because a check that can only answer "no" would report every run as broken, and one that can only answer "yes" would confirm injections that never happened -- worse than not checking. A path that is not a file is an error rather than a confident negative: answering "not injected" for a mistyped path sends the reader to the driver instead of to their own command line. docs/gpu-injection.md is the install guide this issue implied but never had: the mechanism and its two dangerous properties, which shim to build and why the distinction is not cosmetic, that CUPTI is the operator's to supply and where it usually already is, delivery on bare metal, in a container and in Kubernetes, how to verify, and the capability set with the reason CAP_SYS_ADMIN is absent. It also states what is still missing rather than implying completeness: no published image for the init-container pattern, and no attach-to-running-process mode, which is what stops the Kubernetes sidecar shape from working at all.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CUDA_INJECTION64_PATHfails open and silent. A driver that cannot load the library carries onas though the variable were never set: nothing logged, workload runs normally, profile comes out
empty. "The shim is broken" and "this workload launched no kernels" are indistinguishable in the
output — which is how a shim requiring glibc 2.42, unloadable in every mainstream PyTorch image,
went unnoticed.
The check
The mapping is the one observable that separates the two, and it is the same fact enrolment already
keys on.
gpuprobe.ShimIsMappedInexports it; the tool watches for the shim appearing in theworkload's maps while the workload is alive, and when a run samples no launches it says which
cause it was:
perfagent-cuptilinesnvidia-portableTwo limits are stated rather than hidden: it inspects this pid, so a workload launched through a
wrapper script does its CUDA work in a child it cannot see; and it only ever produces a diagnostic,
never a failure. A malformed shim is caught earlier still, at attach, with its own message.
Tested both ways, deliberately
A check that can only answer "no" would report every run as broken; one that can only answer "yes"
would confirm injections that never happened, which is worse than not checking. So the test asserts
both against the same process. A path that is not a file is an error, not a confident
negative — answering "not injected" for a mistyped path sends the reader to the driver instead of
to their own command line.
Docs
docs/gpu-injection.mdis the install guide this issue implied but never had: the mechanism andits two dangerous properties, which shim to build and why that distinction is not cosmetic, that
CUPTI is the operator's to supply and where it usually already is, delivery on bare metal / in a
container / in Kubernetes, how to verify, and the capability set including why
CAP_SYS_ADMINisabsent by design.
It also states what is still missing rather than implying completeness: no published image for
the init-container pattern, and no attach-to-running-process mode — which is what stops the
Kubernetes sidecar shape from working at all (#124).