shim: make the GPU adapter loadable into its targets, and ship it (#121) - #134
Merged
Conversation
The shim is dlopened by the CUDA driver INTO someone else's process, so its requirements are requirements on the TARGET. The shipped one required glibc 2.42 and failed to dlopen on Ubuntu 22.04, 24.04 and 25.04 -- essentially every PyTorch image -- with version `GLIBC_ABI_GNU2_TLS' not found Nothing caught it because it loaded perfectly on the machine that built it, and no release ships it anyway, so GPU mode has never been installable by anyone but its author. `make -C shim nvidia-portable' builds in ubuntu:22.04, because the floor is set by the oldest symbol the linker will reference and no flag lowers it after the fact: __isoc23_*@GLIBC_2.38 comes from g++ predefining _GNU_SOURCE unconditionally, reproduced across every C++ std mode, so the baseline has to BE old. -mtls-dialect=gnu removes GLIBC_ABI_GNU2_TLS, which is a marker rather than a version and was the actual cause; it is not a GCC default but a distro configure choice Fedora makes. -static-libstdc++ makes the shim immune to the target's C++ runtime, which matters because conda and PyTorch ship their own, older one. shipped glibc 2.38 + GNU2_TLS, GLIBCXX_3.4.29, RUNPATH to this machine portable glibc 2.34, no libstdc++, no RUNPATH, soname set ubuntu:22.04 shipped FAILS portable LOADS ubuntu:24.04 shipped FAILS portable LOADS ubuntu:25.04 shipped FAILS portable LOADS elfgate.sh is the part that keeps it true. It checks all THREE version namespaces rather than glibc alone -- a .so can clear a GLIBC_2.28 bar and still require GLIBCXX_3.4.31 -- plus the absence of GLIBC_ABI_GNU2_TLS, no baked RUNPATH pointing at the build machine's CUDA, exactly one exported symbol, no leaked std:: symbols preemptible against the host's C++ runtime, and the presence of the USDT notes the consumer attaches to. Run against the shim we ship today it fails on four of those, which is the point: this could have been caught at any time in the last year by looking.
DT_NEEDED libcupti.so.13 pinned the adapter to one CUDA major version and made
it unloadable wherever that version is absent -- which is the normal case,
because CUPTI ships with the CUDA TOOLKIT, not the driver, and
nvidia-container-toolkit never injects it (zero cupti hits in nvc_info.c).
It also made the shim unbuildable in CI, and that is the reason this is being
done now rather than as a nicety: the runners have no CUDA toolkit, so nothing
could compile the shim, so nothing could gate it or publish it, so GPU mode had
no release artifact at all. Resolving at runtime needs only the HEADERS to
build.
before NEEDED libcupti.so.13, libstdc++, libm, libgcc_s, libc
RUNPATH /usr/local/cuda-13.3/targets/x86_64-linux/lib
after NEEDED libstdc++, libm, libgcc_s, libc
Safe because all twenty entry points are FUNCTIONS -- zero data symbols -- so
nothing needs the linker's participation. One artifact now spans CUDA 12 and
13 and picks up the pip-installed CUPTI that PyTorch already pulls in, which is
frequently the only one present.
Verified end to end, not just linked: the C++ CUDA workload profiles to 8000
executions all exact and all matched with both the local and the
ubuntu:22.04-built portable shim.
DECLINES rather than crashes. This runs inside a process that never asked to be
profiled; killing it because the operator has no toolkit installed would be an
unacceptable way to report that. On a container with no CUDA at all the library
now LOADS (it could not before) and InitializeInjection says which sonames it
tried, that CUPTI comes from the toolkit rather than the driver, and that the
process runs unprofiled -- then returns.
The macros this first used to redirect cuptiX collided with the poison macros
in cupti_guard.h, which turn a direct CUPTI call outside the wrapper into a
compile error. That enforcement is why issue #99 cannot recur -- two of our
threads inside CUPTI at once deadlocked the profiled application permanently --
and shadowing those names would have silently disabled it. The wrappers call
dyn::table() directly instead and the poison stands: check-cupti-guard still
reports the guarded call compiling and all five unguarded ones not.
dlerror() is read once into a variable, because it clears on read and the
`dlerror() ? dlerror() : "..."` in the first version printed (null) for every
real failure.
CI had no shim step at all, and could not have had one: building the adapter
needed libcupti to link against, the runners have no CUDA toolkit, so nothing
compiled the shim, nothing gated it, and no release carried it. GPU mode has
never had a downloadable artifact. Resolving CUPTI with dlopen removed the
link-time dependency and made this job possible.
Four steps, each asserting something that was previously assumed:
1. Install CUPTI HEADERS ONLY -- no runtime, no toolkit, no GPU. Proven in a
clean ubuntu:24.04 container before writing this. cuda-crt and
cuda-cudart-dev come along because CUPTI's headers include
crt/host_defines.h, which the CUPTI package does not itself carry.
13-3 rather than 13-0, because the adapter uses CUpti_ActivityKernel12 and
CUDA 13.0's CUPTI does not define it. That minimum was real and undocumented;
this is now the only place it is written down.
2. Build and gate on the runner. Its glibc is 2.39, so this checks everything
elfgate knows except the floor.
3. Build the PORTABLE artifact in ubuntu:22.04 and gate it at 2.34 -- the
floor is enforced where the artifact that must meet it is produced.
4. Prove it LOADS on ubuntu 22.04, 24.04 and 25.04 by dlopening it there,
rather than inferring loadability from the ELF. The version headers said
the old shim was fine too; only dlopen said otherwise.
The portable shim is uploaded as an artifact, so there is finally something to
put in an init container.
The first CI run failed on the local build for requiring GLIBCXX_3.4.29 -- correctly, because the gate was checking a portability property that build never claimed. The ordinary target links libstdc++ dynamically on purpose; only the artifact a release ships must not. elfgate.sh takes 'any' to permit it, and the CI step uses that for the runner build while the portable artifact is still gated strictly at glibc 2.34 with no libstdc++ at all. A gate that fails a build for a property that build never promised is a gate people learn to switch off.
…ne (#121) GLIBC_ABI_GNU2_TLS is a marker satisfied only by glibc 2.42 however low the numeric requirements are, and the flag that removes it is not a GCC default -- it is a distro configure choice. Fedora makes the opposite one. That divergence is the whole story of this issue: a shim built on a Fedora workstation carried the marker and could not load into any mainstream PyTorch image, while the same source on an Ubuntu runner did not carry it and passed every check. Applying the flag everywhere means a developer's build now differs from the shipped artifact in nothing that affects whether it loads. Makefile becomes a prerequisite of the .so at the same time, because it is not: changing CXXFLAGS left a stale library that looked rebuilt, and the gate duly reported the marker still present after the flag removing it had been added. A build that ignores its own flags is a worse problem than the flag it ignored.
The build and both gates passed; the load proof failed with 'gcc: not found'. Mounting the host's /tmp over the container's made it read-only, so apt-get could not unpack anything, and the install failure was invisible because its output went to /dev/null. tryload.c now goes in the workspace, which is already mounted, and the container writes only to its own /tmp. The apt output is no longer discarded either -- a step that hides why a package did not install spends a CI round telling you the consequence instead of the cause.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GPU mode has never been installable by anyone but its author. No release ships the shim, and the
shim we build could not load into the processes it is injected into.
GLIBC_ABI_GNU2_TLS)NEEDED libcuptiWhy it could not be fixed before
DT_NEEDED libcupti.so.13meant building the adapter required libcupti to link against. Therunners have no CUDA toolkit, so CI could not compile the shim at all — nothing gated it and
no release carried it. "No release ships it" was a consequence, not neglect.
All twenty CUPTI entry points are FUNCTIONS — zero data symbols — so resolving them with
dlopen/dlsymneeds nothing from the linker. Building now needs only the headers, which iswhat makes the CI job possible. One artifact spans CUDA 12 and 13 and picks up the pip-installed
CUPTI that PyTorch already pulls in, frequently the only one present.
The portability fix
Built in
ubuntu:22.04, because the floor is set by the oldest symbol the linker will referenceand no flag lowers it afterwards:
__isoc23_*@GLIBC_2.38comes from g++ predefining_GNU_SOURCEunconditionally, reproduced across every C++ std mode.
-mtls-dialect=gnuremovesGLIBC_ABI_GNU2_TLS, which is a marker rather than a version — satisfied only by glibc 2.42however low the numeric requirements are, and the actual cause of the load failures. It is not a
GCC default; it is a distro configure choice Fedora makes.
shim/elfgate.shkeeps it true: all three version namespaces (a.socan clear aGLIBC_2.28bar and still needGLIBCXX_3.4.31), theGNU2_TLSmarker, no baked RUNPATH, exactlyone exported symbol, no leaked
std::symbols preemptible against the host's C++ runtime, and theUSDT notes the consumer attaches to. Run against the shim we ship today it fails four of them.
It declines instead of crashing
This runs inside a process that never asked to be profiled. On a container with no CUDA the
library now loads — previously impossible — and
InitializeInjectionreports which sonames ittried, that CUPTI ships with the toolkit rather than the driver and is not injected by
nvidia-container-toolkit, and that the process runs unprofiled. Then returns.
Three things worth reviewing closely
The guard nearly got disabled. My first version redefined
cuptiXas macros pointing at theresolved table — colliding with the poison macros in
cupti_guard.hthat make a direct CUPTI calloutside the wrapper a compile error. That enforcement is why #99 cannot recur (two of our threads
inside CUPTI deadlocked the profiled application permanently). The wrappers call
dyn::table()directly instead;
check-cupti-guardstill reports the guarded call compiling and all fiveunguarded ones not.
An undocumented minimum CUPTI version. The adapter uses
CUpti_ActivityKernel12, absent fromCUDA 13.0's CUPTI — the build fails there. Nothing stated this anywhere; the CI job is now the only
place it is written down.
dlerror()clears on read, sodlerror() ? dlerror() : "…"printed(null)for every realfailure. Found by running the missing-CUPTI path in a container rather than reading the code.
The CI job asserts rather than assumes
It installs CUPTI headers only (proven in a clean container first), builds and gates on the
runner, builds the portable artifact in
ubuntu:22.04and gates it at the real floor, and thendlopens it on 22.04, 24.04 and 25.04 rather than inferring loadability from the ELF — the
version headers said the old shim was fine too; only
dlopensaid otherwise.Still open on #121
A published shim image for the init-container pattern (the artifact exists now, the image does
not), and the positive injection self-check —
CUDA_INJECTION64_PATHstill fails open, so a brokeninstall is indistinguishable from a workload that launched no kernels.