feat(cuda): ggml-CUDA EdgeTAM backend (NVIDIA fast-path)#3
Draft
njayj wants to merge 16 commits into
Draft
Conversation
Author
|
Cross-repo platform PR (cgo binding + vendoring, stacked on #908): https://github.com/tryiris-ai/platform/pull/920 |
… SAM3_CUDA - SAM3_CUDA now forces GGML_CUDA_GRAPHS=ON: standalone ggml defaults it OFF (llama.cpp-only flag), so the previously advertised 'CUDA graphs' path was never compiled. ggml auto-falls back at runtime for uncapturable graphs. - New SAM3_CAPI_SHARED option builds the cgo C-ABI as its own shared library (sam3capi.dll). This is the Windows/CUDA toolchain bridge: nvcc requires MSVC as host on Windows, and MSVC-built C++ static libs cannot link into a mingw cgo binary — a C-ABI DLL is toolchain-neutral. ET_API in edgetam_capi.h carries the dllexport (extern "C" alone does not export from an MSVC DLL). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Loader (fix): the CUDA/Vulkan selection logged 'using X backend' BEFORE init
and unconditionally — a machine without the device logged the wrong backend.
Now: device-count guard, log AFTER successful init, explicit fallback warning.
ggml-Vulkan THROWS (vk::SystemError) when the Vulkan loader/driver is absent —
the probe+init are now wrapped so a GPU-less machine falls through to the CPU
backend instead of failing the whole model load (fat-build CUDA->Vulkan->CPU
chain is now safe end to end).
Encoder-ahead (feat): the CoreML-threading equivalent for NVIDIA. The
prefetched-neck seam moves out of #ifdef SAM3_COREML (it is raw floats — no
CoreML dependency); a factored edgetam_load_neck_from_floats() is shared by
the CoreML helper and the new ggml consume path in edgetam_encode_image.
sam3_encoder_ahead_{create,encode,destroy} run the EdgeTAM RepViT+FPN encoder
graph on a SECOND ggml-CUDA backend instance (own stream; shared read-only
weights) so frame N+1's encode overlaps frame N's mem-attn/decoder. The capi
wires it behind SAM3_GGML_ENCODER_AHEAD=1 (default OFF: same-GPU
producer/consumer contention must be A/B-measured on real NVIDIA hardware —
the M4 sweep showed co-location can lose; pool_size()==0 keeps the
synchronous path otherwise).
Verified: -fsyntax-only across {plain, SAM3_COREML, GGML_USE_CUDA,
GGML_USE_VULKAN, CUDA+VULKAN, ET_CAPI_BUILD_DLL}; full Mac cmake build
(SAM3_COREML=ON) + sam3_edgetam_bench link green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…n lacks
ggml-CUDA's fused flash-attention kernels only cover head sizes
{40,64,72,80,96,112,128,256,576} (ggml-cuda/fattn.cu). EdgeTAM's SAM
mask-decoder attention uses head_dim=32, so ggml_cuda_get_best_fattn_kernel
returns BEST_FATTN_KERNEL_NONE and GGML_ABORTs on the first Track() — while
model load and the RepViT encoder run fine on CUDA0. Metal and Vulkan accept
head_dim=32, so this gap is CUDA-specific and never surfaced on those backends.
Add sam3_fa_ext(), a wrapper around ggml_flash_attn_ext. On the CUDA build only
(#ifdef GGML_USE_CUDA) it routes head dims outside the supported set through a
mathematically equivalent manual attention — the exact QK^T -> soft_max_ext ->
.V -> permute(0,2,1,3) idiom already used interchangeably with flash in
sam3_ddec_layer_forward, so the [HD,NH,N_q,B] output layout is identical and no
call site changes. Flash is still used on CUDA for supported head dims (encoder,
memory attention), preserving perf there. On non-CUDA builds sam3_fa_ext forwards
verbatim to ggml_flash_attn_ext, so the Metal/CoreML graph and numerics are
byte-for-byte unchanged. All 17 flash-attn call sites route through the wrapper.
Enables the Windows x64 + CUDA EdgeTAM 512 hold to run per-frame inference on
NVIDIA GPUs (RTX 4060 / Ada sm_89, CUDA 12.9).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ck_trk edges report/CUDA-RESIDENCY-DESIGN.md §5 increment 1. Adds the sam3_transport driver (host default on every backend = today's exact tensor_get/tensor_set sequence; CUDA device opt-in via SAM3_STAGE_TRANSPORT=device = one same-backend D2D ggml_backend_tensor_copy) and converts the four heaviest inter-stage edges: mem-attn curr (with the 4D-leaf + in-graph-reshape fix so D2D layouts match; host graph bit-for-bit unchanged), decoder trk_s0/trk_s1, memenc pix_in_raw. Removes 4 D2H + 4 H2D = ~176.7 MB/frame (76% of PCIe bytes). Gates: host-mode output byte-identical to pre-change baseline; host-vs-device A/B byte-identical over 39 synthetic frames incl. masks; 7.3 -> 9.5-9.7 fps. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ad caches report/CUDA-RESIDENCY-DESIGN.md §5 increment 2. sam3_stagebuf residency handle: the existing host vector stays the value; device mode adds a persistent twin uploaded once per content generation, then D2D into the fresh graph input. Twins: rope_q/rope_k/src_pos (tracker, keyed on pe_gen), sparse/image_pe/ dense_emb (state, keyed on pe_cache_gen). Load-time host caches (device driver only) for the per-frame weight re-reads: obj-ptr MLP W/b, no_obj_ptr, perceiver latents_1d/2d, maskmem tpos. Host driver byte-identical: pushes are verbatim tensor_set, caches stay empty. Gates: host-mode byte-identical to pre-change baseline; host-vs-device A/B byte-identical; 9.6-9.8 fps. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ssembly report/CUDA-RESIDENCY-DESIGN.md §5 increment 3. Ring: bank slots + obj pointers now recycle their device tensors on eviction (sam3_acquire_slot_pair / recycle free-lists) instead of leaking buffers until reset (~263 KB VRAM per credible frame) and accreting tensor metadata in tracker.ctx (exhaustion at ~1365 credible frames). Whole-bank instance eviction recycles too. Host-driver bytes unchanged (same tensor_set order); only the allocation pattern differs. Device prompt assembly (CUDA device driver + EdgeTAM + full steady-state capacity only; ramp frames keep the verbatim host path — no padding): the prompt's spatial region is concatenated IN-GRAPH from fresh slot leaves fed by D2D from the bank ring, with the tpos broadcast add done on device (single fp32 add — bit-exact vs the CPU add). The pointer region is built by the exact host code path (sam3_build_prompt_and_pos with empty slots) and uploaded (~34 KB). Removes the per-frame slot downloads (1.8 MB D2H), the 917 KB prompt/pos uploads, and 16 pointer D2H gets + their syncs. Obj-ptr tpos-proj weights added to the load-time cache; host mirror of pointers maintained alongside ptr_banks. Gates: host-mode byte-identical to pre-change baseline; host-vs-device A/B byte-identical at 40 frames AND a 300-frame soak (19.6 MB, ring wraparound; "device prompt assembly engaged (7 slots x 512 tokens + 64 ptr tokens)" asserted in stderr). Steady-state device fps 10.46 (from 9.6). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
report/CUDA-RESIDENCY-DESIGN.md §5 increment 4 (device driver + EdgeTAM only; SAM2.1 keeps the verbatim host path — its no_obj_embed_spatial branch mutates the memenc output on the host). The perceiver now feeds its two sub-graphs by same-backend D2D from the LIVE memenc output tensor (mo) while its graph memory is still allocated — the 1 MB per-frame download of md disappears. The 1D path uses the 4D-leaf + in-graph-reshape pattern; the 2D path performs the window partition IN-GRAPH (reshape/permute/cont/reshape — a pure byte gather, bit-identical to edgetam_window_partition_cpu, verified by the byte-diff gate). Perceiver pos (cached_sinpe_64) and latents ride persistent twins. Gates: host-mode byte-identical to pre-change baseline; host-vs-device A/B byte-identical at 40 frames and a 300-frame soak; 10.0 fps smoke / 10.45 fps steady-state. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…D feeds
report/CUDA-RESIDENCY-DESIGN.md §5 increment 5 (the severable core). The CUDA
backend now DEFAULTS to the device transport; SAM3_STAGE_TRANSPORT=host is the
runtime escape hatch. The flip sits inside ggml_backend_is_cuda — Metal/
Vulkan/CPU can never take it, so every non-CUDA backend keeps today's host
path unconditionally. All device-path D2D feeds (stage_feed + stagebuf twin
copies) switch to ggml_backend_tensor_copy_async on the backend's compute
stream: same-stream FIFO orders each copy before the graph_compute that
consumes it, removing the per-copy stream fence. Every feed source is
long-lived (state/ring/twins, or a graph output whose gallocr outlives the
consuming compute), so no copy can read recycled memory. Twin UPLOADS stay
synchronous (host-side sources may be transient).
Deferred (severable, per design §5-inc5): per-stage ctx/graph/gallocr reuse
for CUDA-graph replay (needs the CLAUDE.md rule-1 addendum + owner sign-off),
pinned host staging for the frame/mask pair, and the encoder-ahead
cross-instance copy_async handoff (the transport logs a note when
SAM3_GGML_ENCODER_AHEAD is combined with device transport).
Gates: host-mode byte-identical to pre-change baseline; host-vs-device A/B
byte-identical at 40 frames and a 300-frame soak; default engagement asserted
("stage transport = device (CUDA-resident, default)"); 10.0-10.5 fps.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…path) CLAUDE.md graph-isolation addendum 4 (owner-approved): on the CUDA device driver a stage may keep its ggml_context/cgraph/gallocr alive across frames and recompute in place, keyed on every topology-affecting shape/config and released on any key change / reset / compute failure. Applied to the fused mem-attn + mask-decoder graph in sam3_propagate_single (sam3_prop_gcache on the tracker): steady-state frames skip the graph rebuild AND the gallocr reserve+alloc entirely — no per-frame cudaMalloc/free, stable leaf data pointers (the precondition for ggml-CUDA graph capture to reach replay). Ramp frames (growing M_total) miss the key and rebuild as before. Host driver keeps the verbatim build→compute→free per call. Gates: host byte-identical to pre-program baseline; host-vs-device A/B byte-identical at 40 frames + 300-frame soak; 10.45 fps smoke, 11.11 fps steady state (from 10.0/10.5). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Same CLAUDE.md addendum-4 contract as 6a, applied to the three remaining per-credible-frame graphs: the memory encoder (sam3_stage_gcache on the tracker; keyed on H/INTERPOL/D) and the perceiver 1D/2D sub-graphs (keyed on their fixed dims + device flag). Steady state skips build+reserve+alloc for all of them; the cached memenc output tensor (mo) keeps a frame-invariant address, which also stabilizes the perceiver's D2D feed source. Compute failure releases the affected cache. Host driver unchanged. Gates: byte-identity A/B 40f + 300-frame soak; 10.81 fps smoke, 11.28 fps steady state (from 10.45/11.11). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Same addendum-4 contract applied to the largest per-frame graph (RepViT + FPN neck, runs every frame): sam3_stage_gcache on sam3_state, keyed on img_size/n_fpn, released in sam3_free_state / on compute failure. Steady state skips the encoder's build+reserve+alloc entirely; the FPN output tensors keep frame-invariant addresses feeding the existing D2D fpn->neck_trk fast path. Pinned host staging for the 12.6 MB frame upload evaluated and deferred: preprocess returns a fresh vector per frame, so pinning needs a preprocess-into-persistent-buffer restructure for ~1 ms — logged as backlog. Gates: byte-identity A/B 40f + 300-frame soak; 11.49 fps smoke, 12.05 fps steady state (from 10.81/11.28). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The encoder-ahead handoff moved 2×84 MB of neck floats through the host per frame (producer tensor_get -> Go slot buffers -> consumer tensor_set), which capped the overlap win at 1.04x. The device seam keeps everything on-GPU: the producer encodes with a CACHED graph (addendum-4; producer-stream CUDA graph replay) and same-instance-D2D's the three FPN levels into per-slot persistent device tensors (+ one producer sync so writes are complete before the Go channel signals — happens-before across threads); the consumer's new device-prefetch branch in edgetam_encode_image D2D-feeds them straight into state.neck_trk. Zero PCIe on the seam. Falls back to the host-float seam for host transport / CoreML (that path is untouched). capi: encode_slot tries the device seam first; track_slot prefers it per slot. Measured (RTX 4060, device transport): serial 79.2 ms/frame -> threaded 55.7 ms/frame = 18.0 fps (speedup 1.42x, was 1.04x with the host seam). J2-13 gate: threaded-vs-serial outputs BYTE-IDENTICAL over a 299-frame moving-subject A/B (boxes, scores, states, masks) — the second CUDA instance produces bit-identical necks, so encoder-ahead costs zero accuracy here. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
coreml/sam3_preproc.cu: bit-exact CUDA twin of sam3_resize_bilinear (double chain, -fmad=false — load-bearing for the byte gate) + ImageNet normalize, writing f32 CHW straight into the encoder input tensor's device memory. Replaces ~14ms single-threaded CPU preprocess + 12.6MB f32 H2D with a ~2.7MB u8 upload on the device-transport path. SAM3_PREPROC_DEVICE: unset/1 = device (when transport==DEVICE); 0 = shipped CPU path byte-identical; 2 = VERIFY (both paths, memcmp, abort on mismatch). Fail-soft CPU fallback on any CUDA error. Encoder-ahead producer path unchanged (CPU preprocess; default-off).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Wires the already-vendored ggml-CUDA backend into the EdgeTAM engine — the NVIDIA sibling of the existing
SAM3_VULKAN(cross-vendor) and Metal/CoreML (Apple) backends. Pure wiring, zero edits underggml/src/ggml-cuda/(61.cufiles already present).3 commits:
SAM3_CUDACMake option →GGML_CUDA=ON(which auto-definesGGML_USE_CUDAPUBLIC viaggml_add_backend(CUDA)); non-Apple capi-reuse guard widened to(SAM3_VULKAN OR SAM3_CUDA).!model->backendpreference chain → CUDA → Vulkan → CPU fallthrough.sam3_backend_name/edgetam_capi_backend) so the consumer can assert the GPU accelerator actually initialized (catches a silent CPU fallback — Egor "optimizing SAMURAI part 2" lesson RFD 0011: EdgeTAM/sam3.cpp runtime refactor (U0–U4) + CoreML/ANE serving path (hybrid 6 fps shipping; all 4 stages exported, pure-CoreML 20 fps) #1).Why not TensorRT
Egor's part-2 blog optimizes the ONNX/TensorRT EfficientTAM stack; those kernel tricks don't transfer to ggml. This is the ggml-CUDA backend for the EdgeTAM engine the Iris pipeline actually ships — mirrors the Vulkan precedent exactly.
Cross-repo
Pairs with the platform PR (same branch name
claude/cuda-nvidia-edgetam) which adds the cgotracker_cuda.gobinding and vendors this build'slibsam3.a. Platform PR: (linked below).Validation
-DSAM3_METAL=ON -DSAM3_COREML=ON) green at all gates — the CUDA additions are inert whenSAM3_CUDA=OFF.-DSAM3_CUDA=ONconfigure/compile + on-GPU accuracy/FPS are deferred to the platform plan's D1 hardware gate, exactly as the Vulkan backend shipped UNVALIDATED.🤖 Generated with Claude Code
Update 2026-07-04 — review fixes + encoder-ahead producer (commits
2604544,a3194c7)Fork-Reason: guarded runtime backend-fallback chain (CUDA→Vulkan→CPU) — upstream's loader logs before init and lets
ggml_backend_vk_initthrow out ofsam3_load_modelon Vulkan-less machines.Fork-Reason:
SAM3_CAPI_SHAREDC-ABI DLL target +ET_APIexports — required to cross the MSVC(nvcc)/mingw(cgo) toolchain boundary on Windows; upstream has no cgo consumer.Fork-Reason: backend-agnostic prefetched-neck seam +
sam3_encoder_ahead_*ggml-CUDA producer — the CoreML encoder-ahead threading equivalent for NVIDIA, consumed by the platform's-tags sam3cudabinding.Fork-Reason:
SAM3_CUDAnow forcesGGML_CUDA_GRAPHS=ON— standalone-ggml default OFF silently dropped the CUDA-graphs optimization.Details: loader chain now probes device counts, logs AFTER successful init, wraps the Vulkan probe in try/catch (vk::SystemError on driverless machines no longer fails the model load, it falls to CPU). The prefetched-neck seam (
s_prefetch_neck+sam3_coreml_set_prefetched_neck— historic name kept for source compat) moves out of#ifdef SAM3_COREML;edgetam_load_neck_from_floatsis factored and shared by the CoreML helper and the new ggml consume path;sam3_encoder_ahead_{create,encode,destroy}run the RepViT+FPN encoder graph on a second CUDA backend instance (own stream, shared read-only weights). The capi wires the pool behindSAM3_GGML_ENCODER_AHEAD=1, default OFF pending the on-GPU A/B (platformTestEncoderAheadThroughput, now tag-widened tosam3cuda).Validation:
-fsyntax-onlyacross {plain, SAM3_COREML, GGML_USE_CUDA, GGML_USE_VULKAN, CUDA+VULKAN, ET_CAPI_BUILD_DLL}; full Mac cmake build (SAM3_COREML=ON) +sam3_edgetam_benchlink green — the CoreML path is regression-free. CUDA/Vulkan compile+run remain hardware-gated as before.