RFD 0011 U8: native EdgeTAM tracker C-ABI + CoreML accel + encoder-ahead threading (18.55 fps @ 0.98)#2
Draft
njayj wants to merge 14 commits into
Draft
Conversation
edgetam_capi.{h,cpp}: POD-only seed/track/reset over the visual tracker
(sam3_create_visual_tracker / propagate_frame); CoreML stages enabled from
models_dir. Compiles into libsam3.a; verified pure-C link + run against the
ggml dylibs + CoreML/Foundation/Metal/MetalKit/Accelerate + -lc++.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…RFD 0011 U8 Wave 1) sam3_encode_memory: SAM3_COREML_MEMENC gate calls edgetam_coreml_memencode and stores the perceiver slot, skipping the per-frame ggml memencode graph. SPEED VERIFIED: dt0004 7.8→10.86 fps (mem_encoder 24→0ms, full-CoreML 4-stage). ACCURACY BROKEN: 0.000 (107 lost) — memencode slot-parity bug (pix_feat/mask inputs layout-verified correct; output/model-expectation parity is the remaining unknown; needs a cosine-compare vs edgetam_perceiver_forward). Gated OFF by default — the ggml memencode remains the correct path. Not for merge as-is. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…FD 0011 U8) Parity probe (SAM3_COREML_MEMENC_PARITY) cosine-compares CoreML memencode vs the ggml perceiver slot per axis. Result on dt0004: pos cos=1.000, feats cos≈0.85 on IDENTICAL pix_feat+mask (transpose verified: H/W-swap=0.43; mask not the driver). => representation mismatch between ggml perceiver and CoreML spatial_perceiver, not a layout/input bug. The ggml assembly+tpos+memattn are calibrated to the ggml representation, so the 0.85 slot breaks the track. CONCLUSION: 20fps requires a fully pure-CoreML tracker (consistent representation end-to-end), not a CoreML memencode drop-in to the ggml hybrid. memencode gate documented DO-NOT-USE. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… 0011 U8 Wave 1) SAM3_COREML_PAD_BANK pads the mem bank to full capacity (num_maskmem spatial + max_obj_ptrs pointer slots) from frame 1, so the CoreML mem-attn runs EVERY frame (no ggml fallback on early frames). With CoreML enc/memattn/decoder + ggml memencode (correct slots): dt0004 0.952 / dt0006 0.933 (0-2 wrong) @ 11.4 fps — a 4.4x speedup of the real tracker with accuracy HELD/IMPROVED (vs pure-ggml 2.57fps/0.938). Padding also fixed the early-frame accuracy churn (0.882→0.94). 20fps blocker: ggml memencode (23ms) — CoreML memencode is 4.6x faster but its slots are 0.85-off (model-level bug, see prior commit). Also keeps the parity probe (swapHW variant). All env-gated; default ggml path unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…age 0.94 @ ~14.5 fps (RFD 0011 U8) ROOT CAUSE: the memencode export wrapped memory_encoder(skip_mask_sigmoid=False) — sigmoid only — but the tracker feeds sigmoid(mask)*sigmoid_scale_for_mem_enc + sigmoid_bias_for_mem_enc (EdgeTAM 20.0/-10.0). So the CoreML model saw a [0,1] mask vs the tracker's [-10,10] → garbage slot → tracker broke (goldeneval 0.000; Python proto 0.022). The earlier 0.996 "parity" compared against the same scale/bias-less wrapper, hiding it. PyTorch-vs-CoreML check confirmed. FIX: convert_memenc_coreml.py bakes sigmoid*scale+bias (skip_mask_sigmoid=True), + pins stable output names mem_feats/mem_pos (coremltools auto-names var_NNN change per export and silently broke the .mm featureValueForName lookup → ggml fallback). edgetam_coreml.mm updated to the stable names. RESULT (re-exported model): C++ 4-stage CoreML (PAD_BANK + enc/memattn/decoder/ memencode) dt0004 0.970 @ 14.1 fps, dt0006 0.927 @ 15.2 fps — up from 11.4 fps (ggml memencode), accuracy held, 0-wrong. Validated independently in the Python SAM2-predictor proto (1.000 with CoreML enc+memattn+memencode). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…011 U8 Wave 1) Encoder-ahead pipelining: a producer thread runs preprocess + CoreML encode for frame N+1 (own MLModel handle, ANE) while the main thread runs the consumer (assembly + memattn + decode + memencode + bank, via sam3_propagate_frame) for frame N. Library hook sam3_coreml_set_prefetched_neck + sam3_coreml_preprocess_image let the consumer's encode step consume the producer's neck (skipping preprocess + encode). New bench examples/sam3_coreml_tracker_threaded.cpp. RESULT: dt0004 0.941 @ 18.58 fps, dt0006 0.882 @ 18.62 fps — up from 14.5 fps single-threaded (~1.28× overlap), accuracy held, 0-2 wrong. 7.2× over pure-ggml (2.57 fps). At the top of RUNTIME.md's measured 15-20 fps sustained band. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The C-ABI used by platform's in-process cgo tracker (RFD 0011 U8) set only ENCODER/MEMATTN/DECODER. Without MEMENC + PAD_BANK the CoreML mem-attention model (static 3648-token input) can't match the live bank and silently falls back to the ~184ms ggml mem-attention path — correct but ~8x slower. Setting both brings the capi to the full validated config: encoder ANE + memenc ANE + mem-attn GPU + decoder ANE. Measured via platform's cgo hold runtime test: 570ms -> 70ms/frame (dt0004, 29/29 held, 14.3 fps single-thread). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
rope_k_data feeds only the ggml mem-attn fallback graph; the CoreML mem-attn path uses cached_sinpe_256 and never reads it, so building it every frame was ~2ms of pure waste. Moved its construction to just before the ggml graph build (reached only when CoreML did not return). Behavior-preserving: dt0004 goldeneval held 0.979 tracked_fraction, 0 wrong. Honest note: the ~2ms saving is within thermal run-to-run variance — this is a cleanup that documents the finding, NOT a path to 20fps. Per-frame cost is dominated by the CoreML stage compute (mem-attn ~17-19ms GPU + encoder ~11ms + decoder ~8ms + memencode ~5ms) plus preprocess (~5-7ms) and the irreducible host-side memory-bank assembly; trimming CPU glue cannot close the gap to 20. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…go threading RFD 0011 U8 Wave 5. Producer encoder handle + a 3-slot neck pool; Go coordinates slot free/ready via channels (one writer then one reader per slot, no lock). encode_slot (producer thread) preprocesses+encodes into a slot; track_slot (consumer thread) consumes the prefetched neck and propagates. Mirrors examples/sam3_coreml_tracker_threaded.cpp through the C-ABI. Existing create/seed/track/reset/destroy unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…0011 U8 Wave 5) Drives edgetam_capi_encode_slot/track_slot via a producer/consumer thread pair — the exact C-ABI path cgo will use — on dt0004: 18.55 fps (vs 14.3 single-thread), goldeneval tracked_fraction 0.979, 0 wrong. De-risks the Go wiring: the capi split reproduces the standalone threaded bench's speedup with accuracy intact. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
njayj
changed the title
…tooling
RFD 0011 U8. Applies Egor's METHODOLOGY (per-module compute-unit sweep) to the
M4 instead of chasing his M1 result. sam3_coreml_pipeline now parameterizes all
4 stage units (SAM3_{ENC,MA,DEC,MENC}_UNIT_N); the real tracker reads
SAM3_COREML_{ENC,MA,DEC,MENC}_UNIT for per-chip tuning.
M4 Pro sweep (isolated 20-rep median, 0=ALL 1=ANE 2=GPU 3=CPU):
encoder ANE 11.0 | GPU 14.3 | CPU 37.7 | ALL 17.6 -> ANE
mem-attn ANE 26.3 | GPU 17.2 | CPU 24.0 | ALL 33.3 -> GPU
decoder ANE 7.8 | GPU 8.2 | CPU 13.9 | ALL 8.9 -> ANE
memenc ANE 5.0 | GPU 3.9 | CPU 9.9 | ALL 6.7 -> GPU (isolated)
Finding: our M1-era placement (enc ANE, mem-attn GPU, dec ANE, memenc ANE) is
ALREADY M4-optimal — NOT inverted like the ORT/EfficientTAM stack (where the
encoder flips to GPU), because our RepViT encoder is conv-heavy/ANE-friendly and
runs as a standalone .mlpackage (no ORT graph partitioning, no -14). memenc's
isolated GPU edge (3.9 vs 5.0) does NOT survive thermal noise end-to-end (A/B:
GPU 20.5 vs ANE 19.7 at 200 frames, but GPU 13.7 vs ANE 15.3 at 300 frames —
thermal throttling dominates), accuracy identical (0 wrong), so the default
stays ANE. Knobs let a thermally-stable rig re-profile per chip.
Defaults unchanged => behavior-preserving. dt0004 full-clip threaded 0.979, 0 wrong.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…(RFD 0011 W0) Wires GGML_VULKAN behind a SAM3_VULKAN option (non-Apple) so the same ggml EdgeTAM engine runs on AMD/NVIDIA/Intel via Vulkan, and compiles the cgo C-ABI (edgetam_capi.cpp) on the Vulkan path WITHOUT the CoreML bridge: all edgetam_coreml_*/sam3_coreml_* calls + the encoder-ahead producer pool are now #ifdef SAM3_COREML-guarded, so a Vulkan build runs the pure-ggml path (which lands on the Vulkan device) and pool_size()==0 (synchronous Track, no threading). Mac SAM3_COREML build verified unchanged (guards are additive). Build + measure on real AMD/NVIDIA hardware is still required (W0.4 gate) before this is validated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… 0011 W0) The loader was #ifdef GGML_USE_METAL → Metal, else CPU — so a GGML_VULKAN build compiled the Vulkan backend but never used it (model loaded on CPU). Add the GGML_USE_VULKAN branch (ggml_backend_vk_init(0)) + the ggml-vulkan.h include. VALIDATED on Mac via MoltenVK (the only Vulkan device available here): model now loads "using Vulkan backend" on the M4 GPU, runs EdgeTAM end-to-end at 2.88 fps (encoder 102ms, mem-attn 119ms — note ggml-Vulkan mem-attn BEATS ggml-Metal's 184ms). MoltenVK has no matrix cores + translation overhead, so this is a worst-case proxy, NOT representative of native AMD/NVIDIA Vulkan — the real W0.4 per-vendor gate still needs that hardware. Mac SAM3_COREML build unaffected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The U2 fast path derived the bbox from the low-res logit grid and never built a mask bitmap, so the cgo tracker had no mask to ship. Now populate a CHEAP low-res (256x256) binary mask from po.mask_logits on the fast path (no full-res upscale -> no perf regression, ~12.5fps held) and attach it to the result detection unconditionally (was gated behind SAM3_FULLRES_MASK). New edgetam_capi_last_mask(out, cap, *w, *h) copies the last track's mask out. Verified: dt0004 last frame 256x256, 3.8% foreground. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
RFD 0011 U8 engine work — the cgo entry point + CoreML acceleration of the
real EdgeTAM tracker. Stacked on #1 (
rfd-0011-edgetam-runtime-refactor);review/merge #1 first.
Commits
extern "C"tracker C-ABI (coreml/edgetam_capi.{h,cpp}) — POD-onlyseed/track/reset over the visual tracker, compiled into
libsam3.a.Verified: pure-C link + run (
version=1.0.0) against ggml + the CoreMLframeworks. This is what
iris/platformcgo-binds (see platform#885).SAM3_COREML_MEMENC) — speedpath; see feat(cuda): ggml-CUDA EdgeTAM backend (NVIDIA fast-path) #3 for why it's gated off.
SAM3_COREML_MEMENC_PARITY).SAM3_COREML_PAD_BANK— pad the mem bank to full capacity from frame 1 soCoreML mem-attn runs every frame.
Measured (M-series, DanceTrack golden clips, seed-once tf@0.5)
Working native tracker —
PAD_BANK+ CoreML enc/mem-attn/decoder + ggml memencode:dt0004 0.952 / dt0006 0.933 (0–2 wrong) @ 11.4 fps — a 4.4× speedup over
pure-ggml (2.57 fps / 0.938) with accuracy held/improved, 0-wrong-person.
fps ladder: 2.57 → 5.4 → 7.8 → 11.4.
Path to 20 fps (root-caused): CoreML memencode would remove the last ggml stage
(→14.3 fps) but breaks accuracy — proven via the parity harness to be the ggml
perceiver hand-port being ~0.85 off PyTorch (the CoreML export is correct: 0.996
vs PyTorch). The clean 20 fps route is a fully pure-CoreML real-bank tracker;
de-risked recipe in platform#885's
execution-results.md.Companion
Platform-side (cgo binding + plans + results): tryiris-ai/platform#885.
🤖 Generated with Claude Code
Update — Wave 5: encoder-ahead threading + cgo consumer
edgetam_capi_encode_slot/edgetam_capi_track_slot,producer encoder handle + Go-coordinated 3-slot neck pool) + smoke test
sam3_coreml_capi_threaded: 18.55 fps @ 0.979 tracked, 0 wrong on dt0004 (up from14.3 single-thread). Also dropped
rope_k_datafrom the CoreML mem-attn hot path(unused there; accuracy held).
@0df3d64) is vendored aslibsam3.aby platform PRtryiris-ai/platform#886 (the in-process cgo hold runtime). In-bridge: 13.5 → 18.6 fps.
20 fps is the consumer-sequential ceiling; a clean 20+ needs a faster/quantized mem-attn
model or smaller memory bank (real ML, separate work).