cuda: make graph-cache eviction reachable; qwen3_asr: survive concurrent-load memory pressure

[derekja]

Follow-up to #276. That fix made serial qwen3_asr load memory-stable (verified: three identical single-stream sweeps, VRAM byte-identical). Under concurrent /v1/audio/transcriptions load a distinct failure remained: VRAM grows on every run — including identical repeats of the same shapes — until allocation fails, after which every request returns 500 until the process is restarted.

Measured on a 12 GB H100L vGPU slice (no VMM), Qwen3-ASR-0.6B Q8, 8 concurrent streams of mixed 1–8 s utterances posting accumulated audio ~1/s: 6.2 → 8.9 → 10.5 GB (ceiling) across repeated sweeps, then permanent 500s. nemotron_asr under the identical workload is byte-stable, which localized the problem to the qwen3_asr graph lifecycle. With GGML_CUDA_DISABLE_GRAPHS=1 the same workload is also byte-stable, which localized the reservoir.

Four changes, in causal order:

1. Export ggml_backend_cuda_clear_graph through get_proc_address. Every graph destructor already calls engine::core::release_backend_graph_resources, which resolves this name — but the CUDA registry never exported it, so eviction has been a silent no-op since it was added (docs/build/HIP.md claims it works; now it does). Single-threaded servers got away with it because the freed graph arena's address is typically reused by the next same-size ggml_init, overwriting the stale cuda_graphs entry in place; concurrent requests (one detached thread per request) perturb the address space, so each rebuild minted a fresh key and orphaned a cudaGraph_t + cudaGraphExec_t.

2. Release the old graph before constructing its replacement (prefill/decode/classification/encoder). Assigning make_unique over a live unique_ptr holds both arenas at the rebuild peak — and interleaved streams force a rebuild on nearly every request because the prefill/encoder caches are exact-shape single slots.

3. Hold ggml_gallocr_t in a unique_ptr (the existing voxtral_realtime pattern) so the CapacityError/runtime_error throws in the graph constructors stop leaking the partially reserved arena. Previously every failed rebuild after an OOM deepened the OOM.

4. Trim idle pool memory and retry once when graph allocation fails. The legacy pool (no-VMM path) caches every buffer it ever allocated and only flushes when its own cudaMalloc fails; graph arenas allocate through ggml_backend_cuda_buffer_type_alloc_buffer, which flushes nothing — so once the pool ratchets to the ceiling, every graph build fails forever while gigabytes of idle cached buffers sit reclaimable. Adds ggml_cuda_pool::clear(), an exported ggml_backend_cuda_trim_pools(), a framework resolver, and trim-and-retry on the four qwen3_asr allocation-failure paths. The trim costs one device sync and only runs on a previously-fatal failure.

Validation (same slice/model/harness): the workload that previously produced the terminal all-500 state now self-heals — at the ceiling the trim reclaimed 2.3 GB mid-load, 30/32 utterances in the pressure-point sweep still transcribed, and the server returned to full health (WER identical to baseline, finals p50 9 ms) with no restart. Serial behavior is unchanged. Transcripts across the load runs are unchanged (corpus WER 0.05 at every level, same as pre-patch).

Repro used throughout: N concurrent realtime websocket streams → accumulated-utterance POSTs of mixed lengths; happy to share the harness scripts if useful.

[derekja]

RTF data, audiocpp_cli --metrics as in the #276 review. Same slice as the PR description (12 GB H100L vGPU, CUDA backend). Method: one --request-sequence session per family per binary — 6 requests per audio length, first request per length excluded as graph build, p50 (min–max) of the 5 warm runs. Baseline is main @ 54ea3d1, patched is this branch.

qwen3_asr 0.6B Q8 (the family this PR touches):

audio	main	patched
2 s	0.0326 (0.0326–0.0394)	0.0258 (0.0258–0.0285)
6 s	0.0402 (0.0400–0.0429)	0.0354 (0.0354–0.0368)
15 s	0.0243 (0.0242–0.0252)	0.0242 (0.0241–0.0250)
30 s	0.0259 (0.0259–0.0264)	0.0258 (0.0257–0.0262)

nemotron_asr 0.6B Q8 (regression check for the shared ggml/framework changes):

audio	main	patched
2 s	0.0098 (0.0096–0.0157)	0.0096 (0.0095–0.0155)
6 s	0.0097 (0.0094–0.0129)	0.0092 (0.0087–0.0121)
15 s	0.0067 (0.0066–0.0080)	0.0065 (0.0060–0.0071)
30 s	0.0058 (0.0057–0.0072)	0.0065 (0.0064–0.0070)

Long-file RTF is unchanged on both families and nemotron sits inside its baseline spread everywhere. Short/medium qwen is measurably faster on the patched build (2 s: 0.0326 → 0.0258; ranges don't overlap) — plausibly the reset-before-rebuild change relieving allocator pressure, though I haven't isolated it; no length is slower. Consistent with the patch adding no code to the warm path (eviction runs at graph destruction, the pool trim only on a previously-fatal allocation failure).

[0xShug0]

@derekja Could you clean up the churn in the PR caused by the CRLF changes? Changes to GGML are usually high-risk because of their large blast radius and generally require a regression sweep, unless they’re purely additive and opt-in. The current PR shows full-file churn, which makes it difficult to tell what actually changed.

[derekja]

you bet, thanks. running regression tests overnight and will post when complete.

[derekja]

Thanks for the careful look, and apologies for the churn — that was my tooling silently normalizing line endings on the three external/ggml files (they're stored CRLF, so a text-mode edit rewrote every line). I've rebuilt both commits from main's exact bytes with only the real insertions and force-pushed; the PR now shows the actual change: +110/−22 across 7 files.

On blast radius, agreed — here's the breakdown and the regression sweep.

What the ggml side actually changes (+29 lines, no deletions): two proc-address table entries, ggml_backend_cuda_trim_pools() (new, reachable only via the new proc entry), a clear() override on the legacy pool that calls its existing clear_pool(), and a virtual void clear() {} with a default no-op body on the pool base. Nothing existing is modified. The one behavioral change to pre-existing code is the point of the fix: release_backend_graph_resources — already called by every family's graph destructors — stops being a silent no-op on CUDA, so the sweep below deliberately covers multiple families, not just qwen3_asr.

Regression sweep (12 GB H100L vGPU, CUDA backend, this branch):

• ctest (-DENGINE_BUILD_TESTS=ON): 56/56 passed.
• run_audiocpp_cli_path_tests.py + compare_audiocpp_cli_path_results.py, baseline main @ 54ea3d1 CLI vs patched CLI, every family I have local models for: qwen3_asr (3 cases), nemotron_asr (2), citrinet_asr (1), hviske_asr (1), voxcpm2 (2 — a TTS family, included since the eviction change fires in all families' destructors): 9/9 compared cases, 0 differences — artifacts and text byte-match between the two builds. Caveats for reproducibility: the resources/*.wav fixtures aren't in the repo, so I substituted local audio (identical for both runs, which is what the A/B needs); the qwen forced-aligner case exercises the aligner model I don't have installed and the voxcpm2 voice-clone case failed identically on both builds against my substitute reference clip — both failures are byte-identical base-vs-patched and counted in the 0-difference comparison.
• RTF: the --metrics A/B in the comment above (equal-or-faster, no length regressed).
• The concurrent-load soak from the PR description, re-run on the clean branch: previously-fatal workload self-heals; serial byte-stability unchanged.

One pre-existing note from running the unit suite: codec_dequant_parity (tests/moss_tts_local) failed to compile on the first full-tests build against current main headers — unrelated to this PR (neither file is in this diff); mentioning it in case it's already known.

Happy to run any additional sweep you'd like — the concurrent-load harness that produced the leak repro is scriptable and I can share it.