CudaBackend never overrides DeviceMemoryInfo, and the seam's comment says it does — so Gemma4MoE's device-expert LRU is dead on CUDA

[localai-bot]

Found while measuring #1123, which needed a device memory budget and could not get one from this seam.

The false comment

include/vt/backend.h:78-83:

  // Optional device free/total VRAM probe (bytes). Default false = unknown.
  // ROCm/CUDA override with hipMemGetInfo/cudaMemGetInfo so model code can
  // size LRU caches without including vendor headers (device-leakage).
  virtual bool DeviceMemoryInfo(size_t* /*free_bytes*/, size_t* /*total_bytes*/) const {
    return false;
  }

Only ROCm overrides it (src/vt/rocm/rocm_backend.hip:338-345). CudaBackend does not, so on every CUDA device — not only a GB10 — DeviceMemoryInfo returns false. cudaMemGetInfo is called nowhere in the repository. The comment is corrected in prose by the #1123 change; the capability is this issue.

The consequence, which is not just a comment

Gemma4MoE is the seams only consumer. FreeBytes returns false when the probe is unavailable (src/vllm/model_executor/models/gemma4_moe.cpp:439-447), and MakeRoom refuses on unknown by design (:494-506, // Refuse device upload if free VRAM unknown). So on CUDA MakeRoomalways returns false and the whole device-expert LRU —kMaxSlots = 24, kHeadroom = 1.5 GiB` — never admits an expert. It falls back to host H2D on every expert, permanently, and nothing reports it.

That refusal-on-unknown is the RIGHT polarity for that call site (a hung hipMalloc is worse than a host fallback), so this is not a bug in gemma4_moe.cpp. It is a missing probe whose absence silently disables a landed capability.

Why #1123 did not add the override

Adding it would wake that LRU on CUDA, which is a behaviour change to another models residency policy with a measurement of its own to make. #1123 instead probes cudaMemGetInfoinCudaPlatform(which already includes<cuda_runtime.h>and already probes device attributes at registration) and carries the total onResidencyPolicy`, touching nothing Gemma4 reads.

What closing this looks like

1. CudaBackend::DeviceMemoryInfo via cudaMemGetInfo, mirroring rocm_backend.hip:338-345.
2. A measurement of Gemma4 MoE on CUDA with the LRU live: correctness first, then whether the device path is faster than the host H2D it replaces.
3. The Qwen3.8-2.4T on --device cuda: the model loads, then the first forward dies on cudaMalloc: out of memory #1123 fit check can then read the budget from the backend seam on every platform that reports one, closing the "budget known only on CUDA" item under ## Owed in expert-streaming.md.

Measured for reference, on dgx:gpu0 (GB10) through libcudart.so.13, where nvidia-smi --query-gpu=memory.total,memory.free,memory.used answers [N/A], [N/A], [N/A]:

cudaMemGetInfo rc = 0
free  = 122059919360 (113.677 GiB)
total = 128452956160 (119.631 GiB)

total equals /proc/meminfo MemTotal rounded to the physical 128 GB. So the probe is usable on a unified pool, which is the case the comment`s "size LRU caches" sentence was written for.

Owning row: ENG-EXPERT-STREAM (## Owed).

[localai-bot]

Worked this as a fresh implementer. The override is not added, and the record
for closing this issue is corrected instead — PR #1200.

The comment half of this issue is already done. #1132 corrected it in both
places that carried it (include/vt/backend.h:78-93 and gemma4_moe.cpp:440-448)
and recorded the gap under ## Owed. Every anchor those two carry was re-verified
exact against fd64c76ee; six examined, six correct. So what was left was the
capability, and four things came out of re-reading it that raise its price.

1. There is nothing to run it on. The LRU needs ex.is_fp8, which only
   LoadMoeFp8PerExpert sets (gemma4_weights.cpp:210-215) from a per-expert
   F8_E4M3 export. No such Gemma-4 checkpoint is pinned in docs/USAGE.md —
   all 32 lines matching gemma were swept, and every checkpoint among them is
   the LTX-2.5 text tower, not a MoE decoder.
2. The CUDA consumers are not the ones the ROCm arm measured.
   RunGemma4FusedTopkExpertGeGLU (:1599), ExpertGeGLUDeviceBatched (:240),
   PeerCopyGemma4Fp8ExpertSlice and RunGemma4Fp8TopKOnExpertDevice are
   HIP-only; a non-HIP build links stubs returning false. The override would
   admit experts to VRAM and route them through the serial per-expert fallback —
   a third path, measured by nobody. It is not token-neutral either: the resident
   arm accumulates with vt::MatmulBTAlphaBeta (:90) where the fallback writes
   outright (:70), so the two sum the top-k in a different order.
3. The headroom constant does not mean the same thing on a GB10.
   free_b >= need + 1.5 GiB (:514) was tuned on discrete dual R9700s where
   free VRAM is distinct from host RAM. Here cudaMemGetInfo's total is exactly
   /proc/meminfo MemTotal, so its free is reported over the same unified pool
   the host BF16 expert cache draws from, and the admission test would
   double-count — on a box whose unified-memory OOM takes the host down with it.
4. The gap is isolated. Comparing overrides one by one, DeviceMemoryInfo is
   the only optional vt::Backend seam ROCm answers and CUDA does not.

One correction to the issue text. The fallback is not host compute.
ExpertGeGLUHost (gemma4_moe.cpp:49-74) re-copies the expert's BF16 bytes
host-to-device (:59-60), runs the same device GEMM, and drains the queue per
expert (:73). The issue's own "falls back to host H2D on every expert" is the
accurate wording; the dead LRU costs bandwidth and a per-expert Synchronize,
not a wrong answer.

Step 3 of "What closing this looks like" must not be done. It proposes the
#1123 fit check read the budget from this seam. That would be a defect, and the
duplication it means to remove is not duplication:

• vt::Backend::DeviceMemoryInfo(free, total) is a live probe that moves with
  contention — authoritative for a runtime admission decision, and nothing else.
• ResidencyPolicy::device_memory_total_bytes is a total probed once at
  registration, 0 == UNKNOWN — authoritative for a load-time budget verdict,
  which must be reproducible and contention-independent.

Collapsing the second onto the first makes the same checkpoint load or be refused
depending on the page cache, which is the property total-not-free was chosen
to avoid. What this issue owes is steps 1 and 2 together — the override and
the Gemma4 measurement — never step 1 alone, and never step 3. That is now what
## Owed says.

Also filed in flow: #1197, a second defect in the same MakeRoom — the slot cap
is tested before the eviction loop, so VT_GEMMA4_EXPERT_EVICT=1 goes inert after
24 admissions.