NemotronH device MoE re-uploads the router gate and BLOCKING-SYNCS 46 times per decode token

[localai-bot]

Found by a source audit of the decode gap. Not profiled — the counts are from call sites and the byte figures are arithmetic from config.json.

What happens per decode token

src/vllm/model_executor/models/nemotron_h_device.cpp::NemotronHMoeBlockDevice:674-676 calls UploadOwned for mixer.gate.weight and e_score_correction_bias on every call.

UploadOwned (:431-445) ends in d.b.Synchronize(d.q) — a full queue drain.

NemotronH has 23 MoE layers, so per decode token that is:

• 46 H2D copies of weights that never change
• 46 full-device synchronisations
• 30.19 MiB/token of avoidable transfer (the gate alone is [128, 2688] f32 = 1.312 MiB)

46 pipeline drains per token is the clearest mechanical explanation for a GPU that is idle most of the wall clock. The measured decode occupancy on GB10 was 6.31%.

Why it is the cheapest item on the list

The seam already exists in this tree, one line away:

• include/vllm/model_executor/models/dense_attn_block.h::ResidentWeight:177 — upload once, cache in w.d_dev
• ::ResidentWeightF32:208
• and this very file already uses the idiom: ResidentIn<NemotronHMoeMarlinResident> at :665, correctly if (!mr.ready)-guarded

vLLM holds the router as an nn.Parameter; there is no per-step transfer upstream to cite, because none exists.

This is independent of A2-Q1 (#1289) and A2-Q2b — verified against gh pr diff 1289, which does not touch it.

Secondary defect in the same function

DenseMarlinE1 (:581-608) builds a std::vector<float> ones(M) and uploads it per call with no synchronisation, plus a cudaMemset and a MarlinMoeAlignBlockSize, twice per MoE layer. Low cost at M=1, but the un-synced pageable upload is exactly the use-after-free shape that UploadAs (:183-189) documents. A dense GEMV is also being routed through the grouped-MoE alignment machinery.

Confirm or refute

1. Count cudaStreamSynchronize per decode step with nsys.
2. Hoist the two uploads into a resident slot and A/B the same binary.

Refuted if the sync count is already near zero — i.e. if Synchronize is a no-op on this backend. That is worth checking FIRST, since it would make this a bandwidth issue only rather than a serialisation one.