ModelForwardInput::device_token_ids (include/vllm/model_executor/models/model_registry.h:314-324) carries the async runner's device-combined input ids, and its contract is that token_ids is stale for decode rows whenever the pointer is non-null. A registered forward that embeds the host vector instead embeds the same placeholder id on every decode step.
The field's own comment says a model that ignores it "is simply never given one (the runner only sets it on the discrete-CUDA async path)". That is not a mechanism, it is a hope: runner.cpp:1408 sets the pointer for whatever model the step routes to, with no per-model opt-in and no check.
Two models have now been cut from the identical divergence:
- Kimi-Linear, repaired at
src/vllm/model_executor/models/kimi_linear_device.cpp:2270-2280 ("the GB10 9/128 divergence this branch was cut from").
- NemotronH's paged forward, repaired under #1157: its A3 token gate read 4/24 on GB10 while the same binary read 96/96 on CPU, where the pointer is always null.
Nothing can see the next one. The runner sets the pointer only under VLLM_CPP_CUDA with a live device mirror, so no CPU gate reaches the branch at all, and the failure it produces is fluent wrong tokens rather than an error.
Two closes worth considering, and they are not exclusive:
- Make the comment true. Give
ModelFactory an explicit honors_device_token_ids and have the runner fall back to the synchronous host path for a forward that has not declared it. A model that ignores the field is then structurally never given one. Every registered forward except this one already honours it, so the opt-in is a mechanical edit.
- A checker. The unit is the registered
.forward entry point, not the file: several models delegate through detail::DeviceTokenIdsScope or the shared dense block, so a file-level grep flags ~25 false positives. It needs real routing analysis and therefore its own spec and red-before test.
Owned by MODEL-NEMOTRON-H-ABI-A2P and listed under ## Owed in .agents/specs/nemotron-h-a2p-paged-forward.md. Not fixed in the #1157 flow because close (1) changes a shared seam and every model factory, and close (2) changes checker semantics — both are the "needs its own spec" case AGENTS.md names, not the in-flow case.
ModelForwardInput::device_token_ids(include/vllm/model_executor/models/model_registry.h:314-324) carries the async runner's device-combined input ids, and its contract is thattoken_idsis stale for decode rows whenever the pointer is non-null. A registered forward that embeds the host vector instead embeds the same placeholder id on every decode step.The field's own comment says a model that ignores it "is simply never given one (the runner only sets it on the discrete-CUDA async path)". That is not a mechanism, it is a hope:
runner.cpp:1408sets the pointer for whatever model the step routes to, with no per-model opt-in and no check.Two models have now been cut from the identical divergence:
src/vllm/model_executor/models/kimi_linear_device.cpp:2270-2280("the GB10 9/128 divergence this branch was cut from").Nothing can see the next one. The runner sets the pointer only under
VLLM_CPP_CUDAwith a live device mirror, so no CPU gate reaches the branch at all, and the failure it produces is fluent wrong tokens rather than an error.Two closes worth considering, and they are not exclusive:
ModelFactoryan explicithonors_device_token_idsand have the runner fall back to the synchronous host path for a forward that has not declared it. A model that ignores the field is then structurally never given one. Every registered forward except this one already honours it, so the opt-in is a mechanical edit..forwardentry point, not the file: several models delegate throughdetail::DeviceTokenIdsScopeor the shared dense block, so a file-level grep flags ~25 false positives. It needs real routing analysis and therefore its own spec and red-before test.Owned by
MODEL-NEMOTRON-H-ABI-A2Pand listed under## Owedin.agents/specs/nemotron-h-a2p-paged-forward.md. Not fixed in the #1157 flow because close (1) changes a shared seam and every model factory, and close (2) changes checker semantics — both are the "needs its own spec" case AGENTS.md names, not the in-flow case.