Skip to content

The runner hands device_token_ids to forwards that never read it, and nothing detects the ones that do not #1217

Description

@localai-bot

ModelForwardInput::device_token_ids (include/vllm/model_executor/models/model_registry.h:314-324) carries the async runner's device-combined input ids, and its contract is that token_ids is stale for decode rows whenever the pointer is non-null. A registered forward that embeds the host vector instead embeds the same placeholder id on every decode step.

The field's own comment says a model that ignores it "is simply never given one (the runner only sets it on the discrete-CUDA async path)". That is not a mechanism, it is a hope: runner.cpp:1408 sets the pointer for whatever model the step routes to, with no per-model opt-in and no check.

Two models have now been cut from the identical divergence:

  • Kimi-Linear, repaired at src/vllm/model_executor/models/kimi_linear_device.cpp:2270-2280 ("the GB10 9/128 divergence this branch was cut from").
  • NemotronH's paged forward, repaired under #1157: its A3 token gate read 4/24 on GB10 while the same binary read 96/96 on CPU, where the pointer is always null.

Nothing can see the next one. The runner sets the pointer only under VLLM_CPP_CUDA with a live device mirror, so no CPU gate reaches the branch at all, and the failure it produces is fluent wrong tokens rather than an error.

Two closes worth considering, and they are not exclusive:

  1. Make the comment true. Give ModelFactory an explicit honors_device_token_ids and have the runner fall back to the synchronous host path for a forward that has not declared it. A model that ignores the field is then structurally never given one. Every registered forward except this one already honours it, so the opt-in is a mechanical edit.
  2. A checker. The unit is the registered .forward entry point, not the file: several models delegate through detail::DeviceTokenIdsScope or the shared dense block, so a file-level grep flags ~25 false positives. It needs real routing analysis and therefore its own spec and red-before test.

Owned by MODEL-NEMOTRON-H-ABI-A2P and listed under ## Owed in .agents/specs/nemotron-h-a2p-paged-forward.md. Not fixed in the #1157 flow because close (1) changes a shared seam and every model factory, and close (2) changes checker semantics — both are the "needs its own spec" case AGENTS.md names, not the in-flow case.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions