feat(BACKEND-TENSTORRENT-HOST-FREE-FORWARD): capture the Qwen3 decode graph and advance cur_pos on device (#1105)

[lu-zero]

feat(BACKEND-TENSTORRENT-HOST-FREE-FORWARD): capture the Qwen3 decode graph and advance cur_pos on device (#1105)

The Tenstorrent decode path hung after about 38 replays because each
step copied cur_pos and update_idxs into the live trace. This change
captures the Qwen3 dense decode graph and advances cur_pos on-device
with ttnn::plus_one, matching the upstream executor.py pattern.

A P150 run of Qwen3-0.6B "Hello" at 80 tokens completed 79 replays
with no hang, 5.8x vs eager, and 22/22 argmax vs the per-step-copy
baseline. The path is inert unless VT_TT_HOST_FREE_DECODE is set.

A new batch size after the first capture is refused rather than
freezing cur_pos. CPU and Vulkan builds link via header no-ops.
MSVC accepts the MoE refuse path. VT_TT_DUMP_KV is paid only on
kTENSTORRENT.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:grok-4.6 [Grok]

[localai-bot]

Holding this for one change, and it's not the build break.

First the easy part: qwen3.cpp calls eight vt::tenstorrent:: symbols with only a
runtime device.type == kTENSTORRENT guard, never a compile-time one.
VLLM_CPP_TENSTORRENT resolves AUTO -> OFF (CMakeLists.txt:300-301) and
tenstorrent_ops.cpp only compiles under if(VLLM_CPP_TENSTORRENT) (:1468), so every
default build fails to link. That is the single cause of all four fresh reds --
verify, build-test-cpu-arm64, and both sanitize-cpu. About four #ifdef hunks; the
file already uses that shape at :89 for VT_MARLIN_NVFP4.

The reason we are holding is GraphCapturesCounter(). It only ever increments --
NoteGraphCaptured() never decrements and DestroyGraph does not reset it -- and three
sites branch on GraphCapturesDone() > 0. When a DecodePosCache entry is first needed
after the first capture, WarmDecodePos returns early and never seeds it; WarmPaMeta
then only aliases e.cur_pos inside if (!e.allocated), so aliased stays false, a
standalone tensor is created, and if (!r2_steady) skips the per-step copy_to_device
from then on. cur_pos is frozen and sdpa_decode reads the wrong KV length on every
replay. WarmRacIdx has the same shape for update_idxs.

It triggers on a new request, a batch-size change, or a block-boundary crossing that grows
max_blocks -- and there is no VT_CHECK, no log, no divergence signal. It produces
plausible wrong tokens, which is precisely what a token gate cannot see. We suspect it is
also the mechanism behind the multi-prompt engine-gate timeout you already flagged.

We could have added a guard ourselves, but you have the Blackhole silicon and we do not --
a fix we cannot execute is not one we should be writing. The shape we would suggest is
making the discriminator per-cache-entry (a seeded_at_capture / generation field on the
entry) rather than one process-global counter, and resolving the alias on every
WarmPaMeta/WarmRacIdx call instead of only on first allocation.

Two things only you can close: there is no GitHub issue for this work (policy wants it
linked in .agents/issue-index.md, the spec, and the PR body, all agreeing), and the PR
body needs the trailer block -- squash_merge_commit_message = PR_BODY here, so the body
is the landed commit message.

The specs are unusually good, and the measured 5.8x replay result is real. #694 we are
closing as superseded -- all 27 commits are in this branch rebased, and its unique
vllm_c.cpp fix landed as #814.

[localai-bot]

Re-reviewed at c63a9387e. Thanks for the rewrite — you closed the two things
only you could close:

• Tenstorrent host-free decode graph: capture, replay, and on-device state advance #1105 is filed and cited in the title. That was the one gap we could not
  fill for you.
• The PR body now carries the trailer block and passes
  check-commit-trailers.py --filled. That one matters more than it looks:
  this repo squashes with squash_merge_commit_message = PR_BODY, so the body
  is the landed commit message.
• 43 commits squashed to 1.

Both code blockers are still open, though. Checked on c63a9387e, not
inferred from the previous revision:

1. The link break — qwen3.cpp still has no compile-time guard

$ git show c63a9387e:src/vllm/model_executor/models/qwen3.cpp | grep -c 'vt::tenstorrent::'
9
$ git show c63a9387e:src/vllm/model_executor/models/qwen3.cpp | grep -c '#ifdef VLLM_CPP_TENSTORRENT'
0

VLLM_CPP_TENSTORRENT resolves AUTO -> OFF at CMakeLists.txt:300-301, and
tenstorrent_ops.cpp is compiled only inside if(VLLM_CPP_TENSTORRENT) at
:1468. So every default build still fails to link on those nine call sites,
which is the whole of the verify, build-test-cpu-arm64 and both
sanitize-cpu reds. About four #ifdef/#endif pairs; the same file already
uses that shape at :89 for VT_MARLIN_NVFP4.

We are still happy to do this one for you — say the word and we will push it
to your branch rather than have you spin another round trip on it.

2. The frozen cur_pos — unchanged

:3220  std::atomic<int>& GraphCapturesCounter() {
:3225  void NoteGraphCaptured() { GraphCapturesCounter().fetch_add(1); }
:3986  if (GraphCapturesDone() > 0) return;        // WarmDecodePos

The counter still only increments — nothing decrements it and DestroyGraph
does not reset it — and it still gates at :3742, :3921 and :3986.
WarmPaMeta still resolves the alias only inside if (!e.allocated), with
aliased set only when the DecodePosCache entry is already allocated. So when
a DecodePosCache[num_reqs] entry is first needed after the first capture,
WarmDecodePos returns early and never seeds it, aliased stays false, a
standalone tensor is created, and the if (pt_changed || !r2_steady) guard then
skips the per-step copy_to_device from then on. cur_pos is frozen and
sdpa_decode reads the wrong KV length on every subsequent replay.
WarmRacIdx has the same shape for update_idxs.

It fires on a new request, a batch-size change, or a block-boundary crossing that
grows max_blocks — ordinary things. There is still no VT_CHECK, no log and no
divergence signal on that path, so it produces fluent, plausible, wrong tokens.
That is the one defect class our token gates structurally cannot see, which is
why we are holding on it rather than taking it as recorded debt.

We think it is also the mechanism behind the multi-prompt engine-gate timeout you
flagged yourself.

Two shapes would unblock it, and either is fine by us:

• Fix it: make the discriminator per-cache-entry — a seeded_at_capture or
  generation field on DecodePosEntry / PaMetaEntry / the RAC entry — and
  resolve the alias on every WarmPaMeta / WarmRacIdx call rather than only on
  first allocation.
• Refuse it: VT_CHECK loudly when r2_steady is true but the entry was
  never aliased. Then the path stops instead of returning wrong numbers, and the
  real fix can land behind it with an ## Owed entry and an issue.

We cannot pick between those for you, because we have no Blackhole silicon here
and neither option is one we could test.

The specs remain unusually good and the measured 5.8x replay result is real —
this is worth landing once those two are closed.

[localai-bot]

Correction — I was wrong about the link break, and I apologise. You did fix
it, and my previous comment said otherwise.

I grepped qwen3.cpp for #ifdef VLLM_CPP_TENSTORRENT, found none, and
concluded the guards were missing. You put them in the header instead, which is
the better place:

• src/vt/tenstorrent/tenstorrent_device.h wraps all eight symbols in
  #ifdef VLLM_CPP_TENSTORRENT with #else inline …{} no-op fallbacks
  (:77-85, :95-103, :109-116, :123-127, :132-136, :143-147,
  :156-163, :167-173).
• tenstorrent_ops.cpp:24-27 force-defines the macro before the include,
  because the vllm_tenstorrent_ops OBJECT library does not inherit the
  PUBLIC define from the vllm target (CMakeLists.txt:1470).

That chain is sound: the header pulls only <cstdint>/<memory> and an opaque
MeshDevice forward declaration, qwen3.cpp:59 includes it unconditionally,
and src is a PRIVATE include dir. The default build links. So one call site
per symbol stays in the model file and the conditional lives once in the header,
which is cleaner than the four #ifdef pairs I proposed. My grep proved my
search terms wrong, not your fix absent.

I also want to retract a scare before it reaches you: we checked whether this
branch reverts #1069 (an MSVC fix that landed on main a few hours ago in the
same file). It does not. Your qwen3_5_weights.cpp blob differs from main's
by three comment lines only — you carry the same kRequired →
kMoeExpertLayoutHelp change — and it auto-merges to main's exact content.

What is still open is the one blocker: the frozen cur_pos. That part of my
previous comment stands, re-verified on c63a9387e. GraphCapturesCounter()
only fetch_adds, WarmDecodePos still early-returns on
GraphCapturesDone() > 0, and WarmPaMeta still resolves the alias only inside
if (!e.allocated) — so an entry first needed after the first capture is never
seeded and sdpa_decode reads a wrong KV length, silently. Either fix shape from
my last comment works for us; we cannot pick between them without a P150.

Two smaller things worth your attention, and then an offer.

qwen3.cpp:821 and :834 put VT_TT_DUMP_KV getenv calls inside the
if (do_replay) block with no d.q.device.type == kTENSTORRENT guard, so every
Qwen3/Mistral/Llama/InternLM2 decode replay on CUDA now pays two uncached
getenv calls per step. You guarded VT_TT_RECAPTURE_EVERY correctly at
:797-798, so this looks like an omission rather than intent.

And the commit currently contradicts itself: docs/ENVIRONMENT.md still says
VT_TT_HOST_FREE_DECODE is "Incomplete (capture aborts at ReshapeAndCache)"
while docs/BENCHMARKS.md in the same commit records 79 replays at 5.8x.

The offer: everything except the cur_pos fix is records and docs, and we
are happy to push it to your branch so your round trip is only about the code
that needs the card. That would be: the ENVIRONMENT.md row, a ## Owed
section listing the seven constraints your own
tenstorrent-host-free-forward.md:228-251 already enumerates, a ## Now for the
new tenstorrent-trace-runner.md, the missing TRACE-RUNNER row in
docs/STATUS.md, and softening the BENCHMARKS cell from "Measured" to the
framing its sibling rows use until we rerun the gate here — CLAUDE.md is strict
that an implementer's run is an input, not a gate result, and your own ## Now
says the same.

Say the word and we will push those; the cur_pos seeding fix and an on-card
rerun are what we would still need from you.