[Bugfix] PP + MTP + prefix caching corrupted mamba recurrent state (Qwen3.8 "duct" loops)

[Karl0007]

PP + MTP + prefix caching corrupted mamba recurrent state (Qwen3.8-Flash-Next "duct" loops)

Fixes the silent state corruption that made 14–33 % of requests degenerate into a constant-token loop whenever pipeline parallelism, MTP and prefix caching were combined — i.e. exactly the MODE=pp Qwen3.8-Flash-Next recipe this repo enabled in #47. Reproduced here at PP4 13,13,13,9 + MTP4 + --enable-prefix-caching --mamba-cache-mode align + FULL_AND_PIECEWISE, max_num_seqs 8.

Root cause

MambaSpecDecodeGPUContext captures the block tables' raw data_ptr once (idempotent initialize_from_forward_context), and the align copy kernels resolved the table row as batch_idx if HAS_IDX_MAPPING else req_idx — while the V2 runner bound that capture to the per-step gathered input_block_tables, which are batch-ordered and re-gathered every step.

Under async PP, a non-last rank runs postprocess pp_size steps late, so its batch mapping is stale: the kernel walks the current tables with a stale row mapping and copies GDN/PLE state through block ids belonging to (or reallocated to) other requests. The CSA unified layout aliases main KV, GDN conv/SSM and PLE conv inside one page, distinguished only by block-id ownership — so foreign bytes land in live state. Symptomatically that is all-NaN logits; the sampler turns an all-NaN row into token 1023 = duct, forever. On sm_121 the same misdirected read faults outright (vllm#54173 / #54199).

Changes

file	delta
vllm/v1/worker/mamba_utils.py	both copy kernels index the source tables by req_idx; comment corrected to state the real invariant
vllm/v1/worker/gpu/model_runner.py	bind the ctx to the source per-request-slot tables (BlockTables.block_tables[i].gpu), not the gathered views
vllm/v1/worker/gpu/model_states/mamba_hybrid.py	vllm#53142 (no upstream PR): align state-seed divisor must be the mamba group's block size, else a resume over a cached prefix seeds an out-of-range column and the pre-copy reads a garbage block id
vllm/v1/core/single_type_kv_cache_manager.py	vllm#48375 (open, unmerged): MambaManager.find_longest_cache_hit accepted drop_eagle_block and ignored it, so an MTP prefix-cache resume landed on a page whose recurrent-state snapshot was taken over draft tokens verification had rejected
tests/kernels/mamba/test_precopy_mamba_align.py	row-attribution tests: permuted idx_mapping plus per-slot-differing decisions. The existing suite only ever used idx_mapping = arange(n), where batch row == request slot, so it structurally could not see this bug

Measured (4× CMP 170HX, sm_80)

	before	after
8-way soak, same harness	16/48 loops (33 %)	0/56 loops
mean acceptance length	2.80	4.90–5.00
294 K / 509 K needle (bench/qwen_longctx_needle.py)	—	both exact
soak round time	36.8 s	31.6 s

No perf cost by construction: same kernels, same launch counts, same copied bytes — only the row index changed.

Do not "fix" this with a table pool

A rotating pool of gathered-table sets is not a substitute: with full CUDA graphs a captured graph bakes whichever slot's pointer was current at capture, and the other replays run against stale tables — the same corruption by another road. With this fix the ctx never touches gathered tables, so there is nothing to rotate.

Upstream

Filed against vllm-project/vllm as vllm#55506 (core fix + tests) and vllm#55507 (#53142 divisor). #48375 is left to its own open PR; this branch carries the port so production is correct today.

Credit: root cause and fix direction identified first by @zebgop-ops — forensics at https://github.com/zebgop-ops/qwen38-flashnext-pp (FINDINGS.md addenda 11–19, patches 0010/0011/0012; their ablation established that req-indexed source tables are necessary and sufficient).