perf(qwen35): retain speculation during paged prefill

[Graffioh]

Status

This remains a draft. The mixed-row lowering is implemented, and an initial staggered A/B/A exposed a deterministic direct-row numerical error. Commit 312beb19c5b680111a0a21e2af9aa7dae49531ee contains the causal fix and a boundary regression, but the corrected commit has not yet completed GPU qualification because the device was handed back to another user.

• Upstream base: cfa76afe30ade3e53456da86bd957159a121f4bc
• Candidate: 312beb19c5b680111a0a21e2af9aa7dae49531ee
• Campaign: qwen-mixed-prefill-spec
• Trial: mixed-prefill-spec-lanes-0001 (draft)
• Campaign record: Luce Forge draft PR #90

Do not merge this PR until the corrected GPU contract, exact-output A/B/A, steady-state controls, and matched profiles pass.

Mechanism

Qwen now lowers one target step as:

[paged ragged prefill segments][ordinary direct/AR rows][fixed-width speculative tree rows]

select_chain_lanes() no longer disables every eligible chain lane merely because the same common::SeqEngine::StepPlan contains prefill work. The common scheduler and plan types are unchanged.

The Qwen-private builder reuses the existing packed-prefill and fixed-chain machinery. It compacts LM-head rows as completed prompt tails, ordinary direct outputs, and tree verification rows, so long prompt interiors do not reach the LM head. Its replay key contains the ordered prefill topology, direct-row shape, logits shape, tree bucket, and padded KV span. Capacity accounts for prefill, optional AR, and tree recurrence segments.

The attention call is split only when a tree graph has a direct prefix:

• positioned prefill/AR rows use ordinary paged attention;
• the fixed-width suffix uses tree paged attention and ancestry metadata;
• their outputs are concatenated back into the original row order before the rest of the block.

Q/K/V projections and KV writes remain shared across the batch. Pure-tree and ordinary packed paths are unchanged. This preserves ordinary direct-row partitioning while still verifying the tree in the same target graph.

State and commit invariants

• Prefill and AR rows update durable paged KV, convolution state, Gated DeltaNet state, and target features in sequence order.
• Tree rows write scratch KV and tree-local captured convolution, Gated DeltaNet, and feature state.
• Direct features never enter the speculative promotion transaction.
• ggml_backend_cuda_tree_commit_transaction() remains the only tree promotion path and promotes only the accepted prefix across KV, features, convolution state, and Gated DeltaNet state.
• Parent metadata describes only the fixed tree suffix. Prompt and AR rows remain positioned causal reads.
• Runtime failure after durable graph mutation remains cohort-fatal under the existing StepResult contract. The scheduler retires the cohort; slot admission reinitializes recurrent state. Tree promotion still preflights all domains before mutation.

No universal row planner, common scheduler policy, or merged durable/scratch commit path is introduced.

Correctness failure found during qualification

The first candidate retained speculation and produced a large live-decoder speed signal, but one arriving request changed output from its first generated token. Baseline-before and baseline-after were byte-identical, the compact prompt-tail row mapping was correct, and the top-logit gap was large enough to rule out a sampling tie.

The root cause was paged-attention partitioning at the 1,024-token boundary. With 16-token blocks, the ordinary direct call used 64 blocks and one partition. Adding a width-8 tree suffix made the combined launch size 1,032 tokens, or 65 blocks and two partitions. Direct rows were then rounded through the half-precision partial buffer even though their own causal span had not changed. Recurrent state was byte-identical through the preceding layers and first diverged immediately after that full-attention layer.

The current commit separates direct and tree attention calls inside the mixed graph. The new model contract pins this boundary with two established decoders and two ragged prefills, compares the completing prompt-tail token, checks the exact compact row map and W8 tree topology, and requires every durable convolution and SSM slab to remain byte-identical to speculation-disabled execution.

Current verification

Completed on the current commit without using the GPU after it was released:

• git diff --check: pass
• HIP Release build: test_generate, test_model_smoke, and dflash_server: pass
• host sequence-engine, slot-manager, batch-plan, and recurrent-snapshot CTest selection with devices hidden: 7/7 pass
• test_generate --seq-engine-mixed-spec-contract rebuilt after the final sampler-correctness cleanup
• independent static review of row shapes, attention slices, recurrence offsets, cache keys, tree-local feature indices, and promotion ownership: no blocking finding

An earlier short Qwen3.8/DFlash2 contract passed against the split production code, but it predates the final 1,024-token boundary fixture and is not the qualification result. The corrected boundary contract and corrected staggered workload have not run.

Pushing the draft automatically queued upstream self-hosted CI. Two CUDA GPU checks finished before this was noticed; the two ROCm checks were still pending. An attempt to cancel the workflow returned GitHub 403 because the fork identity lacks repository Actions administration. Treat any automatic CI result as incidental, not as the corrected qualification bracket.

Model and machine provenance

The frozen qualification pair is:

• target: Qwen3.8-27B-UD-IQ4_XS.gguf, source unsloth/Qwen3.8-27B-GGUF@4ca720…, 14,252,845,984 bytes, SHA-256 40fac4050e940397dbf13087afd50f4734a11805bf9d65ef8ddd7483470e6199
• draft: Qwen3.8-27B-DFlash2 Q8_0, source incoai/Qwen3.8-27B-DFlash2@dedf8df68adfb1afeaf7b7480c0a0243108177b4, 2,045,471,776 bytes, SHA-256 bb727abc583498aa4deea8b3cd0c34c2d96553954cbff25b5f7bdd469f0f1306
• draft width: 8; captured layers: [5, 19, 33, 47, 61]
• device used for the initial diagnostic bracket: AMD Radeon AI PRO R9700 (gfx1201), ROCm 7.2, Q8 KV

The initial host did not satisfy the frozen clock/cooling topology preflight, so its timings are diagnostic only in addition to failing the output gate.

Quarantined initial A/B/A

Frozen workload: two established HumanEval-raw decoders generating 512 tokens each; after exactly 2.000 seconds, one medium 1,150-word and one long 3,400-word prompt arrived and generated 64 tokens each. Temperature was 0, seed 1, EOS ignored, five slots, 8,192-token context, 40,960 KV tokens, W8, Q8 KV, and graph/prefix caches disabled.

Metric	Baseline A	Initial candidate	Baseline A2
Established live goodput	2.628 tok/s	10.711 tok/s	2.629 tok/s
Effective established TBT	380.571 ms	93.363 ms	380.398 ms
Arriving TTFT, p50	4,661.618 ms	4,827.598 ms	4,664.837 ms
Prompt throughput	840.021 tok/s	805.675 tok/s	840.403 tok/s
Full round	15.250 s	14.398 s	15.264 s

A/A2 drift was 0.02% for live goodput and 0.09% for full-round time. The candidate used speculation in all eight mixed rounds and accepted 68 of 147 verified draft tokens. However, its output-set hash differed (cd4b5b… versus identical A/A2 ded4dec…). These numbers are quarantined and are not a performance result.

Relation to Luce Forge PR #67

PR #67 reports synchronized, steady fixed-concurrency waves: ten HumanEval waves per concurrency with 256 generated tokens per request. Its output-window rates are 101.9, 189.5, 219.5, 268.3, and 298.4 tok/s at C1 through C5.

The 10.711 tok/s value above is established-decoder goodput only inside an overlapping prefill window. It is neither total server throughput nor comparable to PR #67. The corrected commit therefore has no valid improvement result against that baseline yet. Structurally, decode-only execution does not enter this mixed path, but matched C1/C2/C5 controls still have to prove no steady regression.

Qualification handoff

Run no broader HumanEval sweep than the frozen ten-case subset. In order:

1. Run test_generate --seq-engine-mixed-spec-contract <target> <draft> 4. This is the smallest old-exclusion, prompt-tail, W8, 1,024-token partition-boundary, and durable-state regression.
2. Run the adjacent Qwen sequence-engine, target-graph, paged-attention, replay-log, recurrent-state, and server suites, plus one live smoke. Cover rejection, pending-token handling, transaction preflight failure, cancellation, retire/reuse, padded buckets, repeated replay, multiple ragged prefills, and mixed speculative/AR-only lanes.
3. Repeat the exact frozen staggered baseline/candidate/baseline bracket. Require byte-identical complete outputs and transaction behavior before reading performance.
4. Run ordinary packed-prefill and decode-only C1/C2/C5 controls using the same ten-case subset.
5. Capture matched normal and ROCprof/LuceGraph runs for target/draft phase time, launches, dispatch time, graph-cache reuse/rebuilds, acceptance, and peak memory.

The preregistered promotion gate is at least 1% median established-decoder goodput improvement during prefill, greater than two times measured A/A2 drift, with unchanged outputs/transactions and no material steady decode regression.

Remaining limitations

• Mixed full-attention layers add one paged-attention launch and one concatenation. Their cost is unmeasured.
• Paged-attention partition count also depends on query-row occupancy. The 64-to-65-block corruption is addressed, but small prompt tails near other occupancy thresholds need padded-bucket and forced-partition coverage. If that fails, preserve the ordinary direct bucket with dead positioned rows; do not reintroduce the prefill exclusion.
• The corrected output bracket, benchmark, graph-cache/memory evidence, and matched ROCprof/LuceGraph captures remain outstanding.