fix(FIX-FP8-BLOCKWISE-REFUSAL): refuse block-wise FP8 by name at load (#1166)

[localai-bot]

Qwen/Qwen3.8-27B-FP8 is block-wise (fine-grained 128x128) FP8, and this build implements per-tensor FP8 only. The load already stopped, so this was never wrong numerics. It stopped on the wrong sentence.

The dense loader branches on the weight dtype alone (qwen3_5_dense_weights.cpp:479), and a block-wise weight really is F8_E4M3, so the projection entered the per-tensor arm and LoadFp8Raw asked for <proj>.weight_scale (qwen3_5_weights.cpp:458). A block-wise checkpoint spells that tensor weight_scale_inv, so the load died on tensor not found: ...q_proj.weight_scale. Nothing was missing from the checkpoint. The reader was sent after a tensor upstream never writes in this mode instead of being told that the fine-grained arm is absent.

RefuseUnsupportedFp8BlockQuant now reads quantization_config.weight_block_size and refuses with a message naming the key, its value, the arm that is missing, the arm that works, where the scale actually lives, and this issue. It is sited in ModelRegistry::Load after Resolve and before load_weights, so an unsupported architecture still reports the architecture, and the guard covers every architecture and the GGUF arm rather than one loader.

The verdict, since two other outcomes were plausible

Neither a clean refusal nor a silent misload. The third case, and the distinction decides the severity.

The silent-misload mechanism is real and was checked rather than assumed. ReadF32Scalar (src/vllm/model_executor/models/qwen3_5_weights.cpp:312-318) bounds its input with t.nbytes >= sizeof(float), a LOWER bound, then memcpys the first 4 bytes. A [96, 40] block scale passes that check and reads as block (0,0), which LoadFp8Transposed (:471, the dequant arm that needs no input_scale) would then apply to the whole [N, K] weight. That is the shape a token gate cannot see.

It does not fire, and the reason is the NAME rather than the shape guard. The lookup asks for weight_scale, the checkpoint spells it weight_scale_inv, and the resolver (qwen3_5_dense_weights.cpp:682) throws before the scalar read happens. Upstream makes that spelling strictly conditional on block quant ("weight_scale_inv" if self.block_quant else "weight_scale", fp8.py:511), so no upstream block-wise checkpoint reaches it. The hole is therefore latent and unreachable by this route. It is recorded under ## Owed rather than fixed, because hardening it is a separable unit of work.

Measured, not inferred

config.json and one safetensors header read live at revision 017b9c7af6b5689d5dd426a76e0bc077eb5ca20a by HTTP range request, with no checkpoint download and no GPU lease.

Fact	Value
quant_method	fp8
weight_block_size	[128, 128]
activation_scheme	dynamic
self_attn.q_proj.weight	F8_E4M3 [12288, 5120]
self_attn.q_proj.weight_scale_inv	BF16 [96, 40], exactly [12288/128, 5120/128]
input_scale tensors in the shard	0

git grep weight_block_size -- src/ include/ returned nothing before this change, and no load path reads weight_scale_inv.

Upstream, at the parity pin

Mirrors Fp8Config.from_config and Fp8LinearMethod at 5559679229bc961848b121ccdeaa8fa5d79bec98, verified with git rev-parse in the oracle checkout before citing: vllm/model_executor/layers/quantization/fp8.py:161 reads the key, :115-132 validates it, :297-298 makes it the dispatch, and :378-379 and :511 register the block scale as weight_scale_inv.

Evidence

CPU only. RED first, then green.

• Focused RED, before the fix existed: test_fp8_block_quant at 2 of 4 cases and 11 of 19 assertions passing, exit 1.
• Focused GREEN: 4 of 4 cases and 19 of 19 assertions, exit 0. The assertion count is unchanged between the two runs, so nothing became vacuous.
• Reachability, re-run at this head: deleting the RefuseUnsupportedFp8BlockQuant(config) call site compiles cleanly (compile_rc=0, so this is a test failure and not a build failure) and reds the target at 10 of 19 assertions, exit 1. The tree was restored byte-for-byte, verified by sha256 525b242f... before and after, with git status --porcelain empty.
• Full gate at this head: scripts/agent-preflight.sh exit 0, All gates green., and the real per-block counts sliced at the verdict line before .agents/NOW.md is appended are 79 ok, 0 FAIL, 0 SKIP.

Every case enters through ModelRegistry::Load rather than through the predicate, which is why the mutation reds it. A unit test that called the predicate directly would prove the function works and never that a load reaches it.

The negative controls are not decorative. One holds that a per-tensor fp8 config still loads past the guard, without which the gate would pass for a refusal that fires on every fp8 checkpoint. One holds that null and empty are not block quant, mirroring the upstream default of None.

Records

docs/FEATURES.md claimed "Loads BF16/FP8/NVFP4", which overstated what this build reads, and its fp8 weights row was an unqualified check mark. Both are narrowed to per-tensor, and the refused arm gets its own row in the quantization table. docs/USAGE.md documents the refusal beside the checkpoints it applies to, so a reader who hits the message finds it. No checker was weakened and no budget was raised: the diff touches no file under scripts/, and MAX_CELL_CHARS, MAX_ROW_CHARS and UNOWNED_HIGH_WATER are unchanged. The docs/FEATURES.md cell was paid for by shortening the row that owns it.

.agents/quantization-matrix.md is deliberately untouched. QUANT-FP8-GENERIC already records the block axis at PARTIAL and QUANT-FP8-PB-WO at INVENTORIED, and no row changes lifecycle state here.

What stays owed

Block-wise execution itself, meaning reading weight_scale_inv, applying a 128x128 block scale, and the dynamic per-token activation quant upstream pairs with it. That needs a GPU gate and a checkpoint, and it is listed under ## Owed in .agents/specs/fp8-blockwise-refusal.md. Issue #1166 stays open after this lands, because this change makes the gap legible and does not close it.

Closes nothing. Tracked by #1166

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]