feat(MODEL-FP8-BLOCK-WEIGHT): a block-wise FP8 checkpoint now LOADS, and says by name that it cannot run yet (#1189) - #1228
Merged
Conversation
…ung, and its config reader (#1189) Milestone **M3** of #1189, committed before the implementation as the protocol requires. It designs `Fp8BlockWeight`, the `weight_scale_inv` loader rung, and the quantization-config reader that narrows the named refusal landed in `469f38395` (#1166). Three things the design settles rather than assumes. **The BF16 scale dtype.** `Qwen/Qwen3.8-27B-FP8` ships `weight_scale_inv` as `BF16 [96, 40]` while upstream allocates the parameter `float32`. Both are true, and the resolution is that `BlockQuantScaleParameter` loads through `self.data.copy_(loaded_weight)` (`vllm/model_executor/parameter.py:95-108`), which CONVERTS. The parameter is `torch.float32` because `scale_dtype` is `None` unless `is_scale_e8m0`, an attribute `Fp8Config` does not define (`fp8.py:282,376`; `fp8_utils.py:1276`). So upstream widens the scale to f32 once, at load, losslessly. We mirror that, and the resident f32 is not a `.agents/porting.md` widening to justify: f32 is the dtype upstream carries. **Read the config, do not only probe the tensors.** `modules_to_not_convert` is a ~400-entry list a dtype probe reproduces by accident, and a probe cannot see a DISAGREEMENT between the config and the tensors at all. That is where a silent-wrong-scale bug lives, and #1166 recorded the near miss: a `[96, 40]` scale passed the old `nbytes >= sizeof(float)` floor and only the tensor NAME stopped it being applied to the whole weight. The spec tabulates six config/tensor combinations and refuses four of them by name. **Where the M4 gap is refused.** `ModelRegistry::Load` succeeds on a supported block-wise checkpoint, which is what makes the loader rung reachable from a production entry point at its own merge commit. Nothing consumes an `Fp8BlockWeight` yet, so `ModelRegistry::Prepare` refuses by name and quotes #1189 M4 rather than letting the dense `project` lambda fall through to an empty bf16 tensor. The checkpoint loads and declines to run; it never runs wrong. Scope stops at M3. No linear method, no forward wiring, no CUDA kernel, no GPU lease, and no checkpoint download. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…and says by name that it cannot run yet (#1189) Milestone **M3** of #1189: `Fp8BlockWeight`, the `weight_scale_inv` loader rung, and the quantization-config reader. `Qwen/Qwen3.8-27B-FP8` stops being refused at `ModelRegistry::Load` and its weights are read; nothing can execute them yet, so the model declines to be PREPARED rather than running through an empty tensor. Spec: [`.agents/specs/model-fp8-block-weight.md`](.agents/specs/model-fp8-block-weight.md), committed before the implementation as `fd87d4ffa`. Pinned oracle: vLLM `5559679229bc961848b121ccdeaa8fa5d79bec98`, asserted as the local checkout's HEAD before any anchor below was read. It consumes what `ad5f175e7` (M1, `vt::QuantFp8Group`) and `770e49486` (M2, `vt::MatmulFp8BlockScaled`) landed, and stops there. ## The BF16 scale dtype, established rather than assumed The checkpoint ships `weight_scale_inv` as `BF16 [96, 40]`; upstream allocates the parameter `float32`. Both are true, and the resolution is that torch's `copy_` CONVERTS: - the block scale parameter is allocated with `dtype = scale_dtype if scale_dtype is not None else torch.float32` (`utils/fp8_utils.py:1276,1283-1296`); - `scale_dtype` is `torch.float8_e8m0fnu if self.is_scale_e8m0 else None` (`fp8.py:376`), and `is_scale_e8m0` is `getattr(quant_config, "is_scale_e8m0", False)` (`fp8.py:282`) against an `Fp8Config` that defines no such attribute, so it is False and the parameter is f32; - `BlockQuantScaleParameter` loads through `self.data.copy_(loaded_weight)` (`vllm/model_executor/parameter.py:95-108`, inherited at `:397-403`), a dtype-CONVERTING copy; - downstream code then asserts the scale is f32 (`fp8_utils.py:1103-1112`), which is only consistent because the widening already happened. So `Fp8BlockWeight::scale` is f32, and that is the MIRROR rather than a `.agents/porting.md` widening: f32 is the dtype upstream carries resident, `vt::MatmulFp8BlockScaled` refuses anything else, and `bf16 -> f32` is exact. `LoadFp8BlockRaw` switches on the on-disk dtype with NO default branch and refuses any other by name, because `f22c6cc82` (#1181) landed a guard for a reader that memcpy'd four bytes whatever the dtype was. `vt::LoadUnaligned`, because a safetensors offset can be odd (#627). ## Read the config, do not only probe the tensors `modules_to_not_convert` is a ~400-entry list a dtype probe reproduces only by accident, and a probe cannot see a DISAGREEMENT between the config and the tensors at all. That is exactly where a silent wrong-scale bug lives: #1166 measured a `[96, 40]` grid passing a per-tensor reader's byte floor and being applied to the whole weight, stopped only by the tensor NAME. `IsFp8BlockProjection` therefore decides from BOTH sources and refuses four combinations by name: a `weight_scale_inv` with no `weight_block_size` in the config, an `F8_E4M3` weight with `weight_block_size` and no `weight_scale_inv`, a module listed in `modules_to_not_convert` that nevertheless ships one, and an `input_scale` beside `activation_scheme = dynamic` (upstream registers one only when `act_q_static`, `fp8.py:381-384`). The shape check is upstream's own: `cdiv` on BOTH axes (`fp8_utils.py:1283-1296`) and the exact-shape assertion at `parameter.py:95-98`, so a short final block is legal and works. ## What is still refused, and where `RefuseUnsupportedFp8BlockQuant` keeps its call site in `ModelRegistry::Load` and now refuses only what nothing here can execute: a `quant_method` without `fp8`, a `weight_block_size` that is not exactly two dimensions, an `activation_scheme` other than `dynamic` — the first three mirroring upstream's own `ValueError`s at `fp8.py:115-131` — and a block shape other than 128x128, which is OUR limit and says so. STAGED SLICE, named per `.agents/reachability.md`. Nothing CONSUMES an `Fp8BlockWeight` at this merge commit: `layers::MakeLinearMethod` has no block arm and the dense `project` lambda knows only fp4, per-tensor fp8 and bf16. The wiring is owned by **#1189 milestone M4** (`layers::Fp8BlockLinearMethod` and the Qwen3.5 dense forward), it is listed under `## Owed` in the spec, and it is not left silent: `PrepareQwen3_5Dense` — reached from `ModelRegistry::Prepare`, which every runner calls before the first forward and before graph capture (`v1/worker/gpu/runner.cpp:414,455`) — refuses by name and quotes the projection. The checkpoint loads and declines to run; it never runs wrong. The loader rung ITSELF is reached at this commit, from `ModelRegistry::Load` through `LoadQwen3_5DenseModel` to `LoadQwen3_5Dense`, and the reachability mutation is recorded in the spec's `## Evidence`. ## Gates RED first: with the tests present and no implementation, the focused build fails `compile_rc=1` with 50 errors, 6 of them ``'Fp8BlockWeight' does not name a type`` and the rest missing `*_fp8_block` members. GREEN: `test_fp8_block_weight_load` 7 cases / 102 assertions, whose per-block counts sum to the whole-run total; `test_fp8_block_quant` 8 cases / 35 assertions, rewritten from "the scheme is refused" to "the SUPPORTED config reaches the loader and these four do not". `test_ops_quant_fp8_group_cpu`, `test_ops_matmul_fp8_block_cpu`, `test_ops_fp8_cpu`, `test_qwen36_weights`, `test_linear_method`, `test_model_registry` and `test_op_provider` all pass unchanged. No GPU lease, no checkpoint download: the fixture is a complete but tiny synthetic `Qwen3_5ForConditionalGeneration` safetensors checkpoint written to a temp directory. `docs/FEATURES.md` and `docs/USAGE.md` move from "refused at load" to "loads and does not run yet" in the same change. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
`origin/main` advanced to `65d6cdaed` while M3 was in flight: a gcc-16 build repair with its own CI lane, and the ROCm runner's attention-backend selection. Neither touches the fp8 block path, the dense loader, or the quantization config, so the merge is textually clean and the row's gates were rerun on the merged tree rather than on the pre-merge head. `git merge --no-edit` writes a merge commit with no trailer block, so this is the amend that puts one back. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…utations that proved nothing (#1189) Fills the spec's `## Evidence` on the merged tree at `65d6cdaed`: the red-first compile failure, the per-block gate counts, the reachability mutation, and eleven mutation results. Three of those are worth keeping and are written up rather than tabulated. **A mutation that applies cleanly can still change nothing.** Guarding the `if (layer.is_linear_attention)` arm of the `Prepare` refusal with `if (false)` took the `else` arm instead, which runs every attention check, and the MLP checks sit outside the branch entirely. The refusal still fired, the gate stayed green, and `git diff --stat` reported a real two-line diff. Neither `compile_rc` nor `git diff --stat` catches that class. The verdict was re-taken against the actual call site, where it reds. **`-Werror` turns the natural reachability mutation into a non-event.** Deleting the rung call orphans the `block` parameter, `-Werror=unused-parameter` fires, the build fails, and the STALE binary from the previous link prints `SUCCESS!`. Re-run with `block.block_n` kept live, it reds 6 of 7 cases with issue #1166's own sentence, `tensor not found: model.layers.0.self_attn.q_proj.weight_scale`. **A grid of round shapes is blind to the ragged defect.** Replacing `cdiv` with floor left every other block green and failed only G3, which is why `N=576` and `K=3884` are in the grid rather than the target checkpoint's shapes, all of which are multiples of 128. Same measurement M2 recorded one layer down. `test_cpu_x86_llamacpp_floor` is the one failing preflight gate, and it is [#618](#618) rather than this change: a pristine detached worktree at `origin/main` `65d6cdaed`, with no part of this row applied, failed the same two cases with `NO_QUIET_WINDOW after 30s` at `busy=154%` and `busy=177%`. That worktree was removed. The other 81 gates report `ok` and none was skipped. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
FOLLOWING_AGENTS_PROTOCOL Brings the branch up to `origin/main` so the committed-range and trailer gates examine this tree rather than skipping. The incoming commit is records only and cannot affect this row's gates, but a skipped gate reports nothing at all. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
FOLLOWING_AGENTS_PROTOCOL Takes the `test_check_gate_commands` repair so this row gates against a green `main` rather than inheriting a red it did not cause. One comment-only conflict, resolved toward `main`. `c20018f8d` replaced this file's LINE anchor `model_loader.cpp:1706` with the SYMBOL anchor `model_loader.cpp::FromModelDir`, and it landed a gate that enforces exactly that. Keeping this row's side would have reintroduced the anchor form the new gate exists to refuse, so the resolution keeps the symbol anchor and the `RefuseUnsupportedWeightOffload` paragraph from `main`, and keeps this row's sentence naming `test_fp8_block_weight_load` as the gate for the loading half. Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Milestone M3 of #1189:
Fp8BlockWeight, theweight_scale_invloader rung,and the quantization-config reader.
Qwen/Qwen3.8-27B-FP8stops being refusedat
ModelRegistry::Loadand its weights are read; nothing can execute them yet,so the model declines to be PREPARED rather than running through an empty
tensor.
Spec:
.agents/specs/model-fp8-block-weight.md,committed before the implementation as
fd87d4ffa. Pinned oracle: vLLM5559679229bc961848b121ccdeaa8fa5d79bec98, asserted as the local checkout'sHEAD before any anchor below was read. It consumes what
ad5f175e7(M1,vt::QuantFp8Group) and770e49486(M2,vt::MatmulFp8BlockScaled) landed, andstops there.
The BF16 scale dtype, established rather than assumed
The checkpoint ships
weight_scale_invasBF16 [96, 40]; upstream allocatesthe parameter
float32. Both are true, and the resolution is that torch'scopy_CONVERTS:dtype = scale_dtype if scale_dtype is not None else torch.float32(
utils/fp8_utils.py:1276,1283-1296);scale_dtypeistorch.float8_e8m0fnu if self.is_scale_e8m0 else None(
fp8.py:376), andis_scale_e8m0isgetattr(quant_config, "is_scale_e8m0", False)(fp8.py:282) against anFp8Configthat defines no such attribute, so it is False and the parameteris f32;
BlockQuantScaleParameterloads throughself.data.copy_(loaded_weight)(
vllm/model_executor/parameter.py:95-108, inherited at:397-403), adtype-CONVERTING copy;
fp8_utils.py:1103-1112),which is only consistent because the widening already happened.
So
Fp8BlockWeight::scaleis f32, and that is the MIRROR rather than a.agents/porting.mdwidening: f32 is the dtype upstream carries resident,vt::MatmulFp8BlockScaledrefuses anything else, andbf16 -> f32is exact.LoadFp8BlockRawswitches on the on-disk dtype with NO default branch andrefuses any other by name, because
f22c6cc82(#1181) landed a guard for areader that memcpy'd four bytes whatever the dtype was.
vt::LoadUnaligned,because a safetensors offset can be odd (#627).
Read the config, do not only probe the tensors
modules_to_not_convertis a ~400-entry list a dtype probe reproduces only byaccident, and a probe cannot see a DISAGREEMENT between the config and the
tensors at all. That is exactly where a silent wrong-scale bug lives: #1166
measured a
[96, 40]grid passing a per-tensor reader's byte floor and beingapplied to the whole weight, stopped only by the tensor NAME.
IsFp8BlockProjectiontherefore decides from BOTH sources and refuses fourcombinations by name: a
weight_scale_invwith noweight_block_sizein theconfig, an
F8_E4M3weight withweight_block_sizeand noweight_scale_inv, amodule listed in
modules_to_not_convertthat nevertheless ships one, and aninput_scalebesideactivation_scheme = dynamic(upstream registers one onlywhen
act_q_static,fp8.py:381-384). The shape check is upstream's own:cdivon BOTH axes (fp8_utils.py:1283-1296) and the exact-shape assertion atparameter.py:95-98, so a short final block is legal and works. Moduleexclusion mirrors
is_layer_skipped's defaultprefix_full_match(
quant_utils.py:517-518,524,568-569), i.e. exact membership, not a substringtest.
What is still refused, and where
RefuseUnsupportedFp8BlockQuantkeeps its call site inModelRegistry::Loadand now refuses only what nothing here can execute: a
quant_methodwithoutfp8, aweight_block_sizethat is not exactly two dimensions, anactivation_schemeother thandynamic— the first three mirroring upstream'sown
ValueErrors atfp8.py:115-131— and a block shape other than 128x128,which is OUR limit and says so.
Staged slice, named per
.agents/reachability.md. Nothing CONSUMES anFp8BlockWeightat this merge commit:layers::MakeLinearMethodhas no blockarm and the dense
projectlambda knows only fp4, per-tensor fp8 and bf16. Thewiring is owned by #1189 milestone M4 (
layers::Fp8BlockLinearMethodand theQwen3.5 dense forward), it is listed under
## Owedin the spec, and it is notleft silent:
PrepareQwen3_5Dense— reached fromModelRegistry::Prepare, whichevery runner calls before the first forward and before graph capture
(
v1/worker/gpu/runner.cpp:414,455) — refuses by name and quotes theprojection. The checkpoint loads and declines to run; it never runs wrong.
The loader rung ITSELF is reached at this commit, from
ModelRegistry::Loadthrough
LoadQwen3_5DenseModel(qwen3_5_dense.cpp:101) toLoadQwen3_5Dense.Gates
RED first. With both test files present and no implementation, the focused
build fails
compile_rc=1with 50 errors: 6'Fp8BlockWeight' does not name a typeand the rest missing*_fp8_blockmembers onFullAttnLayerWeightsandDenseMlpWeights.GREEN, on the merged tree at
65d6cdaed.test_fp8_block_weight_loadreports 7 cases / 102 assertions / 0 failed, whose per-block counts
(24+17+11+19+17+9+5) sum to the whole-run total, so no block is silently empty.
test_fp8_block_quantreports 8 cases / 35 assertions / 0 failed, rewrittenfrom "the scheme is refused" to "the SUPPORTED config reaches the loader and
these four do not".
test_ops_quant_fp8_group_cpu,test_ops_matmul_fp8_block_cpu,test_ops_fp8_cpu,test_qwen36_weights,test_linear_method,test_model_registryandtest_op_providerall passunchanged, and the whole tree builds clean.
Reachability. Deleting the
LoadFp8BlockRawcall inload_projectionreds6 of 7 cases, with issue #1166's own sentence:
tensor not found: model.layers.0.self_attn.q_proj.weight_scale.Eleven mutations are tabulated in the spec's
## Evidence, each withcompile_rcandgit diff --stat, each restored and verified bysha256sumover
git ls-files -s. Two proved nothing and are reported rather than dropped:-Werror=unused-parameterturned the first reachability mutation into a stalebinary printing
SUCCESS!, and anif (false)on thePreparerefusal'sis_linear_attentionarm took theelsearm instead, so a real two-line diffchanged nothing and the gate stayed green.
scripts/agent-preflight.sh --staged --fail-on-skipreports 81 gatesok, 0skipped, 1 failed:
test_cpu_x86_llamacpp_floor, which is#618 and not this change. A
pristine detached worktree at
origin/main65d6cdaedfailed the same twocases with
NO_QUIET_WINDOW after 30satbusy=154%andbusy=177%; thatworktree was removed.
No GPU lease, no checkpoint download: the fixture is a complete but tiny
synthetic
Qwen3_5ForConditionalGenerationsafetensors checkpoint written to atemp directory.
docs/FEATURES.mdanddocs/USAGE.mdmove from "refused at load" to "loads anddoes not run yet" in the same change.
Closes nothing: #1189 stays open for M4, M5 and M6.
FOLLOWING_AGENTS_PROTOCOL
🤖 Generated with Claude Code
https://claude.ai/code/session_01Tgd6wPDzr5FK4wiEiWB6Mg
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]