From fd87d4ffad367a8097082d248962c482fb4ea2ea Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Tue, 18 Aug 2026 11:23:33 +0000 Subject: [PATCH 1/3] spec(MODEL-FP8-BLOCK-WEIGHT): the block-wise FP8 weight, its loader rung, and its config reader (#1189) Milestone **M3** of #1189, committed before the implementation as the protocol requires. It designs `Fp8BlockWeight`, the `weight_scale_inv` loader rung, and the quantization-config reader that narrows the named refusal landed in `469f38395` (#1166). Three things the design settles rather than assumes. **The BF16 scale dtype.** `Qwen/Qwen3.8-27B-FP8` ships `weight_scale_inv` as `BF16 [96, 40]` while upstream allocates the parameter `float32`. Both are true, and the resolution is that `BlockQuantScaleParameter` loads through `self.data.copy_(loaded_weight)` (`vllm/model_executor/parameter.py:95-108`), which CONVERTS. The parameter is `torch.float32` because `scale_dtype` is `None` unless `is_scale_e8m0`, an attribute `Fp8Config` does not define (`fp8.py:282,376`; `fp8_utils.py:1276`). So upstream widens the scale to f32 once, at load, losslessly. We mirror that, and the resident f32 is not a `.agents/porting.md` widening to justify: f32 is the dtype upstream carries. **Read the config, do not only probe the tensors.** `modules_to_not_convert` is a ~400-entry list a dtype probe reproduces by accident, and a probe cannot see a DISAGREEMENT between the config and the tensors at all. That is where a silent-wrong-scale bug lives, and #1166 recorded the near miss: a `[96, 40]` scale passed the old `nbytes >= sizeof(float)` floor and only the tensor NAME stopped it being applied to the whole weight. The spec tabulates six config/tensor combinations and refuses four of them by name. **Where the M4 gap is refused.** `ModelRegistry::Load` succeeds on a supported block-wise checkpoint, which is what makes the loader rung reachable from a production entry point at its own merge commit. Nothing consumes an `Fp8BlockWeight` yet, so `ModelRegistry::Prepare` refuses by name and quotes #1189 M4 rather than letting the dense `project` lambda fall through to an empty bf16 tensor. The checkpoint loads and declines to run; it never runs wrong. Scope stops at M3. No linear method, no forward wiring, no CUDA kernel, no GPU lease, and no checkpoint download. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/specs/model-fp8-block-weight.md | 357 ++++++++++++++++++++++++ 1 file changed, 357 insertions(+) create mode 100644 .agents/specs/model-fp8-block-weight.md diff --git a/.agents/specs/model-fp8-block-weight.md b/.agents/specs/model-fp8-block-weight.md new file mode 100644 index 000000000..402118ce3 --- /dev/null +++ b/.agents/specs/model-fp8-block-weight.md @@ -0,0 +1,357 @@ +# MODEL-FP8-BLOCK-WEIGHT — the block-wise FP8 weight, its loader rung, and its config reader + +Issue: [#1189](https://github.com/mudler/vllm.cpp/issues/1189), milestone **M3**. +Row: `MODEL-FP8-BLOCK-WEIGHT`. +Pinned oracle: vLLM `5559679229bc961848b121ccdeaa8fa5d79bec98` +(`.agents/upstream-sync.md`), asserted as the HEAD of the local checkout before +any `file:line` below was read. + +## Scope + +Three things, and the refusal they replace. + +1. **`Fp8BlockWeight`**, a sibling of `Fp8Weight` in + `include/vllm/model_executor/models/qwen3_5_weights.h`. +2. **The loader rung**: a probe on `.weight_scale_inv` inserted *before* + the existing `dtype == "F8_E4M3"` per-tensor rung in + `src/vllm/model_executor/models/qwen3_5_dense_weights.cpp`, plus the shared + `dense_loaders::LoadFp8BlockRaw` it calls. +3. **The config reader**, `ReadFp8BlockQuantConfig`, which reads + `weight_block_size`, `activation_scheme` and `modules_to_not_convert`, and + validates them against upstream's own rules before any tensor is touched. + +`RefuseUnsupportedFp8BlockQuant` (landed `469f38395`, #1166) stops refusing the +whole scheme and refuses only what M3 does not cover. The one thing M3 still +cannot do — *consume* the weight — is refused by name at +`ModelRegistry::Prepare`, not left to produce wrong numbers. + +**Out of scope, each owned by a later milestone of #1189**: +`layers::Fp8BlockLinearMethod` and the Qwen3.5 dense forward wiring (M4); the +mainloop-scaled CUTLASS kernel for `sm_121a` (M5); merged `gate_up` and QKV +(M6). No kernel lands here, no GPU is leased, and no checkpoint is downloaded. + +## What the target checkpoint actually is + +`Qwen/Qwen3.8-27B-FP8` at revision `017b9c7a`, measured by HTTP range request +for #1166 and not re-fetched here: + +| Fact | Value | +|---|---| +| `quant_method` | `fp8` | +| `weight_block_size` | `[128, 128]` | +| `activation_scheme` | `dynamic` | +| architecture | `Qwen3_5ForConditionalGeneration` | +| `self_attn.q_proj.weight` | `F8_E4M3` `[12288, 5120]` | +| `self_attn.q_proj.weight_scale_inv` | **`BF16`** `[96, 40]` = `[12288/128, 5120/128]` | +| `input_scale` tensors | **0** | + +Every projection in this checkpoint divides by 128 on both axes. That is a +property of this checkpoint, not of the scheme, and the loader is written and +tested against `cdiv` on both axes for that reason. + +## The BF16 scale dtype, established rather than assumed + +The scale ships `BF16` and upstream allocates the parameter `float32`. Both are +true, and the resolution is that **torch's `copy_` converts**: + +| Step | Where | +|---|---| +| the block scale parameter is allocated `[cdiv(N,block_n), cdiv(K,block_k)]` with `dtype = scale_dtype if scale_dtype is not None else torch.float32` | `vllm/model_executor/layers/quantization/utils/fp8_utils.py:1276,1283-1296` | +| `scale_dtype` is `torch.float8_e8m0fnu if self.is_scale_e8m0 else None` | `fp8.py:376` | +| `is_scale_e8m0` is `getattr(quant_config, "is_scale_e8m0", False)`, and `Fp8Config` defines no such attribute (`fp8.py:96-134`), so it is **False** and the parameter is `float32` | `fp8.py:282` | +| the checkpoint tensor is written into that parameter by `self.data.copy_(loaded_weight)`, a **dtype-converting** copy, not a reinterpretation | `vllm/model_executor/parameter.py:95-108`; `BlockQuantScaleParameter` inherits it at `:397-403` | +| downstream code then asserts the scale is `float32` (or E8M0/uint8 for MXFP8), which is only consistent because the widening already happened at load | `fp8_utils.py:1103-1112` | + +So upstream's answer to a narrower on-disk scale dtype is: **widen it to f32 at +load, once, losslessly.** `bf16 -> f32` is exact — bf16 is the top 16 bits of an +f32 — so nothing is invented and nothing is lost. + +We mirror that. `Fp8BlockWeight::scale` is **f32**, and this is not a +`.agents/porting.md` dtype widening to be justified: f32 *is* the resident dtype +upstream carries. The bytes are negligible (`[96,40]` f32 is 15 360 B per +projection against 60 MiB of weight) and `vt::MatmulFp8BlockScaled` refuses any +scale that is not f32 (`.agents/specs/vt-matmul-fp8-block-ref.md`), so a bf16 +resident scale would need a conversion at every GEMM instead of one at load. + +**What we must not do**, and the reason the rule exists: `f22c6cc82` (#1181) +landed a guard because six copies of the per-tensor scale reader bounded their +input with `nbytes >= sizeof(float)` — a floor — and then `memcpy`d four bytes +whatever the dtype was. `LoadFp8BlockRaw` therefore switches on `t.dtype` +explicitly, decodes `BF16` through `vt::BF16ToF32(vt::LoadUnaligned)` +and `F32` through `vt::LoadUnaligned`, and **refuses every other dtype by +name**. There is no default branch that reinterprets bytes. + +`vt::LoadUnaligned` rather than a typed pointer, because a safetensors tensor's +offset is the running byte total of everything ahead of it and can be odd +(#627), exactly as `dense_loaders::ReadF32Scalar` and `TransposeBf16` already +do. + +## Upstream anchors + +| What | Where | +|---|---| +| `weight_block_size`, `activation_scheme`, `ignored_layers`, and the `modules_to_not_convert` fallback are read together | `vllm/model_executor/layers/quantization/fp8.py:157-172` | +| the validation this mirrors: fp8-serialized, exactly 2 dimensions, `dynamic` only | `fp8.py:115-131` | +| `block_quant = self.weight_block_size is not None` is the whole dispatch | `fp8.py:297-298` | +| the scale registers as `weight_scale_inv`, strictly conditional on block quant | `fp8.py:378-379`, `:511` | +| scale allocation, `cdiv` on **both** axes | `fp8_utils.py:1283-1296` | +| the shape assertion the load performs | `vllm/model_executor/parameter.py:95-98` | +| `is_layer_skipped`: default `prefix_full_match`, i.e. exact membership of the module prefix in the ignore list | `vllm/model_executor/layers/quantization/utils/quant_utils.py:510-524,568-569` | +| the ignore list is rewritten into vLLM module naming before matching | `fp8.py:151-153` (`apply_vllm_mapper`) | +| the activation quant this pairs with (M1, landed `ad5f175e7`) | `.agents/specs/vt-quant-fp8-group.md` | +| the GEMM this pairs with (M2, landed `770e49486`) | `.agents/specs/vt-matmul-fp8-block-ref.md` | + +## Design + +### `Fp8BlockWeight` is a sibling, not an extension + +```c++ +struct Fp8BlockWeight { + OwnedTensor packed; // i8 [N, K] raw fp8-e4m3fn bytes + OwnedTensor scale; // f32 [cdiv(N,bn), cdiv(K,bk)] widened from disk + int64_t n = 0, k = 0; + int64_t block_n = 0, block_k = 0; + bool Empty() const { return packed.Empty(); } + mutable std::shared_ptr d_packed; + mutable std::shared_ptr d_scale; +}; +``` + +`Fp8Weight` (`qwen3_5_weights.h:318-330`) is three host floats — `weight_scale`, +`input_scale`, and the `alpha = input_scale * weight_scale` folded at load. A +block scheme has **no `input_scale` at all** (the activation scheme is dynamic; +the target checkpoint ships zero such tensors) and its weight scale is a 2-D +tensor. There is no value `alpha` could take. + +Adding an optional scale tensor to `Fp8Weight` was rejected. Every existing +reader of `Fp8Weight::alpha` — the cutlass and cuBLASLt fp8 GEMM wrappers, the +merged-QKV alpha vector, `PrepareGdnFp8Resident` — would then need a silent +which-arm branch, and the arm that forgets one produces a plausible number +rather than an error. A distinct type makes the wrong call site fail to compile. + +`Nvfp4Weight` (`qwen3_5_weights.h:243-304`) is the shape that already works +here: an `OwnedTensor scale` beside the packed values, plus lazily-populated +device handles. `Fp8BlockWeight` follows it, minus everything NVFP4-specific. + +`block_n` and `block_k` are carried **on the weight**, not looked up from the +config at use time. The consumer needs them per GEMM, and a weight that knows +its own geometry cannot be paired with the wrong one. + +### The loader rung + +`load_projection` in `LoadAttnDense` (`qwen3_5_dense_weights.cpp:470-479`) +probes NVFP4, then `dtype == "F8_E4M3"`, then bf16. A block-wise weight **is** +`F8_E4M3`, so it fell into the per-tensor arm and asked for a `weight_scale` +that a block-wise checkpoint spells `weight_scale_inv`. That is #1166. + +The block probe goes **before** the per-tensor rung, at every site that probes +`F8_E4M3` today: `LoadAttnDense` (q/k/v/o), `LoadGdnDense` +(`in_proj_qkv`, `in_proj_z`, `out_proj`), and `LoadDenseMlp`, which had no fp8 +rung at all and would otherwise have sent a block-wise MLP into +`LoadMergedBf16RawNK`. + +### Read the config, then cross-check it against the tensors + +A dtype probe alone is not enough, for two measured reasons. + +**`modules_to_not_convert` is a ~400-entry list** that a probe reproduces only +by accident. A projection this checkpoint deliberately left unquantized is +`BF16` on disk and a probe agrees with the config by luck; the moment a +checkpoint ships an `F8_E4M3` tensor for a module it also lists as excluded, the +probe and the config disagree and only one of them is right. + +**A probe cannot see a disagreement at all.** It sees a tensor and picks an arm. +That is precisely where a silent-wrong-scale bug lives, and #1166's own commit +message records the near miss: a `[96,40]` scale passed the old `nbytes >= +sizeof(float)` floor and would have been read as element `(0,0)` and applied to +the whole `[N,K]` weight. Only the tensor *name* stopped it. + +So `ResolveFp8Arm` decides the arm from **both** sources and refuses each +disagreement by name: + +| Config says | Tensors say | Result | +|---|---|---| +| block-wise, module not excluded | `F8_E4M3` + `weight_scale_inv` | block arm | +| block-wise, module not excluded | `F8_E4M3`, no `weight_scale_inv` | **refuse**, naming the tensor that is missing | +| block-wise, module **excluded** | `weight_scale_inv` present | **refuse**, naming the module and the list | +| block-wise, module excluded | no `weight_scale_inv` | falls through to the existing rungs | +| not block-wise | `weight_scale_inv` present | **refuse**, naming the config key that is missing | +| not block-wise | no `weight_scale_inv` | falls through, byte-identical to today | + +Module exclusion mirrors `is_layer_skipped`'s default `prefix_full_match`: the +tensor name with its `.weight` suffix removed, compared for **exact equality** +against each list entry. Upstream first rewrites the list into vLLM module +naming (`fp8.py:151-153`); we match in *checkpoint* naming, which is what our +loader has, and the two coincide for every entry that names a real checkpoint +module. The substring form (`skip_with_substr=True`) is not the default at any +fp8 call site and is not mirrored. + +The shape cross-check lives in `LoadFp8BlockRaw` and is upstream's own: the +scale must be exactly `[cdiv(N, block_n), cdiv(K, block_k)]`, mirroring the +allocation at `fp8_utils.py:1283-1296` and the `param.data.shape == +loaded_weight.shape` assertion at `parameter.py:95-98`. A floor-sized scale, a +transposed scale, and a per-tensor scalar are each refused with both shapes in +the message. + +An `.input_scale` present while the config declares `dynamic` is also a +disagreement, and it is refused: upstream registers an input scale only when +`act_q_static` (`fp8.py:381-384`), which block quant asserts against outright +(`fp8.py:367`). + +### Ragged edges are supported, not refused + +`cdiv` on both axes, everywhere. Upstream's shape contract admits a short final +block (`fp8_utils.py:935-936`) and M2's reference arm already handles one. The +target checkpoint has none, so the tests carry `N=576` (`4*128 + 64`) and +`K=3884` (`30*128 + 44`) — upstream's own ragged shapes from +`tests/kernels/quantization/test_block_fp8.py:49-50` — separately and together. +M2 measured that a grid of round shapes stays green through two different +floor-vs-ceil defects; that measurement is the reason these shapes are here. + +### What is still refused, and where + +| Refused | Where | Why | +|---|---|---| +| `activation_scheme != "dynamic"` | `ReadFp8BlockQuantConfig`, reached from `ModelRegistry::Load` | upstream refuses it too (`fp8.py:127-131`); nothing here quantizes activations statically for a block scheme | +| `weight_block_size` with other than 2 dimensions | same | upstream refuses it (`fp8.py:121-126`) | +| a block shape other than `[128, 128]` | same | M5's kernel is 128x128 and M2's reference is the only other consumer; a `[64, 128]` checkpoint would load into a weight nothing can execute | +| `quant_method` without `fp8` | same | mirrors `is_checkpoint_fp8_serialized` (`fp8.py:117-120`, `:159`) | +| a scale dtype that is neither `BF16` nor `F32` | `LoadFp8BlockRaw` | the #1181 rule: never reinterpret bytes across a dtype | +| a config/tensor disagreement | `ResolveFp8Arm` / `LoadFp8BlockRaw` | the table above | +| **a loaded block weight that nothing consumes** | `PrepareQwen3_5Dense`, reached from `ModelRegistry::Prepare` | M4 owns the linear method. See below | + +The last row is the M3/M4 seam and it is deliberate. `ModelRegistry::Load` now +*succeeds* on a supported block-wise checkpoint, which is what makes the loader +rung reachable from a production entry point at this merge commit. Nothing +consumes an `Fp8BlockWeight` yet, so the dense `project` lambda +(`qwen3_5.cpp:2464-2486`) would fall through to an empty bf16 tensor. Rather +than let that happen, `ModelRegistry::Prepare` — which every runner calls +unconditionally before the first forward and before graph capture +(`src/vllm/v1/worker/gpu/runner.cpp:414,455`) — refuses by name, quotes the +projection, and names #1189 M4. A block-wise checkpoint therefore loads and then +declines to run, and it never runs wrong. + +## Risks + +| Risk | Control | +|---|---| +| the BF16 scale is reinterpreted as f32 rather than converted | `LoadFp8BlockRaw` switches on dtype with no default branch; G2 asserts the decoded VALUES against a hand-computed table, which a reinterpretation cannot pass | +| the scale silently widens or narrows | `Fp8BlockWeight::scale` is f32 by construction and G2 asserts the dtype; the reason it is f32 is recorded above with the upstream anchor | +| a ragged dimension uses floor tiling and misindexes or drops a block | `cdiv` everywhere; G3 runs `N=576`, `K=3884`, and both together | +| a config/tensor disagreement picks an arm silently | G4 asserts four distinct disagreements, each refused by name | +| the block rung is inserted after the per-tensor rung and never selected | G1 loads a block-wise checkpoint through the production loader and asserts the block slot is populated and the per-tensor slot is empty | +| a per-tensor checkpoint regresses into the block arm | G6 is the negative control: a per-tensor fixture still lands in `Fp8Weight` with its `alpha` folded, and the existing `test_fp8_block_quant` cases are retained | +| an unsupported block shape or scheme loads silently | G5 asserts each refusal by name through `ModelRegistry::Load` | +| **nothing consumes the weight, so the whole rung is dead** | acknowledged under `## Owed`; the M4 gap is refused by name at `Prepare` and G7 asserts that refusal | + +## Tests + +`tests/vllm/model_executor/models/test_fp8_block_weight_load.cpp`, registered in +`tests/CMakeLists.txt`. A synthetic safetensors fixture is written to a temp +directory — no checkpoint download, no GPU, no snapshot. The fixture builder +follows `tests/vllm/test_safetensors.cpp:57-89` (u64-LE header length + JSON +header + payload). + +The fixture is a **complete, minimal** `Qwen3_5ForConditionalGeneration` dense +checkpoint: one `full_attention` layer, tiny dimensions, tied `lm_head`. That is +what lets every case enter through the production loader rather than through +`LoadFp8BlockRaw`. + +- **G1** the rung is selected: a block-wise fixture loaded through + `LoadQwen3_5Dense` — the loader `ModelRegistry::Load` reaches at + `qwen3_5_dense.cpp:101` — populates `q_proj_fp8_block` and leaves + `q_proj_fp8` and `q_proj` empty, for attn, GDN and MLP projections alike. +- **G2** the scale tensor: dtype f32, shape `[cdiv(N,128), cdiv(K,128)]`, and + **values** equal to the BF16 bytes written into the fixture, decoded + independently. The fixture writes a value that is exactly representable in + bf16 and one that is not, so a reinterpretation reads as a wildly wrong + number rather than a rounding difference. An `F32` scale fixture is loaded in + the same case, and a `F16` one is refused by name. +- **G3** the ragged grid: `N=576`, `K=3884`, and both together, with the scale + sized by `cdiv`. A floor-sized scale for the same weight is refused. +- **G4** the config/tensor disagreements, each by name: `weight_scale_inv` + present with no `weight_block_size` in the config; `weight_block_size` + declared with no `weight_scale_inv` beside an `F8_E4M3` weight; a module named + in `modules_to_not_convert` that nevertheless ships a `weight_scale_inv`; and + an `input_scale` present under `activation_scheme = dynamic`. +- **G5** the config refusals through `ModelRegistry::Load`, each by name: + `activation_scheme = "static"`, `weight_block_size = [128]`, + `weight_block_size = [64, 128]`, and a `quant_method` that is not fp8. +- **G6** the negative controls: a per-tensor fp8 fixture still loads into + `Fp8Weight` with `alpha = input_scale * weight_scale`, and a bf16 fixture is + untouched. Without this the gate passes for a rung that captures every fp8 + checkpoint. +- **G7** the M4 gap: `ModelRegistry::Prepare` on a loaded block-wise model + refuses by name, quoting the projection and #1189. + +`tests/vllm/model_executor/layers/test_fp8_block_quant.cpp` is updated in the +same change: the cases that asserted the whole scheme was refused now assert the +narrowed refusals, and the negative controls are kept verbatim. + +## Gates + +| Gate | Command | +|---|---| +| focused | `ctest -R test_fp8_block_weight_load --output-on-failure` | +| the narrowed refusal | `ctest -R test_fp8_block_quant --output-on-failure` | +| the M1/M2 siblings, unchanged | `ctest -R "test_ops_quant_fp8_group_cpu\|test_ops_matmul_fp8_block_cpu"` | +| the per-tensor arm, unchanged | `ctest -R "test_ops_fp8_cpu\|test_qwen36_weights\|test_linear_method"` | +| record | `scripts/agent-preflight.sh --fail-on-skip` | + +No GPU lease is taken and none is needed. + +## Owed + +- **Nothing consumes `Fp8BlockWeight`.** At this merge commit the loader + populates it and no forward path reads it: `include/vllm.h` exposes no block + linear method, `layers::MakeLinearMethod` has no block arm, and the dense + `project` lambda (`src/vllm/model_executor/models/qwen3_5.cpp:2464-2486`) + knows only fp4, per-tensor fp8 and bf16. Owed by **#1189 milestone M4** + (`layers::Fp8BlockLinearMethod` and the Qwen3.5 dense forward wiring). This is + the staged-slice exception of `.agents/reachability.md`, named here, in the + commit body, and in the pull request body. It is refused by name at + `ModelRegistry::Prepare` rather than left to produce a wrong number, and G7 + asserts that refusal. +- **The device handles `d_packed` and `d_scale` are declared and never + populated.** They mirror `Nvfp4Weight`'s and `Fp8Weight`'s shape so that M4 + and M5 upload through the same lazily-populated seam every other quantized + weight uses. Owed by #1189 M5, which is the first arm with a device kernel. +- **Merged `gate_up` and QKV.** The MLP rung loads gate, up and down as three + independent block weights. Block scales concatenate losslessly along N, so + merging is *simpler* here than in the per-tensor case, and #1189 M6 owns it. +- **The MoE and GGUF loaders.** `qwen3_5_weights.cpp`'s MoE path and + `qwen3_5_gguf_weights.cpp` are untouched: no block-wise MoE or GGUF checkpoint + is in play for #1189, and `ReadFp8BlockQuantConfig` refuses an unsupported + block config for every architecture at `ModelRegistry::Load` regardless. A + block-wise MoE safetensors checkpoint would reach the MoE loader's per-tensor + rung and fail on the missing `weight_scale` name, which is #1166's original + sentence and is not made worse here. Owed by whichever row ports one. +- **`store_dtype`** (`fp8.py:104`, `:167`) is read by upstream and ignored here. + No checkpoint in play sets it. It becomes owed when one does. + +## Stop conditions + +Stop and report `NEEDS_DECISION` if any of the following holds. + +- The pinned oracle's checkout is not at + `5559679229bc961848b121ccdeaa8fa5d79bec98`. Every anchor above was read at + that revision, asserted before the first read. +- The BF16 scale cannot be shown to widen rather than reinterpret. A byte + reinterpretation that happens to pass a shape check is the #1181 defect and + widening a tolerance is not the fix. +- The loader rung cannot be reached from `ModelRegistry::Load` without wiring a + forward path, which is M4's scope. If reaching it requires M4, the milestone + boundary is wrong and the operator decides where to move it. + +Stop and report `NEEDS_CONTEXT` if the work requires a GPU lease or a checkpoint +download. The row is scoped so that it needs neither. + +## Evidence + +Recorded after the implementation, below the `## Owed` list, so that a reader +who stops at the design has read the design. + +## Now + +`ACTIVE` — M3 of #1189. M1 (`ad5f175e7`) and M2 (`770e49486`) are `DONE`; M4, +M5 and M6 are open. From 5ac877aae0e94fc63cee9095a2b16a9919720278 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Tue, 18 Aug 2026 11:56:14 +0000 Subject: [PATCH 2/3] feat(MODEL-FP8-BLOCK-WEIGHT): a block-wise FP8 checkpoint now LOADS, and says by name that it cannot run yet (#1189) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Milestone **M3** of #1189: `Fp8BlockWeight`, the `weight_scale_inv` loader rung, and the quantization-config reader. `Qwen/Qwen3.8-27B-FP8` stops being refused at `ModelRegistry::Load` and its weights are read; nothing can execute them yet, so the model declines to be PREPARED rather than running through an empty tensor. Spec: [`.agents/specs/model-fp8-block-weight.md`](.agents/specs/model-fp8-block-weight.md), committed before the implementation as `fd87d4ffa`. Pinned oracle: vLLM `5559679229bc961848b121ccdeaa8fa5d79bec98`, asserted as the local checkout's HEAD before any anchor below was read. It consumes what `ad5f175e7` (M1, `vt::QuantFp8Group`) and `770e49486` (M2, `vt::MatmulFp8BlockScaled`) landed, and stops there. ## The BF16 scale dtype, established rather than assumed The checkpoint ships `weight_scale_inv` as `BF16 [96, 40]`; upstream allocates the parameter `float32`. Both are true, and the resolution is that torch's `copy_` CONVERTS: - the block scale parameter is allocated with `dtype = scale_dtype if scale_dtype is not None else torch.float32` (`utils/fp8_utils.py:1276,1283-1296`); - `scale_dtype` is `torch.float8_e8m0fnu if self.is_scale_e8m0 else None` (`fp8.py:376`), and `is_scale_e8m0` is `getattr(quant_config, "is_scale_e8m0", False)` (`fp8.py:282`) against an `Fp8Config` that defines no such attribute, so it is False and the parameter is f32; - `BlockQuantScaleParameter` loads through `self.data.copy_(loaded_weight)` (`vllm/model_executor/parameter.py:95-108`, inherited at `:397-403`), a dtype-CONVERTING copy; - downstream code then asserts the scale is f32 (`fp8_utils.py:1103-1112`), which is only consistent because the widening already happened. So `Fp8BlockWeight::scale` is f32, and that is the MIRROR rather than a `.agents/porting.md` widening: f32 is the dtype upstream carries resident, `vt::MatmulFp8BlockScaled` refuses anything else, and `bf16 -> f32` is exact. `LoadFp8BlockRaw` switches on the on-disk dtype with NO default branch and refuses any other by name, because `f22c6cc82` (#1181) landed a guard for a reader that memcpy'd four bytes whatever the dtype was. `vt::LoadUnaligned`, because a safetensors offset can be odd (#627). ## Read the config, do not only probe the tensors `modules_to_not_convert` is a ~400-entry list a dtype probe reproduces only by accident, and a probe cannot see a DISAGREEMENT between the config and the tensors at all. That is exactly where a silent wrong-scale bug lives: #1166 measured a `[96, 40]` grid passing a per-tensor reader's byte floor and being applied to the whole weight, stopped only by the tensor NAME. `IsFp8BlockProjection` therefore decides from BOTH sources and refuses four combinations by name: a `weight_scale_inv` with no `weight_block_size` in the config, an `F8_E4M3` weight with `weight_block_size` and no `weight_scale_inv`, a module listed in `modules_to_not_convert` that nevertheless ships one, and an `input_scale` beside `activation_scheme = dynamic` (upstream registers one only when `act_q_static`, `fp8.py:381-384`). The shape check is upstream's own: `cdiv` on BOTH axes (`fp8_utils.py:1283-1296`) and the exact-shape assertion at `parameter.py:95-98`, so a short final block is legal and works. ## What is still refused, and where `RefuseUnsupportedFp8BlockQuant` keeps its call site in `ModelRegistry::Load` and now refuses only what nothing here can execute: a `quant_method` without `fp8`, a `weight_block_size` that is not exactly two dimensions, an `activation_scheme` other than `dynamic` — the first three mirroring upstream's own `ValueError`s at `fp8.py:115-131` — and a block shape other than 128x128, which is OUR limit and says so. STAGED SLICE, named per `.agents/reachability.md`. Nothing CONSUMES an `Fp8BlockWeight` at this merge commit: `layers::MakeLinearMethod` has no block arm and the dense `project` lambda knows only fp4, per-tensor fp8 and bf16. The wiring is owned by **#1189 milestone M4** (`layers::Fp8BlockLinearMethod` and the Qwen3.5 dense forward), it is listed under `## Owed` in the spec, and it is not left silent: `PrepareQwen3_5Dense` — reached from `ModelRegistry::Prepare`, which every runner calls before the first forward and before graph capture (`v1/worker/gpu/runner.cpp:414,455`) — refuses by name and quotes the projection. The checkpoint loads and declines to run; it never runs wrong. The loader rung ITSELF is reached at this commit, from `ModelRegistry::Load` through `LoadQwen3_5DenseModel` to `LoadQwen3_5Dense`, and the reachability mutation is recorded in the spec's `## Evidence`. ## Gates RED first: with the tests present and no implementation, the focused build fails `compile_rc=1` with 50 errors, 6 of them ``'Fp8BlockWeight' does not name a type`` and the rest missing `*_fp8_block` members. GREEN: `test_fp8_block_weight_load` 7 cases / 102 assertions, whose per-block counts sum to the whole-run total; `test_fp8_block_quant` 8 cases / 35 assertions, rewritten from "the scheme is refused" to "the SUPPORTED config reaches the loader and these four do not". `test_ops_quant_fp8_group_cpu`, `test_ops_matmul_fp8_block_cpu`, `test_ops_fp8_cpu`, `test_qwen36_weights`, `test_linear_method`, `test_model_registry` and `test_op_provider` all pass unchanged. No GPU lease, no checkpoint download: the fixture is a complete but tiny synthetic `Qwen3_5ForConditionalGeneration` safetensors checkpoint written to a temp directory. `docs/FEATURES.md` and `docs/USAGE.md` move from "refused at load" to "loads and does not run yet" in the same change. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- docs/FEATURES.md | 2 +- docs/USAGE.md | 37 +- .../layers/quantization/fp8_block_quant.h | 99 ++- .../models/dense_weight_loaders.h | 99 +++ .../model_executor/models/qwen3_5_dense.h | 31 + .../model_executor/models/qwen3_5_weights.h | 71 ++ .../layers/quantization/fp8_block_quant.cpp | 145 +++- .../model_executor/models/model_registry.cpp | 22 +- .../model_executor/models/qwen3_5_dense.cpp | 10 + .../models/qwen3_5_dense_weights.cpp | 219 +++++- tests/CMakeLists.txt | 6 + .../layers/test_fp8_block_quant.cpp | 160 ++-- .../models/test_fp8_block_weight_load.cpp | 692 ++++++++++++++++++ 13 files changed, 1457 insertions(+), 136 deletions(-) create mode 100644 tests/vllm/model_executor/models/test_fp8_block_weight_load.cpp diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 121881ea8..da1f014a3 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -79,7 +79,7 @@ are our reading of their documented behavior, not measurements. | GPTQ | ◐ CPU dequant | ✅ | ✅ | ☐ | | MXFP4 compressed-tensors | ◐ W4A16 Marlin, mem 2.63x less. gate_up FUSION + decode-graph default-ON; #44 3/3, 32B 6/6. **`VT_MARLIN_DENSE` DEFAULT-ON** (`KERNEL-MARLIN-DENSE-EXEC`): dense marlin 48-CTA, byte-faithful, beats MoE (c8 0.969) | ✅ | ✅ | ☐ | | fp8 weights, per-tensor scale | ✅ | ✅ | ✅ | ☐ | -| Block-wise (fine-grained 128x128) FP8, the `weight_scale_inv` layout | ☐ REFUSED BY NAME at load (#1166): `Qwen/Qwen3.8-27B-FP8` declares `weight_block_size` [128, 128] and this build is per-tensor FP8 only ([spec](../.agents/specs/fp8-blockwise-refusal.md)) | ✅ | ✅ | ☐ | +| Block-wise (fine-grained 128x128) FP8, the `weight_scale_inv` layout | ◐ LOADS, cannot run (#1189 M3): weight + `cdiv` scale rung + config/tensor cross-check; BF16 scale widened to f32. Linear method is M4, so `Prepare` refuses by name ([spec](../.agents/specs/model-fp8-block-weight.md)) | ✅ | ✅ | ☐ | | Per-tensor FP8 W8A8 linear is a shared seam any model can bind | ✅ `models/dense_fp8_gemm.h` + `layers::Fp8W8A8LinearMethod` (#940), bound via `layers::MakeLinearMethod`. One definition, CUDA only ([spec](../.agents/specs/vt-fp8-shared-seam.md)) | ✅ `Fp8LinearMethod` | ✅ | ☐ | | FP8 W8A8 works on a CUDA arch without `cutlass-fp8` | ✅ `vt::QuantFp8Static` registers from an unconditional TU (#960); sm_110 measured ([spec](../.agents/specs/vt-fp8-quant-arch-gate.md)) | ✅ | ✅ | ☐ | | fp8-tower GDN `in_proj` emits bf16, unlocking packed GDN decode | ◐ `VT_GDN_FP8_IN_BF16` + `VT_GDN_PACKED_DECODE_FP8_TOWER` (inert alone), both default **OFF**, ungated (#339) ([spec](../.agents/specs/perf-fp8-alpha-fold.md)) | ✅ bf16 `out_dtype` | ☐ | ☐ | diff --git a/docs/USAGE.md b/docs/USAGE.md index 1be6fccc3..08fa33536 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -567,13 +567,13 @@ quantizes the activation once; a checkpoint whose scales differ keeps the two separate GEMMs automatically. `VT_GDN_MERGED_QKVZ_FP8=0` restores the two GEMMs in the same binary. -### Block-wise FP8 is refused at load +### Block-wise FP8 loads and does not run yet -This build reads per-tensor FP8, where one scale covers a whole weight. It does -not read block-wise FP8, also called fine-grained FP8, where one scale covers -each 128x128 block of the weight. A block-wise checkpoint declares -`quantization_config.weight_block_size` in its `config.json`, and it stores its -scales under `weight_scale_inv` rather than under `weight_scale`. +Block-wise FP8, also called fine-grained FP8, keeps one scale for each 128x128 +block of a weight rather than one scale for the whole weight. A block-wise +checkpoint declares `quantization_config.weight_block_size` in its +`config.json` and stores its scales under `weight_scale_inv` rather than under +`weight_scale`. `Qwen/Qwen3.8-27B-FP8` is such a checkpoint. At revision `017b9c7af6b5689d5dd426a76e0bc077eb5ca20a` it declares `weight_block_size` @@ -581,18 +581,27 @@ scales under `weight_scale_inv` rather than under `weight_scale`. `self_attn.q_proj.weight` as `F8_E4M3` `[12288, 5120]` beside `self_attn.q_proj.weight_scale_inv` as `BF16` `[96, 40]`. -Loading it stops with a message that names the key: +That checkpoint now LOADS. The weights are read into a block-wise FP8 weight, +the `BF16` scale is widened to `F32` by value the way vLLM widens it, and the +config is cross-checked against the tensors so a disagreement is named rather +than guessed at. Nothing can execute the weight yet, so the model refuses to +finish preparing: ```text -quantization_config.weight_block_size [128, 128] selects block-wise -(fine-grained) FP8, which is not implemented. This build implements per-tensor -FP8 only. +block-wise (fine-grained) 128x128 FP8 weights LOADED for +model.layers.0.self_attn.q_proj and nothing in this build can execute them ``` -The refusal is deliberate. Nothing is wrong with that checkpoint, and the -missing arm is in this project. To run the same model here, use a per-tensor -FP8, BF16, NVFP4, or GGUF checkpoint of it. Issue -[#1166](https://github.com/mudler/vllm.cpp/issues/1166) tracks the port. +Two block-wise configurations are refused earlier, at load, because no build +here implements them: an `activation_scheme` other than `dynamic`, and a +`weight_block_size` other than `[128, 128]`. Both messages name the key and the +value your `config.json` declares. + +Nothing is wrong with those checkpoints; the missing arm is in this project. To +run the same model today, use a per-tensor FP8, BF16, NVFP4, or GGUF checkpoint +of it. Issue [#1189](https://github.com/mudler/vllm.cpp/issues/1189) tracks the +remaining milestones, and +[#1166](https://github.com/mudler/vllm.cpp/issues/1166) is the original report. ### A per-tensor scale has to be one F32 number diff --git a/include/vllm/model_executor/layers/quantization/fp8_block_quant.h b/include/vllm/model_executor/layers/quantization/fp8_block_quant.h index 61db18ce5..b82e31df1 100644 --- a/include/vllm/model_executor/layers/quantization/fp8_block_quant.h +++ b/include/vllm/model_executor/layers/quantization/fp8_block_quant.h @@ -1,34 +1,41 @@ -// Block-wise (fine-grained) FP8 detection and its named refusal. +// Block-wise (fine-grained) FP8: the quantization-config reader, the supported +// shape, and the named refusals for everything else. // // UPSTREAM (ported FROM, ground-every-impl rule), pinned vLLM // `5559679229bc961848b121ccdeaa8fa5d79bec98`: -// vllm/model_executor/layers/quantization/fp8.py:161 -// Fp8Config.from_config — reads `weight_block_size` out of the checkpoint's -// quantization config. Absent means None, which means per-tensor. -// vllm/model_executor/layers/quantization/fp8.py:115-132 +// vllm/model_executor/layers/quantization/fp8.py:157-172 +// Fp8Config.from_config — reads `weight_block_size`, `activation_scheme`, +// `ignored_layers`, and `modules_to_not_convert` as the fallback for the +// ignore list. Absent means None, which means per-tensor. +// vllm/model_executor/layers/quantization/fp8.py:115-131 // Fp8Config.__init__ — validates it: an fp8-serialized checkpoint, exactly -// 2 dimensions, and a dynamic activation scheme. +// 2 dimensions, and a dynamic activation scheme. Each of those three is +// mirrored below as a refusal. // vllm/model_executor/layers/quantization/fp8.py:297-298 // Fp8LinearMethod — `self.block_quant = self.weight_block_size is not None` -// is the whole dispatch, and this tree has no arm to dispatch TO. +// is the whole dispatch. // vllm/model_executor/layers/quantization/fp8.py:378-379, :511 // the block scale registers as `weight_scale_inv`, not `weight_scale`, and // the name is strictly conditional on block quant. +// vllm/model_executor/layers/quantization/utils/quant_utils.py:510-524,568-569 +// is_layer_skipped — the DEFAULT match is `prefix_full_match`, i.e. exact +// membership of the module prefix in the ignore list, not a substring test. // -// WHY THIS FILE EXISTS. `include/.../quantization/fp8.h` mirrors the PER-TENSOR -// arm and says so on its first line. A block-wise checkpoint used to enter that -// arm anyway, because the dense loader branches on the weight dtype alone -// (`qwen3_5_dense_weights.cpp:479`) and the block-wise weight really is -// `F8_E4M3`. The load then asked for `.weight_scale` -// (`qwen3_5_weights.cpp:458`), which a block-wise checkpoint does not have, and -// died on `tensor not found`. That sentence is wrong about the world: the -// checkpoint is complete, and it is this tree that is missing an arm. Issue -// #1166, spec `.agents/specs/fp8-blockwise-refusal.md`. +// HISTORY. `469f38395` (#1166) refused the whole scheme by name here, because +// the dense loader branches on the weight dtype alone and a block-wise weight +// really is `F8_E4M3`: the projection entered the per-tensor arm, asked for +// `.weight_scale`, and died on `tensor not found` — a sentence that is +// wrong about the world, since the checkpoint is complete and it is this tree +// that lacked an arm. MODEL-FP8-BLOCK-WEIGHT (#1189 M3, spec +// `.agents/specs/model-fp8-block-weight.md`) narrows that refusal: a +// `[128, 128]` `dynamic` checkpoint now LOADS, and only the shapes and schemes +// nothing here can execute are still refused. // -// SCOPE. Detect and refuse by name. Reading `weight_scale_inv`, applying a -// 128x128 block scale, and the dynamic per-token activation quant upstream -// pairs with it are OWED, not done, and the refusal names the issue that owes -// them. +// SCOPE. Reading the config and refusing what M3 does not cover. The loader +// rung lives in `qwen3_5_dense_weights.cpp`, the weight in +// `models/qwen3_5_weights.h`, and the linear method does not exist yet — #1189 +// milestone M4 owns it, and `PrepareQwen3_5Dense` refuses a loaded-but-unread +// block weight by name rather than letting the forward produce a number. #pragma once #include @@ -38,6 +45,31 @@ namespace vllm { struct HfConfig; +// The block geometry and ignore list a checkpoint declares, once, validated. +// +// `block_quant` false means the checkpoint declares no `weight_block_size` and +// every other field is unset — the per-tensor world, byte-identical to before +// this row. +struct Fp8BlockQuantConfig { + bool block_quant = false; + int64_t block_n = 0; + int64_t block_k = 0; + // `dynamic` whenever `block_quant` is true; the reader refuses anything else. + std::string activation_scheme; + // `modules_to_not_convert`, or `ignored_layers` when the checkpoint spells it + // that way. `Qwen/Qwen3.8-27B-FP8` ships ~400 entries here, which is why the + // loader reads this list rather than inferring exclusion from a dtype probe. + std::vector modules_to_not_convert; + + // Exact-membership test on the MODULE prefix — the tensor name with its + // trailing `.weight` removed. Mirrors `is_layer_skipped`'s default + // `prefix_full_match` (`quant_utils.py:517-518,524,568-569`). Upstream first + // rewrites the list into vLLM module naming (`fp8.py:151-153`); we match in + // CHECKPOINT naming, which is what this loader has, and the two coincide for + // every entry that names a real checkpoint module. + bool ExcludesModule(const std::string& module_prefix) const; +}; + // The `weight_block_size` a checkpoint declares, empty when it declares none. // // Mirrors `Fp8Config.from_config`: the key is read from `quantization_config`, @@ -46,11 +78,30 @@ struct HfConfig; // the wrapper shape is exactly the one in play on `Qwen3_5ForConditionalGeneration`. std::vector Fp8WeightBlockSizeOf(const HfConfig& config); -// Refuses a block-wise FP8 checkpoint by name, or returns when the checkpoint -// is not block-wise. +// Reads and VALIDATES the block-quant config, or returns a default-constructed +// value when the checkpoint is not block-wise. // -// Throws `std::runtime_error`, the type every other load refusal in this tree -// throws, so the C API surfaces it as `VLLM_ERR_MODEL_LOAD` unchanged. +// Throws `std::runtime_error` — the type every other load refusal in this tree +// throws, so the C API surfaces it as `VLLM_ERR_MODEL_LOAD` unchanged — for a +// `quant_method` that is not fp8, a `weight_block_size` that is not exactly two +// dimensions, an `activation_scheme` other than `dynamic`, and a block shape +// other than 128x128. The first three mirror upstream's own `ValueError`s +// (`fp8.py:115-131`); the fourth is OUR limit and says so, because #1189's +// kernel and its CPU reference are both 128x128 and a `[64, 128]` checkpoint +// would otherwise load into a weight nothing can execute. +Fp8BlockQuantConfig ReadFp8BlockQuantConfig(const HfConfig& config); + +// The pre-load gate, called from `ModelRegistry::Load`. Reads the config for its +// refusals and discards the result; the loader reads it again where it needs the +// geometry. Sited on the registry rather than per loader because +// `weight_block_size` is a property of the checkpoint's quantization config and +// not of one architecture. void RefuseUnsupportedFp8BlockQuant(const HfConfig& config); +// The M3/M4 seam. A block-wise weight LOADS and nothing reads it yet, so the +// model refuses to be prepared rather than letting a forward fall through to an +// empty bf16 tensor and produce a fluent wrong answer. `proj` is the projection +// that carries the weight, so the message names one instead of the class. +[[noreturn]] void RefuseUnconsumedFp8BlockWeight(const std::string& proj); + } // namespace vllm diff --git a/include/vllm/model_executor/models/dense_weight_loaders.h b/include/vllm/model_executor/models/dense_weight_loaders.h index fc032d957..27ac84be0 100644 --- a/include/vllm/model_executor/models/dense_weight_loaders.h +++ b/include/vllm/model_executor/models/dense_weight_loaders.h @@ -117,6 +117,105 @@ inline float ReadF32Scalar(const TensorResolver& get, const std::string& name) { return v; } +// Block-wise (fine-grained) FP8 projection: `.weight` F8_E4M3 [N, K] +// beside `.weight_scale_inv` [cdiv(N, block_n), cdiv(K, block_k)] -> +// `Fp8BlockWeight`. MODEL-FP8-BLOCK-WEIGHT, #1189 M3, spec +// `.agents/specs/model-fp8-block-weight.md`. +// +// The fp8 bytes are kept RAW in the on-disk [N=out, K=in] orientation, as +// `LoadFp8Raw` does for the per-tensor arm: no dequant and no transpose, so the +// projection costs one byte per element and every scale decision stays inside +// the GEMM where upstream applies it (per K-block, in the mainloop -- see +// `.agents/specs/vt-matmul-fp8-block-ref.md`). +// +// THE SCALE IS WIDENED TO F32, NOT REINTERPRETED. Upstream allocates the +// parameter `torch.float32` (`utils/fp8_utils.py:1276,1283-1296`) and loads the +// checkpoint tensor into it with `self.data.copy_()` +// (`vllm/model_executor/parameter.py:97`), which CONVERTS. `Qwen/Qwen3.8-27B-FP8` +// ships the tensor `BF16`, so the resident f32 is the mirror rather than a +// widening: it is the dtype upstream carries, `vt::MatmulFp8BlockScaled` refuses +// anything else, and bf16 -> f32 is exact. The switch below has NO default +// branch that memcpy's bytes, because #1181 landed a guard for exactly that. +// `vt::LoadUnaligned` because a safetensors tensor's offset is the running byte +// total of everything ahead of it and can be odd (#627). +// +// The shape check is upstream's own: the allocation at `fp8_utils.py:1283-1296` +// uses `cdiv` on BOTH axes and `parameter.py:95-98` then asserts the loaded +// tensor matches it exactly. A short final block is legal and must work. +inline Fp8BlockWeight LoadFp8BlockRaw(const TensorResolver& get, + const std::string& proj, int64_t block_n, + int64_t block_k) { + VT_CHECK(block_n > 0 && block_k > 0, + "dense loader: '" + proj + + "' block-wise FP8 needs positive block dimensions, got [" + + std::to_string(block_n) + ", " + std::to_string(block_k) + "]"); + const StTensor& w = get(proj + ".weight"); + VT_CHECK(w.dtype == "F8_E4M3", + "dense loader: '" + proj + ".weight' ships dtype " + w.dtype + + ", not the F8_E4M3 a block-wise FP8 weight is"); + VT_CHECK(w.shape.size() == 2, + "dense loader: '" + proj + ".weight' ships shape " + + ShapeString(w.shape) + + ", not the 2-D [out_features, in_features] a block-wise FP8 " + "weight is"); + Fp8BlockWeight r; + r.n = w.shape[0]; + r.k = w.shape[1]; + r.block_n = block_n; + r.block_k = block_k; + + const std::string scale_name = proj + ".weight_scale_inv"; + const StTensor& s = get(scale_name); + const int64_t rows = (r.n + block_n - 1) / block_n; + const int64_t cols = (r.k + block_k - 1) / block_k; + VT_CHECK( + s.shape.size() == 2 && s.shape[0] == rows && s.shape[1] == cols, + "dense loader: '" + scale_name + "' ships shape " + + ShapeString(s.shape) + ", not the " + + ShapeString(std::vector{rows, cols}) + + " a [" + std::to_string(r.n) + ", " + std::to_string(r.k) + + "] weight quantized in [" + std::to_string(block_n) + ", " + + std::to_string(block_k) + + "] blocks needs. Both dimensions round UP (ceil), so a short final " + "block still owns a scale"); + const int64_t count = rows * cols; + r.scale = MakeOwned(vt::DType::kF32, {rows, cols}); + auto* dst = reinterpret_cast(r.scale.bytes.data()); + if (s.dtype == "BF16") { + VT_CHECK(s.data != nullptr && + s.nbytes == static_cast(count) * sizeof(uint16_t), + "dense loader: '" + scale_name + + "' is a BF16 block scale but does not carry " + + std::to_string(count * 2) + " readable bytes"); + for (int64_t i = 0; i < count; ++i) + dst[i] = vt::BF16ToF32(vt::LoadUnaligned(s.data + i * 2)); + } else if (s.dtype == "F32") { + VT_CHECK(s.data != nullptr && + s.nbytes == static_cast(count) * sizeof(float), + "dense loader: '" + scale_name + + "' is an F32 block scale but does not carry " + + std::to_string(count * 4) + " readable bytes"); + for (int64_t i = 0; i < count; ++i) + dst[i] = vt::LoadUnaligned(s.data + i * 4); + } else { + VT_CHECK(false, + "dense loader: '" + scale_name + "' ships dtype " + s.dtype + + ", and a block-wise FP8 scale is read as BF16 or F32 only. " + "Upstream loads it into an F32 parameter with a CONVERTING " + "copy, so a narrower dtype is widened by VALUE; reading its " + "bytes as another dtype is the defect issue #1181 fixed"); + } + MaybeReleaseSourcePages(s.data, s.nbytes); + + r.packed = MakeOwned(vt::DType::kI8, {r.n, r.k}); + VT_CHECK(w.nbytes == r.packed.bytes.size(), + "dense loader: '" + proj + + ".weight' block-wise FP8 byte-size mismatch"); + std::memcpy(r.packed.bytes.data(), w.data, w.nbytes); + MaybeReleaseSourcePages(w.data, w.nbytes); + return r; +} + // src bf16 [rows, cols] -> dst bf16 [cols, rows]. inline void TransposeBf16(const void* src, int64_t rows, int64_t cols, uint16_t* dst) { diff --git a/include/vllm/model_executor/models/qwen3_5_dense.h b/include/vllm/model_executor/models/qwen3_5_dense.h index 888a91dbd..ed3854589 100644 --- a/include/vllm/model_executor/models/qwen3_5_dense.h +++ b/include/vllm/model_executor/models/qwen3_5_dense.h @@ -33,6 +33,7 @@ #include "vllm/model_executor/models/qwen3_5.h" // PagedKvCache, GdnStateCache + v1 attention metadata #include +#include "vllm/model_executor/layers/quantization/fp8_block_quant.h" #include "vllm/model_executor/models/qwen3_5_weights.h" // OwnedTensor, Gdn/FullAttn weights, TensorResolver #include "vllm/transformers_utils/hf_config.h" #include "vt/device.h" @@ -61,6 +62,16 @@ struct DenseMlpWeights { Nvfp4Weight up_proj_fp4; // [N=I, K=H] Nvfp4Weight down_proj_fp4; // [N=H, K=I] + // MODEL-FP8-BLOCK-WEIGHT (#1189 M3): block-wise (128x128) FP8 MLP + // projections. This block had NO fp8 rung at all before that row, so a + // block-wise MLP fell through to `LoadMergedBf16RawNK` and died on + // "expected BF16". Loaded UNMERGED: block scales concatenate losslessly along + // N, so merging gate+up is simpler here than in the per-tensor case and + // #1189 M6 owns it. + Fp8BlockWeight gate_proj_fp8_block; // [N=I, K=H] + Fp8BlockWeight up_proj_fp8_block; // [N=I, K=H] + Fp8BlockWeight down_proj_fp8_block; // [N=H, K=I] + // CUDA resident for vLLM's MergedColumnParallelLinear gate_up_proj. The // checkpoint stores gate/up separately; production concatenates their packed // rows and linear block scales once, then keeps only the combined packed @@ -201,6 +212,19 @@ Qwen3_5DenseLayerWeights LoadQwen3_5DenseLayer( // or an FP8/BF16 projection next to an NVFP4 one) needs the real probe. Exposed // so the loader gate can drive a whole synthetic layer through the SAME routing // production takes. +// MODEL-FP8-BLOCK-WEIGHT (#1189 M3): `block` carries the checkpoint's declared +// `weight_block_size`, `activation_scheme` and `modules_to_not_convert`, read +// ONCE by `LoadQwen3_5Dense` from the quantization config. A default-constructed +// value means "not block-wise", which is byte-identical to the routing before +// that row. The two seams above default to it; the production loader passes the +// value it read, because a dtype probe alone cannot detect a config/tensor +// DISAGREEMENT and that is where a silent wrong-scale bug lives. +Qwen3_5DenseLayerWeights LoadQwen3_5DenseLayer( + const TensorResolver& get, const std::function& has, + const std::string& layer_type, int64_t layer_idx, + const std::string& backbone_prefix, + const Fp8BlockQuantConfig& block); + Qwen3_5DenseLayerWeights LoadQwen3_5DenseLayer( const TensorResolver& get, const std::function& has, const std::string& layer_type, int64_t layer_idx, @@ -218,6 +242,13 @@ Qwen3_5DenseWeights LoadQwen3_5Dense(const std::vector& shards, // Host-lifetime helpers for ordinary dense CUDA models. The release function // drops only tensors whose authoritative raw/F32 device representation exists; // unused fallbacks stay host-resident. The caller synchronizes first. +// MODEL-FP8-BLOCK-WEIGHT (#1189 M3), the M3/M4 seam. Throws by name when the +// load produced a block-wise FP8 weight, because nothing in this build can read +// one yet. Called from `PrepareQwen3_5Dense`, i.e. `ModelRegistry::Prepare`, so +// a block-wise checkpoint LOADS and declines to run rather than falling through +// to an empty bf16 tensor. Milestone M4 deletes it along with the gap. +void RefuseUnconsumedQwen3_5DenseFp8Block(const Qwen3_5DenseWeights& weights); + bool IsPlainBf16Qwen3_5Dense(const Qwen3_5DenseWeights& weights); size_t ReleaseResidentQwen3_5DenseHostWeights(Qwen3_5DenseWeights& weights); diff --git a/include/vllm/model_executor/models/qwen3_5_weights.h b/include/vllm/model_executor/models/qwen3_5_weights.h index 3c11f21de..caaa9c0ac 100644 --- a/include/vllm/model_executor/models/qwen3_5_weights.h +++ b/include/vllm/model_executor/models/qwen3_5_weights.h @@ -329,6 +329,58 @@ struct Fp8Weight { mutable std::shared_ptr d_packed; }; +// Block-wise (fine-grained) FP8 weight — MODEL-FP8-BLOCK-WEIGHT, #1189 M3, spec +// `.agents/specs/model-fp8-block-weight.md`. One fp8-e4m3fn scale per +// `block_n` x `block_k` tile of the weight, the layout `Qwen/Qwen3.8-27B-FP8` +// ships and vLLM registers as `weight_scale_inv` +// (`vllm/model_executor/layers/quantization/fp8.py:378-379,511` @ `555967922`). +// +// A SIBLING of `Fp8Weight` above, deliberately, not an extension of it. That +// struct is three host floats whose whole point is the `alpha = input_scale * +// weight_scale` folded once at load; a block scheme has NO `input_scale` at all +// (the activation scheme is `dynamic`, and the target checkpoint ships zero such +// tensors) and its weight scale is a 2-D tensor, so there is no value `alpha` +// could take. Adding an optional tensor to `Fp8Weight` would make every existing +// reader of `alpha` — the cutlass and cuBLASLt fp8 wrappers, the merged-QKV alpha +// vector, `PrepareGdnFp8Resident` — carry a silent which-arm branch, and the one +// that forgets it returns a plausible number instead of an error. A distinct type +// makes the wrong call site fail to COMPILE. The shape mirrors `Nvfp4Weight` +// above: an OwnedTensor scale beside the packed values plus lazy device handles. +// +// `scale` is f32 and that is the MIRROR rather than a widening +// (`.agents/porting.md` §"Mirror the memory format"). Upstream allocates the +// parameter `torch.float32` (`utils/fp8_utils.py:1276,1283-1296`; `scale_dtype` +// is None unless `is_scale_e8m0`, which `Fp8Config` does not define) and loads +// the checkpoint tensor into it with `self.data.copy_()` +// (`vllm/model_executor/parameter.py:97`), a CONVERTING copy. A `BF16` tensor on +// disk is therefore widened to f32 once, at load, losslessly. `LoadFp8BlockRaw` +// switches on the on-disk dtype explicitly and has no default branch, because +// #1181 landed a guard for a reader that memcpy'd four bytes whatever the dtype +// was. +// +// `block_n`/`block_k` are carried ON the weight rather than looked up from the +// config at use time: the consumer needs them per GEMM, and a weight that knows +// its own geometry cannot be paired with the wrong one. +// +// NOTHING CONSUMES THIS YET. #1189 milestone M4 owns `Fp8BlockLinearMethod` and +// the forward wiring; `PrepareQwen3_5Dense` refuses a populated one by name so a +// block-wise checkpoint declines to run rather than running wrong. +struct Fp8BlockWeight { + OwnedTensor packed; // i8 [N, K] one fp8-e4m3fn byte per element, verbatim + OwnedTensor scale; // f32 [cdiv(N, block_n), cdiv(K, block_k)] + int64_t n = 0; // out_features + int64_t k = 0; // in_features + int64_t block_n = 0; + int64_t block_k = 0; + bool Empty() const { return packed.Empty(); } + + // Lazily-populated device-resident copies (CUDA forward only; null on host or + // before first use). Declared here so M4/M5 upload through the same seam every + // other quantized weight uses; owed and unpopulated at this merge commit. + mutable std::shared_ptr d_packed; + mutable std::shared_ptr d_scale; +}; + // Gated-DeltaNet (linear_attention) layer weights. Projections in Matmul-B // layout [in, out]; conv1d [conv_dim, K]; a_log/dt_bias f32 [Hv]; norm bf16. struct GdnLayerWeights { @@ -357,6 +409,14 @@ struct GdnLayerWeights { OwnedTensor norm_weight; // bf16 [Dv] (RMSNormGated) OwnedTensor out_proj; // bf16 [value_dim, H] (FP8 dequant + T) + // MODEL-FP8-BLOCK-WEIGHT (#1189 M3): block-wise FP8 GDN projections. The + // target checkpoint lists the GDN small tensors under + // `modules_to_not_convert`, so these stay empty for it; the rung exists + // because the SITE probes `F8_E4M3` and a block-wise weight is one. + Fp8BlockWeight in_proj_qkv_fp8_block; + Fp8BlockWeight in_proj_z_fp8_block; + Fp8BlockWeight out_proj_fp8_block; + // 27B W4A4 fp4-resident variant of out_proj (compressed-tensors NVFP4, notes // §3.6). When populated (real 27B CUDA load) the forward calls vt::MatmulNvfp4 // on it and out_proj above is left EMPTY; the 35B / synthetic loaders populate @@ -431,6 +491,17 @@ struct FullAttnLayerWeights { Fp8Weight v_proj_fp8; // [N=Hkv*Dh, K=H] Fp8Weight o_proj_fp8; // [N=H, K=Hq*Dh] + // MODEL-FP8-BLOCK-WEIGHT (#1189 M3): the block-wise (128x128) FP8 variants, + // populated by the `weight_scale_inv` rung in `qwen3_5_dense_weights.cpp` + // BEFORE the per-tensor rung, because a block-wise weight is also `F8_E4M3` + // (#1166). The bf16, fp4 and per-tensor fp8 slots are left EMPTY when these + // are populated, and vice versa. Nothing reads them yet -- M4 owns the linear + // method and `PrepareQwen3_5Dense` refuses a populated one by name. + Fp8BlockWeight q_proj_fp8_block; // [N=2*Hq*Dh, K=H] + Fp8BlockWeight k_proj_fp8_block; // [N=Hkv*Dh, K=H] + Fp8BlockWeight v_proj_fp8_block; // [N=Hkv*Dh, K=H] + Fp8BlockWeight o_proj_fp8_block; // [N=H, K=Hq*Dh] + // CUDA resident for the FP8 (W8A8) analog of QKVParallelLinear (VT_FP8_MERGED // _QKV, opt-in). The checkpoint owns logical Q/K/V shards separately with a // shared per-tensor input_scale but per-projection weight_scale; production diff --git a/src/vllm/model_executor/layers/quantization/fp8_block_quant.cpp b/src/vllm/model_executor/layers/quantization/fp8_block_quant.cpp index 1ab542ddc..76ac7f078 100644 --- a/src/vllm/model_executor/layers/quantization/fp8_block_quant.cpp +++ b/src/vllm/model_executor/layers/quantization/fp8_block_quant.cpp @@ -1,5 +1,6 @@ #include "vllm/model_executor/layers/quantization/fp8_block_quant.h" +#include #include #include #include @@ -34,8 +35,44 @@ std::string DimensionList(const std::vector& dims) { return out + "]"; } +std::string StringOr(const nlohmann::json& quant, const char* key, + const std::string& fallback) { + const auto it = quant.find(key); + if (it == quant.end() || !it->is_string()) return fallback; + return it->get(); +} + +// `ignored_layers` first, then `modules_to_not_convert`, exactly the order +// `Fp8Config.from_config` reads them in (`fp8.py:160,165-168`): the fallback is +// taken only when the first is absent or empty. +std::vector IgnoreListOf(const nlohmann::json& quant) { + const auto read = [&quant](const char* key) { + std::vector out; + const auto it = quant.find(key); + if (it == quant.end() || !it->is_array()) return out; + for (const nlohmann::json& entry : *it) + if (entry.is_string()) out.push_back(entry.get()); + return out; + }; + std::vector ignored = read("ignored_layers"); + if (!ignored.empty()) return ignored; + return read("modules_to_not_convert"); +} + +constexpr int64_t kSupportedBlockN = 128; +constexpr int64_t kSupportedBlockK = 128; + +const char* kIssue = "https://github.com/mudler/vllm.cpp/issues/1189"; + } // namespace +bool Fp8BlockQuantConfig::ExcludesModule( + const std::string& module_prefix) const { + return std::find(modules_to_not_convert.begin(), + modules_to_not_convert.end(), + module_prefix) != modules_to_not_convert.end(); +} + std::vector Fp8WeightBlockSizeOf(const HfConfig& config) { const nlohmann::json* quant = QuantizationConfigOf(config); if (quant == nullptr) return {}; @@ -52,24 +89,102 @@ std::vector Fp8WeightBlockSizeOf(const HfConfig& config) { return dims; } -void RefuseUnsupportedFp8BlockQuant(const HfConfig& config) { +Fp8BlockQuantConfig ReadFp8BlockQuantConfig(const HfConfig& config) { + Fp8BlockQuantConfig out; const std::vector block = Fp8WeightBlockSizeOf(config); - if (block.empty()) return; + if (block.empty()) return out; + + const nlohmann::json* quant = QuantizationConfigOf(config); + // `Fp8WeightBlockSizeOf` only returns a non-empty list when it found one, so + // the object exists. Asserted rather than assumed because a null deref here + // would be the one failure mode this whole file exists to prevent. + if (quant == nullptr) return out; + + // `is_checkpoint_fp8_serialized = "fp8" in quant_method` (`fp8.py:158-159`), + // and block quant requires it (`fp8.py:117-120`). + const std::string method = StringOr(*quant, "quant_method", ""); + if (method.find("fp8") == std::string::npos) { + throw std::runtime_error( + "quantization_config.weight_block_size " + DimensionList(block) + + " selects block-wise (fine-grained) FP8, but quant_method is \"" + + method + + "\", which is not an fp8-serialized checkpoint. Block-wise weight " + "quantization is defined only for fp8 here, exactly as upstream " + "requires (vllm fp8.py:117-120). Tracked by " + + kIssue); + } + + // `len(weight_block_size) != 2` (`fp8.py:121-126`). + if (block.size() != 2) { + throw std::runtime_error( + "quantization_config.weight_block_size " + DimensionList(block) + + " has " + std::to_string(block.size()) + + " dimensions. A block-wise FP8 quantization block must have exactly " + "2 dimensions, [block_n, block_k], one scale per block of the " + "[out_features, in_features] weight. Tracked by " + + kIssue); + } + + // `activation_scheme != "dynamic"` (`fp8.py:127-131`). The default matches + // `Fp8Config.__init__`'s own default (`fp8.py:102`). + const std::string scheme = StringOr(*quant, "activation_scheme", "dynamic"); + if (scheme != "dynamic") { + throw std::runtime_error( + "quantization_config.activation_scheme is \"" + scheme + + "\" beside weight_block_size " + DimensionList(block) + + ". Block-wise (fine-grained) FP8 supports only the \"dynamic\" " + "activation scheme, which quantizes activations per token and per " + "group at run time; a static per-tensor input_scale cannot express " + "it. Upstream refuses the same combination (vllm fp8.py:127-131). " + "Tracked by " + + kIssue); + } + + // OUR limit, not upstream's, and the message says so. `vt::QuantFp8Group` + // and `vt::MatmulFp8BlockScaled` are general, but the M5 CUTLASS kernel that + // will actually execute this is 128x128 and no gate in this tree has run any + // other shape end to end. Loading a [64, 128] checkpoint would fill a weight + // nothing can consume. + if (block[0] != kSupportedBlockN || block[1] != kSupportedBlockK) { + throw std::runtime_error( + "quantization_config.weight_block_size " + DimensionList(block) + + " selects a block-wise (fine-grained) FP8 block shape this build does " + "not implement. Only [" + + std::to_string(kSupportedBlockN) + ", " + + std::to_string(kSupportedBlockK) + + "] is implemented, which is the shape Qwen3.8-27B-FP8 and DeepSeek-V3 " + "style checkpoints ship. This is a missing arm in vllm.cpp and not a " + "problem with the checkpoint. Tracked by " + + kIssue); + } + + out.block_quant = true; + out.block_n = block[0]; + out.block_k = block[1]; + out.activation_scheme = scheme; + out.modules_to_not_convert = IgnoreListOf(*quant); + return out; +} + +void RefuseUnsupportedFp8BlockQuant(const HfConfig& config) { + const Fp8BlockQuantConfig block = ReadFp8BlockQuantConfig(config); + (void)block; +} - // Named, not merely refused. The key so the reader can grep their own - // config.json, the value so the message is about THIS checkpoint, the arm - // that is missing, the arm that works, where the scale actually lives, and - // the issue that owes the port. +void RefuseUnconsumedFp8BlockWeight(const std::string& proj) { + // Named, not merely refused: the projection so the reader knows this is a + // real loaded weight rather than a config guess, what is missing, what DOES + // work today, and the issue that owes the rest. throw std::runtime_error( - "quantization_config.weight_block_size " + DimensionList(block) + - " selects block-wise (fine-grained) FP8, which is not implemented. This " - "build implements per-tensor FP8 only. A block-wise checkpoint stores one " - "scale for each " + - (block.size() == 2 ? DimensionList(block) : std::string("block")) + - " weight block under `weight_scale_inv`, and nothing here reads that " - "tensor, so the weights cannot be dequantized correctly. This is a " - "missing arm in vllm.cpp and not a problem with the checkpoint. Tracked " - "by https://github.com/mudler/vllm.cpp/issues/1166"); + "block-wise (fine-grained) 128x128 FP8 weights LOADED for " + proj + + " and nothing in this build can execute them: there is no block-wise " + "FP8 linear method, so the checkpoint would run through an empty weight " + "and produce fluent wrong output. The loader, the weight and the CPU " + "reference GEMM are implemented; the linear method and the forward " + "wiring are milestone M4 of " + + std::string(kIssue) + + ". Per-tensor FP8 and NVFP4 checkpoints of the same architecture run " + "today."); } } // namespace vllm diff --git a/src/vllm/model_executor/models/model_registry.cpp b/src/vllm/model_executor/models/model_registry.cpp index 73bf3a272..c524b9018 100644 --- a/src/vllm/model_executor/models/model_registry.cpp +++ b/src/vllm/model_executor/models/model_registry.cpp @@ -323,16 +323,24 @@ ModelRegistry::OutOfTreeSupportedModels() { std::unique_ptr ModelRegistry::Load(const HfConfig& config, const ModelSource& source) { const ModelRegistration& registration = Resolve(config); - // FIX-FP8-BLOCKWISE-REFUSAL (#1166): a block-wise (fine-grained) FP8 - // checkpoint is refused BY NAME here, before any weight loader runs. + // MODEL-FP8-BLOCK-WEIGHT (#1189 M3), narrowing FIX-FP8-BLOCKWISE-REFUSAL + // (#1166): the block-wise (fine-grained) FP8 config is READ and VALIDATED + // here, before any weight loader runs, and only what this build cannot + // execute is refused BY NAME -- a `quant_method` that is not fp8, a + // `weight_block_size` that is not exactly two dimensions, an + // `activation_scheme` other than `dynamic`, and a block shape other than + // 128x128. A supported `[128, 128]` `dynamic` checkpoint now passes and its + // projections load through the `weight_scale_inv` rung in + // `qwen3_5_dense_weights.cpp`. Nothing READS the resulting weight yet, and + // that gap is refused by name one step later, at `ModelRegistry::Prepare`. // // AFTER `Resolve`, so an unsupported architecture still reports the // architecture rather than its quantization. BEFORE `load_weights`, because - // that is what makes the message about the missing ARM instead of about the - // first tensor whose name does not resolve: the dense loader branches on the - // weight dtype alone and pulls a block-wise projection into the per-tensor - // arm, which then asks for a `weight_scale` a block-wise checkpoint spells - // `weight_scale_inv` and dies on `tensor not found`. + // that is what makes the message about the unsupported SHAPE instead of about + // the first tensor whose name does not resolve: the dense loader branches on + // the weight dtype alone and a block-wise weight really is `F8_E4M3`, so it + // used to enter the per-tensor arm, ask for a `weight_scale` the checkpoint + // spells `weight_scale_inv`, and die on `tensor not found`. // // Sited on the registry rather than per model loader on purpose. // `weight_block_size` is a property of the checkpoint's quantization config diff --git a/src/vllm/model_executor/models/qwen3_5_dense.cpp b/src/vllm/model_executor/models/qwen3_5_dense.cpp index 07d9abcba..2f8f1690c 100644 --- a/src/vllm/model_executor/models/qwen3_5_dense.cpp +++ b/src/vllm/model_executor/models/qwen3_5_dense.cpp @@ -103,6 +103,16 @@ std::unique_ptr LoadQwen3_5DenseModel( void PrepareQwen3_5Dense(LoadedModel& model, const HfConfig& config, vt::Queue& queue) { + // MODEL-FP8-BLOCK-WEIGHT (#1189 M3). FIRST, before any resident is built: the + // loader can now produce `Fp8BlockWeight`s and no forward path reads one, so + // an unwired block-wise checkpoint is refused BY NAME here instead of running + // through an empty bf16 tensor. `ModelRegistry::Prepare` is called by every + // runner before the first forward and before graph capture. Inert on every + // other checkpoint. Milestone M4 removes this along with the gap. + RefuseUnconsumedQwen3_5DenseFp8Block( + ModelAs(model, + "Qwen3_5ForConditionalGeneration") + .weights()); // PERF-27B-LMHEAD-FP4 (issue #213): build the packed lm_head's resident HERE — // on CUDA before the runner captures a decode graph, elsewhere before the first // forward pays the dequant. Inert on every BF16/FP8/GGUF/tied checkpoint. diff --git a/src/vllm/model_executor/models/qwen3_5_dense_weights.cpp b/src/vllm/model_executor/models/qwen3_5_dense_weights.cpp index af0caea59..36c6a9e9e 100644 --- a/src/vllm/model_executor/models/qwen3_5_dense_weights.cpp +++ b/src/vllm/model_executor/models/qwen3_5_dense_weights.cpp @@ -14,6 +14,7 @@ #include "vllm/model_executor/layers/quantization/compressed_tensors/nvfp4_emulation.h" #include "vllm/model_executor/model_loader/nvfp4_dequant.h" +#include "vllm/model_executor/layers/quantization/fp8_block_quant.h" #include "vllm/model_executor/models/dense_weight_loaders.h" #include "vllm/platforms/interface.h" #include "vt/backend.h" @@ -404,8 +405,77 @@ Nvfp4Weight LoadNvfp4AnyNaming(const TensorResolver& get, const TensorExists& ha return r; } +// MODEL-FP8-BLOCK-WEIGHT (#1189 M3), spec +// `.agents/specs/model-fp8-block-weight.md`. Does this projection take the +// block-wise (fine-grained 128x128) FP8 arm? +// +// READ THE CONFIG AND THE TENSORS, AND REFUSE WHEN THEY DISAGREE. A dtype probe +// alone answers "which arm" and can never answer "do these two sources agree", +// and the disagreement is where a silent wrong-scale bug lives: #1166 measured a +// `[96, 40]` block grid passing a per-tensor scale reader's byte floor and being +// applied to the whole `[N, K]` weight, stopped only by the tensor NAME. The +// checkpoint's `modules_to_not_convert` is the other half -- ~400 entries on +// `Qwen/Qwen3.8-27B-FP8` -- which a probe reproduces only by accident. +// +// Six combinations, four of them refused by name; the table is in the spec. +bool IsFp8BlockProjection(const TensorExists& has, const std::string& proj, + const std::string& weight_dtype, + const Fp8BlockQuantConfig& block) { + const bool has_scale_inv = has(proj + ".weight_scale_inv"); + const bool excluded = block.ExcludesModule(proj); + + if (!block.block_quant) { + VT_CHECK(!has_scale_inv, + "qwen3_5 dense: '" + proj + + ".weight_scale_inv' is present, which is the block-wise " + "(fine-grained) FP8 scale, but the checkpoint's " + "quantization_config declares no weight_block_size. The " + "tensors and the config disagree and there is no block " + "geometry to read the scale with; refusing rather than " + "guessing 128x128"); + return false; + } + + if (excluded) { + VT_CHECK(!has_scale_inv, + "qwen3_5 dense: '" + proj + + "' is listed in quantization_config.modules_to_not_convert, " + "so it must be unquantized, yet it ships '" + proj + + ".weight_scale_inv'. The tensors and the config disagree; " + "refusing rather than picking one of them"); + return false; + } + + if (weight_dtype == "F8_E4M3") { + VT_CHECK(has_scale_inv, + "qwen3_5 dense: the checkpoint declares " + "quantization_config.weight_block_size and '" + + proj + + ".weight' is F8_E4M3, but '" + proj + + ".weight_scale_inv' is missing. Block-wise (fine-grained) FP8 " + "stores its scale under that name and only that name " + "(vllm fp8.py:378-379,511), so this projection cannot be " + "dequantized. Either the module belongs in " + "modules_to_not_convert or the shard is incomplete"); + } + if (!has_scale_inv) return false; + + // An `input_scale` cannot coexist with a dynamic activation scheme: upstream + // registers one only when `act_q_static` (`fp8.py:381-384`), which block quant + // asserts against outright (`fp8.py:367`). `Qwen/Qwen3.8-27B-FP8` ships zero. + VT_CHECK(!has(proj + ".input_scale"), + "qwen3_5 dense: '" + proj + + ".input_scale' is present beside a block-wise (fine-grained) " + "FP8 weight whose quantization_config declares " + "activation_scheme \"dynamic\". A dynamic scheme quantizes " + "activations at run time and has no static input scale; the " + "tensors and the config disagree"); + return true; +} + GdnLayerWeights LoadGdnDense(const TensorResolver& get, const TensorExists& has, - const std::string& base) { + const std::string& base, + const Fp8BlockQuantConfig& block) { const std::string la = base + "linear_attn."; GdnLayerWeights g; // in_proj_{qkv,z,a,b}: bf16 (ignore list, notes §3.6). Kept raw [N,K] @@ -421,7 +491,15 @@ GdnLayerWeights LoadGdnDense(const TensorResolver& get, const TensorExists& has, // decode and starves the KV pool. `ProjectGdnQkvz` already carries the // separate-fp8 arm the 35B runs and selects it when the merged owner is empty. const StTensor& qkv_probe = get(la + "in_proj_qkv.weight"); - if (qkv_probe.dtype == "F8_E4M3") { + // MODEL-FP8-BLOCK-WEIGHT (#1189 M3): the block-wise rung goes BEFORE the + // per-tensor one at every site that probes `F8_E4M3`, because a block-wise + // weight IS `F8_E4M3` and fell into the per-tensor arm (#1166). + if (IsFp8BlockProjection(has, la + "in_proj_qkv", qkv_probe.dtype, block)) { + g.in_proj_qkv_fp8_block = dense_loaders::LoadFp8BlockRaw( + get, la + "in_proj_qkv", block.block_n, block.block_k); + g.in_proj_z_fp8_block = dense_loaders::LoadFp8BlockRaw( + get, la + "in_proj_z", block.block_n, block.block_k); + } else if (qkv_probe.dtype == "F8_E4M3") { g.in_proj_qkv_fp8 = LoadFp8RawShared(get, la + "in_proj_qkv"); g.in_proj_z_fp8 = LoadFp8RawShared(get, la + "in_proj_z"); } else { @@ -434,6 +512,10 @@ GdnLayerWeights LoadGdnDense(const TensorResolver& get, const TensorExists& has, // torch-Linear BF16 [N,K]. if (IsNvfp4Projection(has, la + "out_proj")) { g.out_proj_fp4 = LoadNvfp4AnyNaming(get, has, la + "out_proj"); + } else if (IsFp8BlockProjection(has, la + "out_proj", + get(la + "out_proj.weight").dtype, block)) { + g.out_proj_fp8_block = dense_loaders::LoadFp8BlockRaw( + get, la + "out_proj", block.block_n, block.block_k); } else if (get(la + "out_proj.weight").dtype == "F8_E4M3") { // Same rule as the in_proj shards above, and for the same measured reason: // the bf16 arm dequantizes this tower and then runs it as a cuBLAS `gemvx`, @@ -457,7 +539,8 @@ GdnLayerWeights LoadGdnDense(const TensorResolver& get, const TensorExists& has, FullAttnLayerWeights LoadAttnDense(const TensorResolver& get, const TensorExists& has, - const std::string& base) { + const std::string& base, + const Fp8BlockQuantConfig& block) { const std::string sa = base + "self_attn."; FullAttnLayerWeights a; // Three forms, not two. `modelopt_mixed` checkpoints quantize this tower to @@ -467,20 +550,35 @@ FullAttnLayerWeights LoadAttnDense(const TensorResolver& get, // 1.562 GiB of FP8 became 3.12 GiB of BF16 re-read every decode step and // executed as cuBLAS `gemvx`. The `*_fp8` slots and their `MatmulFp8Cutlass*` // consumers already exist and are what the 35B runs. + // FOUR forms. MODEL-FP8-BLOCK-WEIGHT (#1189 M3) inserts the block-wise rung + // BEFORE the per-tensor one, because a block-wise weight is also `F8_E4M3` + // and therefore entered the per-tensor arm, which then asked for a + // `weight_scale` the checkpoint spells `weight_scale_inv` (#1166). const auto load_projection = [&](const std::string& name, Nvfp4Weight& fp4, - Fp8Weight& fp8, OwnedTensor& plain) { + Fp8BlockWeight& fp8_block, Fp8Weight& fp8, + OwnedTensor& plain) { if (IsNvfp4Projection(has, name)) { fp4 = LoadNvfp4AnyNaming(get, has, name); - } else if (get(name + ".weight").dtype == "F8_E4M3") { + return; + } + const std::string dtype = get(name + ".weight").dtype; + if (IsFp8BlockProjection(has, name, dtype, block)) { + fp8_block = dense_loaders::LoadFp8BlockRaw(get, name, block.block_n, + block.block_k); + } else if (dtype == "F8_E4M3") { fp8 = LoadFp8RawShared(get, name); } else { plain = LoadBf16RawNK(get, name + ".weight"); } }; - load_projection(sa + "q_proj", a.q_proj_fp4, a.q_proj_fp8, a.q_proj); - load_projection(sa + "k_proj", a.k_proj_fp4, a.k_proj_fp8, a.k_proj); - load_projection(sa + "v_proj", a.v_proj_fp4, a.v_proj_fp8, a.v_proj); - load_projection(sa + "o_proj", a.o_proj_fp4, a.o_proj_fp8, a.o_proj); + load_projection(sa + "q_proj", a.q_proj_fp4, a.q_proj_fp8_block, a.q_proj_fp8, + a.q_proj); + load_projection(sa + "k_proj", a.k_proj_fp4, a.k_proj_fp8_block, a.k_proj_fp8, + a.k_proj); + load_projection(sa + "v_proj", a.v_proj_fp4, a.v_proj_fp8_block, a.v_proj_fp8, + a.v_proj); + load_projection(sa + "o_proj", a.o_proj_fp4, a.o_proj_fp8_block, a.o_proj_fp8, + a.o_proj); a.q_norm = LoadModelBf16Direct(get, sa + "q_norm.weight"); a.k_norm = LoadModelBf16Direct(get, sa + "k_norm.weight"); return a; @@ -488,13 +586,27 @@ FullAttnLayerWeights LoadAttnDense(const TensorResolver& get, // Dense SwiGLU MLP: gate/up/down all W4A4-quantized -> fp4-resident (§5 6a). DenseMlpWeights LoadDenseMlp(const TensorResolver& get, const TensorExists& has, - const std::string& base) { + const std::string& base, + const Fp8BlockQuantConfig& block) { const std::string mlp = base + "mlp."; DenseMlpWeights m; if (IsNvfp4Projection(has, mlp + "gate_proj")) { m.gate_proj_fp4 = LoadNvfp4AnyNaming(get, has, mlp + "gate_proj"); m.up_proj_fp4 = LoadNvfp4AnyNaming(get, has, mlp + "up_proj"); m.down_proj_fp4 = LoadNvfp4AnyNaming(get, has, mlp + "down_proj"); + } else if (IsFp8BlockProjection(has, mlp + "gate_proj", + get(mlp + "gate_proj.weight").dtype, block)) { + // MODEL-FP8-BLOCK-WEIGHT (#1189 M3). This block had NO fp8 rung at all, so + // a block-wise MLP fell through to `LoadMergedBf16RawNK` and died on + // "expected BF16". Loaded UNMERGED: block scales concatenate losslessly + // along N, so merging gate+up is simpler here than in the per-tensor case + // and #1189 M6 owns it rather than this row guessing at the layout. + m.gate_proj_fp8_block = dense_loaders::LoadFp8BlockRaw( + get, mlp + "gate_proj", block.block_n, block.block_k); + m.up_proj_fp8_block = dense_loaders::LoadFp8BlockRaw( + get, mlp + "up_proj", block.block_n, block.block_k); + m.down_proj_fp8_block = dense_loaders::LoadFp8BlockRaw( + get, mlp + "down_proj", block.block_n, block.block_k); } else { m.gate_up_proj = dense_loaders::LoadMergedBf16RawNK( get, {mlp + "gate_proj.weight", mlp + "up_proj.weight"}); @@ -560,9 +672,11 @@ OwnedTensor LoadMergedBf16RawNK(const TensorResolver& get, GdnLayerWeights LoadQwen3_5DenseGdn(const TensorResolver& get, const std::string& layer_base) { - // Public focused-loader seam historically describes the 27B checkpoint. + // Public focused-loader seam historically describes the 27B checkpoint, which + // is NVFP4 and declares no `weight_block_size`, so the default-constructed + // block config here is the truthful one and the routing is unchanged. const TensorExists has = [](const std::string&) { return true; }; - return LoadGdnDense(get, has, layer_base); + return LoadGdnDense(get, has, layer_base, Fp8BlockQuantConfig{}); } bool IsQwen27QuantizedLinear(const std::string& name) { @@ -625,7 +739,7 @@ OwnedTensor MaterializeCtNvfp4Bf16Transposed(const TensorResolver& get, Qwen3_5DenseLayerWeights LoadQwen3_5DenseLayer( const TensorResolver& get, const TensorExists& has, const std::string& layer_type, int64_t layer_idx, - const std::string& backbone_prefix) { + const std::string& backbone_prefix, const Fp8BlockQuantConfig& block) { const std::string base = backbone_prefix + "layers." + std::to_string(layer_idx) + "."; Qwen3_5DenseLayerWeights layer; @@ -635,17 +749,25 @@ Qwen3_5DenseLayerWeights LoadQwen3_5DenseLayer( LoadModelBf16Direct(get, base + "post_attention_layernorm.weight"); if (layer_type == "linear_attention") { layer.is_linear_attention = true; - layer.gdn = LoadGdnDense(get, has, base); + layer.gdn = LoadGdnDense(get, has, base, block); } else if (layer_type == "full_attention") { layer.is_linear_attention = false; - layer.attn = LoadAttnDense(get, has, base); + layer.attn = LoadAttnDense(get, has, base, block); } else { VT_CHECK(false, "qwen3_5 dense: unknown layer_type " + layer_type); } - layer.mlp = LoadDenseMlp(get, has, base); + layer.mlp = LoadDenseMlp(get, has, base, block); return layer; } +Qwen3_5DenseLayerWeights LoadQwen3_5DenseLayer( + const TensorResolver& get, const TensorExists& has, + const std::string& layer_type, int64_t layer_idx, + const std::string& backbone_prefix) { + return LoadQwen3_5DenseLayer(get, has, layer_type, layer_idx, backbone_prefix, + Fp8BlockQuantConfig{}); +} + Qwen3_5DenseLayerWeights LoadQwen3_5DenseLayer( const TensorResolver& get, const std::string& layer_type, int64_t layer_idx, const std::string& backbone_prefix) { @@ -686,6 +808,14 @@ Qwen3_5DenseWeights LoadQwen3_5Dense(const std::vector& shards, config.num_hidden_layers, "qwen3_5 dense: layer_types size must equal num_hidden_layers"); + // MODEL-FP8-BLOCK-WEIGHT (#1189 M3): ONE read of the quantization config for + // the whole checkpoint, validated here rather than per projection. It carries + // the block geometry the rung needs and the `modules_to_not_convert` list the + // rung cross-checks against, which a per-tensor dtype probe cannot supply. + // Default-constructed (`block_quant == false`) on every non-block checkpoint, + // which leaves the routing below byte-identical to before this row. + const Fp8BlockQuantConfig block = ReadFp8BlockQuantConfig(config); + Qwen3_5DenseWeights w; w.embed_tokens = LoadBf16Direct(get, backbone + "embed_tokens.weight"); w.final_norm = LoadModelBf16Direct(get, backbone + "norm.weight"); @@ -701,7 +831,8 @@ Qwen3_5DenseWeights LoadQwen3_5Dense(const std::vector& shards, bool direct_device = DirectDeviceLoadEligible(load_queue); for (int64_t l = 0; l < config.num_hidden_layers; ++l) { w.layers.push_back(LoadQwen3_5DenseLayer( - get, has, config.layer_types[static_cast(l)], l, backbone)); + get, has, config.layer_types[static_cast(l)], l, backbone, + block)); if (direct_device) { direct_device = IsPlainBf16Qwen3_5Dense(w); if (direct_device) StageAndReleaseLoadedDense(w, *load_queue); @@ -710,6 +841,41 @@ Qwen3_5DenseWeights LoadQwen3_5Dense(const std::vector& shards, return w; } +// MODEL-FP8-BLOCK-WEIGHT (#1189 M3), the M3/M4 seam. The loader now BUILDS +// `Fp8BlockWeight`s and no forward path reads one: `layers::MakeLinearMethod` +// has no block arm and the dense `project` lambda +// (`src/vllm/model_executor/models/qwen3_5.cpp`) knows only fp4, per-tensor fp8 +// and bf16, so an unwired block-wise checkpoint would fall through to an EMPTY +// bf16 tensor and produce fluent wrong output. This runs from +// `PrepareQwen3_5Dense`, i.e. `ModelRegistry::Prepare`, which every runner calls +// unconditionally before the first forward and before any graph capture +// (`src/vllm/v1/worker/gpu/runner.cpp:414,455`). The checkpoint therefore loads +// and DECLINES to run; it never runs wrong. Deleted by #1189 milestone M4, which +// makes the weight readable. +void RefuseUnconsumedQwen3_5DenseFp8Block(const Qwen3_5DenseWeights& weights) { + for (size_t l = 0; l < weights.layers.size(); ++l) { + const Qwen3_5DenseLayerWeights& layer = weights.layers[l]; + const std::string base = "model.layers." + std::to_string(l) + "."; + const auto check = [&base](const Fp8BlockWeight& w, + const std::string& suffix) { + if (!w.Empty()) RefuseUnconsumedFp8BlockWeight(base + suffix); + }; + if (layer.is_linear_attention) { + check(layer.gdn.in_proj_qkv_fp8_block, "linear_attn.in_proj_qkv"); + check(layer.gdn.in_proj_z_fp8_block, "linear_attn.in_proj_z"); + check(layer.gdn.out_proj_fp8_block, "linear_attn.out_proj"); + } else { + check(layer.attn.q_proj_fp8_block, "self_attn.q_proj"); + check(layer.attn.k_proj_fp8_block, "self_attn.k_proj"); + check(layer.attn.v_proj_fp8_block, "self_attn.v_proj"); + check(layer.attn.o_proj_fp8_block, "self_attn.o_proj"); + } + check(layer.mlp.gate_proj_fp8_block, "mlp.gate_proj"); + check(layer.mlp.up_proj_fp8_block, "mlp.up_proj"); + check(layer.mlp.down_proj_fp8_block, "mlp.down_proj"); + } +} + bool IsPlainBf16Qwen3_5Dense(const Qwen3_5DenseWeights& weights) { // A PACKED head (PERF-27B-LMHEAD-FP4) is not plain bf16: this staging path // only knows how to stage OwnedTensors. @@ -719,14 +885,29 @@ bool IsPlainBf16Qwen3_5Dense(const Qwen3_5DenseWeights& weights) { !layer.mlp.down_proj_fp4.Empty()) { return false; } + // MODEL-FP8-BLOCK-WEIGHT (#1189 M3): a block-wise FP8 projection is not + // plain bf16 either, and the direct-device staging path only knows how to + // stage OwnedTensors. + if (!layer.mlp.gate_proj_fp8_block.Empty() || + !layer.mlp.up_proj_fp8_block.Empty() || + !layer.mlp.down_proj_fp8_block.Empty()) { + return false; + } if (layer.is_linear_attention) { if (!layer.gdn.out_proj_fp4.Empty() || !layer.gdn.in_proj_qkv_fp8.Empty() || !layer.gdn.in_proj_z_fp8.Empty() || - !layer.gdn.out_proj_fp8.Empty()) { + !layer.gdn.out_proj_fp8.Empty() || + !layer.gdn.in_proj_qkv_fp8_block.Empty() || + !layer.gdn.in_proj_z_fp8_block.Empty() || + !layer.gdn.out_proj_fp8_block.Empty()) { return false; } - } else if (!layer.attn.q_proj_fp4.Empty() || + } else if (!layer.attn.q_proj_fp8_block.Empty() || + !layer.attn.k_proj_fp8_block.Empty() || + !layer.attn.v_proj_fp8_block.Empty() || + !layer.attn.o_proj_fp8_block.Empty() || + !layer.attn.q_proj_fp4.Empty() || !layer.attn.k_proj_fp4.Empty() || !layer.attn.v_proj_fp4.Empty() || !layer.attn.o_proj_fp4.Empty() || diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index ffddf5fc5..8799a7544 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -400,6 +400,12 @@ vllm_cpp_add_test(test_gguf_device_fit_reach # reachability proof and not a class test. vllm_cpp_add_test(test_fp8_block_quant vllm/model_executor/layers/test_fp8_block_quant.cpp) +# MODEL-FP8-BLOCK-WEIGHT #1189 M3: `Fp8BlockWeight`, the `weight_scale_inv` +# loader rung, and the config reader. Every case enters through +# `ModelRegistry::Load` or `LoadQwen3_5Dense` -- the loader that call reaches -- +# and never through `LoadFp8BlockRaw`, so deleting the rung reds this target. +vllm_cpp_add_test(test_fp8_block_weight_load + vllm/model_executor/models/test_fp8_block_weight_load.cpp) vllm_cpp_add_test(test_linear_method vllm/model_executor/layers/test_linear_method.cpp) target_include_directories(test_linear_method PRIVATE ${CMAKE_SOURCE_DIR}/src) vllm_cpp_add_test(test_qwen3_break_point vllm/models/test_qwen3_break_point.cpp) diff --git a/tests/vllm/model_executor/layers/test_fp8_block_quant.cpp b/tests/vllm/model_executor/layers/test_fp8_block_quant.cpp index 7fec946a8..4b5958ba9 100644 --- a/tests/vllm/model_executor/layers/test_fp8_block_quant.cpp +++ b/tests/vllm/model_executor/layers/test_fp8_block_quant.cpp @@ -1,30 +1,24 @@ -// Block-wise (fine-grained) FP8 is refused BY NAME at load — issue #1166, spec -// `.agents/specs/fp8-blockwise-refusal.md`. +// Block-wise (fine-grained) FP8: the CONFIG gate at `ModelRegistry::Load`. // -// `Qwen/Qwen3.8-27B-FP8` declares `quantization_config.weight_block_size` -// `[128, 128]` and stores one scale per 128x128 block under `weight_scale_inv`. -// This tree implements PER-TENSOR fp8 only. Before this gate the load still -// stopped, so the defect was never wrong numerics — it stopped on the wrong -// sentence. `LoadFp8Raw` asks for `.weight_scale` -// (`src/vllm/model_executor/models/qwen3_5_weights.cpp:458`), the checkpoint -// spells that tensor `weight_scale_inv`, and the resolver raised -// `qwen3_5 dense: tensor not found: ...q_proj.weight_scale`. Nothing was missing -// from the checkpoint. The reader was sent after a tensor upstream never writes -// in this mode instead of being told the fine-grained arm is absent. +// This file used to assert that the whole scheme was refused (issue #1166, +// `469f38395`, spec `.agents/specs/fp8-blockwise-refusal.md`). MODEL-FP8-BLOCK- +// WEIGHT (#1189 milestone M3, spec `.agents/specs/model-fp8-block-weight.md`) +// narrowed that: `Qwen/Qwen3.8-27B-FP8`'s `[128, 128]` `dynamic` config now +// PASSES this gate and its projections load through the `weight_scale_inv` rung +// in `qwen3_5_dense_weights.cpp`. What is still refused here is what no build in +// this tree can execute, and each refusal names the part. // -// EVERY case here enters through `ModelRegistry::Load`, the production loader -// `src/vllm/entrypoints/model_loader.cpp:1613` calls, and NOT through the +// EVERY case enters through `ModelRegistry::Load`, the production loader +// `src/vllm/entrypoints/model_loader.cpp:1706` calls, and NOT through the // predicate. That is deliberate and it is the reachability proof AGENTS.md // `## Nothing lands dead` asks for: a unit test that called the predicate // directly would prove the function works and never that a load reaches it. // Deleting the call site in `ModelRegistry::Load` must red this file. // -// No checkpoint, no GPU, and no model directory: the refusal fires before -// `factory.load_weights`, so an EMPTY shard vector is all a load needs to reach -// it. That is why the guard sits in `ModelRegistry::Load` rather than at the -// other pre-load refusal site, `RefuseUnsupportedWeightOffload` -// (`src/vllm/entrypoints/model_loader.cpp:1536`), which needs a directory on -// disk. +// The LOADING half — the rung, the weight, the scale dtype, the config/tensor +// cross-check — is gated by `test_fp8_block_weight_load`, which needs a +// synthetic checkpoint. This file needs none: the config gate runs before +// `factory.load_weights`, so an EMPTY shard vector reaches it. #include #include @@ -63,7 +57,7 @@ vllm::HfConfig ConfigWithQuant(const nlohmann::json& quant, bool nested) { // The message `ModelRegistry::Load` fails with, or "" when it does not throw. // The load is EXPECTED to throw in every case here: a config carrying no // weights cannot produce a model. What each case asserts is WHICH sentence -// comes back, which is the whole subject of issue #1166. +// comes back, which is the whole subject of issues #1166 and #1189. std::string LoadFailureMessage(const vllm::HfConfig& config) { const std::vector shards; const vllm::ModelSource source = vllm::ModelSource::FromSafetensors(shards); @@ -80,6 +74,12 @@ bool Names(const std::string& haystack, const std::string& needle) { return haystack.find(needle) != std::string::npos; } +// The sentence a config that PASSES the quantization gate fails with instead: +// the load reached `factory.load_weights` and found no tensors. Asserting this +// is what proves the gate did not fire, and it is the same marker the negative +// controls below use. +constexpr const char* kReachedLoader = "backbone tensors found"; + // The block-wise config the real checkpoint ships, measured from // `Qwen/Qwen3.8-27B-FP8` at revision `017b9c7af6b5689d5dd426a76e0bc077eb5ca20a` // on 2026-08-17: `quant_method` fp8, `weight_block_size` [128, 128], @@ -95,47 +95,93 @@ nlohmann::json BlockWiseQuantConfig() { } // namespace -TEST_CASE("fp8 block quant: a block-wise checkpoint is refused by name") { +TEST_CASE("fp8 block quant: the SUPPORTED block-wise config reaches the loader") { + // The whole point of #1189 M3. Before it this config was refused by name; + // now it passes the quantization gate and the load fails only because this + // case ships no weights. Without this assertion the file passes for a gate + // that refuses every block-wise checkpoint, which is the state M3 replaces. const std::string message = LoadFailureMessage(ConfigWithQuant(BlockWiseQuantConfig(), false)); REQUIRE_FALSE(message.empty()); + MESSAGE("supported block-wise fp8 load failed with: " << message); + CHECK(Names(message, kReachedLoader)); + CHECK_FALSE(Names(message, "not implemented")); + CHECK_FALSE(Names(message, "does not implement")); +} + +TEST_CASE("fp8 block quant: a non-dynamic activation scheme is refused by name") { + // Upstream refuses the same combination + // (`vllm/model_executor/layers/quantization/fp8.py:127-131` @ `555967922`): + // a static per-tensor input scale cannot express a per-token per-group + // dynamic quantization. + nlohmann::json quant = BlockWiseQuantConfig(); + quant["activation_scheme"] = "static"; + const std::string message = LoadFailureMessage(ConfigWithQuant(quant, false)); + REQUIRE_FALSE(message.empty()); + CHECK(Names(message, "activation_scheme")); + CHECK(Names(message, "static")); // the VALUE this checkpoint declares + CHECK(Names(message, "dynamic")); // the value that works + CHECK(Names(message, "1189")); // the owing pointer + // THE REFUSAL MUST PREEMPT THE WEIGHT LOADER. Asserting the absence of the + // real checkpoint's `tensor not found` would be a MUTE SWITCH: this config + // carries no shards, so an unguarded load does not reach that sentence + // either. It reaches a DIFFERENT one, `kReachedLoader`, which means + // `factory.load_weights` RAN. Requiring that sentence to be absent proves the + // guard fires before the loader, and it is exactly what reds when the call + // site in `ModelRegistry::Load` is deleted. + CHECK_FALSE(Names(message, kReachedLoader)); +} + +TEST_CASE("fp8 block quant: a block shape other than 128x128 is refused by name") { + // OUR limit rather than upstream's, and the message says so: #1189's CUTLASS + // kernel (M5) is 128x128 and nothing here has run any other shape end to end, + // so a [64, 128] checkpoint would fill a weight nothing can consume. + nlohmann::json quant = BlockWiseQuantConfig(); + quant["weight_block_size"] = nlohmann::json::array({64, 128}); + const std::string message = LoadFailureMessage(ConfigWithQuant(quant, false)); + REQUIRE_FALSE(message.empty()); + CHECK(Names(message, "weight_block_size")); + CHECK(Names(message, "[64, 128]")); // the VALUE quoted back + CHECK(Names(message, "1189")); + CHECK_FALSE(Names(message, kReachedLoader)); +} - // The KEY, so a reader can grep their own config.json for it. +TEST_CASE("fp8 block quant: a weight_block_size that is not 2-D is refused by name") { + // Upstream refuses it too (`fp8.py:121-126`). + nlohmann::json quant = BlockWiseQuantConfig(); + quant["weight_block_size"] = nlohmann::json::array({128}); + const std::string message = LoadFailureMessage(ConfigWithQuant(quant, false)); + REQUIRE_FALSE(message.empty()); CHECK(Names(message, "weight_block_size")); - // The VALUE quoted back, so the message is about THIS checkpoint. - CHECK(Names(message, "128")); - // WHAT is missing, named. "unsupported" alone makes the next person - // re-derive it. - CHECK(Names(message, "block-wise")); - // WHAT WOULD work, so the refusal points somewhere. - CHECK(Names(message, "per-tensor")); - // The owing pointer. Deleting it reds this line. - CHECK(Names(message, "1166")); - - // THE REFUSAL MUST PREEMPT THE WEIGHT LOADER, and this is the assertion that - // can actually fail. Asserting the absence of the real checkpoint's - // `tensor not found: ...weight_scale` would be a MUTE SWITCH here: this - // config carries no shards, so without a guard the load does not reach that - // sentence either. It reaches a different one. Measured on the RED run of - // this same file, an unguarded load of this config fails with - // `no Qwen3.5 backbone tensors found` from `qwen3_5_weights.cpp:1190`, which - // means `factory.load_weights` RAN. So requiring that sentence to be absent - // proves the guard fires before the loader rather than after it, and it is - // exactly what reds when the call site is deleted. - CHECK_FALSE(Names(message, "backbone tensors found")); + CHECK(Names(message, "2 dimensions")); + CHECK_FALSE(Names(message, kReachedLoader)); +} + +TEST_CASE("fp8 block quant: a non-fp8 quant_method is refused by name") { + // Mirrors `is_checkpoint_fp8_serialized = "fp8" in quant_method` + // (`fp8.py:158-159`) and the `__init__` refusal it feeds (`fp8.py:117-120`). + // Block-wise weight quantization is defined only for fp8 here. + nlohmann::json quant = BlockWiseQuantConfig(); + quant["quant_method"] = "awq"; + const std::string message = LoadFailureMessage(ConfigWithQuant(quant, false)); + REQUIRE_FALSE(message.empty()); + CHECK(Names(message, "quant_method")); + CHECK(Names(message, "awq")); + CHECK_FALSE(Names(message, kReachedLoader)); } -TEST_CASE("fp8 block quant: the nested text_config spelling is refused too") { +TEST_CASE("fp8 block quant: the nested text_config spelling is read too") { // A multimodal wrapper can nest `quantization_config` under `text_config`. // `Qwen/Qwen3.8-27B-FP8` uses the TOP-LEVEL spelling (measured: `text_config` - // carries no `quantization_config`), so this case covers the shape the next - // checkpoint can arrive in and would otherwise load straight past the guard. - const std::string message = - LoadFailureMessage(ConfigWithQuant(BlockWiseQuantConfig(), true)); + // carries no `quantization_config`), so this covers the shape the next + // checkpoint can arrive in and would otherwise load straight past the gate. + nlohmann::json quant = BlockWiseQuantConfig(); + quant["activation_scheme"] = "static"; + const std::string message = LoadFailureMessage(ConfigWithQuant(quant, true)); REQUIRE_FALSE(message.empty()); - CHECK(Names(message, "weight_block_size")); - CHECK(Names(message, "block-wise")); - CHECK(Names(message, "1166")); + CHECK(Names(message, "activation_scheme")); + CHECK(Names(message, "1189")); + CHECK_FALSE(Names(message, kReachedLoader)); } TEST_CASE("fp8 block quant: a per-tensor fp8 config is not refused as block-wise") { @@ -151,27 +197,29 @@ TEST_CASE("fp8 block quant: a per-tensor fp8 config is not refused as block-wise // it is what the negative controls are asserting the ABSENCE of markers in, // and a control that never says what it saw cannot be falsified. MESSAGE("per-tensor fp8 load failed with: " << message); + CHECK(Names(message, kReachedLoader)); CHECK_FALSE(Names(message, "weight_block_size")); CHECK_FALSE(Names(message, "block-wise")); - CHECK_FALSE(Names(message, "1166")); } TEST_CASE("fp8 block quant: a null or empty weight_block_size is not block-wise") { // Upstream treats the key as absent when it is null // (`Fp8Config.from_config`, `vllm/model_executor/layers/quantization/fp8.py:161` // reads it with a default of None), so a null must not refuse a checkpoint - // that is really per-tensor. + // that is really per-tensor. A `[64, 128]` value is used for the OTHER keys + // here so that a reader that mistook null for a real list would be refused + // and this case would notice. nlohmann::json null_quant = BlockWiseQuantConfig(); null_quant["weight_block_size"] = nullptr; const std::string null_message = LoadFailureMessage(ConfigWithQuant(null_quant, false)); REQUIRE_FALSE(null_message.empty()); - CHECK_FALSE(Names(null_message, "block-wise")); + CHECK(Names(null_message, kReachedLoader)); nlohmann::json empty_quant = BlockWiseQuantConfig(); empty_quant["weight_block_size"] = nlohmann::json::array(); const std::string empty_message = LoadFailureMessage(ConfigWithQuant(empty_quant, false)); REQUIRE_FALSE(empty_message.empty()); - CHECK_FALSE(Names(empty_message, "block-wise")); + CHECK(Names(empty_message, kReachedLoader)); } diff --git a/tests/vllm/model_executor/models/test_fp8_block_weight_load.cpp b/tests/vllm/model_executor/models/test_fp8_block_weight_load.cpp new file mode 100644 index 000000000..531dfe8a8 --- /dev/null +++ b/tests/vllm/model_executor/models/test_fp8_block_weight_load.cpp @@ -0,0 +1,692 @@ +// MODEL-FP8-BLOCK-WEIGHT — #1189 milestone M3, spec +// `.agents/specs/model-fp8-block-weight.md`. +// +// Block-wise (fine-grained 128x128) FP8 LOADS: `Fp8BlockWeight`, the +// `weight_scale_inv` loader rung, and the quantization-config reader that +// cross-checks the config against the tensors. +// +// EVERY case here enters through a PRODUCTION entry point and never through +// `LoadFp8BlockRaw`. `ModelRegistry::Load` is the registry seam +// `src/vllm/entrypoints/model_loader.cpp:1706` calls; `LoadQwen3_5Dense` is the +// dense loader it reaches at `src/vllm/model_executor/models/qwen3_5_dense.cpp:101` +// and is the only one of the two that hands back an inspectable +// `Qwen3_5DenseWeights`. Deleting the block rung in `load_projection` +// (`qwen3_5_dense_weights.cpp`) must red this file: without it a block-wise +// projection falls into the per-tensor arm and the load dies on +// `tensor not found: ...q_proj.weight_scale`, which is exactly issue #1166. +// +// No checkpoint download, no GPU, no snapshot. The fixture is a complete but +// tiny `Qwen3_5ForConditionalGeneration` dense checkpoint written to a temp +// directory, following the safetensors byte layout pinned by +// `tests/vllm/test_safetensors.cpp:57-89`. +#include + +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "vllm/model_executor/model_loader/safetensors_reader.h" +#include "vllm/model_executor/models/model_registry.h" +#include "vllm/model_executor/models/qwen3_5_dense.h" +#include "vllm/model_executor/models/qwen3_5_weights.h" +#include "vllm/transformers_utils/hf_config.h" +#include "vt/backend.h" +#include "vt/dtype.h" + +namespace { + +using vllm::Fp8BlockWeight; +using vllm::HfConfig; +using vllm::ModelSource; +using vllm::OwnedTensor; +using vllm::Qwen3_5DenseWeights; +using vllm::SafetensorsFile; + +constexpr const char* kArch = "Qwen3_5ForConditionalGeneration"; + +// --------------------------------------------------------------------------- +// The synthetic checkpoint +// --------------------------------------------------------------------------- + +struct FixtureTensor { + std::string name; + std::string dtype; + std::vector shape; + std::vector bytes; +}; + +int64_t Numel(const std::vector& shape) { + int64_t n = 1; + for (const int64_t d : shape) n *= d; + return n; +} + +std::string U64Le(uint64_t v) { + std::string s(8, '\0'); + for (int i = 0; i < 8; ++i) s[i] = static_cast((v >> (8 * i)) & 0xff); + return s; +} + +// u64-LE header length + JSON header + payload, the safetensors container. +std::string BuildSafetensors(const std::vector& tensors) { + nlohmann::json header = nlohmann::json::object(); + std::string payload; + for (const FixtureTensor& t : tensors) { + const size_t begin = payload.size(); + payload.append(reinterpret_cast(t.bytes.data()), + t.bytes.size()); + nlohmann::json entry = nlohmann::json::object(); + entry["dtype"] = t.dtype; + entry["shape"] = t.shape; + entry["data_offsets"] = nlohmann::json::array({begin, payload.size()}); + header[t.name] = std::move(entry); + } + const std::string head = header.dump(); + return U64Le(head.size()) + head + payload; +} + +class TempCheckpoint { + public: + explicit TempCheckpoint(const std::vector& tensors) { + static std::atomic counter{0}; + static const uint64_t nonce = [] { + std::random_device rd; + return (static_cast(rd()) << 32) ^ rd(); + }(); + dir_ = std::filesystem::temp_directory_path() / + ("vllm_fp8_block_" + std::to_string(nonce) + "_" + + std::to_string(counter.fetch_add(1))); + std::filesystem::create_directories(dir_); + path_ = dir_ / "model.safetensors"; + const std::string bytes = BuildSafetensors(tensors); + std::ofstream out(path_, std::ios::binary); + out.write(bytes.data(), static_cast(bytes.size())); + if (!out) throw std::runtime_error("failed to write fixture checkpoint"); + } + ~TempCheckpoint() { + std::error_code ignored; + std::filesystem::remove_all(dir_, ignored); + } + TempCheckpoint(const TempCheckpoint&) = delete; + TempCheckpoint& operator=(const TempCheckpoint&) = delete; + std::string path() const { return path_.string(); } + + private: + std::filesystem::path dir_; + std::filesystem::path path_; +}; + +std::vector Bf16Filled(const std::vector& shape, + uint16_t pattern) { + std::vector bytes(static_cast(Numel(shape)) * 2); + for (size_t i = 0; i + 1 < bytes.size(); i += 2) { + bytes[i] = static_cast(pattern & 0xff); + bytes[i + 1] = static_cast(pattern >> 8); + } + return bytes; +} + +// Raw fp8-e4m3fn bytes, a walking pattern so a truncated or misaligned copy is +// visible rather than uniform. +std::vector Fp8Walk(const std::vector& shape) { + const size_t n = static_cast(Numel(shape)); + std::vector bytes(n); + for (size_t i = 0; i < n; ++i) bytes[i] = static_cast((i * 7) & 0x7f); + return bytes; +} + +std::vector Bytes16(const std::vector& values) { + std::vector bytes(values.size() * 2); + for (size_t i = 0; i < values.size(); ++i) { + bytes[2 * i] = static_cast(values[i] & 0xff); + bytes[2 * i + 1] = static_cast(values[i] >> 8); + } + return bytes; +} + +std::vector Bytes32(const std::vector& values) { + std::vector bytes(values.size() * 4); + for (size_t i = 0; i < values.size(); ++i) + std::memcpy(bytes.data() + 4 * i, &values[i], 4); + return bytes; +} + +int64_t CDiv(int64_t a, int64_t b) { return (a + b - 1) / b; } + +// The four bf16 scale bit patterns every block fixture uses. Each is an EXACT +// f32 value with its low 16 bits zero, so the expected f32 is a literal and a +// four-byte REINTERPRETATION of two adjacent bf16 halves lands nowhere near it. +// 0x3F80 -> 1.0 0x3E00 -> 0.125 +// 0xBF00 -> -0.5 0x3DCD -> 0.10009765625 (bf16(0.1), deliberately not 0.1) +const std::vector& BlockScaleBf16Pattern() { + static const std::vector p = {0x3F80, 0x3E00, 0xBF00, 0x3DCD}; + return p; +} +const std::vector& BlockScaleExpected() { + static const std::vector v = {1.0F, 0.125F, -0.5F, 0.10009765625F}; + return v; +} + +std::vector BlockScaleBytesBf16(int64_t rows, int64_t cols) { + const std::vector& pat = BlockScaleBf16Pattern(); + std::vector out(static_cast(rows * cols)); + for (size_t i = 0; i < out.size(); ++i) out[i] = pat[i % pat.size()]; + return Bytes16(out); +} + +std::vector BlockScaleBytesF32(int64_t rows, int64_t cols) { + const std::vector& pat = BlockScaleExpected(); + std::vector out(static_cast(rows * cols)); + for (size_t i = 0; i < out.size(); ++i) out[i] = pat[i % pat.size()]; + return Bytes32(out); +} + +float ExpectedScaleAt(int64_t index) { + return BlockScaleExpected()[static_cast(index) % + BlockScaleExpected().size()]; +} + +// How a projection's tensors are written into the fixture. +enum class ProjArm { + kBlockBf16Scale, // F8_E4M3 weight + BF16 weight_scale_inv + kBlockF32Scale, // F8_E4M3 weight + F32 weight_scale_inv + kBlockF16Scale, // F8_E4M3 weight + F16 weight_scale_inv (must refuse) + kBlockFloorScale, // scale sized by FLOOR instead of cdiv (must refuse) + kBlockNoScale, // F8_E4M3 weight with NO weight_scale_inv + kBlockPlusInput, // block-wise plus a stray input_scale (must refuse) + kPerTensor, // F8_E4M3 weight + F32 weight_scale + F32 input_scale + kBf16, // plain BF16 weight +}; + +void AppendProjection(std::vector& out, const std::string& proj, + int64_t n, int64_t k, ProjArm arm, int64_t block_n = 128, + int64_t block_k = 128) { + const std::vector wshape = {n, k}; + if (arm == ProjArm::kBf16) { + out.push_back({proj + ".weight", "BF16", wshape, Bf16Filled(wshape, 0x3F80)}); + return; + } + out.push_back({proj + ".weight", "F8_E4M3", wshape, Fp8Walk(wshape)}); + if (arm == ProjArm::kPerTensor) { + out.push_back({proj + ".weight_scale", "F32", {}, Bytes32({0.25F})}); + out.push_back({proj + ".input_scale", "F32", {}, Bytes32({0.5F})}); + return; + } + if (arm == ProjArm::kBlockNoScale) return; + int64_t rows = CDiv(n, block_n); + int64_t cols = CDiv(k, block_k); + if (arm == ProjArm::kBlockFloorScale) { + rows = n / block_n; + cols = k / block_k; + } + const std::vector sshape = {rows, cols}; + switch (arm) { + case ProjArm::kBlockF32Scale: + out.push_back({proj + ".weight_scale_inv", "F32", sshape, + BlockScaleBytesF32(rows, cols)}); + break; + case ProjArm::kBlockF16Scale: + out.push_back({proj + ".weight_scale_inv", "F16", sshape, + BlockScaleBytesBf16(rows, cols)}); + break; + default: + out.push_back({proj + ".weight_scale_inv", "BF16", sshape, + BlockScaleBytesBf16(rows, cols)}); + break; + } + if (arm == ProjArm::kBlockPlusInput) + out.push_back({proj + ".input_scale", "F32", {}, Bytes32({0.5F})}); +} + +// Geometry of the fixture model. Small, and every projection's N and K are +// independent because the loader loads tensors rather than validating a model. +struct FixtureShape { + int64_t hidden = 256; + int64_t vocab = 32; + int64_t q_n = 256; + int64_t q_k = 256; + int64_t kv_n = 128; + int64_t inter = 512; +}; + +// One `full_attention` layer, tied lm_head. `arm` selects how the four +// self_attn projections and the three MLP projections are written. +std::vector DenseFixture(ProjArm arm, + const FixtureShape& s = {}) { + std::vector t; + t.push_back({"model.embed_tokens.weight", "BF16", {s.vocab, s.hidden}, + Bf16Filled({s.vocab, s.hidden}, 0x3F80)}); + t.push_back({"model.norm.weight", "BF16", {s.hidden}, + Bf16Filled({s.hidden}, 0x3F80)}); + const std::string base = "model.layers.0."; + t.push_back({base + "input_layernorm.weight", "BF16", {s.hidden}, + Bf16Filled({s.hidden}, 0x3F80)}); + t.push_back({base + "post_attention_layernorm.weight", "BF16", {s.hidden}, + Bf16Filled({s.hidden}, 0x3F80)}); + AppendProjection(t, base + "self_attn.q_proj", s.q_n, s.q_k, arm); + AppendProjection(t, base + "self_attn.k_proj", s.kv_n, s.hidden, arm); + AppendProjection(t, base + "self_attn.v_proj", s.kv_n, s.hidden, arm); + AppendProjection(t, base + "self_attn.o_proj", s.hidden, s.q_n, arm); + t.push_back({base + "self_attn.q_norm.weight", "BF16", {64}, + Bf16Filled({64}, 0x3F80)}); + t.push_back({base + "self_attn.k_norm.weight", "BF16", {64}, + Bf16Filled({64}, 0x3F80)}); + AppendProjection(t, base + "mlp.gate_proj", s.inter, s.hidden, arm); + AppendProjection(t, base + "mlp.up_proj", s.inter, s.hidden, arm); + AppendProjection(t, base + "mlp.down_proj", s.hidden, s.inter, arm); + return t; +} + +// --------------------------------------------------------------------------- +// The config +// --------------------------------------------------------------------------- + +nlohmann::json BlockQuantJson(const std::vector& block = {128, 128}, + const std::string& scheme = "dynamic", + const std::string& method = "fp8") { + nlohmann::json q = nlohmann::json::object(); + q["quant_method"] = method; + q["fmt"] = "e4m3"; + q["activation_scheme"] = scheme; + q["weight_block_size"] = block; + return q; +} + +// `quant` is null for a checkpoint that declares no quantization config. +HfConfig DenseConfig(const nlohmann::json& quant) { + HfConfig config; + config.architectures = {kArch}; + config.model_type = "qwen3_5"; + config.num_hidden_layers = 1; + config.layer_types = {"full_attention"}; + config.vocab_size = 32; + nlohmann::json doc = nlohmann::json::object(); + doc["architectures"] = nlohmann::json::array({kArch}); + if (!quant.is_null()) doc["quantization_config"] = quant; + config.raw = std::move(doc); + return config; +} + +// --------------------------------------------------------------------------- +// Driving the production paths +// --------------------------------------------------------------------------- + +// `LoadQwen3_5Dense` is the loader `ModelRegistry::Load` reaches through +// `LoadQwen3_5DenseModel` (`qwen3_5_dense.cpp:101`). It is used where a case +// has to INSPECT what was loaded, which the type-erased `LoadedModel` cannot +// hand back. +Qwen3_5DenseWeights LoadDense(const TempCheckpoint& ckpt, + const HfConfig& config) { + std::vector shards; + shards.push_back(SafetensorsFile::Open(ckpt.path())); + return vllm::LoadQwen3_5Dense(shards, config, /*load_queue=*/nullptr); +} + +std::string LoadDenseFailure(const TempCheckpoint& ckpt, + const HfConfig& config) { + try { + const Qwen3_5DenseWeights weights = LoadDense(ckpt, config); + (void)weights; + return ""; + } catch (const std::exception& e) { + return e.what(); + } +} + +// The registry seam, the entry point a user actually arrives through. +std::string RegistryLoadFailure(const TempCheckpoint& ckpt, + const HfConfig& config) { + std::vector shards; + shards.push_back(SafetensorsFile::Open(ckpt.path())); + const ModelSource source = ModelSource::FromSafetensors(shards); + try { + std::unique_ptr model = + vllm::ModelRegistry::Load(config, source); + return ""; + } catch (const std::exception& e) { + return e.what(); + } +} + +bool Names(const std::string& haystack, const std::string& needle) { + return haystack.find(needle) != std::string::npos; +} + +float ScaleAt(const Fp8BlockWeight& w, int64_t r, int64_t c) { + const auto* p = reinterpret_cast(w.scale.bytes.data()); + return p[r * w.scale.shape[1] + c]; +} + +} // namespace + +// G1 ----------------------------------------------------------------------- +TEST_CASE("fp8 block weight: the loader rung is selected for a block-wise checkpoint") { + const TempCheckpoint ckpt(DenseFixture(ProjArm::kBlockBf16Scale)); + const HfConfig config = DenseConfig(BlockQuantJson()); + + // Through the REGISTRY first. Without the rung this throws + // `tensor not found: ...q_proj.weight_scale`, which is issue #1166 and is the + // RED this case was written against. + const std::string registry = RegistryLoadFailure(ckpt, config); + CHECK_MESSAGE(registry.empty(), "ModelRegistry::Load failed: " << registry); + + const Qwen3_5DenseWeights w = LoadDense(ckpt, config); + REQUIRE(w.layers.size() == 1); + const vllm::Qwen3_5DenseLayerWeights& layer = w.layers[0]; + REQUIRE_FALSE(layer.is_linear_attention); + + // The block slot is populated and the two arms it must not have taken are + // empty. Checking only the block slot would pass for a rung that ALSO ran the + // per-tensor arm. + CHECK_FALSE(layer.attn.q_proj_fp8_block.Empty()); + CHECK(layer.attn.q_proj_fp8.Empty()); + CHECK(layer.attn.q_proj.Empty()); + CHECK(layer.attn.q_proj_fp4.Empty()); + CHECK_FALSE(layer.attn.k_proj_fp8_block.Empty()); + CHECK_FALSE(layer.attn.v_proj_fp8_block.Empty()); + CHECK_FALSE(layer.attn.o_proj_fp8_block.Empty()); + // The MLP had NO fp8 rung before this row; without one a block-wise MLP goes + // to LoadMergedBf16RawNK and dies on "expected BF16". + CHECK_FALSE(layer.mlp.gate_proj_fp8_block.Empty()); + CHECK_FALSE(layer.mlp.up_proj_fp8_block.Empty()); + CHECK_FALSE(layer.mlp.down_proj_fp8_block.Empty()); + CHECK(layer.mlp.gate_up_proj.Empty()); + + // The geometry travels ON the weight, so a consumer cannot pair it with a + // block shape from somewhere else. + const Fp8BlockWeight& q = layer.attn.q_proj_fp8_block; + CHECK(q.n == 256); + CHECK(q.k == 256); + CHECK(q.block_n == 128); + CHECK(q.block_k == 128); + CHECK(q.packed.dtype == vt::DType::kI8); + REQUIRE(q.packed.rank == 2); + CHECK(q.packed.shape[0] == 256); + CHECK(q.packed.shape[1] == 256); + // The fp8 bytes are copied verbatim, not dequantized. + const std::vector expect = Fp8Walk({256, 256}); + REQUIRE(q.packed.bytes.size() == expect.size()); + CHECK(std::memcmp(q.packed.bytes.data(), expect.data(), expect.size()) == 0); +} + +// G2 ----------------------------------------------------------------------- +TEST_CASE("fp8 block weight: the BF16 scale is WIDENED to f32 rather than reinterpreted") { + // Upstream allocates the scale parameter f32 (`fp8_utils.py:1276`) and loads + // the checkpoint tensor with `self.data.copy_()` (`parameter.py:97`), which + // CONVERTS. `Qwen/Qwen3.8-27B-FP8` ships the tensor BF16. So the resident + // scale is f32 and the conversion is a value conversion. + // + // The expected values are literals with their low 16 bits zero. Reading four + // bytes as one f32 — the #1181 defect — splices two adjacent bf16 halves and + // cannot land on any of them. + SUBCASE("BF16 on disk") { + const TempCheckpoint ckpt(DenseFixture(ProjArm::kBlockBf16Scale)); + const Qwen3_5DenseWeights w = LoadDense(ckpt, DenseConfig(BlockQuantJson())); + const Fp8BlockWeight& q = w.layers[0].attn.q_proj_fp8_block; + CHECK(q.scale.dtype == vt::DType::kF32); + REQUIRE(q.scale.rank == 2); + CHECK(q.scale.shape[0] == 2); // cdiv(256, 128) + CHECK(q.scale.shape[1] == 2); // cdiv(256, 128) + REQUIRE(q.scale.bytes.size() == 4u * 4u); + CHECK(ScaleAt(q, 0, 0) == 1.0F); + CHECK(ScaleAt(q, 0, 1) == 0.125F); + CHECK(ScaleAt(q, 1, 0) == -0.5F); + // bf16(0.1) is 0.10009765625 exactly, NOT 0.1. A loader that rounded or + // rewrote the value would produce 0.1 and fail here. + CHECK(ScaleAt(q, 1, 1) == 0.10009765625F); + } + SUBCASE("F32 on disk") { + // The other dtype upstream's converting copy accepts without change. + const TempCheckpoint ckpt(DenseFixture(ProjArm::kBlockF32Scale)); + const Qwen3_5DenseWeights w = LoadDense(ckpt, DenseConfig(BlockQuantJson())); + const Fp8BlockWeight& q = w.layers[0].attn.q_proj_fp8_block; + CHECK(q.scale.dtype == vt::DType::kF32); + CHECK(ScaleAt(q, 0, 0) == 1.0F); + CHECK(ScaleAt(q, 1, 1) == 0.10009765625F); + } + SUBCASE("any other dtype is refused BY NAME rather than reinterpreted") { + const TempCheckpoint ckpt(DenseFixture(ProjArm::kBlockF16Scale)); + const std::string message = + LoadDenseFailure(ckpt, DenseConfig(BlockQuantJson())); + REQUIRE_FALSE(message.empty()); + CHECK(Names(message, "weight_scale_inv")); + CHECK(Names(message, "F16")); + CHECK(Names(message, "BF16")); + CHECK(Names(message, "F32")); + } +} + +// G3 ----------------------------------------------------------------------- +TEST_CASE("fp8 block weight: a ragged dimension tiles by cdiv on both axes") { + // `N=576` is `4*128 + 64` and `K=3884` is `30*128 + 44`, upstream's own + // non-round shapes (`tests/kernels/quantization/test_block_fp8.py:49-50`). + // The target checkpoint has no ragged edge; the SCHEME does, and M2 measured + // that a grid of round shapes stays green through two floor-vs-ceil defects. + FixtureShape s; + s.q_n = 576; // ragged N alone + s.q_k = 256; + s.kv_n = 128; + s.inter = 512; + const TempCheckpoint ragged_n(DenseFixture(ProjArm::kBlockBf16Scale, s)); + const Qwen3_5DenseWeights wn = + LoadDense(ragged_n, DenseConfig(BlockQuantJson())); + const Fp8BlockWeight& qn = wn.layers[0].attn.q_proj_fp8_block; + CHECK(qn.scale.shape[0] == 5); // cdiv(576, 128) == 5, floor would be 4 + CHECK(qn.scale.shape[1] == 2); + CHECK(qn.n == 576); + + FixtureShape both; + both.q_n = 576; + both.q_k = 3884; // ragged K as well + both.hidden = 256; + both.kv_n = 128; + both.inter = 512; + const TempCheckpoint ragged_both(DenseFixture(ProjArm::kBlockBf16Scale, both)); + const Qwen3_5DenseWeights wb = + LoadDense(ragged_both, DenseConfig(BlockQuantJson())); + const Fp8BlockWeight& qb = wb.layers[0].attn.q_proj_fp8_block; + CHECK(qb.scale.shape[0] == 5); + CHECK(qb.scale.shape[1] == 31); // cdiv(3884, 128) == 31, floor would be 30 + CHECK(qb.k == 3884); + // The last row and column of the scale still carry their fixture value, so a + // short final block was not dropped. + CHECK(ScaleAt(qb, 4, 30) == + ExpectedScaleAt(4 * 31 + 30)); + + SUBCASE("a FLOOR-sized scale for the same weight is refused by name") { + const TempCheckpoint floor(DenseFixture(ProjArm::kBlockFloorScale, both)); + const std::string message = + LoadDenseFailure(floor, DenseConfig(BlockQuantJson())); + REQUIRE_FALSE(message.empty()); + CHECK(Names(message, "weight_scale_inv")); + // BOTH shapes, so the reader does not have to derive the expected one. + CHECK(Names(message, "[4, 30]")); + CHECK(Names(message, "[5, 31]")); + } +} + +// G4 ----------------------------------------------------------------------- +TEST_CASE("fp8 block weight: a config and tensor disagreement is refused by name") { + // A dtype probe alone cannot see a DISAGREEMENT: it sees a tensor and picks + // an arm. This is where a silent-wrong-scale bug lives. + SUBCASE("weight_scale_inv present but the config declares no weight_block_size") { + const TempCheckpoint ckpt(DenseFixture(ProjArm::kBlockBf16Scale)); + nlohmann::json quant = BlockQuantJson(); + quant.erase("weight_block_size"); + const std::string message = LoadDenseFailure(ckpt, DenseConfig(quant)); + REQUIRE_FALSE(message.empty()); + CHECK(Names(message, "weight_scale_inv")); + CHECK(Names(message, "weight_block_size")); + CHECK(Names(message, "q_proj")); + } + SUBCASE("the config declares weight_block_size but the tensor has no scale") { + const TempCheckpoint ckpt(DenseFixture(ProjArm::kBlockNoScale)); + const std::string message = + LoadDenseFailure(ckpt, DenseConfig(BlockQuantJson())); + REQUIRE_FALSE(message.empty()); + CHECK(Names(message, "weight_scale_inv")); + CHECK(Names(message, "weight_block_size")); + CHECK(Names(message, "q_proj")); + } + SUBCASE("a module listed in modules_to_not_convert still ships a block scale") { + const TempCheckpoint ckpt(DenseFixture(ProjArm::kBlockBf16Scale)); + nlohmann::json quant = BlockQuantJson(); + quant["modules_to_not_convert"] = + nlohmann::json::array({"model.layers.0.self_attn.q_proj"}); + const std::string message = LoadDenseFailure(ckpt, DenseConfig(quant)); + REQUIRE_FALSE(message.empty()); + CHECK(Names(message, "modules_to_not_convert")); + CHECK(Names(message, "q_proj")); + CHECK(Names(message, "weight_scale_inv")); + } + SUBCASE("an input_scale beside a dynamic activation scheme") { + // The target checkpoint ships ZERO input_scale tensors, and upstream + // registers one only when `act_q_static` (`fp8.py:381-384`), which block + // quant asserts against at `fp8.py:367`. + const TempCheckpoint ckpt(DenseFixture(ProjArm::kBlockPlusInput)); + const std::string message = + LoadDenseFailure(ckpt, DenseConfig(BlockQuantJson())); + REQUIRE_FALSE(message.empty()); + CHECK(Names(message, "input_scale")); + CHECK(Names(message, "dynamic")); + CHECK(Names(message, "q_proj")); + } + SUBCASE("a module genuinely excluded and genuinely unquantized loads") { + // The other half of the exclusion rule. Without this the gate passes for a + // reader that refuses every listed module outright. + FixtureShape s; + std::vector t = DenseFixture(ProjArm::kBlockBf16Scale, s); + // Rewrite o_proj as plain BF16 and list it as not converted. + std::vector filtered; + for (FixtureTensor& e : t) { + if (e.name.find("o_proj") == std::string::npos) + filtered.push_back(std::move(e)); + } + AppendProjection(filtered, "model.layers.0.self_attn.o_proj", s.hidden, + s.q_n, ProjArm::kBf16); + const TempCheckpoint ckpt(filtered); + nlohmann::json quant = BlockQuantJson(); + quant["modules_to_not_convert"] = + nlohmann::json::array({"model.layers.0.self_attn.o_proj"}); + const Qwen3_5DenseWeights w = LoadDense(ckpt, DenseConfig(quant)); + CHECK(w.layers[0].attn.o_proj_fp8_block.Empty()); + CHECK_FALSE(w.layers[0].attn.o_proj.Empty()); + CHECK_FALSE(w.layers[0].attn.q_proj_fp8_block.Empty()); + } +} + +// G5 ----------------------------------------------------------------------- +TEST_CASE("fp8 block weight: an unsupported block config is refused by name at the registry") { + const TempCheckpoint ckpt(DenseFixture(ProjArm::kBlockBf16Scale)); + + SUBCASE("a static activation scheme") { + // Upstream refuses it too (`fp8.py:127-131`). + const std::string message = RegistryLoadFailure( + ckpt, DenseConfig(BlockQuantJson({128, 128}, "static"))); + REQUIRE_FALSE(message.empty()); + CHECK(Names(message, "activation_scheme")); + CHECK(Names(message, "static")); + CHECK(Names(message, "dynamic")); + CHECK(Names(message, "1189")); + } + SUBCASE("a weight_block_size that is not two dimensions") { + // Upstream refuses it too (`fp8.py:121-126`). + const std::string message = + RegistryLoadFailure(ckpt, DenseConfig(BlockQuantJson({128}))); + REQUIRE_FALSE(message.empty()); + CHECK(Names(message, "weight_block_size")); + CHECK(Names(message, "2 dimensions")); + } + SUBCASE("a block shape other than 128x128") { + // OUR limit, not upstream's: M5's kernel is 128x128 and nothing here can + // execute a 64x128 weight. + const std::string message = + RegistryLoadFailure(ckpt, DenseConfig(BlockQuantJson({64, 128}))); + REQUIRE_FALSE(message.empty()); + CHECK(Names(message, "weight_block_size")); + CHECK(Names(message, "[64, 128]")); + CHECK(Names(message, "128")); + CHECK(Names(message, "1189")); + } + SUBCASE("a quant_method that is not fp8") { + // Mirrors `is_checkpoint_fp8_serialized` (`fp8.py:117-120`). + const std::string message = RegistryLoadFailure( + ckpt, DenseConfig(BlockQuantJson({128, 128}, "dynamic", "awq"))); + REQUIRE_FALSE(message.empty()); + CHECK(Names(message, "quant_method")); + CHECK(Names(message, "awq")); + } + SUBCASE("the supported config is NOT refused") { + // Without this the gate passes for a reader that refuses every block-wise + // checkpoint, which is exactly the state this row replaces. + const std::string message = + RegistryLoadFailure(ckpt, DenseConfig(BlockQuantJson())); + CHECK_MESSAGE(message.empty(), "supported block config refused: " << message); + } +} + +// G6 ----------------------------------------------------------------------- +TEST_CASE("fp8 block weight: the per-tensor and bf16 arms are unchanged") { + SUBCASE("a per-tensor fp8 checkpoint still lands in Fp8Weight") { + const TempCheckpoint ckpt(DenseFixture(ProjArm::kPerTensor)); + const Qwen3_5DenseWeights w = + LoadDense(ckpt, DenseConfig(nlohmann::json())); + const vllm::FullAttnLayerWeights& a = w.layers[0].attn; + CHECK(a.q_proj_fp8_block.Empty()); + CHECK_FALSE(a.q_proj_fp8.Empty()); + CHECK(a.q_proj_fp8.weight_scale == 0.25F); + CHECK(a.q_proj_fp8.input_scale == 0.5F); + CHECK(a.q_proj_fp8.alpha == 0.125F); + } + SUBCASE("a bf16 checkpoint is untouched") { + const TempCheckpoint ckpt(DenseFixture(ProjArm::kBf16)); + const Qwen3_5DenseWeights w = + LoadDense(ckpt, DenseConfig(nlohmann::json())); + const vllm::FullAttnLayerWeights& a = w.layers[0].attn; + CHECK(a.q_proj_fp8_block.Empty()); + CHECK(a.q_proj_fp8.Empty()); + CHECK_FALSE(a.q_proj.Empty()); + CHECK_FALSE(w.layers[0].mlp.gate_up_proj.Empty()); + } +} + +// G7 ----------------------------------------------------------------------- +TEST_CASE("fp8 block weight: nothing consumes it yet and Prepare says so by name") { + // The M3/M4 seam. `ModelRegistry::Load` succeeds so the loader rung is + // reachable from a production entry point at this merge commit; nothing reads + // an `Fp8BlockWeight` yet, so `ModelRegistry::Prepare` — which every runner + // calls before the first forward (`src/vllm/v1/worker/gpu/runner.cpp:414`) — + // refuses rather than letting the dense `project` lambda fall through to an + // empty bf16 tensor. + const TempCheckpoint ckpt(DenseFixture(ProjArm::kBlockBf16Scale)); + const HfConfig config = DenseConfig(BlockQuantJson()); + std::vector shards; + shards.push_back(SafetensorsFile::Open(ckpt.path())); + const ModelSource source = ModelSource::FromSafetensors(shards); + std::unique_ptr model = + vllm::ModelRegistry::Load(config, source); + REQUIRE(model != nullptr); + + vt::Queue queue = vt::GetBackend(vt::DeviceType::kCPU).CreateQueue(); + std::string message; + try { + vllm::ModelRegistry::Prepare(*model, config, queue); + } catch (const std::exception& e) { + message = e.what(); + } + REQUIRE_FALSE(message.empty()); + CHECK(Names(message, "block-wise")); + CHECK(Names(message, "q_proj")); + CHECK(Names(message, "1189")); +} From 1b1104421d327d2d36581a1acaa2bc0c8015fcf9 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Tue, 18 Aug 2026 12:19:27 +0000 Subject: [PATCH 3/3] spec(MODEL-FP8-BLOCK-WEIGHT): record the M3 evidence, including two mutations that proved nothing (#1189) Fills the spec's `## Evidence` on the merged tree at `65d6cdaed`: the red-first compile failure, the per-block gate counts, the reachability mutation, and eleven mutation results. Three of those are worth keeping and are written up rather than tabulated. **A mutation that applies cleanly can still change nothing.** Guarding the `if (layer.is_linear_attention)` arm of the `Prepare` refusal with `if (false)` took the `else` arm instead, which runs every attention check, and the MLP checks sit outside the branch entirely. The refusal still fired, the gate stayed green, and `git diff --stat` reported a real two-line diff. Neither `compile_rc` nor `git diff --stat` catches that class. The verdict was re-taken against the actual call site, where it reds. **`-Werror` turns the natural reachability mutation into a non-event.** Deleting the rung call orphans the `block` parameter, `-Werror=unused-parameter` fires, the build fails, and the STALE binary from the previous link prints `SUCCESS!`. Re-run with `block.block_n` kept live, it reds 6 of 7 cases with issue #1166's own sentence, `tensor not found: model.layers.0.self_attn.q_proj.weight_scale`. **A grid of round shapes is blind to the ragged defect.** Replacing `cdiv` with floor left every other block green and failed only G3, which is why `N=576` and `K=3884` are in the grid rather than the target checkpoint's shapes, all of which are multiples of 128. Same measurement M2 recorded one layer down. `test_cpu_x86_llamacpp_floor` is the one failing preflight gate, and it is [#618](https://github.com/mudler/vllm.cpp/issues/618) rather than this change: a pristine detached worktree at `origin/main` `65d6cdaed`, with no part of this row applied, failed the same two cases with `NO_QUIET_WINDOW after 30s` at `busy=154%` and `busy=177%`. That worktree was removed. The other 81 gates report `ok` and none was skipped. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/specs/model-fp8-block-weight.md | 121 +++++++++++++++++++++++- 1 file changed, 119 insertions(+), 2 deletions(-) diff --git a/.agents/specs/model-fp8-block-weight.md b/.agents/specs/model-fp8-block-weight.md index 402118ce3..bdd7679f8 100644 --- a/.agents/specs/model-fp8-block-weight.md +++ b/.agents/specs/model-fp8-block-weight.md @@ -348,8 +348,125 @@ download. The row is scoped so that it needs neither. ## Evidence -Recorded after the implementation, below the `## Owed` list, so that a reader -who stops at the design has read the design. +Taken on the merged tree, `origin/main` at `65d6cdaed`. No GPU lease, no +checkpoint download, no snapshot. + +**RED.** With both test files present and no implementation, the focused build +fails, `compile_rc=1`, with **50 errors**: 6 ``'Fp8BlockWeight' does not name a +type``, 4 + 2 + 1 + 1 ``has no member named 'q_proj_fp8_block'`` / +`'o_proj_fp8_block'` / `'v_proj_fp8_block'` / `'k_proj_fp8_block'` on +`FullAttnLayerWeights`, 3 more on `DenseMlpWeights`, and the cascade of +undeclared locals that follows. `test_fp8_block_quant` compiled but reported +**4 cases, 2 failed, 19 assertions, 9 failed** against the narrowed refusal, +which is the same red seen from the other side. + +**GREEN.** `test_fp8_block_weight_load` reports **7 cases, 102 assertions, 0 +failed**; `test_fp8_block_quant` reports **8 cases, 35 assertions, 0 failed**. +Per block, each run through a `-tc` prefix filter that contains no comma, +because doctest splits `-tc` on commas and a name that contains one yields +`0 cases ran` under a `SUCCESS!` banner: + +| Block | Cases | Assertions | +|---|---:|---:| +| G1 the rung is selected | 1 | 24 | +| G2 the scale is widened not reinterpreted | 1 | 17 | +| G3 the ragged grid | 1 | 11 | +| G4 the config/tensor disagreements | 1 | 19 | +| G5 the unsupported configs | 1 | 17 | +| G6 the per-tensor and bf16 negative controls | 1 | 9 | +| G7 the M4 gap at `Prepare` | 1 | 5 | +| **sum** | **7** | **102** | + +The buckets sum to the whole-run count, so no block is silently empty and no +filter selected nothing. + +The other declared gates on the same tree, all passing: +`test_ops_quant_fp8_group_cpu` (M1 unchanged), `test_ops_matmul_fp8_block_cpu` +(M2 unchanged), `test_ops_fp8_cpu`, `test_qwen36_weights`, `test_linear_method`, +`test_model_registry`, `test_op_provider`. The whole tree builds clean, 609 +targets, `build_rc=0`. + +### The reachability mutation + +`.agents/reachability.md`: delete the **production call site**, not the +implementation, and rerun the focused gate. + +The chain is `ModelRegistry::Load` -> `LoadQwen3_5DenseModel` +(`qwen3_5_dense.cpp:101`) -> `LoadQwen3_5Dense` -> `LoadQwen3_5DenseLayer` -> +`LoadAttnDense`'s `load_projection`. Deleting the `LoadFp8BlockRaw` call there +reds **6 of 7 cases**, and the sentence it reds with is issue #1166 verbatim: + +```text +vt: qwen3_5 dense: tensor not found: model.layers.0.self_attn.q_proj.weight_scale +``` + +That is the whole point of the rung. Without it the block-wise projection falls +into the per-tensor arm and asks for a tensor the checkpoint spells +`weight_scale_inv`. + +### Mutation results + +Every mutation printed `git diff --stat` **and** `compile_rc` before the run, +because a mutation that fails to build and a mutation that never applied both +read as a passing test, and M1 and M2 of #1189 each hit one. Each was restored +with `git checkout -- .` and the restore verified by comparing +`git ls-files -s | sha256sum` before and after; every row below restored to +`fdb713f0...`. + +| Mutation | `compile_rc` | Result | +|---|---|---| +| the attn block rung call site deleted | **1** | proves nothing: `-Werror=unused-parameter` on `block` | +| the same with `block` kept live | 0 | **6 of 7 cases fail**, 10 assertions; the failure is `tensor not found: ...q_proj.weight_scale` | +| the block rung moved AFTER the per-tensor rung, i.e. the #1166 ordering | 0 | 6 of 7 cases fail, 10 assertions, same sentence. Ordering is load-bearing and measured | +| the BF16 scale `memcpy`'d instead of converted, i.e. the #1181 defect | 0 | 2 cases fail, 5 assertions, all in G2 | +| the scale-dtype refusal widened to a silent f32 read for any dtype | 0 | 1 case fails, 1 assertion (G2's F16 refusal) | +| `cdiv` replaced by floor on both axes | 0 | **only G3 fails**, and by THROWING: the run reports 91 assertions instead of 102 with `Status: FAILURE!` and `run_rc=1`. G1, G2 and G6 stay green, because every dimension in them is a multiple of 128 | +| the config/tensor cross-check reduced to a probe | 0 | **only G4 fails**, 5 assertions | +| the 128x128 and `dynamic` refusals removed | 0 | G5 fails 1 assertion and `test_fp8_block_quant` fails **3 of 8 cases, 12 assertions** | +| the `Prepare` refusal's `is_linear_attention` branch flipped to `if (false)` | 0 | **everything stays green** — see below | +| the `Prepare` call site deleted | 0 | 1 case fails (G7), 1 assertion | +| the packed fp8 bytes zeroed instead of copied | 0 | 1 case fails, 1 assertion (G1's verbatim-bytes check) | + +Three results are worth keeping. + +**A mutation that applies cleanly can still change nothing.** Guarding the +`if (layer.is_linear_attention)` arm with `if (false)` looked like deleting the +refusal. It took the `else` arm instead, which runs every attention check, and +the MLP checks sit outside the branch entirely — so the refusal still fired and +the gate stayed green over a two-line diff that `git diff --stat` happily +reported. The verdict was re-taken against the actual call site, where it reds. +`compile_rc` and `git diff --stat` do not catch this class; only asking what the +mutated code now does catches it. + +**`-Werror` turns the natural reachability mutation into a non-event.** Deleting +the rung call orphans the `block` parameter, `-Werror=unused-parameter` fires, +the build fails, and the STALE binary from the previous link then prints +`SUCCESS!`. That is a green over a mutation that never ran. The re-run keeps +`block.block_n` live. + +**A grid of round shapes is blind to the ragged defect.** Replacing `cdiv` with +floor left G1, G2, G4, G5, G6 and both of `test_fp8_block_quant`'s halves +entirely green and failed only G3, which is why `N=576` and `K=3884` are in the +grid rather than the shapes the target checkpoint happens to use — all of which +are multiples of 128. This is the same measurement M2 recorded for its own +kernel, reproduced one layer up. + +### The known-red gate on this host + +`scripts/agent-preflight.sh --staged --fail-on-skip` reports +`test_cpu_x86_llamacpp_floor` FAIL: +`NO_QUIET_WINDOW after 30s (busy=114% builders=0 load=134.83 146.63 122.42)`, +the harness exiting 4 where the case expects 2. That is +[#618](https://github.com/mudler/vllm.cpp/issues/618): the harness refuses to +measure while the box is loaded, and the refusal itself reads as a defect in +whatever diff is in flight. + +It is a property of the host and not of this change, and that was **measured +rather than argued**. A pristine detached worktree at `origin/main` +`65d6cdaed`, with no part of this row applied, ran the same suite: `Ran 10 +tests`, `FAILED (failures=2)`, both `NO_QUIET_WINDOW after 30s` at `busy=154%` +and `busy=177%`, `builders=0`, 1-minute load 50 to 60. The worktree was removed +afterwards. Every other preflight gate is `ok`. ## Now