diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 6ee7e5bc4..a6f04a141 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -347,6 +347,7 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1150](https://github.com/mudler/vllm.cpp/issues/1150) | `LTX25-TI2VID-RECIPE` | The sigma SHIFT is derived from the target latent on every arm, where six of upstream's seven `LTX2Scheduler.execute` call sites pass NO latent and take `default_number_of_tokens` = `MAX_SHIFT_ANCHOR` = 4096 (`schedulers.py:11,:29,:31`). `grep -rn '\.execute(' packages/ltx-pipelines/src/ltx_pipelines/` at `fd4ded7f` returns seven and that grep is the whole population: only `ti2vid_two_stages_hq.py:267` passes `latent=empty_latent`. This engine passes `target_tokens` at `src/vllm/multimodal/ltx2_video.cpp:3442-3443`, so it mirrors the exception and diverges from the rule. Correct today: `t2a_one_stage` (passes 0 at `src/vllm/model_executor/models/ltx2_t2a.cpp:178`) and `res2s_two_stage`. DIVERGENT: `one_stage` at four version keys (`ti2vid_one_stage.py:207`), `a2vid_two_stage` stage 1 (`a2vid_two_stage.py:226`) and `retake`'s non-distilled arm (`retake.py:287`). Recipes carrying explicit `sigmas` never reach the derivation and are unaffected. Not a rounding difference: at the recipe default geometry the target latent is 6144 tokens, giving `sigma_shift` 2.78 against upstream's 2.05, so every sigma moves. Invisible because the trajectory changes while the frame count, shapes, sample rate and errors do not, and our goldens were captured from this engine so they PIN it rather than detect it. `.agents/specs/ltx25-res2s-loop.md:80-88` saw the HQ/plain split and concluded the divergence was on the plain two-stage arm alone; that is right about HQ and wrong about the blast radius. Found by row `LTX25-TI2VID-RECIPE`, which added the seam — `Ltx2PhaseRecipe::schedule_tokens`, defaulted to today's behaviour so nothing moves — and set it on the one phase it ships. Not fixed in flow because flipping the other three re-samples five shipped, gated arms and rewrites their goldens, which needs its own spec and fresh review. Listed under `## Owed` in [`ltx25-ti2vid-recipe.md`](specs/ltx25-ti2vid-recipe.md) | bug | | [#1151](https://github.com/mudler/vllm.cpp/issues/1151) | `LTX25-TI2VID-RECIPE` | The `requires_distilled_lora` refusal still advertised [#1118](https://github.com/mudler/vllm.cpp/issues/1118) as OPEN, and cited `a2vid_two_stage.py`'s line numbers to every other pipeline. #1118 closed at `4ae0f54ab` (row `LTX25-PHASE-LORA`, PR [#1140](https://github.com/mudler/vllm.cpp/pull/1140)), which added `Ltx2PhaseRecipe::loras` and `Ltx2RebindDitLoras`; the message at `src/vllm/multimodal/ltx2_video.cpp:1039-1041` still ended "upstream fuses that adapter into stage 2 ALONE and this engine fuses once at load, so stage 1 sees it too", every clause of which had become false, and the comment above the refusal said the same. `ltx25-phase-lora.md` repaired the REFERENCE-CONDITIONING refusal, which carried the identical claim ~1100 lines away, and named only that one in its port map, so this site and the `ltx2-gen --help` text (`examples/ltx2_gen/main.cpp:210-212`) were both missed. Second defect at the same site: the refusal is deliberately keyed on the FLAG rather than on the kind string so the next recipe inherits it (the comment names #1093 and #1096 as waiting), yet its body interpolated `im.pipeline_kind` into the first sentence and hard-coded a2vid's `:164`, `:114`, `:107` into the rest — so the first arm to inherit it would be told its own name and then a different pipeline's source lines. `--distilled-lora required=True` lives on `default_2_stage_arg_parser` (`utils/args.py:1123`, `:1140-1155`), which all of these pipelines select, and that shared anchor is what the message now cites. Found and fixed IN FLOW by row `LTX25-TI2VID-RECIPE`, the second user of the flag; `tests/vllm/multimodal/test_ltx2_video.cpp` asserted the string `1118` was PRESENT and now asserts it is absent | bug | | [#1152](https://github.com/mudler/vllm.cpp/issues/1152) | `LTX25-TI2VID-RECIPE` | `Ltx2PipelineRecipe::allow_request_latents` (`include/vllm/model_executor/models/ltx2_pipeline.h:705`) is WRITTEN by every recipe and READ by nothing — the "a parameter no caller passes" shape AGENTS.md `## Nothing lands dead` names. Measured at `c83b96934`: `grep -rn allow_request_latents src include examples` minus the declaration returns FIVE lines and all five are assignments (`ltx2_pipeline.cpp:1264` false, `:1345` false, `:1473` true, `:1604` false). Positive control, the field declared one line above and set in the same blocks: `allow_request_sigmas` returns its assignments PLUS a real reader at `src/vllm/multimodal/ltx2_video.cpp:3476`, so the grep is well-formed and the absence is the finding. Consequence: `Res2sTwoStageRecipe` carries `true` where every other recipe carries `false` and nothing can tell the difference — no upstream `__call__` among these pipelines takes an initial-latent parameter, so `false` is what the signatures support and the `true` looks like an oversight, but it is unfalsifiable while nothing reads the field. The tests assert the VALUES, so they gate the record against itself and cannot see that nothing consumes it, which is the tautology shape [#911](https://github.com/mudler/vllm.cpp/issues/911) recorded on the anchor checker. Two closes: give it a reader (a refusal on a request supplying a latent to a recipe whose upstream signature has none) or delete it and its assertions — deleting is defensible, since no request surface carries a latent at all so the refusal could never fire either. Found by row `LTX25-TI2VID-RECIPE` while deriving the same field for a sixth recipe; not fixed in flow because both closes touch five landed recipes and one deletes gated assertions. Listed under `## Owed` in [`ltx25-ti2vid-recipe.md`](specs/ltx25-ti2vid-recipe.md) | bug | +| [#1166](https://github.com/mudler/vllm.cpp/issues/1166) | `FIX-FP8-BLOCKWISE-REFUSAL` | `Qwen/Qwen3.8-27B-FP8` is block-wise (fine-grained 128x128) FP8 and this tree implements per-tensor FP8 only, so the load stops on a message that names the wrong thing. Measured live at revision `017b9c7af6b5689d5dd426a76e0bc077eb5ca20a` on 2026-08-17: the config declares `quant_method` `fp8`, `weight_block_size` `[128, 128]` and `activation_scheme` `dynamic`, and the safetensors header of `layers-3.safetensors`, read by RANGE REQUEST rather than downloaded, gives `self_attn.q_proj.weight` `F8_E4M3` `[12288, 5120]` beside `self_attn.q_proj.weight_scale_inv` `BF16` `[96, 40]`, which is exactly `[12288/128, 5120/128]`, with ZERO `input_scale` tensors in the shard. `LoadAttnDense` branches on the weight dtype alone (`qwen3_5_dense_weights.cpp:479`) so the block-wise projection enters the per-tensor arm at `:480`, and `LoadFp8Raw` (`qwen3_5_weights.cpp:449`) asks for `.weight_scale` at `:458`, which this checkpoint spells `weight_scale_inv`, so the resolver at `qwen3_5_dense_weights.cpp:682` raises `tensor not found: ...q_proj.weight_scale`. Nothing is missing from the checkpoint. The reader is sent after a tensor upstream never writes in this mode instead of being told the fine-grained arm is absent. NOT the silently-wrong-numerics case, and the check that rules it out is recorded rather than assumed: `ReadF32Scalar` (`qwen3_5_weights.cpp:312`) bounds its input with `t.nbytes >= sizeof(float)`, a LOWER bound, so a `[96, 40]` scale would pass and read as block `(0,0)`, but the NAME miss stops the load before that scalar read, and upstream makes the spelling strictly conditional on block quant (`weight_scale_inv if self.block_quant else weight_scale`, `fp8.py:511` at pin `555967922`), so no upstream block-wise checkpoint reaches it. FIXED IN FLOW as a named refusal at `ModelRegistry::Load`; the block-wise arm itself stays owed | bug | | [#1161](https://github.com/mudler/vllm.cpp/issues/1161) | `ENG-CUDAGRAPH` | SGLang landed **Breakable CUDA Graph (BCG)** and the whole surface was absent from `.agents/sglang-matrix.md`, while `ENG-CUDAGRAPH` carried no verdict on it. Grounded at pin `v0.5.15`/`f63458b5be`: BCG captures one forward as a SEQUENCE of `torch.cuda.CUDAGraph` segments split at eager break points on the attention and recurrent calls, sharing one mempool with weak-ref'd intermediates, with **no `torch.compile`** (`breakable_cuda_graph.py:204-241,244-260,14-23,156-169`; `breakable_cuda_graph_backend.py:14-17`; break points `radix_attention.py:256`, `forward_mla.py:1092`, `radix_linear_attention.py:159`, `nemotron_h.py:1240`, `dsa_indexer.py:2414`). **The public priority claim is narrower than it reads:** vLLM's v1 default is ALREADY `FULL_AND_PIECEWISE`, splitting at `splitting_ops` (`vllm/config/compilation.py:60-63,517,615,630` @ `555967922`), so the capability has been in our benchmark denominator all along — what SGLang landed first is the CONSTRUCTION (piecewise coverage from runtime stream capture plus a decorator, no Dynamo/Inductor/FX), which matters to us only because we have no compiler either. **The load-bearing outcome is a REFUTATION:** prefill graph capture is not a lever on our CUDA gate models, because GB10 2026-07-09 measured steady-state prefill GPU-idle-between-launches at **3.8%** with GPU-busy >96% on both arms, and attributed the 27B prefill gap to **92.5% non-GEMM glue GPU work** with the dominant GEMM the same symbol at the same call count (+0.17%) and attention AHEAD. There are no launch bubbles in our prefill for a graph to collapse, and vLLM's piecewise prefill graph is already inside the denominator we lose to, so it does not explain the gap either. Decode is a separate and already-banked case (conc-64 launch overhead was ~24%, already one `cudaGraphLaunch`/step). FIXED IN FLOW: `SGLANG-BCG` row added under a new "Graph capture and runtime execution" section of `sglang-matrix.md`, the refutation recorded on `ENG-CUDAGRAPH` (which stays `PARTIAL`, because nothing shipped), roadmap track `C12` added, and spec [`sglang-breakable-cuda-graph.md`](specs/sglang-breakable-cuda-graph.md) written to carry the analysis. NOT fixed here and owned by the three issues below: no engine code, no measurement taken, no oracle run, and no advance of the SGLang pin | record | | [#1162](https://github.com/mudler/vllm.cpp/issues/1162) | `ENG-CUDAGRAPH-DEDUP` | We instantiate **one `cudaGraphExec` per padded decode bucket, per model**, and `grep -rn "cudaGraphExecUpdate" src include` returns nothing. `src/vt/cuda/cuda_backend.cu:222-232` instantiates a fresh exec per capture; `include/vllm/model_executor/models/decode_graph_sizes.h:32-41` yields 7 buckets at `max_num_seqs=32` and 11 at 64; eight drivers each build their own set. SGLang folds compatible captures onto one executable by hashing graph topology and calling `cudaGraphExecUpdate` on a signature hit (`cuda_graph_dedup_mixin.py:219-242`, logging "captured %d CUDA graphs, deduped to %d execs" at `:358`). Portable to us unchanged, because it is driver-level rather than PyTorch-level. It is a **memory and capture-time** change, NOT a throughput change — a deduped replay launches the same nodes — and it matters because on GB10 unified memory an OOM reboots the box, capture time is startup latency (a recorded gate axis), and bucket count is exactly what widening graph coverage would raise. Owed: the dedup registry behind the `vt` seam, a capture-count/exec-count log line, and a same-binary A/B proving a deduped replay is byte-identical rather than asserting it. Hazard already recorded: capture bakes host source addresses and a clean `compute-sanitizer` run is NOT evidence a capture path is safe (`specs/decode-graph-scratch-uaf-2026-07-18.md`). Spec [`sglang-breakable-cuda-graph.md`](specs/sglang-breakable-cuda-graph.md) `## Owed`. Analysis: [#1161](https://github.com/mudler/vllm.cpp/issues/1161) | perf | | [#1163](https://github.com/mudler/vllm.cpp/issues/1163) | `ENG-CUDAGRAPH-BREAK` | Two defects, stated separately. **(1) Capture has no break points:** `src/vllm/v1/worker/gpu/runner.cpp:1338-1341` routes only `pure_decode` batches to a graph, so prefill, mixed batches, and anything whose metadata is computed on the host stay eager for the WHOLE step. There is no way to express "capture this forward except for these three calls", so coverage is a cliff rather than a slope; [#1020](https://github.com/mudler/vllm.cpp/issues/1020) is one instance (a spec verify step whose actual draft depth differs from the configured `k` silently falls out to eager). **(2) Eight hand-rolled drivers:** `Qwen3_5DecodeGraph` (`qwen3_5.h:275`), `Qwen3_5DenseDecodeGraph` (`qwen3_5_dense.h:391`), `Qwen3MoeDecodeGraph` (`qwen3_moe.h:117`), `Qwen3DenseDecodeGraph` (`qwen3.h:243`), `DeepseekV2DecodeGraph` (`deepseek_v2.h:324`), `VoxtralDecodeGraph` (`voxtral.h:126`), plus graph code in `deepseek_v4.cpp` and `laguna.cpp` — each re-deriving capture, bucket padding, persistent-input threading and the pure-decode predicate. AGENTS.md names this shape: a parallel path written by hand instead of a shared seam, and every new model that wants a decode graph writes a ninth one. Mirror vLLM's `CUDAGraphMode.PIECEWISE` boundary (`splitting_ops`, i.e. the attention ops); take the CONSTRUCTION from SGLang BCG, because vLLM gets its split from `torch.compile` and we have no compiler. **Explicitly NOT a throughput row and must not be sold as one** — our prefill has 3.8% host idle and is >96% GPU-busy, and decode already banked its launch-overhead win; the value is coverage and one seam instead of eight. Any speed claim must first name and measure a path that is currently eager AND currently host-bound. Owed: the seam, break-point registration, a reachability mutation (delete the production call site, rerun the focused gate), and bit-exactness vs eager on every migrated model over MORE than one replay. Large and structural; spike first. Spec [`sglang-breakable-cuda-graph.md`](specs/sglang-breakable-cuda-graph.md) `## Owed`. Analysis: [#1161](https://github.com/mudler/vllm.cpp/issues/1161) | feature | diff --git a/.agents/specs/fp8-blockwise-refusal.md b/.agents/specs/fp8-blockwise-refusal.md new file mode 100644 index 000000000..cef22a13b --- /dev/null +++ b/.agents/specs/fp8-blockwise-refusal.md @@ -0,0 +1,203 @@ +# Refuse block-wise FP8 by name + +Row `FIX-FP8-BLOCKWISE-REFUSAL`. Issue +[#1166](https://github.com/mudler/vllm.cpp/issues/1166). + +## Scope + +Detect a block-wise (fine-grained) FP8 checkpoint at load and refuse it with a +message that names `weight_block_size`, states that block-wise FP8 is not +implemented, and names issue #1166. + +Out of scope, and recorded under `## Owed`: reading `weight_scale_inv`, +dequantizing a 128x128 block scale, and the block-wise GEMM. That work needs a +GPU gate and its own row. + +## 0. What is wrong today, measured + +`Qwen/Qwen3.8-27B-FP8` at revision `017b9c7af6b5689d5dd426a76e0bc077eb5ca20a`, +read over HTTP on 2026-08-17. The config declares `quant_method` `fp8`, +`weight_block_size` `[128, 128]`, and `activation_scheme` `dynamic`, with +`quantization_config` at the top level and none under `text_config`. + +The safetensors header of `layers-3.safetensors`, read by range request, gives +`self_attn.q_proj.weight` as `F8_E4M3` `[12288, 5120]` and +`self_attn.q_proj.weight_scale_inv` as `BF16` `[96, 40]`. `[96, 40]` equals +`[12288/128, 5120/128]`, so the scale carries one value for each 128x128 block. +The shard holds zero `input_scale` tensors, which is what the dynamic activation +scheme means. + +This tree stops on that checkpoint, so the defect is not wrong numerics. It +stops on the wrong sentence. `LoadAttnDense` branches on the weight dtype alone +(`src/vllm/model_executor/models/qwen3_5_dense_weights.cpp:479`), so an +`F8_E4M3` block-wise projection enters the per-tensor arm at `:480`, and +`LoadFp8Raw` (`src/vllm/model_executor/models/qwen3_5_weights.cpp:449`) asks for +`.weight_scale` at `:458`. The checkpoint spells that tensor +`weight_scale_inv`, so the resolver lambda at `qwen3_5_dense_weights.cpp:682` +raises + +``` +qwen3_5 dense: tensor not found: model.language_model.layers.N.self_attn.q_proj.weight_scale +``` + +Nothing is missing from the checkpoint. The reader is sent to look for a tensor +that upstream never writes in this mode, instead of being told that the +fine-grained arm is absent. + +`git grep weight_block_size -- src/ include/` returns nothing. +`git grep weight_scale_inv` returns one hit in an unrelated name predicate +(`src/vllm/model_executor/models/minimax_music3_quant.cpp:276`) and no reader in +any load path. + +## 1. What upstream does, with anchors + +Pinned vLLM `5559679229bc961848b121ccdeaa8fa5d79bec98`, the parity pin in +[`upstream-sync.md`](../upstream-sync.md), verified with `git rev-parse` in the +oracle checkout before citing. + +| Anchor | What it does | +|---|---| +| `vllm/model_executor/layers/quantization/fp8.py:161` | `Fp8Config.from_config` reads `weight_block_size` out of the quantization config | +| `vllm/model_executor/layers/quantization/fp8.py:115-132` | `Fp8Config.__init__` validates it. It requires an fp8-serialized checkpoint, exactly 2 dimensions, and a dynamic activation scheme | +| `vllm/model_executor/layers/quantization/fp8.py:297-298` | `Fp8LinearMethod` sets `self.block_quant = self.weight_block_size is not None`, which is the dispatch | +| `vllm/model_executor/layers/quantization/fp8.py:378-379` | the block scale registers as `weight_scale_inv`, with the comment that the name is intentional for deepseekv3 | +| `vllm/model_executor/layers/quantization/fp8.py:511` | `"weight_scale_inv" if self.block_quant else "weight_scale"`, the MoE spelling of the same rule | + +The last anchor is why the failure is a name miss rather than a shape miss. +Upstream makes the scale name strictly conditional on block quant, so a +block-wise checkpoint that spells its scale `weight_scale` is not a shape +upstream produces. + +## 2. Our baseline + +`include/vllm/model_executor/layers/quantization/fp8.h:1` describes itself as +"Per-tensor FP8 (W8A8)" and mirrors the per-tensor arm only. The quantization +matrix already records the gap. `QUANT-FP8-GENERIC` +([`quantization-matrix.md:125`](../quantization-matrix.md)) covers +"static/dynamic, tensor/channel/token/block" at `PARTIAL`, and +`QUANT-FP8-PB-WO` at `:126` is `INVENTORIED`. No matrix edit is owed, because +neither row changes lifecycle state here. + +## 3. Design + +Refuse at the one place every model load passes through with an `HfConfig` in +hand. `ModelRegistry::Load` +(`src/vllm/model_executor/models/model_registry.cpp:322`) resolves the +architecture, then calls `factory.parse_config` and `factory.load_weights`. The +check goes after `Resolve` and before `parse_config`, so an unsupported +architecture still reports the architecture, and a block-wise checkpoint of a +supported architecture reports the quantization before any tensor lookup runs. + +This placement is deliberately architecture-independent. `weight_block_size` is +a property of the checkpoint's quantization config, not of one model, so a +refusal per model loader would have to be written again for every architecture +and would be absent from whichever one is added next. + +The tree has a second pre-load refusal site. `LoadedEngine::FromModelDir` calls +`RefuseUnsupportedWeightOffload` at +`src/vllm/entrypoints/model_loader.cpp:1536-1541`, after `LoadHfConfig` at +`:1529` and before the shards are mapped at `:1550`. It was rejected for two +reasons. It covers the safetensors path only, so the GGUF arm that reaches +`ModelRegistry::Load` at `:1448` would keep the old message. It also needs an +on-disk model directory to reach, which would make the gate depend on a +checkpoint this row is not allowed to download. `ModelRegistry::Load` takes the +`HfConfig` and the `ModelSource` directly, so the test drives the real +production function with no model directory and no weights. + +The predicate mirrors `Fp8Config.from_config`. It reads +`quantization_config.weight_block_size` from `HfConfig::raw`, treats a non-empty +array as block quant, and ignores the key when it is absent or null. `raw` holds +the full top-level document (`src/vllm/transformers_utils/hf_config.cpp:564`), +so the top-level `quantization_config` this checkpoint uses is reachable. The +nested `text_config.quantization_config` spelling is read as well, because a +multimodal wrapper is exactly the shape in play. + +New files, mirroring the upstream module that owns the key: + +- `include/vllm/model_executor/layers/quantization/fp8_block_quant.h` +- `src/vllm/model_executor/layers/quantization/fp8_block_quant.cpp` + +The header forward-declares `HfConfig` rather than including +`transformers_utils/hf_config.h`, so no JSON dependency enters the quantization +headers. + +## 4. Port map + +| Upstream | Here | +|---|---| +| `fp8.py:161` `from_config` reads the key | `Fp8BlockQuantOf` in `fp8_block_quant.cpp` | +| `fp8.py:115-132` `__init__` validates 2 dimensions | `RefuseUnsupportedFp8BlockQuant` reports the dimensions it found | +| `fp8.py:297-298` `block_quant` dispatch | the refusal. This tree has no block-wise arm to dispatch to | + +## 5. Tests to port + +Upstream has no test for refusing block-wise FP8, because upstream implements +it. There is nothing to port, and this section records that rather than leaving +it blank. The tests below are this tree's own, and they are written red first. + +`tests/vllm/model_executor/layers/test_fp8_block_quant.cpp`, registered in +`tests/CMakeLists.txt` with `vllm_cpp_add_test`: + +1. The refusal fires through `ModelRegistry::Load` on a config carrying + `quantization_config.weight_block_size` `[128, 128]` and a registered + architecture. This is the reachability case. It enters at the production + entry point rather than at the predicate. +2. The message names `weight_block_size`, names block-wise FP8, and names + `1166`. +3. A per-tensor FP8 config, meaning `quant_method` `fp8` with no + `weight_block_size`, does NOT trip the refusal. Without this case the gate + passes for a refusal that fires on every FP8 checkpoint. +4. The nested `text_config.quantization_config` spelling trips it too. +5. A null or empty `weight_block_size` does not trip it. + +## 6. Gates + +CPU only. No GPU lease, and no checkpoint download. + +- Focused: the new target, red before the change and green after. +- Full: `scripts/agent-preflight.sh` with real per-block counts. +- Reachability mutation: delete the call site in `ModelRegistry::Load` in a + scratch copy, rerun the focused target, and require red. Restore + byte-for-byte. + +## 7. Dependencies + +None. The change adds two files, one call site, one test target, and the record +edits the change makes stale. + +## 8. Work breakdown + +One unit. Spec, then the refusal with its tests, then the record edits, in one +pull request with the spec committed first. + +## 9. Risks and decisions + +| Risk | Decision | +|---|---| +| The refusal fires on a checkpoint that loads today | Measured false. `weight_block_size` appears in no test and no example, and no load path reads `weight_scale_inv`, so nothing in the tree can be loading a block-wise checkpoint now | +| Refusing before `parse_config` changes error precedence | Accepted and bounded. The check sits AFTER `Resolve`, so an unsupported architecture still reports the architecture. Only a supported architecture with a block-wise config changes its message, which is the point | +| A GGUF load carries no `quantization_config` | Inert by construction. The key is absent, so the predicate is false | +| The message drifts from the issue it names | The test asserts the issue number, so deleting it reds the gate | + +## Owed + +- Block-wise FP8 execution itself. Reading `weight_scale_inv`, dequantizing or + applying a 128x128 block scale, and the dynamic per-token activation quant + upstream pairs with it (`fp8.py:301-310`). This is the arm the refusal names, + it needs a GPU gate and a checkpoint, and it has no row yet. Tracked by + [#1166](https://github.com/mudler/vllm.cpp/issues/1166), which stays open + after this change lands, because this change makes the gap legible and does + not close it. +- `ReadF32Scalar` (`src/vllm/model_executor/models/qwen3_5_weights.cpp:312`) + bounds its input with `t.nbytes >= sizeof(float)`, a lower bound, and returns + the first 4 bytes. A multi-element scale passes that check and reads as block + `(0,0)`. It is unreachable by the block-wise route, because the name miss + described in section 0 stops the load first, and it is recorded here rather + than fixed so a later change that adds a `weight_scale` alias cannot + reintroduce it silently. Tracked under + [#1166](https://github.com/mudler/vllm.cpp/issues/1166). + +## Now + +`ACTIVE`. The refusal and its tests land in this change. The block-wise arm +stays owed. diff --git a/CMakeLists.txt b/CMakeLists.txt index 0d080eb56..b284eeaf1 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -974,6 +974,7 @@ add_library(vllm STATIC src/vllm/model_executor/models/phi_weights.cpp src/vllm/model_executor/models/phi.cpp src/vllm/model_executor/models/model_registry.cpp + src/vllm/model_executor/layers/quantization/fp8_block_quant.cpp src/vllm/plugins/plugins.cpp src/vllm/model_executor/models/qwen3_vl_vision.cpp src/vllm/model_executor/models/gemma4_vision.cpp diff --git a/docs/FEATURES.md b/docs/FEATURES.md index e57dabf5b..3931b54eb 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -78,7 +78,8 @@ are our reading of their documented behavior, not measurements. | AWQ | ◐ CPU dequant | ✅ | ✅ | ☐ | | GPTQ | ◐ CPU dequant | ✅ | ✅ | ☐ | | MXFP4 compressed-tensors | ◐ W4A16 Marlin, mem 2.63x less. gate_up FUSION + decode-graph default-ON; #44 3/3, 32B 6/6. **`VT_MARLIN_DENSE` DEFAULT-ON** (`KERNEL-MARLIN-DENSE-EXEC`): dense marlin 48-CTA, byte-faithful, beats MoE (c8 0.969) | ✅ | ✅ | ☐ | -| fp8 weights | ✅ | ✅ | ✅ | ☐ | +| fp8 weights, per-tensor scale | ✅ | ✅ | ✅ | ☐ | +| Block-wise (fine-grained 128x128) FP8, the `weight_scale_inv` layout | ☐ REFUSED BY NAME at load (#1166): `Qwen/Qwen3.8-27B-FP8` declares `weight_block_size` [128, 128] and this build is per-tensor FP8 only ([spec](../.agents/specs/fp8-blockwise-refusal.md)) | ✅ | ✅ | ☐ | | Per-tensor FP8 W8A8 linear is a shared seam any model can bind | ✅ `models/dense_fp8_gemm.h` + `layers::Fp8W8A8LinearMethod` (#940), bound via `layers::MakeLinearMethod`. One definition, CUDA only ([spec](../.agents/specs/vt-fp8-shared-seam.md)) | ✅ `Fp8LinearMethod` | ✅ | ☐ | | FP8 W8A8 works on a CUDA arch without `cutlass-fp8` | ✅ `vt::QuantFp8Static` registers from an unconditional TU (#960); sm_110 measured ([spec](../.agents/specs/vt-fp8-quant-arch-gate.md)) | ✅ | ✅ | ☐ | | fp8-tower GDN `in_proj` emits bf16, unlocking packed GDN decode | ◐ `VT_GDN_FP8_IN_BF16` + `VT_GDN_PACKED_DECODE_FP8_TOWER` (inert alone), both default **OFF**, ungated (#339) ([spec](../.agents/specs/perf-fp8-alpha-fold.md)) | ✅ bf16 `out_dtype` | ☐ | ☐ | @@ -114,7 +115,7 @@ speed-pending, which [BENCHMARKS.md](BENCHMARKS.md) tracks. | Architecture | Tested checkpoint(s) | Correctness gate | Speed vs reference | |---|---|---|---| -| `Qwen3_5ForConditionalGeneration` | Qwen3.6-27B NVFP4 (`unsloth` @`890bdef7`, `nvidia` @`0893e160`); Qwen3.5-4B BF16; **Qwen3.8-27B BF16** @`1d4bf0f2` | 27B strict 235/235 text + 32/32 image/video; 4B cached 3/3; Qwen3.8-27B 4/7 strict, 3 exact fp32 ties in band (#915) | `unsloth` 27B at/above vLLM, ModelOpt 0.85x; 4B 1.021x; 3.8-27B c4 **0.963x**, c1/c8 withheld (#931). Loads BF16/FP8/NVFP4 (CT + ModelOpt); `modelopt_mixed` FP8 tower NATIVE (#164), GDN `in_proj_qkvz` merged. CUDA/CPU | +| `Qwen3_5ForConditionalGeneration` | Qwen3.6-27B NVFP4 (`unsloth` @`890bdef7`, `nvidia` @`0893e160`); Qwen3.5-4B BF16; **Qwen3.8-27B BF16** @`1d4bf0f2` | 27B strict 235/235 text + 32/32 image/video; 4B cached 3/3; Qwen3.8-27B 4/7 strict, 3 exact fp32 ties in band (#915) | `unsloth` 27B at/above vLLM, ModelOpt 0.85x; 4B 1.021x; 3.8-27B c4 **0.963x**, c1/c8 withheld (#931). Loads BF16/per-tensor FP8/NVFP4 (CT + ModelOpt); `modelopt_mixed` FP8 tower NATIVE (#164), GDN qkvz merged. CUDA/CPU | | `Qwen3_5MoeForConditionalGeneration` | Qwen3.6-35B-A3B (NVFP4 text; published BF16 text + vision tower) | NVFP4 strict 315/315 vs vLLM 0.25.0; published BF16 6/7 prompts strict 16/16 vs the pin, 7th an exact tie (#910). Image/video IMPLEMENTED, NOT GATED (#891): the tower loads and runs, mm gate OWED | gate model: 0.93x to 1.03x grid; NO BF16 or mm speed claim | | `Qwen3_5ForCausalLM`, `Qwen3_5MoeForCausalLM` | none: no text-only Qwen3.5 checkpoint fits this hardware | **NO RUN GATE, OWED.** Gated on `test_qwen3_8_text_only.cpp`; NO token claim. Loader reads stacked BF16 experts (#740) plus BF16 towers, shared expert and `lm_head` (#864), so both published indices satisfy the load plan | not measured | | `Qwen3ForCausalLM` | Qwen3 dense 0.6B/1.7B/4B/32B, NVFP4A16 | near-tie strict 16/16 vs vLLM 0.25.0 | c1 every-axis parity, c8 decode residual | diff --git a/docs/USAGE.md b/docs/USAGE.md index 43ce7053f..c0c7ac302 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -524,6 +524,33 @@ quantizes the activation once; a checkpoint whose scales differ keeps the two separate GEMMs automatically. `VT_GDN_MERGED_QKVZ_FP8=0` restores the two GEMMs in the same binary. +### Block-wise FP8 is refused at load + +This build reads per-tensor FP8, where one scale covers a whole weight. It does +not read block-wise FP8, also called fine-grained FP8, where one scale covers +each 128x128 block of the weight. A block-wise checkpoint declares +`quantization_config.weight_block_size` in its `config.json`, and it stores its +scales under `weight_scale_inv` rather than under `weight_scale`. + +`Qwen/Qwen3.8-27B-FP8` is such a checkpoint. At revision +`017b9c7af6b5689d5dd426a76e0bc077eb5ca20a` it declares `weight_block_size` +`[128, 128]` with `activation_scheme` `dynamic`, and it stores +`self_attn.q_proj.weight` as `F8_E4M3` `[12288, 5120]` beside +`self_attn.q_proj.weight_scale_inv` as `BF16` `[96, 40]`. + +Loading it stops with a message that names the key: + +```text +quantization_config.weight_block_size [128, 128] selects block-wise +(fine-grained) FP8, which is not implemented. This build implements per-tensor +FP8 only. +``` + +The refusal is deliberate. Nothing is wrong with that checkpoint, and the +missing arm is in this project. To run the same model here, use a per-tensor +FP8, BF16, NVFP4, or GGUF checkpoint of it. Issue +[#1166](https://github.com/mudler/vllm.cpp/issues/1166) tracks the port. + ### Architectures that resolve but refuse to run A few architectures are registered so their config and weight layout are diff --git a/include/vllm/model_executor/layers/quantization/fp8_block_quant.h b/include/vllm/model_executor/layers/quantization/fp8_block_quant.h new file mode 100644 index 000000000..61db18ce5 --- /dev/null +++ b/include/vllm/model_executor/layers/quantization/fp8_block_quant.h @@ -0,0 +1,56 @@ +// Block-wise (fine-grained) FP8 detection and its named refusal. +// +// UPSTREAM (ported FROM, ground-every-impl rule), pinned vLLM +// `5559679229bc961848b121ccdeaa8fa5d79bec98`: +// vllm/model_executor/layers/quantization/fp8.py:161 +// Fp8Config.from_config — reads `weight_block_size` out of the checkpoint's +// quantization config. Absent means None, which means per-tensor. +// vllm/model_executor/layers/quantization/fp8.py:115-132 +// Fp8Config.__init__ — validates it: an fp8-serialized checkpoint, exactly +// 2 dimensions, and a dynamic activation scheme. +// vllm/model_executor/layers/quantization/fp8.py:297-298 +// Fp8LinearMethod — `self.block_quant = self.weight_block_size is not None` +// is the whole dispatch, and this tree has no arm to dispatch TO. +// vllm/model_executor/layers/quantization/fp8.py:378-379, :511 +// the block scale registers as `weight_scale_inv`, not `weight_scale`, and +// the name is strictly conditional on block quant. +// +// WHY THIS FILE EXISTS. `include/.../quantization/fp8.h` mirrors the PER-TENSOR +// arm and says so on its first line. A block-wise checkpoint used to enter that +// arm anyway, because the dense loader branches on the weight dtype alone +// (`qwen3_5_dense_weights.cpp:479`) and the block-wise weight really is +// `F8_E4M3`. The load then asked for `.weight_scale` +// (`qwen3_5_weights.cpp:458`), which a block-wise checkpoint does not have, and +// died on `tensor not found`. That sentence is wrong about the world: the +// checkpoint is complete, and it is this tree that is missing an arm. Issue +// #1166, spec `.agents/specs/fp8-blockwise-refusal.md`. +// +// SCOPE. Detect and refuse by name. Reading `weight_scale_inv`, applying a +// 128x128 block scale, and the dynamic per-token activation quant upstream +// pairs with it are OWED, not done, and the refusal names the issue that owes +// them. +#pragma once + +#include +#include + +namespace vllm { + +struct HfConfig; + +// The `weight_block_size` a checkpoint declares, empty when it declares none. +// +// Mirrors `Fp8Config.from_config`: the key is read from `quantization_config`, +// and absent, null, or empty all mean per-tensor. Both spellings are read, the +// top level and the `text_config` nesting a multimodal wrapper can use, because +// the wrapper shape is exactly the one in play on `Qwen3_5ForConditionalGeneration`. +std::vector Fp8WeightBlockSizeOf(const HfConfig& config); + +// Refuses a block-wise FP8 checkpoint by name, or returns when the checkpoint +// is not block-wise. +// +// Throws `std::runtime_error`, the type every other load refusal in this tree +// throws, so the C API surfaces it as `VLLM_ERR_MODEL_LOAD` unchanged. +void RefuseUnsupportedFp8BlockQuant(const HfConfig& config); + +} // namespace vllm diff --git a/src/vllm/model_executor/layers/quantization/fp8_block_quant.cpp b/src/vllm/model_executor/layers/quantization/fp8_block_quant.cpp new file mode 100644 index 000000000..1ab542ddc --- /dev/null +++ b/src/vllm/model_executor/layers/quantization/fp8_block_quant.cpp @@ -0,0 +1,75 @@ +#include "vllm/model_executor/layers/quantization/fp8_block_quant.h" + +#include +#include +#include + +#include "vllm/transformers_utils/hf_config.h" + +namespace vllm { +namespace { + +// `quantization_config`, top level first and then the `text_config` nesting. +// `HfConfig::raw` holds the FULL top-level document +// (`src/vllm/transformers_utils/hf_config.cpp:564`), so both are reachable from +// here. `Qwen/Qwen3.8-27B-FP8` uses the top-level spelling, measured: its +// `text_config` carries no `quantization_config`. +const nlohmann::json* QuantizationConfigOf(const HfConfig& config) { + if (!config.raw.is_object()) return nullptr; + const auto top = config.raw.find("quantization_config"); + if (top != config.raw.end() && top->is_object()) return &*top; + const auto text = config.raw.find("text_config"); + if (text == config.raw.end() || !text->is_object()) return nullptr; + const auto nested = text->find("quantization_config"); + if (nested != text->end() && nested->is_object()) return &*nested; + return nullptr; +} + +std::string DimensionList(const std::vector& dims) { + std::string out = "["; + for (size_t i = 0; i < dims.size(); ++i) { + if (i != 0) out += ", "; + out += std::to_string(dims[i]); + } + return out + "]"; +} + +} // namespace + +std::vector Fp8WeightBlockSizeOf(const HfConfig& config) { + const nlohmann::json* quant = QuantizationConfigOf(config); + if (quant == nullptr) return {}; + const auto it = quant->find("weight_block_size"); + // Absent and null are both "None" upstream (`fp8.py:161` reads it with a + // default of None), so neither is block quant. + if (it == quant->end() || it->is_null() || !it->is_array()) return {}; + std::vector dims; + dims.reserve(it->size()); + for (const nlohmann::json& dim : *it) { + if (!dim.is_number_integer()) return {}; + dims.push_back(dim.get()); + } + return dims; +} + +void RefuseUnsupportedFp8BlockQuant(const HfConfig& config) { + const std::vector block = Fp8WeightBlockSizeOf(config); + if (block.empty()) return; + + // Named, not merely refused. The key so the reader can grep their own + // config.json, the value so the message is about THIS checkpoint, the arm + // that is missing, the arm that works, where the scale actually lives, and + // the issue that owes the port. + throw std::runtime_error( + "quantization_config.weight_block_size " + DimensionList(block) + + " selects block-wise (fine-grained) FP8, which is not implemented. This " + "build implements per-tensor FP8 only. A block-wise checkpoint stores one " + "scale for each " + + (block.size() == 2 ? DimensionList(block) : std::string("block")) + + " weight block under `weight_scale_inv`, and nothing here reads that " + "tensor, so the weights cannot be dequantized correctly. This is a " + "missing arm in vllm.cpp and not a problem with the checkpoint. Tracked " + "by https://github.com/mudler/vllm.cpp/issues/1166"); +} + +} // namespace vllm diff --git a/src/vllm/model_executor/models/model_registry.cpp b/src/vllm/model_executor/models/model_registry.cpp index 1b6c329df..73bf3a272 100644 --- a/src/vllm/model_executor/models/model_registry.cpp +++ b/src/vllm/model_executor/models/model_registry.cpp @@ -12,6 +12,7 @@ // rather than a fixed in-file array. #include "vllm/model_executor/models/model_registry.h" +#include "vllm/model_executor/layers/quantization/fp8_block_quant.h" #include "vllm/model_executor/weight_offloader.h" #include @@ -322,6 +323,24 @@ ModelRegistry::OutOfTreeSupportedModels() { std::unique_ptr ModelRegistry::Load(const HfConfig& config, const ModelSource& source) { const ModelRegistration& registration = Resolve(config); + // FIX-FP8-BLOCKWISE-REFUSAL (#1166): a block-wise (fine-grained) FP8 + // checkpoint is refused BY NAME here, before any weight loader runs. + // + // AFTER `Resolve`, so an unsupported architecture still reports the + // architecture rather than its quantization. BEFORE `load_weights`, because + // that is what makes the message about the missing ARM instead of about the + // first tensor whose name does not resolve: the dense loader branches on the + // weight dtype alone and pulls a block-wise projection into the per-tensor + // arm, which then asks for a `weight_scale` a block-wise checkpoint spells + // `weight_scale_inv` and dies on `tensor not found`. + // + // Sited on the registry rather than per model loader on purpose. + // `weight_block_size` is a property of the checkpoint's quantization config + // and not of one architecture, so a per-loader refusal would have to be + // written again for every architecture and would be missing from whichever + // one is added next. This also covers the GGUF arm, which reaches this + // function too and where the key is simply absent. + RefuseUnsupportedFp8BlockQuant(config); const ModelFactory& factory = *registration.factory; factory.parse_config(config); std::unique_ptr model = diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 63706934f..c9f2b15c0 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -394,6 +394,12 @@ vllm_cpp_add_test(test_gguf_device_fit vllm/model_executor/test_gguf_device_fit.cpp) vllm_cpp_add_test(test_gguf_device_fit_reach vllm/entrypoints/test_gguf_device_fit_reach.cpp) +# FIX-FP8-BLOCKWISE-REFUSAL #1166: block-wise (128x128) FP8 is refused BY NAME +# at load. Every case enters through `ModelRegistry::Load` rather than through +# the predicate, so deleting that call site reds this target. That is the +# reachability proof and not a class test. +vllm_cpp_add_test(test_fp8_block_quant + vllm/model_executor/layers/test_fp8_block_quant.cpp) vllm_cpp_add_test(test_linear_method vllm/model_executor/layers/test_linear_method.cpp) target_include_directories(test_linear_method PRIVATE ${CMAKE_SOURCE_DIR}/src) vllm_cpp_add_test(test_decode_graph_sizes vllm/models/test_decode_graph_sizes.cpp) diff --git a/tests/vllm/model_executor/layers/test_fp8_block_quant.cpp b/tests/vllm/model_executor/layers/test_fp8_block_quant.cpp new file mode 100644 index 000000000..7fec946a8 --- /dev/null +++ b/tests/vllm/model_executor/layers/test_fp8_block_quant.cpp @@ -0,0 +1,177 @@ +// Block-wise (fine-grained) FP8 is refused BY NAME at load — issue #1166, spec +// `.agents/specs/fp8-blockwise-refusal.md`. +// +// `Qwen/Qwen3.8-27B-FP8` declares `quantization_config.weight_block_size` +// `[128, 128]` and stores one scale per 128x128 block under `weight_scale_inv`. +// This tree implements PER-TENSOR fp8 only. Before this gate the load still +// stopped, so the defect was never wrong numerics — it stopped on the wrong +// sentence. `LoadFp8Raw` asks for `.weight_scale` +// (`src/vllm/model_executor/models/qwen3_5_weights.cpp:458`), the checkpoint +// spells that tensor `weight_scale_inv`, and the resolver raised +// `qwen3_5 dense: tensor not found: ...q_proj.weight_scale`. Nothing was missing +// from the checkpoint. The reader was sent after a tensor upstream never writes +// in this mode instead of being told the fine-grained arm is absent. +// +// EVERY case here enters through `ModelRegistry::Load`, the production loader +// `src/vllm/entrypoints/model_loader.cpp:1613` calls, and NOT through the +// predicate. That is deliberate and it is the reachability proof AGENTS.md +// `## Nothing lands dead` asks for: a unit test that called the predicate +// directly would prove the function works and never that a load reaches it. +// Deleting the call site in `ModelRegistry::Load` must red this file. +// +// No checkpoint, no GPU, and no model directory: the refusal fires before +// `factory.load_weights`, so an EMPTY shard vector is all a load needs to reach +// it. That is why the guard sits in `ModelRegistry::Load` rather than at the +// other pre-load refusal site, `RefuseUnsupportedWeightOffload` +// (`src/vllm/entrypoints/model_loader.cpp:1536`), which needs a directory on +// disk. +#include + +#include +#include + +// `model_registry.h` only forward-declares `SafetensorsFile`, and building the +// empty shard vector the loads below hand to `ModelSource` needs the complete +// type. +#include "vllm/model_executor/model_loader/safetensors_reader.h" +#include "vllm/model_executor/models/model_registry.h" +#include "vllm/transformers_utils/hf_config.h" + +namespace { + +// A registered architecture, so `ModelRegistry::Resolve` succeeds and the load +// gets far enough to be refused for the QUANTIZATION rather than for the +// architecture. Pinned by `tests/vllm/models/test_model_registry.cpp:88`. +constexpr const char* kArch = "Qwen3_5ForConditionalGeneration"; + +vllm::HfConfig ConfigWithQuant(const nlohmann::json& quant, bool nested) { + vllm::HfConfig config; + config.architectures = {kArch}; + config.model_type = "qwen3_5"; + nlohmann::json doc = nlohmann::json::object(); + doc["architectures"] = nlohmann::json::array({kArch}); + if (nested) { + doc["text_config"] = nlohmann::json::object(); + doc["text_config"]["quantization_config"] = quant; + } else { + doc["quantization_config"] = quant; + } + config.raw = std::move(doc); + return config; +} + +// The message `ModelRegistry::Load` fails with, or "" when it does not throw. +// The load is EXPECTED to throw in every case here: a config carrying no +// weights cannot produce a model. What each case asserts is WHICH sentence +// comes back, which is the whole subject of issue #1166. +std::string LoadFailureMessage(const vllm::HfConfig& config) { + const std::vector shards; + const vllm::ModelSource source = vllm::ModelSource::FromSafetensors(shards); + try { + std::unique_ptr model = + vllm::ModelRegistry::Load(config, source); + return ""; + } catch (const std::exception& e) { + return e.what(); + } +} + +bool Names(const std::string& haystack, const std::string& needle) { + return haystack.find(needle) != std::string::npos; +} + +// The block-wise config the real checkpoint ships, measured from +// `Qwen/Qwen3.8-27B-FP8` at revision `017b9c7af6b5689d5dd426a76e0bc077eb5ca20a` +// on 2026-08-17: `quant_method` fp8, `weight_block_size` [128, 128], +// `activation_scheme` dynamic, `fmt` e4m3. +nlohmann::json BlockWiseQuantConfig() { + nlohmann::json quant = nlohmann::json::object(); + quant["quant_method"] = "fp8"; + quant["fmt"] = "e4m3"; + quant["activation_scheme"] = "dynamic"; + quant["weight_block_size"] = nlohmann::json::array({128, 128}); + return quant; +} + +} // namespace + +TEST_CASE("fp8 block quant: a block-wise checkpoint is refused by name") { + const std::string message = + LoadFailureMessage(ConfigWithQuant(BlockWiseQuantConfig(), false)); + REQUIRE_FALSE(message.empty()); + + // The KEY, so a reader can grep their own config.json for it. + CHECK(Names(message, "weight_block_size")); + // The VALUE quoted back, so the message is about THIS checkpoint. + CHECK(Names(message, "128")); + // WHAT is missing, named. "unsupported" alone makes the next person + // re-derive it. + CHECK(Names(message, "block-wise")); + // WHAT WOULD work, so the refusal points somewhere. + CHECK(Names(message, "per-tensor")); + // The owing pointer. Deleting it reds this line. + CHECK(Names(message, "1166")); + + // THE REFUSAL MUST PREEMPT THE WEIGHT LOADER, and this is the assertion that + // can actually fail. Asserting the absence of the real checkpoint's + // `tensor not found: ...weight_scale` would be a MUTE SWITCH here: this + // config carries no shards, so without a guard the load does not reach that + // sentence either. It reaches a different one. Measured on the RED run of + // this same file, an unguarded load of this config fails with + // `no Qwen3.5 backbone tensors found` from `qwen3_5_weights.cpp:1190`, which + // means `factory.load_weights` RAN. So requiring that sentence to be absent + // proves the guard fires before the loader rather than after it, and it is + // exactly what reds when the call site is deleted. + CHECK_FALSE(Names(message, "backbone tensors found")); +} + +TEST_CASE("fp8 block quant: the nested text_config spelling is refused too") { + // A multimodal wrapper can nest `quantization_config` under `text_config`. + // `Qwen/Qwen3.8-27B-FP8` uses the TOP-LEVEL spelling (measured: `text_config` + // carries no `quantization_config`), so this case covers the shape the next + // checkpoint can arrive in and would otherwise load straight past the guard. + const std::string message = + LoadFailureMessage(ConfigWithQuant(BlockWiseQuantConfig(), true)); + REQUIRE_FALSE(message.empty()); + CHECK(Names(message, "weight_block_size")); + CHECK(Names(message, "block-wise")); + CHECK(Names(message, "1166")); +} + +TEST_CASE("fp8 block quant: a per-tensor fp8 config is not refused as block-wise") { + // WITHOUT this case the gate passes for a refusal that fires on EVERY fp8 + // checkpoint, which would break the per-tensor arm this tree does implement + // and gates on the 27B and 35B. The load still fails, because there are no + // weights, but it must fail for a different reason. + nlohmann::json quant = BlockWiseQuantConfig(); + quant.erase("weight_block_size"); + const std::string message = LoadFailureMessage(ConfigWithQuant(quant, false)); + REQUIRE_FALSE(message.empty()); + // The sentence a NON-block-wise load fails with here is evidence, not noise: + // it is what the negative controls are asserting the ABSENCE of markers in, + // and a control that never says what it saw cannot be falsified. + MESSAGE("per-tensor fp8 load failed with: " << message); + CHECK_FALSE(Names(message, "weight_block_size")); + CHECK_FALSE(Names(message, "block-wise")); + CHECK_FALSE(Names(message, "1166")); +} + +TEST_CASE("fp8 block quant: a null or empty weight_block_size is not block-wise") { + // Upstream treats the key as absent when it is null + // (`Fp8Config.from_config`, `vllm/model_executor/layers/quantization/fp8.py:161` + // reads it with a default of None), so a null must not refuse a checkpoint + // that is really per-tensor. + nlohmann::json null_quant = BlockWiseQuantConfig(); + null_quant["weight_block_size"] = nullptr; + const std::string null_message = + LoadFailureMessage(ConfigWithQuant(null_quant, false)); + REQUIRE_FALSE(null_message.empty()); + CHECK_FALSE(Names(null_message, "block-wise")); + + nlohmann::json empty_quant = BlockWiseQuantConfig(); + empty_quant["weight_block_size"] = nlohmann::json::array(); + const std::string empty_message = + LoadFailureMessage(ConfigWithQuant(empty_quant, false)); + REQUIRE_FALSE(empty_message.empty()); + CHECK_FALSE(Names(empty_message, "block-wise")); +}