feat(MODEL-FP8-BLOCK-WEIGHT): a block-wise FP8 checkpoint now LOADS, and says by name that it cannot run yet (#1189)

docs/FEATURES.md

@@ -79,7 +79,7 @@ are our reading of their documented behavior, not measurements.
 | GPTQ | ◐ CPU dequant | ✅ | ✅ | ☐ |
 | MXFP4 compressed-tensors | ◐ W4A16 Marlin, mem 2.63x less. gate_up FUSION + decode-graph default-ON; #44 3/3, 32B 6/6. **`VT_MARLIN_DENSE` DEFAULT-ON** (`KERNEL-MARLIN-DENSE-EXEC`): dense marlin 48-CTA, byte-faithful, beats MoE (c8 0.969) | ✅ | ✅ | ☐ |
 | fp8 weights, per-tensor scale | ✅ | ✅ | ✅ | ☐ |
-| Block-wise (fine-grained 128x128) FP8, the `weight_scale_inv` layout | ☐ REFUSED BY NAME at load (#1166): `Qwen/Qwen3.8-27B-FP8` declares `weight_block_size` [128, 128] and this build is per-tensor FP8 only ([spec](../.agents/specs/fp8-blockwise-refusal.md)) | ✅ | ✅ | ☐ |
+| Block-wise (fine-grained 128x128) FP8, the `weight_scale_inv` layout | ◐ LOADS, cannot run (#1189 M3): weight + `cdiv` scale rung + config/tensor cross-check; BF16 scale widened to f32. Linear method is M4, so `Prepare` refuses by name ([spec](../.agents/specs/model-fp8-block-weight.md)) | ✅ | ✅ | ☐ |
 | Per-tensor FP8 W8A8 linear is a shared seam any model can bind | ✅ `models/dense_fp8_gemm.h` + `layers::Fp8W8A8LinearMethod` (#940), bound via `layers::MakeLinearMethod`. One definition, CUDA only ([spec](../.agents/specs/vt-fp8-shared-seam.md)) | ✅ `Fp8LinearMethod` | ✅ | ☐ |
 | FP8 W8A8 works on a CUDA arch without `cutlass-fp8` | ✅ `vt::QuantFp8Static` registers from an unconditional TU (#960); sm_110 measured ([spec](../.agents/specs/vt-fp8-quant-arch-gate.md)) | ✅ | ✅ | ☐ |
 | fp8-tower GDN `in_proj` emits bf16, unlocking packed GDN decode | ◐ `VT_GDN_FP8_IN_BF16` + `VT_GDN_PACKED_DECODE_FP8_TOWER` (inert alone), both default **OFF**, ungated (#339) ([spec](../.agents/specs/perf-fp8-alpha-fold.md)) | ✅ bf16 `out_dtype` | ☐ | ☐ |

docs/USAGE.md

@@ -595,32 +595,41 @@ quantizes the activation once; a checkpoint whose scales differ keeps the two
 separate GEMMs automatically. `VT_GDN_MERGED_QKVZ_FP8=0` restores the two GEMMs
 in the same binary.
 
-### Block-wise FP8 is refused at load
+### Block-wise FP8 loads and does not run yet
 
-This build reads per-tensor FP8, where one scale covers a whole weight. It does
-not read block-wise FP8, also called fine-grained FP8, where one scale covers
-each 128x128 block of the weight. A block-wise checkpoint declares
-`quantization_config.weight_block_size` in its `config.json`, and it stores its
-scales under `weight_scale_inv` rather than under `weight_scale`.
+Block-wise FP8, also called fine-grained FP8, keeps one scale for each 128x128
+block of a weight rather than one scale for the whole weight. A block-wise
+checkpoint declares `quantization_config.weight_block_size` in its
+`config.json` and stores its scales under `weight_scale_inv` rather than under
+`weight_scale`.
 
 `Qwen/Qwen3.8-27B-FP8` is such a checkpoint. At revision
 `017b9c7af6b5689d5dd426a76e0bc077eb5ca20a` it declares `weight_block_size`
 `[128, 128]` with `activation_scheme` `dynamic`, and it stores
 `self_attn.q_proj.weight` as `F8_E4M3` `[12288, 5120]` beside
 `self_attn.q_proj.weight_scale_inv` as `BF16` `[96, 40]`.
 
-Loading it stops with a message that names the key:
+That checkpoint now LOADS. The weights are read into a block-wise FP8 weight,
+the `BF16` scale is widened to `F32` by value the way vLLM widens it, and the
+config is cross-checked against the tensors so a disagreement is named rather
+than guessed at. Nothing can execute the weight yet, so the model refuses to
+finish preparing:
 
 ```text
-quantization_config.weight_block_size [128, 128] selects block-wise
-(fine-grained) FP8, which is not implemented. This build implements per-tensor
-FP8 only.
+block-wise (fine-grained) 128x128 FP8 weights LOADED for
+model.layers.0.self_attn.q_proj and nothing in this build can execute them
 ```
 
-The refusal is deliberate. Nothing is wrong with that checkpoint, and the
-missing arm is in this project. To run the same model here, use a per-tensor
-FP8, BF16, NVFP4, or GGUF checkpoint of it. Issue
-[#1166](https://github.com/mudler/vllm.cpp/issues/1166) tracks the port.
+Two block-wise configurations are refused earlier, at load, because no build
+here implements them: an `activation_scheme` other than `dynamic`, and a
+`weight_block_size` other than `[128, 128]`. Both messages name the key and the
+value your `config.json` declares.
+
+Nothing is wrong with those checkpoints; the missing arm is in this project. To
+run the same model today, use a per-tensor FP8, BF16, NVFP4, or GGUF checkpoint
+of it. Issue [#1189](https://github.com/mudler/vllm.cpp/issues/1189) tracks the
+remaining milestones, and
+[#1166](https://github.com/mudler/vllm.cpp/issues/1166) is the original report.
 
 ### A per-tensor scale has to be one F32 number
 

include/vllm/model_executor/layers/quantization/fp8_block_quant.h

@@ -1,34 +1,41 @@
-// Block-wise (fine-grained) FP8 detection and its named refusal.
+// Block-wise (fine-grained) FP8: the quantization-config reader, the supported
+// shape, and the named refusals for everything else.
 //
 // UPSTREAM (ported FROM, ground-every-impl rule), pinned vLLM
 // `5559679229bc961848b121ccdeaa8fa5d79bec98`:
-//   vllm/model_executor/layers/quantization/fp8.py:161
-//     Fp8Config.from_config — reads `weight_block_size` out of the checkpoint's
-//     quantization config. Absent means None, which means per-tensor.
-//   vllm/model_executor/layers/quantization/fp8.py:115-132
+//   vllm/model_executor/layers/quantization/fp8.py:157-172
+//     Fp8Config.from_config — reads `weight_block_size`, `activation_scheme`,
+//     `ignored_layers`, and `modules_to_not_convert` as the fallback for the
+//     ignore list. Absent means None, which means per-tensor.
+//   vllm/model_executor/layers/quantization/fp8.py:115-131
 //     Fp8Config.__init__ — validates it: an fp8-serialized checkpoint, exactly
-//     2 dimensions, and a dynamic activation scheme.
+//     2 dimensions, and a dynamic activation scheme. Each of those three is
+//     mirrored below as a refusal.
 //   vllm/model_executor/layers/quantization/fp8.py:297-298
 //     Fp8LinearMethod — `self.block_quant = self.weight_block_size is not None`
-//     is the whole dispatch, and this tree has no arm to dispatch TO.
+//     is the whole dispatch.
 //   vllm/model_executor/layers/quantization/fp8.py:378-379, :511
 //     the block scale registers as `weight_scale_inv`, not `weight_scale`, and
 //     the name is strictly conditional on block quant.
+//   vllm/model_executor/layers/quantization/utils/quant_utils.py:510-524,568-569
+//     is_layer_skipped — the DEFAULT match is `prefix_full_match`, i.e. exact
+//     membership of the module prefix in the ignore list, not a substring test.
 //
-// WHY THIS FILE EXISTS. `include/.../quantization/fp8.h` mirrors the PER-TENSOR
-// arm and says so on its first line. A block-wise checkpoint used to enter that
-// arm anyway, because the dense loader branches on the weight dtype alone
-// (`qwen3_5_dense_weights.cpp:479`) and the block-wise weight really is
-// `F8_E4M3`. The load then asked for `<proj>.weight_scale`
-// (`qwen3_5_weights.cpp:458`), which a block-wise checkpoint does not have, and
-// died on `tensor not found`. That sentence is wrong about the world: the
-// checkpoint is complete, and it is this tree that is missing an arm. Issue
-// #1166, spec `.agents/specs/fp8-blockwise-refusal.md`.
+// HISTORY. `469f38395` (#1166) refused the whole scheme by name here, because
+// the dense loader branches on the weight dtype alone and a block-wise weight
+// really is `F8_E4M3`: the projection entered the per-tensor arm, asked for
+// `<proj>.weight_scale`, and died on `tensor not found` — a sentence that is
+// wrong about the world, since the checkpoint is complete and it is this tree
+// that lacked an arm. MODEL-FP8-BLOCK-WEIGHT (#1189 M3, spec
+// `.agents/specs/model-fp8-block-weight.md`) narrows that refusal: a
+// `[128, 128]` `dynamic` checkpoint now LOADS, and only the shapes and schemes
+// nothing here can execute are still refused.
 //
-// SCOPE. Detect and refuse by name. Reading `weight_scale_inv`, applying a
-// 128x128 block scale, and the dynamic per-token activation quant upstream
-// pairs with it are OWED, not done, and the refusal names the issue that owes
-// them.
+// SCOPE. Reading the config and refusing what M3 does not cover. The loader
+// rung lives in `qwen3_5_dense_weights.cpp`, the weight in
+// `models/qwen3_5_weights.h`, and the linear method does not exist yet — #1189
+// milestone M4 owns it, and `PrepareQwen3_5Dense` refuses a loaded-but-unread
+// block weight by name rather than letting the forward produce a number.
 #pragma once
 
 #include <string>
@@ -38,6 +45,31 @@ namespace vllm {
 
 struct HfConfig;
 
+// The block geometry and ignore list a checkpoint declares, once, validated.
+//
+// `block_quant` false means the checkpoint declares no `weight_block_size` and
+// every other field is unset — the per-tensor world, byte-identical to before
+// this row.
+struct Fp8BlockQuantConfig {
+  bool block_quant = false;
+  int64_t block_n = 0;
+  int64_t block_k = 0;
+  // `dynamic` whenever `block_quant` is true; the reader refuses anything else.
+  std::string activation_scheme;
+  // `modules_to_not_convert`, or `ignored_layers` when the checkpoint spells it
+  // that way. `Qwen/Qwen3.8-27B-FP8` ships ~400 entries here, which is why the
+  // loader reads this list rather than inferring exclusion from a dtype probe.
+  std::vector<std::string> modules_to_not_convert;
+
+  // Exact-membership test on the MODULE prefix — the tensor name with its
+  // trailing `.weight` removed. Mirrors `is_layer_skipped`'s default
+  // `prefix_full_match` (`quant_utils.py:517-518,524,568-569`). Upstream first
+  // rewrites the list into vLLM module naming (`fp8.py:151-153`); we match in
+  // CHECKPOINT naming, which is what this loader has, and the two coincide for
+  // every entry that names a real checkpoint module.
+  bool ExcludesModule(const std::string& module_prefix) const;
+};
+
 // The `weight_block_size` a checkpoint declares, empty when it declares none.
 //
 // Mirrors `Fp8Config.from_config`: the key is read from `quantization_config`,
@@ -46,11 +78,30 @@ struct HfConfig;
 // the wrapper shape is exactly the one in play on `Qwen3_5ForConditionalGeneration`.
 std::vector<int64_t> Fp8WeightBlockSizeOf(const HfConfig& config);
 
-// Refuses a block-wise FP8 checkpoint by name, or returns when the checkpoint
-// is not block-wise.
+// Reads and VALIDATES the block-quant config, or returns a default-constructed
+// value when the checkpoint is not block-wise.
 //
-// Throws `std::runtime_error`, the type every other load refusal in this tree
-// throws, so the C API surfaces it as `VLLM_ERR_MODEL_LOAD` unchanged.
+// Throws `std::runtime_error` — the type every other load refusal in this tree
+// throws, so the C API surfaces it as `VLLM_ERR_MODEL_LOAD` unchanged — for a
+// `quant_method` that is not fp8, a `weight_block_size` that is not exactly two
+// dimensions, an `activation_scheme` other than `dynamic`, and a block shape
+// other than 128x128. The first three mirror upstream's own `ValueError`s
+// (`fp8.py:115-131`); the fourth is OUR limit and says so, because #1189's
+// kernel and its CPU reference are both 128x128 and a `[64, 128]` checkpoint
+// would otherwise load into a weight nothing can execute.
+Fp8BlockQuantConfig ReadFp8BlockQuantConfig(const HfConfig& config);
+
+// The pre-load gate, called from `ModelRegistry::Load`. Reads the config for its
+// refusals and discards the result; the loader reads it again where it needs the
+// geometry. Sited on the registry rather than per loader because
+// `weight_block_size` is a property of the checkpoint's quantization config and
+// not of one architecture.
 void RefuseUnsupportedFp8BlockQuant(const HfConfig& config);
 
+// The M3/M4 seam. A block-wise weight LOADS and nothing reads it yet, so the
+// model refuses to be prepared rather than letting a forward fall through to an
+// empty bf16 tensor and produce a fluent wrong answer. `proj` is the projection
+// that carries the weight, so the message names one instead of the class.
+[[noreturn]] void RefuseUnconsumedFp8BlockWeight(const std::string& proj);
+
 }  // namespace vllm

include/vllm/model_executor/models/dense_weight_loaders.h

@@ -117,6 +117,105 @@ inline float ReadF32Scalar(const TensorResolver& get, const std::string& name) {
   return v;
 }
 
+// Block-wise (fine-grained) FP8 projection: `<proj>.weight` F8_E4M3 [N, K]
+// beside `<proj>.weight_scale_inv` [cdiv(N, block_n), cdiv(K, block_k)] ->
+// `Fp8BlockWeight`. MODEL-FP8-BLOCK-WEIGHT, #1189 M3, spec
+// `.agents/specs/model-fp8-block-weight.md`.
+//
+// The fp8 bytes are kept RAW in the on-disk [N=out, K=in] orientation, as
+// `LoadFp8Raw` does for the per-tensor arm: no dequant and no transpose, so the
+// projection costs one byte per element and every scale decision stays inside
+// the GEMM where upstream applies it (per K-block, in the mainloop -- see
+// `.agents/specs/vt-matmul-fp8-block-ref.md`).
+//
+// THE SCALE IS WIDENED TO F32, NOT REINTERPRETED. Upstream allocates the
+// parameter `torch.float32` (`utils/fp8_utils.py:1276,1283-1296`) and loads the
+// checkpoint tensor into it with `self.data.copy_()`
+// (`vllm/model_executor/parameter.py:97`), which CONVERTS. `Qwen/Qwen3.8-27B-FP8`
+// ships the tensor `BF16`, so the resident f32 is the mirror rather than a
+// widening: it is the dtype upstream carries, `vt::MatmulFp8BlockScaled` refuses
+// anything else, and bf16 -> f32 is exact. The switch below has NO default
+// branch that memcpy's bytes, because #1181 landed a guard for exactly that.
+// `vt::LoadUnaligned` because a safetensors tensor's offset is the running byte
+// total of everything ahead of it and can be odd (#627).
+//
+// The shape check is upstream's own: the allocation at `fp8_utils.py:1283-1296`
+// uses `cdiv` on BOTH axes and `parameter.py:95-98` then asserts the loaded
+// tensor matches it exactly. A short final block is legal and must work.
+inline Fp8BlockWeight LoadFp8BlockRaw(const TensorResolver& get,
+                                      const std::string& proj, int64_t block_n,
+                                      int64_t block_k) {
+  VT_CHECK(block_n > 0 && block_k > 0,
+           "dense loader: '" + proj +
+               "' block-wise FP8 needs positive block dimensions, got [" +
+               std::to_string(block_n) + ", " + std::to_string(block_k) + "]");
+  const StTensor& w = get(proj + ".weight");
+  VT_CHECK(w.dtype == "F8_E4M3",
+           "dense loader: '" + proj + ".weight' ships dtype " + w.dtype +
+               ", not the F8_E4M3 a block-wise FP8 weight is");
+  VT_CHECK(w.shape.size() == 2,
+           "dense loader: '" + proj + ".weight' ships shape " +
+               ShapeString(w.shape) +
+               ", not the 2-D [out_features, in_features] a block-wise FP8 "
+               "weight is");
+  Fp8BlockWeight r;
+  r.n = w.shape[0];
+  r.k = w.shape[1];
+  r.block_n = block_n;
+  r.block_k = block_k;
+
+  const std::string scale_name = proj + ".weight_scale_inv";
+  const StTensor& s = get(scale_name);
+  const int64_t rows = (r.n + block_n - 1) / block_n;
+  const int64_t cols = (r.k + block_k - 1) / block_k;
+  VT_CHECK(
+      s.shape.size() == 2 && s.shape[0] == rows && s.shape[1] == cols,
+      "dense loader: '" + scale_name + "' ships shape " +
+          ShapeString(s.shape) + ", not the " +
+          ShapeString(std::vector<int64_t>{rows, cols}) +
+          " a [" + std::to_string(r.n) + ", " + std::to_string(r.k) +
+          "] weight quantized in [" + std::to_string(block_n) + ", " +
+          std::to_string(block_k) +
+          "] blocks needs. Both dimensions round UP (ceil), so a short final "
+          "block still owns a scale");
+  const int64_t count = rows * cols;
+  r.scale = MakeOwned(vt::DType::kF32, {rows, cols});
+  auto* dst = reinterpret_cast<float*>(r.scale.bytes.data());
+  if (s.dtype == "BF16") {
+    VT_CHECK(s.data != nullptr &&
+                 s.nbytes == static_cast<size_t>(count) * sizeof(uint16_t),
+             "dense loader: '" + scale_name +
+                 "' is a BF16 block scale but does not carry " +
+                 std::to_string(count * 2) + " readable bytes");
+    for (int64_t i = 0; i < count; ++i)
+      dst[i] = vt::BF16ToF32(vt::LoadUnaligned<uint16_t>(s.data + i * 2));
+  } else if (s.dtype == "F32") {
+    VT_CHECK(s.data != nullptr &&
+                 s.nbytes == static_cast<size_t>(count) * sizeof(float),
+             "dense loader: '" + scale_name +
+                 "' is an F32 block scale but does not carry " +
+                 std::to_string(count * 4) + " readable bytes");
+    for (int64_t i = 0; i < count; ++i)
+      dst[i] = vt::LoadUnaligned<float>(s.data + i * 4);
+  } else {
+    VT_CHECK(false,
+             "dense loader: '" + scale_name + "' ships dtype " + s.dtype +
+                 ", and a block-wise FP8 scale is read as BF16 or F32 only. "
+                 "Upstream loads it into an F32 parameter with a CONVERTING "
+                 "copy, so a narrower dtype is widened by VALUE; reading its "
+                 "bytes as another dtype is the defect issue #1181 fixed");
+  }
+  MaybeReleaseSourcePages(s.data, s.nbytes);
+
+  r.packed = MakeOwned(vt::DType::kI8, {r.n, r.k});
+  VT_CHECK(w.nbytes == r.packed.bytes.size(),
+           "dense loader: '" + proj +
+               ".weight' block-wise FP8 byte-size mismatch");
+  std::memcpy(r.packed.bytes.data(), w.data, w.nbytes);
+  MaybeReleaseSourcePages(w.data, w.nbytes);
+  return r;
+}
+
 // src bf16 [rows, cols] -> dst bf16 [cols, rows].
 inline void TransposeBf16(const void* src, int64_t rows, int64_t cols,
                           uint16_t* dst) {