feat(VT-QUANT-FP8-GROUP): add vt::QuantFp8Group, the dynamic per-token per-group fp8 activation quant, CPU and CUDA (#1189)

[localai-bot]

Block-wise FP8 needs an activation quantizer before it needs anything else, and
this is it: milestone M1 of #1189, the only one of the six that is gateable
without a GPU. vt::QuantFp8Group takes x [M,K] f32 or bf16 and emits the
fp8-e4m3fn bytes plus an f32 [M, K/group_size] scale, one scale per contiguous
run of group_size elements inside a row. It refuses K % group_size != 0 by
name.

Spec: .agents/specs/vt-quant-fp8-group.md,
committed before the implementation. Pinned oracle: vLLM
5559679229bc961848b121ccdeaa8fa5d79bec98, verified as the local checkout's HEAD
before any anchor below was read.

Which upstream arm this mirrors

per_token_group_quant_fp8 reads like a Triton kernel with a C++ fast path and
it is the other way round. On a CUDA-alike platform with a contiguous input it
calls torch.ops._C.per_token_group_fp8_quant and returns
(fp8_utils.py:635-650), so the Triton kernel below it never executes there. The
executing kernel is
csrc/libtorch_stable/quantization/w8a8/fp8/per_token_group_quant.cu:

:47  float local_absmax = eps                       eps SEEDS the reduction
:53  fmaxf(local_absmax, fabsf((float)src))
:68  float y_s = local_absmax / max_8bit            a DIVIDE
:85  fminf(fmaxf((float)src / y_s, min_8bit), max_8bit)   a DIVIDE
:86  DST_DTYPE(q)                                   hardware e4m3 RNE

Two divides. The Triton fallback instead forms _absmax * (1.0 / fp8_max)
(fp8_utils.py:145) under an upstream comment that names the 1-ULP difference it
opens. This is the opposite polarity from vt::QuantFp8Static, which multiplies
by a hoisted reciprocal because that is upstream's shipped form for the static
per-tensor path (common.cuh:62, with the inverse formed by the caller at
common.cu:31). Both kernels now carry the reason beside the code, because each
looks like a defect from the other's point of view.

eps seeds the reduction rather than clamping it afterwards. The two are
numerically identical, and upstream's form makes it visible that an all-zero
group yields 1e-10/448 instead of dividing by zero.

Why the ported test is not enough, measured rather than argued

G2 ports tests/kernels/quantization/test_block_fp8.py:82-118 with its grid
(num_tokens in {7, 2050}, d in {512, 4096, 5120, 13824}, group_size in
{64, 128, 512}, seed 0) and upstream's tolerances: values at rtol=0.15, the
scale at torch.allclose's rtol=1e-5. Mutating the CPU kernel to each of the
Triton arm's two forms, both built with compile_rc=0:

Mutation	G1 (bytes)	G2 (upstream tolerances)
y_s = amax * (1/448)	fails 49/146	fails 6 of 48 shape checks, all at num_tokens=2050, all the value check, never the scale check
x * (1/y_s)	fails 14/146	passes 50/50

So upstream's tolerances catch one of the two forms, on the large shapes only,
by luck of which element lands on an e4m3 boundary, and miss the other entirely.
G1 catches both on every shape because it compares bytes, against a reference
derived from the e4m3fn format by exhaustive nearest-value scan. That is a
different algorithm from this tree's F32ToFp8 rather than a restatement of it,
so agreement is evidence.

Evidence

RED, before the implementation existed: the focused build failed with
'QuantFp8Group' is not a member of 'vt' and 'kQuantFp8Group' is not a member of 'vt::OpId' at 12 sites.

GREEN: test_ops_quant_fp8_group_cpu reports 6 cases, 476 assertions, 0
failed. ctest -R "test_ops_quant_fp8_group_cpu|test_op_provider|test_ops_fp8_cpu"
reports 3/3 passed.

Full preflight (scripts/agent-preflight.sh --fail-on-skip) at the pushed
head: 27 record gates ok, 45 mutation suites ok and 1 FAIL, 3
committed-range gates ok, 2 commit-trailer gates ok, no SKIP — 78 ok
in total. Read the verdict line, not the exit status: this script exits 0 on a
SKIP, which is why --fail-on-skip is on.

The one FAIL is test_cpu_x86_llamacpp_floor, the known load flake of #618,
and it failed with that issue's exact signature rather than a novel one:
NO_QUIET_WINDOW after 30s (busy=111% builders=0 load=102.07 81.60 61.68),
exit 4 instead of 0. The box was carrying a load average of 94 from other
sessions at that moment. Re-run serially once the load fell to 32: 10 tests,
OK, 200.7s. Nothing in this diff is read by that harness — it touches no
llama.cpp path, no benchmark record and no published figure.

Mutation evidence, ten runs. Every run prints git diff --stat and
compile_rc, because a mutation that never applied and a mutation that failed to
build both read as a passing test. Two of the ten did exactly that and are
reported rather than dropped: deleting the eps seed orphans kEps and deleting
the CPU registration orphans the kernel, so both die on -Werror and prove
nothing until re-run in a form that keeps the symbol live. Three results are
worth keeping beyond the table above:

• with the CPU registration deleted, G5 still passed — the refusals live in
  the dispatcher's validation and fire before dispatch, so a refusal test cannot
  stand in for a registration test. Only G1's REQUIRE(OpRegistered(...)) caught
  it;
• removing the eps seed fails G3 at 260 of 269 assertions and leaves G1 green,
  which is why the all-zero row is its own case;
• collapsing the per-group scale to a per-row scale fails both G1 and G3.

The full table is in the spec under ## Evidence.

What is owed, named rather than implied

Nothing reaches this op yet, and that is deliberate. No production entry point
dispatches vt::QuantFp8Group at this merge commit: include/vllm.h does not
expose it, no loader builds an Fp8BlockWeight, and ModelRegistry::Forward has
no block-FP8 linear method to call it from. The wiring is owned by #1189
milestone M4 (Fp8BlockLinearMethod and the Qwen3.5 dense forward), which needs
M2's GEMM and M3's loader rung first. This is the staged-slice exception of
.agents/reachability.md; the row's spec lists it
under ## Owed.

The CUDA arm's on-hardware leg is owed, and it is worth saying exactly where
the line falls. The implementing host has no CUDA toolkit, so this row did not
compile the kernel locally; the cuda-fat-build job
(.github/workflows/ci.yml:669-710) does, building the vllm target with
-DVLLM_CPP_CUDA=ON for ten architectures on every pull request, and
cuda_quant_fp8.cu is in the unconditional CUDA source list. So the compile leg
is gated — by this pull request's own CI, not by a measurement taken before it
opened. Nothing runs the kernel: that job states it uses no GPU and no other
lane executes a CUDA kernel. G6 prints a PENDING banner naming what was not
measured instead of skipping silently, and the run belongs to M5.

Also owed and listed in the spec: the column-major and TMA-aligned scale layouts
(fp8_utils.py:610-628), owed to M5, which is their first consumer; and
use_ue8m0 scale rounding, which the target architecture never selects because
upstream excludes qwen3_5_text from DeepGEMM on family 120
(vllm/utils/deep_gemm.py:27-46).

Also in this change

scripts/check-cuda-op-arch-gate.py now pins kQuantFp8Group beside
kQuantFp8Static. The new kernel has the same property and the same consequence
if it moves: it is a max, two divides and a hardware convert, with no cutlass
dependency, so a CUDA queue must never fall through to the host reference tier
and dereference device pointers — the defect #960 and #844 record. Its miniature
fixture gains the matching registration, because the fixture describes the real
translation unit and the real one now has two. That change was red first: adding
the REQUIRED entry alone made test_baseline_miniature_is_green fail with
expected exactly ONE live RegisterOp(OpId::kQuantFp8Group, ...) found 0, which
is the checker proving the new entry is live.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5-1m [claude-code]