feat(test): add kernel qualification coverage for GDN, MoE, and DeepSeek4

[Graffioh]

What this adds

• A reusable kernel qualification result core with finite checks, maximum absolute error and normalized MSE metrics, explicit pass, fail, and unsupported states, CTest-compatible exit codes, and a stable lucebox.kernel_qualification.v1 JSON report.
• A stateful GDN active-inplace contract. It runs three deterministic steps through the scalar reference and the grouped-column candidate. Each path carries its own state. The contract compares output and mutated state, checks allocation guards, and proves both dispatch routes with launch counters.
• A fused MoE gate/up GLU contract for the ROCmFP2 mixed-expert path. It compares the fused launch with two unfused launches plus a host SwiGLU reference. The inputs include invalid expert IDs. Allocation guards catch writes outside the output.
• A compact DeepSeek4 grouped-MMID contract at speculative-verification width. Separate child processes run the disabled-grouped reference and the grouped candidate because the dispatch flags are process-cached. The contract compares their outputs at the repository's MMQ tolerance and requires exactly one grouped-MMID launch.
• A grouped-MMID launch counter for route proof.
• Build and test coverage in the existing NVIDIA and ROCm jobs. This PR does not add a workflow or service.

Scope

This draft covers ggml-level kernel correctness and dispatch proof. It is separate from #636, which owns model and release qualification through profiles, baselines, evidence, reports, and dashboards.

The runner follows the useful shape of llama.cpp's tests/test-backend-ops.cpp: deterministic inputs, reference and candidate execution, non-finite rejection, and explicit tolerances. Lucebox also checks state changes, allocation guards, route counters, and unsupported hardware.

The current contracts are representative, not a model-by-model test matrix. GDN covers the recurrent kernel used by the Qwen and speculative paths. The MoE and DeepSeek4 contracts cover one focused optimized route each.

What remains

• Run the CUDA and HIP builds and all three GPU contracts on the RTX 3090, GB10, R9700, and Strix runners. This host has no CUDA or HIP toolkit and no GPU.
• Add a FlashPrefill contract after the backend exposes a reliable route witness. Numerical comparison without dispatch proof would allow a silent fallback to pass.
• Qualify full speculative decoding in a separate runtime-level change. That work must cover draft generation, acceptance, KV rollback, sampling, cache state, and multi-step commit behavior. These checks do not belong in a single-kernel contract.
• Add LuceForge orchestration after the GPU contracts pass. Forge can consume the JSON report and combine it with the model profiles and release evidence from feat(test): add manual Qwen 3.6 production qualification #636.
• Keep Qwen model profiles in feat(test): add manual Qwen 3.6 production qualification #636. Current main has Qwen 3.5 and Qwen 3.6 paths, but no Qwen 3.8 backend. This PR does not claim Qwen 3.8 model coverage.

Local validation

Passed on the CPU-only macOS host:

• The core qualification self-test under AddressSanitizer and UndefinedBehaviorSanitizer.
• JSON parsing plus schema, status, and metric assertions.
• Strict C++17 syntax and warning checks for the GDN and MoE/DeepSeek4 qualification sources. The latter used the real project headers and minimal declarations for the unavailable CUDA runtime.
• Workflow YAML parsing.
• git diff --check.

Not run here:

• CUDA or HIP compilation of the modified backend translation units.
• GDN, fused MoE gate/up GLU, or grouped DeepSeek4 MMID execution on a GPU.

The draft stays open until the remote GPU runners are available and these checks pass.

[Graffioh]

@cubic-dev-ai review this PR

[cubic-dev-ai]

@cubic-dev-ai review this PR

@Graffioh I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

All reported issues were addressed across 9 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

4 issues found across 9 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="server/test/kernel_qualification.h">

<violation number="1" location="server/test/kernel_qualification.h:143">
P2: When a qualification produces no output or no metrics, `evaluate` reports a pass because `std::all_of` is true for an empty range. Reject empty comparisons and empty metric lists before declaring a case passed, otherwise missing measurements can produce a false-positive qualification.</violation>
</file>

<file name="server/test/test_kernel_qualification_core.cpp">

<violation number="1" location="server/test/test_kernel_qualification_core.cpp:73">
P3: The precision assertion is too weak to detect loss of double precision. Match the complete expected serialized value so reduced-precision JSON output fails the regression test.</violation>
</file>

<file name="server/test/test_kernel_qualification_gdn.cpp">

<violation number="1" location="server/test/test_kernel_qualification_gdn.cpp:162">
P2: The allocation guards do not surround the in-place `state` buffer, so a state-region overwrite can corrupt adjacent tensors without either guard failing. Place sentinels directly around each guarded allocation, or use a guarded backing buffer for the state and output regions.</violation>

<violation number="2" location="server/test/test_kernel_qualification_gdn.cpp:308">
P2: When the initialized GPU cannot provide the grouped-column specialization, this contract reports `fail` and exits nonzero rather than `unsupported`. Detect the device capability before qualification and return an unsupported case, or register this test only for devices with the required route.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: When a qualification produces no output or no metrics, evaluate reports a pass because std::all_of is true for an empty range. Reject empty comparisons and empty metric lists before declaring a case passed, otherwise missing measurements can produce a false-positive qualification.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/test/kernel_qualification.h, line 143:

<comment>When a qualification produces no output or no metrics, `evaluate` reports a pass because `std::all_of` is true for an empty range. Reject empty comparisons and empty metric lists before declaring a case passed, otherwise missing measurements can produce a false-positive qualification.</comment>

<file context>
@@ -0,0 +1,257 @@
+    result.status = result.route.matched &&
+            std::all_of(result.metrics.begin(), result.metrics.end(),
+                        [](const Metric & metric) { return metric.passed; })
+        ? Status::pass
+        : Status::fail;
+    if (!result.route.matched) {
</file context>

[cubic-dev-ai]

P2: The allocation guards do not surround the in-place state buffer, so a state-region overwrite can corrupt adjacent tensors without either guard failing. Place sentinels directly around each guarded allocation, or use a guarded backing buffer for the state and output regions.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/test/test_kernel_qualification_gdn.cpp, line 162:

<comment>The allocation guards do not surround the in-place `state` buffer, so a state-region overwrite can corrupt adjacent tensors without either guard failing. Place sentinels directly around each guarded allocation, or use a guarded backing buffer for the state and output regions.</comment>

<file context>
@@ -0,0 +1,390 @@
+    ggml_context * context = ggml_init(params);
+    if (!context) return result;
+
+    ggml_tensor * guard_before =
+        ggml_new_tensor_1d(context, GGML_TYPE_F32, GUARD_ELEMENTS);
+    ggml_tensor * q = ggml_new_tensor_4d(
</file context>

[cubic-dev-ai]

P2: When the initialized GPU cannot provide the grouped-column specialization, this contract reports fail and exits nonzero rather than unsupported. Detect the device capability before qualification and return an unsupported case, or register this test only for devices with the required route.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/test/test_kernel_qualification_gdn.cpp, line 308:

<comment>When the initialized GPU cannot provide the grouped-column specialization, this contract reports `fail` and exits nonzero rather than `unsupported`. Detect the device capability before qualification and return an unsupported case, or register this test only for devices with the required route.</comment>

<file context>
@@ -0,0 +1,390 @@
+    const PathRun candidate = run_path(backend, inputs, initial_state, true);
+    ggml_backend_cuda_set_graphs_disabled_override(previous_graphs_disabled);
+
+    const bool route_matched = reference.computed && candidate.computed &&
+        reference.scalar_launches == N_STEPS &&
+        reference.grouped_launches == 0 &&
</file context>

[cubic-dev-ai]

P3: The precision assertion is too weak to detect loss of double precision. Match the complete expected serialized value so reduced-precision JSON output fails the regression test.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/test/test_kernel_qualification_core.cpp, line 73:

<comment>The precision assertion is too weak to detect loss of double precision. Match the complete expected serialized value so reduced-precision JSON output fails the regression test.</comment>

<file context>
@@ -0,0 +1,74 @@
+    REQUIRE_TRUE(
+        json.find("1.2345678901234567e-12") != std::string::npos);
+    REQUIRE_TRUE(
+        json.find("9.87654321098765") != std::string::npos);
+}
</file context>