[Engine] gfx1100 + gfx1201 layer split: prefill dies with gr_mix_r: no kernel image is available on engine 0.1.52 (HIP) #4431

The problem

On a dual-GPU AMD system (RX 7900 XTX, gfx1100, + RX 9070 XT, gfx1201), a 2-stage layer split starts and loads both cards, then the first prompt kills the process during prefill:

strata: prompt experts on the fused int8 kernels (STRATA_PF_FUSED=1, #136)
prefill gr_mix_r: no kernel image is available for execution on the device
prefill mmq: this build has no pointer-table MMQ (STRATA_MMQ_TAB)

The engine exits before generating a single token.

The result

A gfx1100 + gfx1201 pair that works fine on a single card each (and is benchmarked as supported in #3933) cannot run the Qwen3.8-Flash-Next native pack with --layer-split auto: every multi-chunk prompt on the mixed split fails the same way, and there is no flag combination I have found that keeps it alive. Two identical reproductions are in the same log (runs of 2026-10-27 21:03 and 21:05): the first ends with both lines (log 248-250), the second with the gr_mix_r line (log 325) where the capture stops.

Reproduction

Local source build, engine 0.1.52, backend = hip, archs = ["gfx1100", "gfx1201"] (engine/BUILD.json), Linux.

Command (no STRATA_* environment variables set):

strata --serve --pack <packs/iq3_xxs> \
  --native <.../Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf> \
  --ple-gguf <.../Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00002-of-00002.gguf> \
  --expert-profile <.../expert-profile.bin> --expert-cache auto --prefill auto \
  --spec 4 --spec-min-p 0.5 --mtp <.../mtp/rt> \
  --max-context 131072 --kv int8 --vision --layer-split auto

What the log shows before the crash:

• layer split across 2 GPUs: CUDA0, then CUDA1 (split auto) — the two stages (the log labels them CUDA0 32 SMs @ 2.40 GHz and CUDA1 42 SMs @ 2.02 GHz; these are the engine's device labels, one gfx1100 and one gfx1201) take layers 0-12 and 13-47 + head respectively, one hand-off per window.
• prompt experts on the fused int8 kernels (STRATA_PF_FUSED=1, #136) — the default for this build.
• Then the two fatal lines above, and the process exits.

Validation (what I ran):

• Passed: the same model on one card (either card) completes 10K-13K-token prompts (e.g. prompt 13816 tokens = 0 reused + 13816 read in 12698 ms (1088.0 tok/s), 87 generated), so each card alone is fine.
• Failed: both --layer-split auto runs above, identical error.
• Not run: reversing the stage order (9070 XT first), STRATA_PF_FUSED=0, STRATA_HIP_WMMA=0, and a single-chunk prompt (< 8192 tokens) — I can run those if they would help narrow it down.

The approach (static reading of 0.1.52 source, unconfirmed)

This looks like the same class as the now-closed #690 and #1772 (both "no kernel image is available" on a mixed-arch split), plus the mechanism described in still-open #2706: per-stage architecture decisions are cached process-wide.

• The mixed-arch prefill kernels make their per-stage arch decision process-globally: gr_upmix's dispatch reads the current device's arch into a static const bool gfx11 (src/prefill/kernels.cu:2584), so whichever device initializes it first defines the value for every stage — the exact mechanism [Engine] STRATA_HC_UPMIX on a gfx11 + gfx12 layer split corrupts prefill on the gfx12 stage (gr_upmix caches its arch check from the first device); still on 0.2.0-beta #2706 describes for STRATA_HC_UPMIX.
• native PLE postops launch: no kernel image is available #1772's maintainer note: "'no kernel image' is the HIP/ROCm runtime saying a GPU kernel was not compiled for the card that runs it, and the PLE post-ops kernel is simply the first one that reports it (the check is a late cudaGetLastError, so it can report an earlier launch that failed)." gr_mix_r's check("gr_mix_r") is such a late error check, so the line naming gr_mix_r is the first sync point after whatever failed, not proof that gr_mix_r is the bad launch.
• The fatal STRATA_MMQ_TAB line is printed only by the pointer-table MMQ stubs (src/prefill/moe_mmq.cu:288 and :637), which std::exit(1). I could not confirm how that path is reached on this build: with STRATA_MMQ_DIRECT unset, direct_mode() is 0, gu.tab is never set (prefill.cpp:5330/5347), and the direct_mode() announcement ("prompt MMQ reads resident experts in place") never appears in the log. So either (a) the "no kernel image" failure cascades into this stub through a path I have not traced, or (b) something in my launch environment enables the pointer-table path despite the missing announcement. I have checked ~/.bashrc, ~/.profile and the launch config for STRATA_MMQ_* and found none.

The limits

• Static analysis plus one log from a single machine; I have no gfx11 + gfx12 hardware beyond this pair and have not instrumented which stage or which kernel is the real first failure, nor which code path takes the fatal exit.
• CUDA0/CUDA1 in the log are the engine's device labels; both are AMD cards under the HIP backend (no NVIDIA hardware involved).
• The full strata-iq3_xxs.log is available if you want it; I have left it out of the issue body.

Related: #690, #703, #1208, #1772 (closed; still reproducing on gfx1102 + gfx1201 per its last comment), #2706, #2230, #3933.