vllm: alt-stack community recipe for Qwen3.8-27B-FP8 on 2x Arc Pro B70 (vLLM 0.27.2rc1, #53059-lineage scheduler gate, 0.1.14.dev544 kernels) — DRAFT, pending week-long soak

[dominick253]

Alt-stack community recipe: Qwen3.8-27B-FP8 on 2x Arc Pro B70 (vLLM 0.27.2rc1 fork)

DRAFT / EXPERIMENTAL — supersets intel/llm-scaler-vllm:0.26.0-b2 as a community-provided alternative runtime assembly. Pending a week-long live-traffic soak test; soak has NOT completed. Not evidence of stability until this PR is un-drafted with soak results.

This PR does not replace Intel's packaged images and does not re-submit the #677/#683 pointer fix (already merged on main). It contributes, under vllm/community/alt-recipes/qwen38-27b-fp8-r272/:

• A documented runtime built from vLLM 0.27.2rc1.dev77+gac7509e2b.xpu / torch 2.13.0+xpu — a newer fork point than 0.26.0-b2 (0.26.1.dev0 / torch 2.12.0) — with

  • the merged pointer-fix content (mamba_utils unsigned->signed pointer reinterpretation) regenerated as a standalone verified diff against this fork point, plus its upstream regression test;
  • a per-request prefill/decode classification in _is_uniform_decode (shape-alias prefill batches no longer dispatch to the uniform-decode path — upstream vllm-project#53059 lineage, vllm-project#53051), as gpu_model_runner_53059.diff + RED/GREEN test;
  • vllm-xpu-kernels 0.1.14.dev544 (upstream gemma4-31B api call failed with intel/llm-scaler-vllm:0.21.0-b1 on 4 x B70 with both fp8 and sym_int4 #544 sliding-window conv state, Refactor/omni keep compiler #552 mixed spec/non-spec GDN batch) with untouched stock kernel families preserved.

• A production-identical launch script (launcher.sh), captured live and reduced (no LAN IPs, no paths outside the recipe, no secrets), full env list and docker run flags included.

• A matched A/A direct speed comparison (pointer-patch vs no-pointer-patch) at fixed prompt bytes (8063→589 tokens), seed, sampling, reasoning effort, verified cached_tokens=0, one-request-verified concurrency, and identical output hashes across both arms:

  Median (3 runs each arm)	no-pointer	with-pointer	Δ
  Server prefill tok/s	2419.88	2417.88	-0.08%
  Streamed decode tok/s	50.02	50.08	+0.11%
  TTFT s	3.344	3.346	+0.05%

• Why the fork matters: the 0.26.x fork lacks the #53059-lineage scheduler fix. We recorded deterministic first-token corruption from the 0.26.0-b2 image with this checkpoint on 2026-09-08 across MTP on/off, f16/bf16 SSM cache, and explicit/auto FP8 combinations. This PR does not re-run that evaluation; it is documented so maintainers can reproduce independently if desired.

• Explicit qualification table with PASS/NOT-Run entries. Explicitly NOT COLLECTED: MTP acceptance-rate study, vision qualification, matched counting/capitals probes. Explicitly pending: 168-hour live-traffic soak with an automated corruption-watcher client; the soak hasn't started accumulating under this exact image tag yet (paused deliberately for the isolated A/A run, contributing to nothing yet).

Why a draft

The 168h soak client (a separate stdlib streaming recorder, not a giant benchmark) has not completed against this exact image. Everything here is reproducible now, but nothing should be interpreted as "this image is production-stable" until the soak exits clean. Drafting keeps upstream reviewers Pastoroo to review methodology without reading it as a readiness claim.

Test plan

• Week-long live-traffic soak (!!!!!-detector + degenerate-output hashes) on the ptr683 image; expect 0 incidents across the 168h window when run continuously, plus daily health and known-answer (17*23=391) probes.
• Vision (mmproj) qualification, currently NOT RUN.
• Matched MTP acceptance-rate study, currently NOT COLLECTED.
• Cross-check on a 1-GPU (B60) host, currently NOT RUN.
• Re-run reviewer request for any probe in the checklist, on your own hardware, using only launcher.sh + patches in this directory.

Signed-off-by: Dominick Pescetto dominick253@gmail.com (dominick253)

[noopduck]

I will definiitely test this! I am just delving into the insane memory utilization of vLLM, isn't i fair to assume that 64GB SYSTEM RAM should be enough, i mean the model is going into VRAM right :)

But regardless, I will test this for sure man <3

Started building the docker image of this pr branch

[dominick253]

I will definiitely test this! I am just delving into the insane memory utilization of vLLM, isn't i fair to assume that 64GB SYSTEM RAM should be enough, i mean the model is going into VRAM right :)

But regardless, I will test this for sure man <3

Started building the docker image of this pr branch

I understand your frustration. Still in the early days compared to Nvidia stack. But the performance is there most definitely and the price per GB is un matched.

I would say maybe make sure to set the offload flag to 0 instead of leaving it out?

Not sure. Personally I've uses all my Astra useage and heavily lean on the GitHub repo Intel b70 optimization lab. Please point your AI to that repo and find a good recipe there. They do great work for the recipes themselves. This repo is more for the inference stack itself.

Trust me once you get it setup right it'll be worth the hassle.

[noopduck]

I will definiitely test this! I am just delving into the insane memory utilization of vLLM, isn't i fair to assume that 64GB SYSTEM RAM should be enough, i mean the model is going into VRAM right :)
But regardless, I will test this for sure man <3
Started building the docker image of this pr branch

I understand your frustration. Still in the early days compared to Nvidia stack. But the performance is there most definitely and the price per GB is un matched.

I would say maybe make sure to set the offload flag to 0 instead of leaving it out?

Not sure. Personally I've uses all my Astra useage and heavily lean on the GitHub repo Intel b70 optimization lab. Please point your AI to that repo and find a good recipe there. They do great work for the recipes themselves. This repo is more for the inference stack itself.

Trust me once you get it setup right it'll be worth the hassle.

I alreay got a good feel for the potential, but on the sergiioB stuff which absolutely flies on a single stream but on crashes on a double stream even in TP=2. So for me personally it's just a matter of having to buy more CPU RAM vs. something that should be solved in software :) It's hard to tell at times, but i dont regret these two B70's at all, they are good!! I will investigate the offload flag for sure, it sounds like a potential game changer!

[noopduck]

Thanks for the detailed writeup and for including gpu_model_runner_53059.diff as a standalone artifact — it made this quick to try in isolation.

What we tested: whether gpu_model_runner_53059.diff alone, backported directly onto the stock intel/llm-scaler-vllm:0.26.0-b2 image (i.e. without the accompanying vllm-xpu-kernels 0.1.14.dev544 rebuild or the 0.27.2rc1 fork point), is enough to fix the corruption on 0.26.0-b2's own kernel stack (Intel's ESIMD GDN ENSEMBLE integration).

Setup:

• FROM intel/llm-scaler-vllm:0.26.0-b2, patch applied directly to /opt/venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py using the diff from this PR.
• The diff applied cleanly (Hunk #2 succeeded ... with fuzz 1, offset only), the patched file parses, and the referenced input_batch.num_computed_tokens_cpu / .num_prompt_tokens fields exist under those exact names in this vLLM build (0.26.1.dev0+g568afb3a1.d20260907), so this isn't a version-mismatch no-op.
• Serving Qwen/Qwen3.8-27B-FP8, TP2 on 2x Arc Pro B70, --gpu-memory-utilization 0.70 --max-model-len 8192 --max-num-seqs 2 --max-num-batched-tokens 2048, cudagraph capture enabled (VLLM_XPU_ENABLE_XPU_GRAPH=1, no --enforce-eager), no speculative decoding.

Result: still corrupted. Repeated raw /v1/completions calls on a plain prompt ("The capital of France is") returned garbled/degenerate text on all 3 attempts, and /v1/chat/completions returned content: null with completion_tokens: 1 — the same failure signature as the unpatched image (and as described in this PR's own motivation section).

For comparison, on the unmodified 0.26.0-b2 image, --enforce-eager (disabling cudagraph capture entirely, no scheduler patch) reliably produces correct output across repeated prompts, which is consistent with the corruption path running through a captured graph rather than being purely an artifact of the scheduler misclassification alone.

Takeaway (happy to be corrected): on 0.26.0-b2's own kernel stack, the _is_uniform_decode gating fix by itself doesn't appear sufficient to eliminate the corruption — which suggests it may be interdependent with the vllm-xpu-kernels 0.1.14.dev544 rebuild (or something else in the newer fork point) rather than a standalone fix that also patches forward onto 0.26.0-b2. Wanted to flag this before anyone assumes a simple backport of just this diff would remediate 0.26.0-b2 directly. Happy to share full logs/repro steps if useful for the soak test writeup.