Skip to content

docs: pin and reproduce Qwen3.8-2.4T-A95B UD-Q1_0 on one GB10 (#1194) - #1211

Open
localai-bot wants to merge 1 commit into
mainfrom
docs/qwen38-spark-recipe
Open

docs: pin and reproduce Qwen3.8-2.4T-A95B UD-Q1_0 on one GB10 (#1194)#1211
localai-bot wants to merge 1 commit into
mainfrom
docs/qwen38-spark-recipe

Conversation

@localai-bot

Copy link
Copy Markdown
Collaborator

The capability landed across #912, #1119 and #1132, and nothing a user reads
said how to reach it. AGENTS.md binds the checkpoint pin to docs/USAGE.md, and
revision 567d3e6ac26c5474b18311e619c04350fb9a5556 lived only in the agent
record: USAGE.md named this checkpoint three times and pinned it zero times, so
nobody outside .agents/ could reproduce a single number the project publishes
about it.

This lands the recipe as a fifth per-model section beside Muse Glimmer 30B,
Nemotron-3.5-Lightning-30B, MiniMax-H3 and MiniMax-Music3, and it LINKS the
"Streaming routed experts from disk" section rather than restating it, so the
schema, the precedence rule, the #1135 reach limits and the #1123 CUDA refusal
each keep one home.

No docs/examples/ directory. examples/ in this tree is runnable C++ CMake
targets and docs/ is one topic per file listed in the README index, so a new
markdown genre would have had exactly one member and would have duplicated facts
that already have owners.

Every pin was verified against the HuggingFace API at that revision rather than
transcribed: ten shards under UD-Q1_0/, 397,256,393,248 bytes on disk, which
reconciles with the 397,245,341,184-byte weight footprint gguf_device_fit.h
records, and two X-Linked-ETag sha256 digests. Shard 1 is pinned as well as
shard 2 because shard 1 carries no weights - it is the split declaration, so it
is the file that decides what the other nine are.

Two defects found while reading the section this one sits beside, fixed here.
Both streaming examples named -00001-of-00008.gguf, a file that does not exist
at that revision. And two invocations wrote the server as ./build/vllm-server;
examples/ is added with add_subdirectory and nothing overrides
RUNTIME_OUTPUT_DIRECTORY, so it is build/examples/vllm-server, which is the form
the same file already used at lines 1401 and 2634.

The arithmetic was re-derived rather than copied, because the draft this came
from carried GiB values wearing GB labels. 2790 slices at 2,490,368 bytes is
6.95 GB (6.47 GiB) per token, so 44.2 s/token implies 157 MB/s and a ~5 GB/s
NVMe puts the ceiling at 1.39 s/token, about 0.72 tok/s - not the 1.29 s and
0.77 tok/s a GiB-for-GB substitution produces.

No new measurement is published. The warm 667.0 s TTFT and 44.2 s/token are the
figures #1123 already records; the cold 3318 s and 66.7 s/token are what
BENCHMARKS.md and the spec record; and the expert slot cache's steady-decode
figure stays VOID per #912 F1, stated as void rather than promoted to a loss.

Closes #1194.

Gates

scripts/agent-preflight.sh --staged green except test_cpu_x86_llamacpp_floor,
which is the box and not this change. Its harness refuses to measure under
contention and reported NO_QUIET_WINDOW after 30s (busy=110% builders=0 load=26.72 41.76 50.61) with three sibling agents building; the test reads
neither changed file (grep -c 'USAGE.md|issue-index' on it is 0) and this diff
is two markdown files.

Individually, from this worktree: check-public-doc-tables OK, check-doc-checkpoint
--staged OK, check-issue-index-append-only OK, check-agent-record OK
(ENGINE=162 MODEL=377 QUANT=82 KERNEL=52 BACKEND=85), check-readme-structure OK,
check-env-doc OK, check-supported-models OK, check-oracle-pins OK,
check-commit-trailers OK, check-commit-style OK.

No build was run and none is needed: no src/, include/, tests/ or CMake
file is touched.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]

The capability landed across #912, #1119 and #1132, and nothing a user reads
said how to reach it. AGENTS.md binds the checkpoint pin to docs/USAGE.md, and
revision 567d3e6ac26c5474b18311e619c04350fb9a5556 lived only in the agent
record: USAGE.md named this checkpoint three times and pinned it zero times, so
nobody outside .agents/ could reproduce a single number the project publishes
about it.

This lands the recipe as a fifth per-model section beside Muse Glimmer 30B,
Nemotron-3.5-Lightning-30B, MiniMax-H3 and MiniMax-Music3, and it LINKS the
"Streaming routed experts from disk" section rather than restating it, so the
schema, the precedence rule, the #1135 reach limits and the #1123 CUDA refusal
each keep one home.

No docs/examples/ directory. `examples/` in this tree is runnable C++ CMake
targets and `docs/` is one topic per file listed in the README index, so a new
markdown genre would have had exactly one member and would have duplicated facts
that already have owners.

Every pin was verified against the HuggingFace API at that revision rather than
transcribed: ten shards under UD-Q1_0/, 397,256,393,248 bytes on disk, which
reconciles with the 397,245,341,184-byte weight footprint gguf_device_fit.h
records, and two X-Linked-ETag sha256 digests. Shard 1 is pinned as well as
shard 2 because shard 1 carries no weights - it is the split declaration, so it
is the file that decides what the other nine are.

Two defects found while reading the section this one sits beside, fixed here.
Both streaming examples named -00001-of-00008.gguf, a file that does not exist
at that revision. And two invocations wrote the server as ./build/vllm-server;
examples/ is added with add_subdirectory and nothing overrides
RUNTIME_OUTPUT_DIRECTORY, so it is build/examples/vllm-server, which is the form
the same file already used at lines 1401 and 2634.

The arithmetic was re-derived rather than copied, because the draft this came
from carried GiB values wearing GB labels. 2790 slices at 2,490,368 bytes is
6.95 GB (6.47 GiB) per token, so 44.2 s/token implies 157 MB/s and a ~5 GB/s
NVMe puts the ceiling at 1.39 s/token, about 0.72 tok/s - not the 1.29 s and
0.77 tok/s a GiB-for-GB substitution produces.

No new measurement is published. The warm 667.0 s TTFT and 44.2 s/token are the
figures #1123 already records; the cold 3318 s and 66.7 s/token are what
BENCHMARKS.md and the spec record; and the expert slot cache's steady-decode
figure stays VOID per #912 F1, stated as void rather than promoted to a loss.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

docs: Qwen3.8-2.4T-A95B UD-Q1_0 has no reproduction recipe and no checkpoint pin in USAGE.md

2 participants