docs: pin and reproduce Qwen3.8-2.4T-A95B UD-Q1_0 on one GB10 (#1194) - #1211
Open
localai-bot wants to merge 1 commit into
Open
docs: pin and reproduce Qwen3.8-2.4T-A95B UD-Q1_0 on one GB10 (#1194)#1211localai-bot wants to merge 1 commit into
localai-bot wants to merge 1 commit into
Conversation
The capability landed across #912, #1119 and #1132, and nothing a user reads said how to reach it. AGENTS.md binds the checkpoint pin to docs/USAGE.md, and revision 567d3e6ac26c5474b18311e619c04350fb9a5556 lived only in the agent record: USAGE.md named this checkpoint three times and pinned it zero times, so nobody outside .agents/ could reproduce a single number the project publishes about it. This lands the recipe as a fifth per-model section beside Muse Glimmer 30B, Nemotron-3.5-Lightning-30B, MiniMax-H3 and MiniMax-Music3, and it LINKS the "Streaming routed experts from disk" section rather than restating it, so the schema, the precedence rule, the #1135 reach limits and the #1123 CUDA refusal each keep one home. No docs/examples/ directory. `examples/` in this tree is runnable C++ CMake targets and `docs/` is one topic per file listed in the README index, so a new markdown genre would have had exactly one member and would have duplicated facts that already have owners. Every pin was verified against the HuggingFace API at that revision rather than transcribed: ten shards under UD-Q1_0/, 397,256,393,248 bytes on disk, which reconciles with the 397,245,341,184-byte weight footprint gguf_device_fit.h records, and two X-Linked-ETag sha256 digests. Shard 1 is pinned as well as shard 2 because shard 1 carries no weights - it is the split declaration, so it is the file that decides what the other nine are. Two defects found while reading the section this one sits beside, fixed here. Both streaming examples named -00001-of-00008.gguf, a file that does not exist at that revision. And two invocations wrote the server as ./build/vllm-server; examples/ is added with add_subdirectory and nothing overrides RUNTIME_OUTPUT_DIRECTORY, so it is build/examples/vllm-server, which is the form the same file already used at lines 1401 and 2634. The arithmetic was re-derived rather than copied, because the draft this came from carried GiB values wearing GB labels. 2790 slices at 2,490,368 bytes is 6.95 GB (6.47 GiB) per token, so 44.2 s/token implies 157 MB/s and a ~5 GB/s NVMe puts the ceiling at 1.39 s/token, about 0.72 tok/s - not the 1.29 s and 0.77 tok/s a GiB-for-GB substitution produces. No new measurement is published. The warm 667.0 s TTFT and 44.2 s/token are the figures #1123 already records; the cold 3318 s and 66.7 s/token are what BENCHMARKS.md and the spec record; and the expert slot cache's steady-decode figure stays VOID per #912 F1, stated as void rather than promoted to a loss. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The capability landed across #912, #1119 and #1132, and nothing a user reads
said how to reach it. AGENTS.md binds the checkpoint pin to docs/USAGE.md, and
revision 567d3e6ac26c5474b18311e619c04350fb9a5556 lived only in the agent
record: USAGE.md named this checkpoint three times and pinned it zero times, so
nobody outside .agents/ could reproduce a single number the project publishes
about it.
This lands the recipe as a fifth per-model section beside Muse Glimmer 30B,
Nemotron-3.5-Lightning-30B, MiniMax-H3 and MiniMax-Music3, and it LINKS the
"Streaming routed experts from disk" section rather than restating it, so the
schema, the precedence rule, the #1135 reach limits and the #1123 CUDA refusal
each keep one home.
No docs/examples/ directory.
examples/in this tree is runnable C++ CMaketargets and
docs/is one topic per file listed in the README index, so a newmarkdown genre would have had exactly one member and would have duplicated facts
that already have owners.
Every pin was verified against the HuggingFace API at that revision rather than
transcribed: ten shards under UD-Q1_0/, 397,256,393,248 bytes on disk, which
reconciles with the 397,245,341,184-byte weight footprint gguf_device_fit.h
records, and two X-Linked-ETag sha256 digests. Shard 1 is pinned as well as
shard 2 because shard 1 carries no weights - it is the split declaration, so it
is the file that decides what the other nine are.
Two defects found while reading the section this one sits beside, fixed here.
Both streaming examples named -00001-of-00008.gguf, a file that does not exist
at that revision. And two invocations wrote the server as ./build/vllm-server;
examples/ is added with add_subdirectory and nothing overrides
RUNTIME_OUTPUT_DIRECTORY, so it is build/examples/vllm-server, which is the form
the same file already used at lines 1401 and 2634.
The arithmetic was re-derived rather than copied, because the draft this came
from carried GiB values wearing GB labels. 2790 slices at 2,490,368 bytes is
6.95 GB (6.47 GiB) per token, so 44.2 s/token implies 157 MB/s and a ~5 GB/s
NVMe puts the ceiling at 1.39 s/token, about 0.72 tok/s - not the 1.29 s and
0.77 tok/s a GiB-for-GB substitution produces.
No new measurement is published. The warm 667.0 s TTFT and 44.2 s/token are the
figures #1123 already records; the cold 3318 s and 66.7 s/token are what
BENCHMARKS.md and the spec record; and the expert slot cache's steady-decode
figure stays VOID per #912 F1, stated as void rather than promoted to a loss.
Closes #1194.
Gates
scripts/agent-preflight.sh --stagedgreen excepttest_cpu_x86_llamacpp_floor,which is the box and not this change. Its harness refuses to measure under
contention and reported
NO_QUIET_WINDOW after 30s (busy=110% builders=0 load=26.72 41.76 50.61)with three sibling agents building; the test readsneither changed file (
grep -c 'USAGE.md|issue-index'on it is 0) and this diffis two markdown files.
Individually, from this worktree: check-public-doc-tables OK, check-doc-checkpoint
--staged OK, check-issue-index-append-only OK, check-agent-record OK
(ENGINE=162 MODEL=377 QUANT=82 KERNEL=52 BACKEND=85), check-readme-structure OK,
check-env-doc OK, check-supported-models OK, check-oracle-pins OK,
check-commit-trailers OK, check-commit-style OK.
No build was run and none is needed: no
src/,include/,tests/or CMakefile is touched.
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]