feat(serving): Qwen3.8-27B frontier lane — catalog row + native MTP spec-decode (#440) - #2310
Merged
Merged
Conversation
…pec-decode via artifact-sibling resolution (#440) Joel's directive 2026-08-15: ggml-org/Qwen3.8-27B-GGUF beats Opus-class on agentic coding (SWE-bench Pro 61.7 vs Opus 4.6 Max 53.4, QwenSWEBench 79.0 vs 63.8) and serves at frontier speed on consumer hardware — get benchmarks killing it. Three pieces, all data-driven, zero per-machine paths: 1. artifacts.rs — `is_main_model_gguf` now excludes `mtp-*.gguf` draft heads. Without this the fix is a live outage: the ggml-org snapshot ships main + mtp + mmproj in ONE dir, the draft downloads AFTER the main weights (live ordering 02:48/02:49), and mtime-newest candidate selection would serve the 1.6GB DRAFT HEAD as the 27B model — the exact #106 clip failure shape its own comment glass-boxes. New `resolve_mtp_draft_for_model` + `find_mtp_draft_beside` are the draft-POSITIVE scan, the same pattern `resolve_mmproj_for_model` established: artifact presence IS the capability signal. 2. llama_server.rs — when a draft head resolves, the spawn adds `--spec-type draft-mtp --spec-draft-model <head> --spec-draft-n-max 4 --spec-draft-p-min 0.7`. MTP heads are trained WITH the model: field-measured decode 40.7 → 60.1 t/s (RTX 4090) for ~0.1GB extra state. No draft file → no flags → byte-identical serving. 3. catalog.rs — the Qwen3.8-27B row: Arch::Qwen35 (fork carries LLM_ARCH_QWEN35 + draft-mtp + the mmproj vision path; installed llama-server build 10196 verified to expose all three), 262k native context, Vision (mmproj ships in-snapshot), hf_source Qwen/Qwen3.8-27B (verified live: repo exists, arch qwen3_5), embedded template + --jinja. Regression test pins the resolution split: draft head never wins main-model resolution regardless of mtime, stays discoverable as the draft, and its absence resolves None (pre-#440 byte-identical serving). 46 model_registry tests green incl. catalog anchor validation with the new row. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Wires ggml-org/Qwen3.8-27B-GGUF (Joel's directive: open models now beat Opus-class on agentic coding — SWE-bench Pro 61.7 vs Opus 4.6 Max 53.4) into serving, with native MTP speculative decode.
artifacts.rs—is_main_model_ggufexcludesmtp-*.ggufdraft heads. Without this the change is a live outage: the snapshot ships main+mtp+mmproj in one dir, the draft downloads AFTER the main weights, and mtime-newest selection would serve the 1.6GB draft head as the 27B model (the exact Build(deps-dev): Bump @types/node from 22.14.0 to 24.2.1 #106 clip failure shape). Newresolve_mtp_draft_for_modelmirrors theresolve_mmproj_for_modelpattern — artifact presence IS the capability signal.llama_server.rs— draft resolves → spawn adds--spec-type draft-mtp --spec-draft-model … --spec-draft-n-max 4 --spec-draft-p-min 0.7(field-measured 40.7→60.1 t/s decode on 4090). No draft → no flags → byte-identical.catalog.rs— Qwen3.8-27B row: Arch::Qwen35, 262k native context, Vision, verifiedhf_source. Installed fork llama-server (build 10196) verified to exposedraft-mtp+ qwen35 + mmproj.Verification
cargo check -p continuum-core --features metal,acceleratecleanmtp_draft_head_never_wins_main_model_resolution_but_resolves_as_draftand catalog anchor validation with the new rowLive proof owed after the next reboot: a lane serving Qwen3.8-27B with the MTP flags visible in its cmdline.
🤖 Generated with Claude Code
https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo