Same tokens as vLLM. Same throughput. 140x less to install.
- Continuous batching, paged KV, 37 registered architectures, CUDA / CPU / Metal / Vulkan. No Python anywhere.
+ Continuous batching, paged KV, 38 registered architectures, CUDA / CPU / Metal / Vulkan. No Python anywhere.
@@ -78,7 +78,7 @@ Where that stands today:
band; the other five, 0.7% to 1.7%, are ties. Also **1.18x llama.cpp's prefill** on the same
GGUF file, and **ahead of MLX-LM on prefill** on Apple Silicon. Most other architectures are
correct but speed-pending, and each one says so.
-- **Everything.** 37 registered architectures, 36 tool-parser families, structured output including
+- **Everything.** 38 registered architectures, 37 tool-parser families, structured output including
GBNF, three speculative decoders, image and video and audio input, external KV offload, Prometheus
metrics, and the SGLang knobs, all in a library you can `dlopen`.
@@ -216,7 +216,7 @@ you get on top, most of it borrowed from whichever engine does it best:
sample logprobs.
- **Structured output.** JSON schema, JSON object, regex, choice, and GBNF grammar, enforced in the
engine with a per-step logits bitmask.
-- **Tool calling and reasoning.** 36 tool-parser families (40 accepted names) and 12 reasoning
+- **Tool calling and reasoning.** 37 tool-parser families (41 accepted names) and 12 reasoning
parser names, streaming, selectable with `--tool-call-parser` / `--reasoning-parser`. Chat templates
render through the vendored google/minja engine, the same renderer llama.cpp ships.
- **Multimodal.** Image, video, and audio to text, correctness-complete. Image chat requests are
@@ -249,7 +249,7 @@ InternLM2/3, MiniCPM and MiniCPM3, Yi, OPT, plus Qwen3-VL and Qwen3.6-27B vision
and Voxtral (audio).
The full architecture matrix (37 registered architectures grouped by family)
+The full architecture matrix (38 registered architectures grouped by family)
| Architecture | Example checkpoint | GGUF | Correctness | Speed |
|---|---|:---:|---|---|
@@ -292,7 +292,7 @@ sampler, no logits); upstream is `vllm-project/vllm-omni`. Five conditioning mod
Compressed-tensors NVFP4A16 (W4A16) dense weights also load and compute natively
(RedHatAI/Qwen3-32B-NVFP4A16). Long-context RoPE (YaRN, Llama-3, LongRoPE, dynamic-NTK) and
sliding-window attention are gated feature-positive. The authoritative per-architecture list, bound
-to the C++ registry (all 37 registered architectures with their tested checkpoint and gate, plus the
+to the C++ registry (all 38 registered architectures with their tested checkpoint and gate, plus the
standalone audio/diffusion lanes and the inventoried-but-blocked archs), is in
[docs/FEATURES.md](docs/FEATURES.md); family-by-family lifecycle detail, including what is
hardware-blocked and why, is in [docs/STATUS.md](docs/STATUS.md).
diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md
index 8ce7e32ae..5738c3cd2 100644
--- a/docs/BENCHMARKS.md
+++ b/docs/BENCHMARKS.md
@@ -436,6 +436,7 @@ built on it rather than keeping the flattering one.
| vLLM 0.26 re-benchmark | Pending | Re-run the binding grids on the advanced pin |
| MiniMax-H3 FP4 speed (W-FP4a) | **Measured GB10 (`row/H3-FP4-GPU-E2E`).** Marlin W4A16 byte-exact vs bf16; fp4 a memory win, 0.8x bf16/forward. Real-ckpt fp4-resident e2e RUNS (mp4/wav) | fp4 speed CLOSED. bf16-vs-quant A/B: ENCODER half MEASURED (§8.15), DiT half NOT (no bf16 render exists). Detail: benchmark-record + spec §8 |
| LTX-2.5 axes | Speed `PENDING` (vllm-omni#6066 has no native 2.5), binding oracle too. **SIZE: 320x192/25f completes on GB10, 448x256 does not**; that render was REGISTER-conditioned, not prompted | Wall is the HOST VAE decode, not the pool: drain returns 0.11 GiB, byte-inert. 2 baselines UNRESOLVED (lock). A real-checkpoint PROMPTED render is OWED |
+| IndexTTS-2.5 speech generation | `PENDING`: component correctness only. No loadable model, synthesized audio, or speed number exists | Finish the conditioning, codec, S2Mel, composition, routes, and C ABI. Pin the vLLM-Omni oracle before an end-to-end comparison (#633, #634) |
| MiniMax-H3 render coherence (`row/H3-RENDER-CLOSE` #77) | **CLOSED: a COHERENT scene on GB10.** #70/#74 white was wrong-PARTITION usage (t2va on the ref2va ckpt); t2va on the FL2VA GGUF renders a prompt-matched orange cat (adj-cos 0.95 vs 0.06, no patch-grid) | Verified first: t2va inputs byte-exact vs upstream; CUDA device==host at seq 1920. Follow-up `H3-TASK-PARTITION-GUARD`: the task/partition mismatch now RAISES 1:1 with `_resolve_task` (spec §8.6-8.7) |
| MiniMax-H3 image conditioning (`row/H3-CONDITIONED-E2E`, `row/H3-VISION-SCATTER`, `row/H3-REF2VA-ASSEMBLY`) | **fl2va COHERENT; ref2va assembly bug FIXED+gated.** vision→cond scatter gated; ref2va block-dim double-division fixed + RED-first gated (128 vs 512) + a permanent ref2va DiT-forward rung (§8.10) | grid RE-ATTRIBUTED: with the fix ref2va grids in fp4 AND bf16, and t2va with no refs on the ref2va NVFP4 also grids while FL2VA-GGUF renders, so it is the **NVFP4 checkpoint/loader**, NOT assembly/fp4 (§8.10) |
| MiniMax-H3 Thor render speed (sm_110, no FA2) | **34.6 s/step** at 864x480/124f/50 steps on Q4_K_M, **16.6x** off 574.5 (render ~28 min, was ~8 h). Landed: warp-per-query, chunked warp reduce-scatter (1.76x), bf16 `mma.sync` (9.82x) | Shared-memory K/V tiling (23% SLOWER) and register Q-blocking (-0.8%) both measured and REVERTED: memory traffic is not the bound (one head's K+V is 3.9 MB against 32 MB of L2) |
diff --git a/docs/FEATURES.md b/docs/FEATURES.md
index 26317660c..e11a7233a 100644
--- a/docs/FEATURES.md
+++ b/docs/FEATURES.md
@@ -194,7 +194,7 @@ on the committed fixture); reranking/classify models are not yet registered.
| Video | ✅ correctness-gated | ✅ | ✅ | ☐ |
| Audio | ✅ correctness-gated | ✅ | ◐ | ◐ |
| Video+audio GENERATION (MiniMax-H3 DiT, LTX-2.5 DiT) | ◐ H3: all three modalities COHERENT on Q4_K_M (t2va, fl2va, ref2va; §8.20); the NVFP4 arm carries the patch grid; GGUF/NVFP4/bf16 loaders, pruned too (§8.21). LTX-2.5: a second lane, `SPIKE`, gated at reduced dims | ✅ H3 (vllm-omni, BF16-only, no quantized arm); LTX-2.5 only through the generic diffusers adapter, no native recipe ([vllm-omni#6066](https://github.com/vllm-project/vllm-omni/issues/6066)) | ☐ | ☐ |
-| Speech / audio GENERATION (TTS, vLLM-Omni lane) | ☐ not started. Eight architectures inventoried in `.agents/model-matrix.md`: six from the recipes sweep (#610) plus IndexTTS-2.5 (#634). Inventoried is not supported; all block on the absent `vllm-omni` pin (#633) | ✅ (vllm-omni: MOSS-TTS, Qwen3-TTS, Higgs Audio v3, Voxtral TTS, IndexTTS-2.5) | not assessed | not assessed |
+| Speech / audio GENERATION (TTS, vLLM-Omni lane) | ◐ IndexTTS-2.5 foundations only: speech seam, GPT-2, BigVGAN, partial CAMPPlus. Load refuses because the remaining stages, routes, and C ABI are absent. No model synthesizes audio (#633, #634) | ✅ (vllm-omni: MOSS-TTS, Qwen3-TTS, Higgs Audio v3, Voxtral TTS, IndexTTS-2.5) | not assessed | not assessed |
| MUSIC generation (MiniMax-Music3) | ☐ scoped, not started ([spec](../.agents/specs/minimax-music3.md), #672). Lyrics plus a structured description in, a multi-minute stereo song out | ☐ absent from the pin, from vLLM `main` and from `vllm-omni` alike | ◐ served by SGLang-Omni, a third repository, which loads the NATIVE checkpoint layout | ☐ |
| Multimodal over the OpenAI server | ◐ image request path wired, forward pending | ✅ | ✅ | ◐ |
@@ -203,9 +203,8 @@ API the image **request** path is wired end to end (`ROAD-V1-MM` W1-W3): the
production server attaches the seam at `server_main.cpp:826`. Two residuals keep
it from ✅: the model runner has no mm-forward consuming `Request.mm_features`,
and no image codec is vendored (raw RGB only). Video, audio and multi-image over
-HTTP are not started. Audio **in** is gated; audio **out** does not exist: we
-ship no TTS or speech-generation path on any surface, which is why that row is
-the only ☐ in our column here.
+HTTP are not started. Audio **in** is gated. The TTS lane has reusable components
+and a family seam, but no loadable model, synthesis path, route, or C ABI.
## Speculative decoding
diff --git a/docs/STATUS.md b/docs/STATUS.md
index c93061da2..b4e54fc3b 100644
--- a/docs/STATUS.md
+++ b/docs/STATUS.md
@@ -93,7 +93,7 @@ are `VOID`; no product behavior changed.
Supported-model registry guard (2026-08-06): the public per-architecture list in
[FEATURES](FEATURES.md) is CI-bound to the C++ registry by
-`scripts/check-supported-models.py` (+ mutation test), so the 37
+`scripts/check-supported-models.py` (+ mutation test), so the 38
`REGISTER_VLLM_MODEL` architectures and the FEATURES rows cannot drift.
GCC 12 production-library maintenance (2026-07-31): the two known `-Werror`
@@ -164,6 +164,7 @@ token-for-token correctness against the pinned oracle.
| Structured output | Supported (subset), engine-enforced; xgrammar backend W1 (CPU, not yet production-wired) | JSON schema, JSON object, regex, choice, GBNF grammar. Constrained decoding runs in the production engine (native grammar backend, per-step logits bitmask) and is reachable from OpenAI `response_format` and the C ABI (ABI v2 `structured_*` fields). A second, xgrammar-faithful backend (`XgrammarStructuredOutputBackend`, vLLM's default `auto`) is built behind the same seam: it reuses the native pushdown-FSM/trie matcher (xgrammar's own algorithm) and adds the xgrammar JSON-schema→EBNF converter that preserves property declaration order + `any_whitespace` + the `basic_*` grammar, closing the key-order/whitespace/exotic-schema parity gap. CPU-gated (`test_backend_xgrammar` 6/6, RED-first); production wiring + the `auto` fallback + GPU oracle parity are the named residuals (see `.agents/specs/xgrammar-backend.md`) |
| Tool-call parsing (`TOOLS-PARSER-BREADTH`, PARTIAL) | 37 parser families / 41 accepted names, streaming | 39 of upstream's 44 registered names at the pin, plus 2 that upstream's registry does not carry there (`qwen3`, our alias for the Hermes-JSON Qwen dialect, and `muse_glimmer`) = 41 accepted names; pure-text parsers ported 1:1, the engine-backed families reimplemented from their wire formats. FIVE upstream names are NOT ported, and only ONE of them is a port we could make from vLLM source: `inkling`, a ParserEngine adapter. `minimax_m3` is backed by upstream's Rust crate; `openai` (`GptOssToolParser`) is a declared Harmony stub that raises on both methods; `cohere_command3` and `cohere_command4` are shims over the out-of-tree `cohere_melody` package. So for four of the five there is no grammar in vLLM source to mirror — tracked as W1/W2 in `.agents/specs/tool-parser-breadth.md`, which records the decision each one owes. Upstream's shared `ToolParserTestConfig` harness is also unported, so the per-parser test floor is set file by file rather than enforced (W3). Selection via `--tool-call-parser` (server), `tool_parser` (C ABI), or template auto-detection over a 27-row ordered marker table; native-syntax forced tool_choice where expressible. Tables: docs/BENCHMARKS.md |
| Reasoning parsing (`SAMPLE-REASONING`, ACTIVE, partial coverage) | 12 names, streaming | think_auto (auto-detect default: content unless markers appear), deepseek_r1, deepseek_v3 (passthrough) / holo2 (thinking→R1), mistral ([THINK]), minimax_m2 (+append_think), step3, olmo3, muse_glimmer, and qwen3 / mimo - reasoning split engine-side BEFORE tool parsing, streamed as `reasoning` deltas in the chat chunks. qwen3+mimo are the first ENGINE-BACKED adapter (one reasoning face over the shared `src/vllm/parser/engine/` parser, so `