Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@

<p align="center">
<b>Same tokens as vLLM. Same throughput. 140x less to install.</b><br>
<sub>Continuous batching, paged KV, 37 registered architectures, CUDA / CPU / Metal / Vulkan. No Python anywhere.</sub>
<sub>Continuous batching, paged KV, 38 registered architectures, CUDA / CPU / Metal / Vulkan. No Python anywhere.</sub>
</p>

<p align="center">
Expand Down Expand Up @@ -78,7 +78,7 @@ Where that stands today:
band; the other five, 0.7% to 1.7%, are ties. Also **1.18x llama.cpp's prefill** on the same
GGUF file, and **ahead of MLX-LM on prefill** on Apple Silicon. Most other architectures are
correct but speed-pending, and each one says so.
- **Everything.** 37 registered architectures, 36 tool-parser families, structured output including
- **Everything.** 38 registered architectures, 37 tool-parser families, structured output including
GBNF, three speculative decoders, image and video and audio input, external KV offload, Prometheus
metrics, and the SGLang knobs, all in a library you can `dlopen`.

Expand Down Expand Up @@ -216,7 +216,7 @@ you get on top, most of it borrowed from whichever engine does it best:
sample logprobs.
- **Structured output.** JSON schema, JSON object, regex, choice, and GBNF grammar, enforced in the
engine with a per-step logits bitmask.
- **Tool calling and reasoning.** 36 tool-parser families (40 accepted names) and 12 reasoning
- **Tool calling and reasoning.** 37 tool-parser families (41 accepted names) and 12 reasoning
parser names, streaming, selectable with `--tool-call-parser` / `--reasoning-parser`. Chat templates
render through the vendored google/minja engine, the same renderer llama.cpp ships.
- **Multimodal.** Image, video, and audio to text, correctness-complete. Image chat requests are
Expand Down Expand Up @@ -249,7 +249,7 @@ InternLM2/3, MiniCPM and MiniCPM3, Yi, OPT, plus Qwen3-VL and Qwen3.6-27B vision
and Voxtral (audio).

<details>
<summary><b>The full architecture matrix</b> (37 registered architectures grouped by family)</summary>
<summary><b>The full architecture matrix</b> (38 registered architectures grouped by family)</summary>

| Architecture | Example checkpoint | GGUF | Correctness | Speed |
|---|---|:---:|---|---|
Expand Down Expand Up @@ -292,7 +292,7 @@ sampler, no logits); upstream is `vllm-project/vllm-omni`. Five conditioning mod
Compressed-tensors NVFP4A16 (W4A16) dense weights also load and compute natively
(RedHatAI/Qwen3-32B-NVFP4A16). Long-context RoPE (YaRN, Llama-3, LongRoPE, dynamic-NTK) and
sliding-window attention are gated feature-positive. The authoritative per-architecture list, bound
to the C++ registry (all 37 registered architectures with their tested checkpoint and gate, plus the
to the C++ registry (all 38 registered architectures with their tested checkpoint and gate, plus the
standalone audio/diffusion lanes and the inventoried-but-blocked archs), is in
[docs/FEATURES.md](docs/FEATURES.md); family-by-family lifecycle detail, including what is
hardware-blocked and why, is in [docs/STATUS.md](docs/STATUS.md).
Expand Down
1 change: 1 addition & 0 deletions docs/BENCHMARKS.md
Original file line number Diff line number Diff line change
Expand Up @@ -436,6 +436,7 @@ built on it rather than keeping the flattering one.
| vLLM 0.26 re-benchmark | Pending | Re-run the binding grids on the advanced pin |
| MiniMax-H3 FP4 speed (W-FP4a) | **Measured GB10 (`row/H3-FP4-GPU-E2E`).** Marlin W4A16 byte-exact vs bf16; fp4 a memory win, 0.8x bf16/forward. Real-ckpt fp4-resident e2e RUNS (mp4/wav) | fp4 speed CLOSED. bf16-vs-quant A/B: ENCODER half MEASURED (§8.15), DiT half NOT (no bf16 render exists). Detail: benchmark-record + spec §8 |
| LTX-2.5 axes | Speed `PENDING` (vllm-omni#6066 has no native 2.5), binding oracle too. **SIZE: 320x192/25f completes on GB10, 448x256 does not**; that render was REGISTER-conditioned, not prompted | Wall is the HOST VAE decode, not the pool: drain returns 0.11 GiB, byte-inert. 2 baselines UNRESOLVED (lock). A real-checkpoint PROMPTED render is OWED |
| IndexTTS-2.5 speech generation | `PENDING`: component correctness only. No loadable model, synthesized audio, or speed number exists | Finish the conditioning, codec, S2Mel, composition, routes, and C ABI. Pin the vLLM-Omni oracle before an end-to-end comparison (#633, #634) |
| MiniMax-H3 render coherence (`row/H3-RENDER-CLOSE` #77) | **CLOSED: a COHERENT scene on GB10.** #70/#74 white was wrong-PARTITION usage (t2va on the ref2va ckpt); t2va on the FL2VA GGUF renders a prompt-matched orange cat (adj-cos 0.95 vs 0.06, no patch-grid) | Verified first: t2va inputs byte-exact vs upstream; CUDA device==host at seq 1920. Follow-up `H3-TASK-PARTITION-GUARD`: the task/partition mismatch now RAISES 1:1 with `_resolve_task` (spec §8.6-8.7) |
| MiniMax-H3 image conditioning (`row/H3-CONDITIONED-E2E`, `row/H3-VISION-SCATTER`, `row/H3-REF2VA-ASSEMBLY`) | **fl2va COHERENT; ref2va assembly bug FIXED+gated.** vision→cond scatter gated; ref2va block-dim double-division fixed + RED-first gated (128 vs 512) + a permanent ref2va DiT-forward rung (§8.10) | grid RE-ATTRIBUTED: with the fix ref2va grids in fp4 AND bf16, and t2va with no refs on the ref2va NVFP4 also grids while FL2VA-GGUF renders, so it is the **NVFP4 checkpoint/loader**, NOT assembly/fp4 (§8.10) |
| MiniMax-H3 Thor render speed (sm_110, no FA2) | **34.6 s/step** at 864x480/124f/50 steps on Q4_K_M, **16.6x** off 574.5 (render ~28 min, was ~8 h). Landed: warp-per-query, chunked warp reduce-scatter (1.76x), bf16 `mma.sync` (9.82x) | Shared-memory K/V tiling (23% SLOWER) and register Q-blocking (-0.8%) both measured and REVERTED: memory traffic is not the bound (one head's K+V is 3.9 MB against 32 MB of L2) |
Expand Down
7 changes: 3 additions & 4 deletions docs/FEATURES.md
Original file line number Diff line number Diff line change
Expand Up @@ -194,7 +194,7 @@ on the committed fixture); reranking/classify models are not yet registered.
| Video | ✅ correctness-gated | ✅ | ✅ | ☐ |
| Audio | ✅ correctness-gated | ✅ | ◐ | ◐ |
| Video+audio GENERATION (MiniMax-H3 DiT, LTX-2.5 DiT) | ◐ H3: all three modalities COHERENT on Q4_K_M (t2va, fl2va, ref2va; §8.20); the NVFP4 arm carries the patch grid; GGUF/NVFP4/bf16 loaders, pruned too (§8.21). LTX-2.5: a second lane, `SPIKE`, gated at reduced dims | ✅ H3 (vllm-omni, BF16-only, no quantized arm); LTX-2.5 only through the generic diffusers adapter, no native recipe ([vllm-omni#6066](https://github.com/vllm-project/vllm-omni/issues/6066)) | ☐ | ☐ |
| Speech / audio GENERATION (TTS, vLLM-Omni lane) | ☐ not started. Eight architectures inventoried in `.agents/model-matrix.md`: six from the recipes sweep (#610) plus IndexTTS-2.5 (#634). Inventoried is not supported; all block on the absent `vllm-omni` pin (#633) | ✅ (vllm-omni: MOSS-TTS, Qwen3-TTS, Higgs Audio v3, Voxtral TTS, IndexTTS-2.5) | not assessed | not assessed |
| Speech / audio GENERATION (TTS, vLLM-Omni lane) | ◐ IndexTTS-2.5 foundations only: speech seam, GPT-2, BigVGAN, partial CAMPPlus. Load refuses because the remaining stages, routes, and C ABI are absent. No model synthesizes audio (#633, #634) | ✅ (vllm-omni: MOSS-TTS, Qwen3-TTS, Higgs Audio v3, Voxtral TTS, IndexTTS-2.5) | not assessed | not assessed |
| MUSIC generation (MiniMax-Music3) | ☐ scoped, not started ([spec](../.agents/specs/minimax-music3.md), #672). Lyrics plus a structured description in, a multi-minute stereo song out | ☐ absent from the pin, from vLLM `main` and from `vllm-omni` alike | ◐ served by SGLang-Omni, a third repository, which loads the NATIVE checkpoint layout | ☐ |
| Multimodal over the OpenAI server | ◐ image request path wired, forward pending | ✅ | ✅ | ◐ |

Expand All @@ -203,9 +203,8 @@ API the image **request** path is wired end to end (`ROAD-V1-MM` W1-W3): the
production server attaches the seam at `server_main.cpp:826`. Two residuals keep
it from ✅: the model runner has no mm-forward consuming `Request.mm_features`,
and no image codec is vendored (raw RGB only). Video, audio and multi-image over
HTTP are not started. Audio **in** is gated; audio **out** does not exist: we
ship no TTS or speech-generation path on any surface, which is why that row is
the only ☐ in our column here.
HTTP are not started. Audio **in** is gated. The TTS lane has reusable components
and a family seam, but no loadable model, synthesis path, route, or C ABI.

## Speculative decoding

Expand Down
3 changes: 2 additions & 1 deletion docs/STATUS.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ are `VOID`; no product behavior changed.

Supported-model registry guard (2026-08-06): the public per-architecture list in
[FEATURES](FEATURES.md) is CI-bound to the C++ registry by
`scripts/check-supported-models.py` (+ mutation test), so the 37
`scripts/check-supported-models.py` (+ mutation test), so the 38
`REGISTER_VLLM_MODEL` architectures and the FEATURES rows cannot drift.

GCC 12 production-library maintenance (2026-07-31): the two known `-Werror`
Expand Down Expand Up @@ -164,6 +164,7 @@ token-for-token correctness against the pinned oracle.
| Structured output | Supported (subset), engine-enforced; xgrammar backend W1 (CPU, not yet production-wired) | JSON schema, JSON object, regex, choice, GBNF grammar. Constrained decoding runs in the production engine (native grammar backend, per-step logits bitmask) and is reachable from OpenAI `response_format` and the C ABI (ABI v2 `structured_*` fields). A second, xgrammar-faithful backend (`XgrammarStructuredOutputBackend`, vLLM's default `auto`) is built behind the same seam: it reuses the native pushdown-FSM/trie matcher (xgrammar's own algorithm) and adds the xgrammar JSON-schema→EBNF converter that preserves property declaration order + `any_whitespace` + the `basic_*` grammar, closing the key-order/whitespace/exotic-schema parity gap. CPU-gated (`test_backend_xgrammar` 6/6, RED-first); production wiring + the `auto` fallback + GPU oracle parity are the named residuals (see `.agents/specs/xgrammar-backend.md`) |
| Tool-call parsing (`TOOLS-PARSER-BREADTH`, PARTIAL) | 37 parser families / 41 accepted names, streaming | 39 of upstream's 44 registered names at the pin, plus 2 that upstream's registry does not carry there (`qwen3`, our alias for the Hermes-JSON Qwen dialect, and `muse_glimmer`) = 41 accepted names; pure-text parsers ported 1:1, the engine-backed families reimplemented from their wire formats. FIVE upstream names are NOT ported, and only ONE of them is a port we could make from vLLM source: `inkling`, a ParserEngine adapter. `minimax_m3` is backed by upstream's Rust crate; `openai` (`GptOssToolParser`) is a declared Harmony stub that raises on both methods; `cohere_command3` and `cohere_command4` are shims over the out-of-tree `cohere_melody` package. So for four of the five there is no grammar in vLLM source to mirror — tracked as W1/W2 in `.agents/specs/tool-parser-breadth.md`, which records the decision each one owes. Upstream's shared `ToolParserTestConfig` harness is also unported, so the per-parser test floor is set file by file rather than enforced (W3). Selection via `--tool-call-parser` (server), `tool_parser` (C ABI), or template auto-detection over a 27-row ordered marker table; native-syntax forced tool_choice where expressible. Tables: docs/BENCHMARKS.md |
| Reasoning parsing (`SAMPLE-REASONING`, ACTIVE, partial coverage) | 12 names, streaming | think_auto (auto-detect default: content unless markers appear), deepseek_r1, deepseek_v3 (passthrough) / holo2 (thinking→R1), mistral ([THINK]), minimax_m2 (+append_think), step3, olmo3, muse_glimmer, and qwen3 / mimo - reasoning split engine-side BEFORE tool parsing, streamed as `reasoning` deltas in the chat chunks. qwen3+mimo are the first ENGINE-BACKED adapter (one reasoning face over the shared `src/vllm/parser/engine/` parser, so `<tool_call>` ends reasoning with no `</think>`); the rest are text parsers. Coverage: 12 of upstream's ~28 registered names (remaining engine-backed adapters + text families tracked as W3/W2 in specs/reasoning-parsers.md); each ported parser doctest-gated vs its tests/reasoning case |
| IndexTTS-2.5 speech generation (`MODEL-MM-indextts2`, ACTIVE, unavailable) | CPU-gated foundations only; load refuses by name | Speech seam, detector, GPT-2, BigVGAN, and partial CAMPPlus landed. Conditioning, codec, S2Mel, composition, routes, C ABI, and the oracle pin remain open. No synthesis or speed claim (#633, #634) |
| Unified streaming parser engine | Core, assembly, serving-SSE dispatch landed, gated; all 10 engine-backed families ported (family parity closed); JSON-schema tool-arg type coercion landed | The vLLM 0.26 declarative `parser/engine/` (shared state machine plus all 10 configs: qwen3, seed_oss, kimi_k2, minimax_m2, glm47_moe, deepseek_v4/v32, nemotron_v3, gemma4, inkling) and assembly layer, gated field-for-field vs vLLM 0.26. An engine-backed `--tool-call-parser` name drives the live chat SSE chunks, off by default. When a request's tools declare typed parameters, the assembled tool-call arguments are coerced to the declared JSON types (int/number/bool/string/array/null) 1:1 with vLLM `_fix_arg_types`, in both streaming and one-shot; no schema means the arguments pass through as strings unchanged. Details: .agents/specs/parser-assembly-c8.md |
| OpenAI server | Subset; v0.0.2 publishes eight server bundles; Windows v0.0.3-pre.1 pending | Completion/chat (SSE), models, health/version/ping, metrics, tokenize/detokenize, tokenizer/server info, prefix-cache reset, abort, and Sora-shaped video creation/content. Tokenizer info and abort are flag-gated; cache reset lacks live async backing. Details: docs/USAGE.md |
| Pooling task class (embeddings / classify / score / rerank) | **EMBEDDINGS LIVE ON THE ONE SURFACE (ROW 6)**: `LlamaModel` registered, `PoolingRunner` in the engine step, `vllm_embed` (ABI v15) + live `/v1/embeddings`; classify/score/rerank engine-side only | The non-generative task class. W0 spike over the whole vLLM pooling surface (`.agents/specs/pooling-task-class.md`, `CLAIM-POOLING`). W1 landed the pooler OP (CLS/LAST/MEAN + Identity/Normalize/MultiLabelClassify/Classify activations, double-precision-gated). **W2 landed the pooler HEADS composite** (`EmbeddingPoolerHead`, `ClassifierPoolerHead`, `SequencePooler` + factories, `DispatchPooler` routing, `PoolerConfig`/`PoolingParams`; `test_pooler_heads` 27/27-240, RED-first). **W3 landed the pooling RUNNER path** (`PoolingRunner`: pooled embeddings instead of sampled tokens, structural cosine gate vs an f64 LAST+normalize reference, `test_pooling_runner` 5/5-14, RED-first). **ROW 6 (2026-08-08): embeddings LIVE** — fold gate `test_llama_embedding_fold` 4/4-231 (engine path == direct registry path, f64 LAST+normalize ref, chunked is_valid arm); residuals: REAL checkpoint + `LLM(task="embed")` oracle cosine (no number fabricated), score/rerank/classify endpoints, matryoshka/base64/token-array inputs, tokwise (W5). Detail: `.agents/specs/embeddings-one-surface.md` |
Expand Down
Loading