diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 0b4cc3063..eabf4a3e0 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -347,6 +347,7 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1150](https://github.com/mudler/vllm.cpp/issues/1150) | `LTX25-TI2VID-RECIPE` | The sigma SHIFT is derived from the target latent on every arm, where six of upstream's seven `LTX2Scheduler.execute` call sites pass NO latent and take `default_number_of_tokens` = `MAX_SHIFT_ANCHOR` = 4096 (`schedulers.py:11,:29,:31`). `grep -rn '\.execute(' packages/ltx-pipelines/src/ltx_pipelines/` at `fd4ded7f` returns seven and that grep is the whole population: only `ti2vid_two_stages_hq.py:267` passes `latent=empty_latent`. This engine passes `target_tokens` at `src/vllm/multimodal/ltx2_video.cpp:3442-3443`, so it mirrors the exception and diverges from the rule. Correct today: `t2a_one_stage` (passes 0 at `src/vllm/model_executor/models/ltx2_t2a.cpp:178`) and `res2s_two_stage`. DIVERGENT: `one_stage` at four version keys (`ti2vid_one_stage.py:207`), `a2vid_two_stage` stage 1 (`a2vid_two_stage.py:226`) and `retake`'s non-distilled arm (`retake.py:287`). Recipes carrying explicit `sigmas` never reach the derivation and are unaffected. Not a rounding difference: at the recipe default geometry the target latent is 6144 tokens, giving `sigma_shift` 2.78 against upstream's 2.05, so every sigma moves. Invisible because the trajectory changes while the frame count, shapes, sample rate and errors do not, and our goldens were captured from this engine so they PIN it rather than detect it. `.agents/specs/ltx25-res2s-loop.md:80-88` saw the HQ/plain split and concluded the divergence was on the plain two-stage arm alone; that is right about HQ and wrong about the blast radius. Found by row `LTX25-TI2VID-RECIPE`, which added the seam — `Ltx2PhaseRecipe::schedule_tokens`, defaulted to today's behaviour so nothing moves — and set it on the one phase it ships. Not fixed in flow because flipping the other three re-samples five shipped, gated arms and rewrites their goldens, which needs its own spec and fresh review. Listed under `## Owed` in [`ltx25-ti2vid-recipe.md`](specs/ltx25-ti2vid-recipe.md) | bug | | [#1151](https://github.com/mudler/vllm.cpp/issues/1151) | `LTX25-TI2VID-RECIPE` | The `requires_distilled_lora` refusal still advertised [#1118](https://github.com/mudler/vllm.cpp/issues/1118) as OPEN, and cited `a2vid_two_stage.py`'s line numbers to every other pipeline. #1118 closed at `4ae0f54ab` (row `LTX25-PHASE-LORA`, PR [#1140](https://github.com/mudler/vllm.cpp/pull/1140)), which added `Ltx2PhaseRecipe::loras` and `Ltx2RebindDitLoras`; the message at `src/vllm/multimodal/ltx2_video.cpp:1039-1041` still ended "upstream fuses that adapter into stage 2 ALONE and this engine fuses once at load, so stage 1 sees it too", every clause of which had become false, and the comment above the refusal said the same. `ltx25-phase-lora.md` repaired the REFERENCE-CONDITIONING refusal, which carried the identical claim ~1100 lines away, and named only that one in its port map, so this site and the `ltx2-gen --help` text (`examples/ltx2_gen/main.cpp:210-212`) were both missed. Second defect at the same site: the refusal is deliberately keyed on the FLAG rather than on the kind string so the next recipe inherits it (the comment names #1093 and #1096 as waiting), yet its body interpolated `im.pipeline_kind` into the first sentence and hard-coded a2vid's `:164`, `:114`, `:107` into the rest — so the first arm to inherit it would be told its own name and then a different pipeline's source lines. `--distilled-lora required=True` lives on `default_2_stage_arg_parser` (`utils/args.py:1123`, `:1140-1155`), which all of these pipelines select, and that shared anchor is what the message now cites. Found and fixed IN FLOW by row `LTX25-TI2VID-RECIPE`, the second user of the flag; `tests/vllm/multimodal/test_ltx2_video.cpp` asserted the string `1118` was PRESENT and now asserts it is absent | bug | | [#1152](https://github.com/mudler/vllm.cpp/issues/1152) | `LTX25-TI2VID-RECIPE` | `Ltx2PipelineRecipe::allow_request_latents` (`include/vllm/model_executor/models/ltx2_pipeline.h:705`) is WRITTEN by every recipe and READ by nothing — the "a parameter no caller passes" shape AGENTS.md `## Nothing lands dead` names. Measured at `c83b96934`: `grep -rn allow_request_latents src include examples` minus the declaration returns FIVE lines and all five are assignments (`ltx2_pipeline.cpp:1264` false, `:1345` false, `:1473` true, `:1604` false). Positive control, the field declared one line above and set in the same blocks: `allow_request_sigmas` returns its assignments PLUS a real reader at `src/vllm/multimodal/ltx2_video.cpp:3476`, so the grep is well-formed and the absence is the finding. Consequence: `Res2sTwoStageRecipe` carries `true` where every other recipe carries `false` and nothing can tell the difference — no upstream `__call__` among these pipelines takes an initial-latent parameter, so `false` is what the signatures support and the `true` looks like an oversight, but it is unfalsifiable while nothing reads the field. The tests assert the VALUES, so they gate the record against itself and cannot see that nothing consumes it, which is the tautology shape [#911](https://github.com/mudler/vllm.cpp/issues/911) recorded on the anchor checker. Two closes: give it a reader (a refusal on a request supplying a latent to a recipe whose upstream signature has none) or delete it and its assertions — deleting is defensible, since no request surface carries a latent at all so the refusal could never fire either. Found by row `LTX25-TI2VID-RECIPE` while deriving the same field for a sixth recipe; not fixed in flow because both closes touch five landed recipes and one deletes gated assertions. Listed under `## Owed` in [`ltx25-ti2vid-recipe.md`](specs/ltx25-ti2vid-recipe.md) | bug | +| [#1165](https://github.com/mudler/vllm.cpp/issues/1165) | `FIX-GPU-MEM-UTIL-INERT` | `--gpu-memory-utilization` is parsed (`src/vllm/entrypoints/openai/server_main.cpp:440-441`), threaded to both engines (`:952`, `:1039`), carried on the C ABI (`include/vllm.h:486`, `src/capi/vllm_c.cpp:577-580`), spelled by `examples/cli/main.cpp:118-119`, defaulted to 0.92 (`include/vllm/entrypoints/model_loader.h:90`) and then read by NOTHING: `LoadedEngine::ResolveNumBlocks` falls through knob 1 (`num_blocks`) and knob 2 (`kv_cache_memory_bytes`) to a bare `return 256` under a `TODO(ROAD-V1-MEM M3)` (`src/vllm/entrypoints/model_loader.cpp:954-959`), so a user who passes `--gpu-memory-utilization 0.85` believes they sized the KV pool and sized nothing. DISTINCT from [#83](https://github.com/mudler/vllm.cpp/issues/83), which owns IMPLEMENTING the utilization path (`ROAD-V1-MEM` M3, dgx-gated on a profile run and an oracle-matched pool). This row owns not lying about it: accept the flag, keeping vLLM's exact name and fraction semantics per `.agents/roadmap_v1.md:71`, and emit one notice per engine load when the caller set it explicitly AND the utilization path is the one that resolved the pool. Fixed in flow. Spec [`gpu-mem-util-inert.md`](specs/gpu-mem-util-inert.md) | bug | | [#1166](https://github.com/mudler/vllm.cpp/issues/1166) | `FIX-FP8-BLOCKWISE-REFUSAL` | `Qwen/Qwen3.8-27B-FP8` is block-wise (fine-grained 128x128) FP8 and this tree implements per-tensor FP8 only, so the load stops on a message that names the wrong thing. Measured live at revision `017b9c7af6b5689d5dd426a76e0bc077eb5ca20a` on 2026-08-17: the config declares `quant_method` `fp8`, `weight_block_size` `[128, 128]` and `activation_scheme` `dynamic`, and the safetensors header of `layers-3.safetensors`, read by RANGE REQUEST rather than downloaded, gives `self_attn.q_proj.weight` `F8_E4M3` `[12288, 5120]` beside `self_attn.q_proj.weight_scale_inv` `BF16` `[96, 40]`, which is exactly `[12288/128, 5120/128]`, with ZERO `input_scale` tensors in the shard. `LoadAttnDense` branches on the weight dtype alone (`qwen3_5_dense_weights.cpp:479`) so the block-wise projection enters the per-tensor arm at `:480`, and `LoadFp8Raw` (`qwen3_5_weights.cpp:449`) asks for `.weight_scale` at `:458`, which this checkpoint spells `weight_scale_inv`, so the resolver at `qwen3_5_dense_weights.cpp:682` raises `tensor not found: ...q_proj.weight_scale`. Nothing is missing from the checkpoint. The reader is sent after a tensor upstream never writes in this mode instead of being told the fine-grained arm is absent. NOT the silently-wrong-numerics case, and the check that rules it out is recorded rather than assumed: `ReadF32Scalar` (`qwen3_5_weights.cpp:312`) bounds its input with `t.nbytes >= sizeof(float)`, a LOWER bound, so a `[96, 40]` scale would pass and read as block `(0,0)`, but the NAME miss stops the load before that scalar read, and upstream makes the spelling strictly conditional on block quant (`weight_scale_inv if self.block_quant else weight_scale`, `fp8.py:511` at pin `555967922`), so no upstream block-wise checkpoint reaches it. FIXED IN FLOW as a named refusal at `ModelRegistry::Load`; the block-wise arm itself stays owed | bug | | [#1161](https://github.com/mudler/vllm.cpp/issues/1161) | `ENG-CUDAGRAPH` | SGLang landed **Breakable CUDA Graph (BCG)** and the whole surface was absent from `.agents/sglang-matrix.md`, while `ENG-CUDAGRAPH` carried no verdict on it. Grounded at pin `v0.5.15`/`f63458b5be`: BCG captures one forward as a SEQUENCE of `torch.cuda.CUDAGraph` segments split at eager break points on the attention and recurrent calls, sharing one mempool with weak-ref'd intermediates, with **no `torch.compile`** (`breakable_cuda_graph.py:204-241,244-260,14-23,156-169`; `breakable_cuda_graph_backend.py:14-17`; break points `radix_attention.py:256`, `forward_mla.py:1092`, `radix_linear_attention.py:159`, `nemotron_h.py:1240`, `dsa_indexer.py:2414`). **The public priority claim is narrower than it reads:** vLLM's v1 default is ALREADY `FULL_AND_PIECEWISE`, splitting at `splitting_ops` (`vllm/config/compilation.py:60-63,517,615,630` @ `555967922`), so the capability has been in our benchmark denominator all along — what SGLang landed first is the CONSTRUCTION (piecewise coverage from runtime stream capture plus a decorator, no Dynamo/Inductor/FX), which matters to us only because we have no compiler either. **The load-bearing outcome is a REFUTATION:** prefill graph capture is not a lever on our CUDA gate models, because GB10 2026-07-09 measured steady-state prefill GPU-idle-between-launches at **3.8%** with GPU-busy >96% on both arms, and attributed the 27B prefill gap to **92.5% non-GEMM glue GPU work** with the dominant GEMM the same symbol at the same call count (+0.17%) and attention AHEAD. There are no launch bubbles in our prefill for a graph to collapse, and vLLM's piecewise prefill graph is already inside the denominator we lose to, so it does not explain the gap either. Decode is a separate and already-banked case (conc-64 launch overhead was ~24%, already one `cudaGraphLaunch`/step). FIXED IN FLOW: `SGLANG-BCG` row added under a new "Graph capture and runtime execution" section of `sglang-matrix.md`, the refutation recorded on `ENG-CUDAGRAPH` (which stays `PARTIAL`, because nothing shipped), roadmap track `C12` added, and spec [`sglang-breakable-cuda-graph.md`](specs/sglang-breakable-cuda-graph.md) written to carry the analysis. NOT fixed here and owned by the three issues below: no engine code, no measurement taken, no oracle run, and no advance of the SGLang pin | record | | [#1162](https://github.com/mudler/vllm.cpp/issues/1162) | `ENG-CUDAGRAPH-DEDUP` | We instantiate **one `cudaGraphExec` per padded decode bucket, per model**, and `grep -rn "cudaGraphExecUpdate" src include` returns nothing. `src/vt/cuda/cuda_backend.cu:222-232` instantiates a fresh exec per capture; `include/vllm/model_executor/models/decode_graph_sizes.h:32-41` yields 7 buckets at `max_num_seqs=32` and 11 at 64; eight drivers each build their own set. SGLang folds compatible captures onto one executable by hashing graph topology and calling `cudaGraphExecUpdate` on a signature hit (`cuda_graph_dedup_mixin.py:219-242`, logging "captured %d CUDA graphs, deduped to %d execs" at `:358`). Portable to us unchanged, because it is driver-level rather than PyTorch-level. It is a **memory and capture-time** change, NOT a throughput change — a deduped replay launches the same nodes — and it matters because on GB10 unified memory an OOM reboots the box, capture time is startup latency (a recorded gate axis), and bucket count is exactly what widening graph coverage would raise. Owed: the dedup registry behind the `vt` seam, a capture-count/exec-count log line, and a same-binary A/B proving a deduped replay is byte-identical rather than asserting it. Hazard already recorded: capture bakes host source addresses and a clean `compute-sanitizer` run is NOT evidence a capture path is safe (`specs/decode-graph-scratch-uaf-2026-07-18.md`). Spec [`sglang-breakable-cuda-graph.md`](specs/sglang-breakable-cuda-graph.md) `## Owed`. Analysis: [#1161](https://github.com/mudler/vllm.cpp/issues/1161) | perf | diff --git a/.agents/specs/gpu-mem-util-inert.md b/.agents/specs/gpu-mem-util-inert.md new file mode 100644 index 000000000..297b0ac05 --- /dev/null +++ b/.agents/specs/gpu-mem-util-inert.md @@ -0,0 +1,211 @@ +# Stop `--gpu-memory-utilization` from lying + +Row: `FIX-GPU-MEM-UTIL-INERT`. Issue: +[#1165](https://github.com/mudler/vllm.cpp/issues/1165). Parity pin: vLLM +`555967922` (0.26.0.dev0). + +## Scope + +This row makes an already-recorded gap audible. It does not close the gap. + +`--gpu-memory-utilization` is parsed, threaded, carried on the C application +binary interface (ABI), defaulted to 0.92, and then read by nothing. A user who +passes `--gpu-memory-utilization 0.85` believes they sized the key value (KV) +pool. They sized nothing, and the engine reports no difference from a run that +never set the flag. + +### The boundary against `ROAD-V1-MEM` M3, and why it is not moved here + +[#83](https://github.com/mudler/vllm.cpp/issues/83) owns making the flag work. +That is `ROAD-V1-MEM` M3, designed in [`kv-sizing.md`](kv-sizing.md) and +recorded on the roadmap at `.agents/roadmap_v1.md:71`. M3 needs a device profile +run that measures the non-KV footprint before a free-memory fraction can become +a block count, and its gate is our KV pool matching vLLM's own at a matched +`--gpu-memory-utilization`, which needs a working pinned oracle on `dgx.casa`. + +M3 is therefore dgx-gated and cannot land on the CPU tier. **This row does not +close #83, does not implement the profile run, and does not stub one.** A later +reader who finds this spec must not read it as evidence that the utilization +path works. The fallback is still 256 blocks. The only change is that the engine +now says so. + +## Anchors + +| What | Where | +|---|---| +| Flag parsed | `src/vllm/entrypoints/openai/server_main.cpp:440-441` | +| Threaded to both engines | `src/vllm/entrypoints/openai/server_main.cpp:952,1039` | +| Command-line client | `examples/cli/main.cpp:118-119,209-212` | +| C ABI field | `include/vllm.h:474-486` | +| C ABI mapping | `src/capi/vllm_c.cpp:577-580` | +| Engine field and default | `include/vllm/entrypoints/model_loader.h:87-90` | +| The place it is discarded | `src/vllm/entrypoints/model_loader.cpp:954-959` | +| Upstream knob this mirrors | `vllm/config/cache.py:68` @ `555967922` | +| Upstream override precedence | `vllm/config/cache.py:189` @ `555967922` | + +## Design + +### Accept the flag. Never refuse it + +`.agents/roadmap_v1.md:71` records the design intent that this engine keeps +vLLM's exact flag name and fraction semantics so an existing vLLM launch line +ports unchanged. Refusing the flag would break that intent and contradict the +mirror rule in `AGENTS.md`, `## vLLM is the reference`. The value stays +accepted, and its resolution stays exactly what it is today. + +### Warn only when the caller chose the value + +A notice on every start for a default nobody set is noise. A missing notice for +a value somebody deliberately chose is the defect this row exists to fix. So the +engine needs to tell an explicit ask apart from the built-in default, and a +plain `double` that defaults to 0.92 cannot. + +`EngineParams::gpu_memory_utilization` becomes `std::optional`, where +`nullopt` means unset and resolves to vLLM's 0.92. This mirrors the tri-state +`enable_prefix_caching` already in the same struct +(`include/vllm/entrypoints/model_loader.h:120`, resolved at +`src/vllm/entrypoints/model_loader.cpp:718-728`). Nothing in the tree reads the +field today, so the type change moves no behavior. + +Each surface then says what it means: + +- The server sets the optional only when `--gpu-memory-utilization` was parsed. +- `examples/cli` sets the ABI field only when the flag was parsed, and otherwise + passes the ABI's documented unset spelling, `0.0`. +- The C ABI mapping is unchanged. `> 0.0` still means explicit. + +### The C ABI asymmetry, stated rather than fixed here + +`vllm_model_params_default()` pre-fills `gpu_memory_utilization` with `0.92` +(`src/capi/vllm_c.cpp:524`), so a C caller who never touched the field is +indistinguishable from one who typed `0.92`. Such a caller gets the notice. + +That is left alone deliberately. Changing the value `vllm_model_params_default()` +returns is an observable change to the ABI surface, it is not needed for the +notice to be true, and a struct that carries `0.92` into an engine that ignores +it is a caller who benefits from hearing that. `include/vllm.h` gains one +sentence saying the notice fires and that `0.0` is the spelling that suppresses +it. The command-line path, which does have a real typed-or-not distinction, is +silent by default after this row. + +### Where the notice fires, and how often + +The notice fires inside `LoadedEngine::ResolveNumBlocks`, at step 3, on the line +that discards the value. That is the one seam every entry point reaches: the +server, `examples/cli` through the C ABI, and any `include/vllm.h` client all +build a `LoadedEngine`, whose constructor calls `MakeKVCacheResolved`, which +calls `ResolveNumBlocks` +(`src/vllm/entrypoints/model_loader.cpp:1081-1083,972`). + +It fires once per engine load, not once per process. A server that loads a text +engine and an embedding engine reports twice, because there are two pools and +the value was discarded twice. A process-wide latch was rejected: it would make +the second engine silent, and it would make the test order dependent, which is +the shape recorded in `a-test-class-after-the-main-guard-never-runs` and +`the-state-was-not-the-one-you-believed`. + +Steps 1 and 2 return before step 3, so the notice cannot fire when +`--num-blocks` or `--kv-cache-memory` sized the pool. That is correct and not +incidental. vLLM's `cache.py:189` ignores `gpu_memory_utilization` under +`kv_cache_memory_bytes`, so a caller who set both gets vLLM's exact semantics +and has nothing to be warned about. + +### What the notice says + +It names four things: that the value did not size the pool, the count that +resolved instead, the flags that do bind today, and the row and issue that own +the real fix. + +```text +vllm.cpp: WARNING --gpu-memory-utilization 0.85 was accepted but did NOT size the KV cache. +vllm.cpp: The profile run that turns a free-memory fraction into a block count is not +vllm.cpp: implemented yet (ROAD-V1-MEM M3, https://github.com/mudler/vllm.cpp/issues/83). +vllm.cpp: The pool fell back to 256 blocks. To size it today, pass +vllm.cpp: --kv-cache-memory for an absolute KV budget, or --num-blocks for an +vllm.cpp: exact block count. +``` + +### The GB10 hazard belongs in the document, not in the notice + +`gpu_memory_utilization` reserves host random access memory (RAM) on GB10's +unified 119 GiB pool, and a 0.85-class value has hard-rebooted that box +(`.agents/roadmap_v1.md:71`, [`mtp-k-gt-1.md`](mtp-k-gt-1.md) around `:523-542`). +The hazard is real and load-bearing. + +It is recorded in `docs/USAGE.md` and kept out of the runtime notice, for two +reasons. The flag is inert today, so it cannot reboot anything today, and a +hazard warning attached to a value nothing reads misdirects the reader from the +sentence that matters. Second, the notice disappears when M3 lands, which is +exactly when the hazard becomes real, so putting the hazard there would delete +it at the moment it starts to bite. No platform detection is added, per the +row's constraint. + +## Records + +`docs/USAGE.md` is owed, and `scripts/check-doc-checkpoint.py` demands it: this +change touches `src/vllm/entrypoints/`, `include/vllm/`, `include/vllm.h`, and +`examples/cli/`, all of which classify as `user_usage`. + +The page does not document `--gpu-memory-utilization` or `--kv-cache-memory` at +all today. Its server-flag table lists `--num-blocks N | 256 | KV blocks`, which +is stale in its own right: `num_blocks` defaults to 0, meaning auto, and 256 is +the resolved fallback rather than the default value. All three entries are +corrected here, and the GB10 hazard lands beside the utilization row. + +`.agents/feature-matrix.md:92` and `docs/FEATURES.md:365` were both checked and +are **not** edited. Both already record the exact state: the row is `PARTIAL`, +M1 and M2 landed, and `--gpu-memory-utilization` needs the dgx-gated M3 profile +run. Neither states anything this row makes false. Editing the matrix would pull +`docs/FEATURES.md` in behind it through the `feature_surface` class with nothing +new to say, which is the shape `check-doc-checkpoint.py` records at its +`registration_changes` docstring as the defect that blocked every branch under +[#1055](https://github.com/mudler/vllm.cpp/issues/1055). + +## Tests + +`tests/vllm/entrypoints/test_loaded_engine_dense.cpp`, already registered at +`tests/CMakeLists.txt:1208`. It builds a real `LoadedEngine` over synthetic +dense weights, which is the loader entry point, so the test enters the change +through production rather than by calling the resolver. `ResolveNumBlocks` is +private, so no test can reach it directly. + +Four cases, using the `CerrRedirect` idiom from +`tests/vllm/v1/test_async_llm.cpp:130-144`: + +1. An explicit `gpu_memory_utilization` prints the notice, and the notice names + the value, the 256-block fallback, both binding flags, and issue 83. +2. An unset `gpu_memory_utilization` prints nothing. +3. An explicit `gpu_memory_utilization` beside `kv_cache_memory_bytes` prints + nothing, because knob 2 sized the pool and vLLM ignores the fraction there. +4. An explicit `gpu_memory_utilization` beside a `num_blocks` override prints + nothing, for the same reason at knob 1. + +Case 2 is the one that fails if the notice is made unconditional, and cases 3 +and 4 are the ones that fail if it is moved above the early returns. + +## Gates + +CPU only. No GPU lease, and no `rc` call. + +- Focused: `ctest -R test_loaded_engine_dense`. +- Full: `scripts/agent-preflight.sh`. +- Reachability: delete the `ResolveNumBlocks` call in `MakeKVCacheResolved` in a + scratch copy, rerun the focused gate, and require red. Restore byte for byte. + +## Stop conditions + +Stop and report `NEEDS_DECISION` if closing this needs the M3 profile run, if it +needs a GPU, or if the only way to make the notice fire is to refuse the flag. + +## Owed + +Nothing. `ROAD-V1-MEM` M3 still owns the utilization path itself, and this row +leaves that debt exactly where it was. Its issue is deliberately not linked in +this section: `owed_issues()` in `scripts/check-agent-record.py` reads every +issue number under a `## Owed` heading as a claim of ownership, and this row +does not own it. + +## Now + +Landed on `row/FIX-GPU-MEM-UTIL-INERT`. The next step for the utilization path +is `ROAD-V1-MEM` M3 on `dgx.casa`, unchanged by this row. diff --git a/docs/USAGE.md b/docs/USAGE.md index c0c7ac302..35164e786 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -2184,7 +2184,9 @@ a stop token early. | `--served-model-name N` | model dir basename | Model id in `/v1/models` and responses | | `--tokenizer-config F` | `/tokenizer_config.json` | Chat template / tokenizer config | | `--block-size N` | `32` | KV block size | -| `--num-blocks N` | `256` | KV blocks | +| `--num-blocks N` | `0` (auto, resolves to `256`) | KV block count, and vLLM's `num_gpu_blocks_override`. It wins over every other sizing knob. `0` means auto, which uses `--kv-cache-memory` when that is set and otherwise falls back to `256` blocks | +| `--kv-cache-memory BYTES` | `0` (unset) | Absolute KV-pool size in bytes, vLLM's `kv_cache_memory_bytes`. The block count is this budget divided by the model's own bytes per block, summed across its KV groups, so it is correct on MLA and heterogeneous-KV architectures too. It ignores `--gpu-memory-utilization`, as vLLM does. A budget smaller than one KV block is refused at startup | +| `--gpu-memory-utilization F` | `0.92` | **Accepted, and it does not size anything yet.** See [What `--gpu-memory-utilization` does not do yet](#what---gpu-memory-utilization-does-not-do-yet) | | `--max-model-len N` | `0` (config default) | Max sequence length | | `--max-num-seqs N` | `32` | Max concurrent sequences (also sizes the HTTP worker pool). Was `8`, which put a c8 client exactly on the batch ceiling; vLLM's own default is 1024, which we do not mirror because this also caps the padded decode-graph set. On a GDN/Mamba model under speculative decoding this also multiplies the recurrent state, which is sized `max-num-seqs x (k+1)`; an unservable budget is refused at load with the arithmetic | | `--max-num-batched-tokens N` | `0` (per-arch default) | Per-step token budget | @@ -2245,6 +2247,43 @@ only because the capability is missing (`--tensor-parallel-size` and the other parallelism flags) — silently accepting those would let you believe you got tensor parallelism when you did not. +#### What `--gpu-memory-utilization` does not do yet + +The flag is accepted, keeps vLLM's exact name and fraction semantics, and is +then discarded. It does not size the KV pool. Passing +`--gpu-memory-utilization 0.85` gives the same 256-block pool as passing +nothing. + +Turning a free-memory fraction into a block count needs a profile run that +measures what the weights and activations cost on the device first. That run is +not implemented. It is `ROAD-V1-MEM` M3, tracked by +[issue #83](https://github.com/mudler/vllm.cpp/issues/83), and it needs a GPU to +gate. + +The flag is accepted rather than refused so that a published `vllm serve` +command line runs here unchanged. Setting it prints this warning at startup, so +a log never implies it took effect: + +```text +vllm.cpp: WARNING --gpu-memory-utilization 0.85 was accepted but did NOT size the KV cache. +vllm.cpp: The profile run that turns a free-memory fraction into a block count is not +vllm.cpp: implemented yet (ROAD-V1-MEM M3, https://github.com/mudler/vllm.cpp/issues/83). +vllm.cpp: The pool fell back to 256 blocks. To size it today, pass +vllm.cpp: --kv-cache-memory for an absolute KV budget, or --num-blocks for an +vllm.cpp: exact block count. +``` + +To size the pool today, use `--kv-cache-memory` for an absolute byte budget or +`--num-blocks` for an exact count. A run that never sets the flag prints +nothing. + +**Warning.** On a unified-memory board such as NVIDIA GB10, a fraction of +"device" memory is a fraction of the one pool the host shares, so it reserves +host RAM as well. A value of 0.85 has hard-rebooted a GB10 box. When M3 lands +and this flag starts to bind, choose the fraction on such a board against the +whole 119 GiB pool and leave the host its headroom. Until then the flag reserves +nothing, on any board. + #### Context length vs the KV pool The KV pool holds `--num-blocks × --block-size` tokens — `256 × 32 = 8192` by diff --git a/examples/cli/main.cpp b/examples/cli/main.cpp index 1645fe283..b45685319 100644 --- a/examples/cli/main.cpp +++ b/examples/cli/main.cpp @@ -48,7 +48,13 @@ struct Args { // --gpu-memory-utilization / --kv-cache-memory (ABI v16): KV-pool sizing. // gpu_memory_utilization is inert until the M3 profile run lands; // kv_cache_memory_bytes (> 0) sizes the block count directly. - double gpu_memory_utilization = 0.92; + // + // 0.0, NOT 0.92 (FIX-GPU-MEM-UTIL-INERT, #1165). 0.0 is the ABI's documented + // "unset" spelling (vllm.h: 0.0 => 0.92), so this stays 0.0 until the user + // types --gpu-memory-utilization. The engine now warns that a CHOSEN fraction + // sized nothing, and pre-filling 0.92 here would make every plain `vllm-cli` + // run look like an explicit ask and print that warning. + double gpu_memory_utilization = 0.0; long long kv_cache_memory_bytes = 0; // --max-num-seqs: max concurrent sequences. Exposed because it is the knob the // recurrent-state budget check names (issue #371): under speculative decoding @@ -207,6 +213,10 @@ int main(int argc, char** argv) { mp.device = args.device; // KV-pool sizing knobs (ABI v16). Defaults leave the historical 256-block // behaviour; --kv-cache-memory sizes the pool from an absolute byte budget. + // The assignment OVERWRITES vllm_model_params_default()'s pre-filled 0.92 + // with 0.0 unless --gpu-memory-utilization was typed, which is what keeps a + // plain run out of the #1165 warning. Do not guard it with `> 0.0`: that + // would leave the 0.92 in place and warn on every start. mp.gpu_memory_utilization = args.gpu_memory_utilization; mp.kv_cache_memory_bytes = args.kv_cache_memory_bytes; if (args.max_num_seqs > 0) mp.max_num_seqs = args.max_num_seqs; diff --git a/include/vllm.h b/include/vllm.h index a8d8842ea..a41ebdf0f 100644 --- a/include/vllm.h +++ b/include/vllm.h @@ -482,7 +482,20 @@ typedef struct vllm_model_params { * the non-KV footprint; that profile run is not implemented yet * (ROAD-V1-MEM M3), so until it lands a struct with both other knobs unset * still falls back to the historical 256-block default — the zero-initialized - * struct's behaviour is unchanged from pre-v16. */ + * struct's behaviour is unchanged from pre-v16. + * + * Since #1165 that fallback is no longer SILENT: a value > 0.0 here, with + * num_blocks and kv_cache_memory_bytes both unset, prints one warning per + * vllm_engine_load naming the block count that resolved instead and the two + * knobs that do bind today. Accepting a fraction and sizing nothing without + * saying so left callers believing they had sized the pool. + * + * Note that vllm_model_params_default() pre-fills this field with 0.92, so a + * caller who never touched it is indistinguishable from one who chose 0.92 + * and does get the warning. That is deliberate: on this ABI there is no + * "flag not typed" state, and a struct carrying 0.92 into an engine that + * ignores it is exactly the case the warning is for. To opt out, spell the + * unset sentinel: set the field to 0.0. */ double gpu_memory_utilization; /* kv_cache_memory_bytes: an ABSOLUTE KV-pool size in bytes. When > 0 it sizes * the block count directly (num_blocks = kv_cache_memory_bytes / bytes-per- diff --git a/include/vllm/entrypoints/model_loader.h b/include/vllm/entrypoints/model_loader.h index 3a979bea9..c9d47b1ae 100644 --- a/include/vllm/entrypoints/model_loader.h +++ b/include/vllm/entrypoints/model_loader.h @@ -65,6 +65,12 @@ struct DflashDraft { int k = 0; }; +// vLLM CacheConfig.gpu_memory_utilization's own default (vllm/config/cache.py:68 +// @ 555967922). Named because EngineParams carries the knob as an optional, so +// "the caller chose nothing" and "the caller chose 0.92" are different states +// and only one of them has a value to fall back to. +inline constexpr double kDefaultGpuMemoryUtilization = 0.92; + // Knobs that size the engine stack. Zero/negative fields fall back to the // documented defaults (see below), so a default-constructed EngineParams is // valid. @@ -86,8 +92,17 @@ struct EngineParams { int num_blocks = 0; // 0 => auto (resolved: override > bytes > 256). // Fraction of free device memory the whole engine may consume (weights + // activations + KV), mirroring vLLM CacheConfig.gpu_memory_utilization - // (cache.py:68). Used only by the M3 profile path; inert until that lands. - double gpu_memory_utilization = 0.92; + // (cache.py:68). Still INERT: knob 3 needs the M3 profile run. + // + // TRI-STATE (FIX-GPU-MEM-UTIL-INERT, #1165), mirroring enable_prefix_caching + // below. nullopt means the caller never named a fraction and resolves to + // vLLM's kDefaultGpuMemoryUtilization; a value means the caller CHOSE one. + // The distinction exists because the value is discarded: ResolveNumBlocks + // warns on a chosen value, and stays silent on a default nobody set. Before + // this was tri-state the field defaulted to 0.92 and no surface could tell + // the two apart, so the engine accepted `--gpu-memory-utilization 0.85`, + // sized nothing, and said nothing. + std::optional gpu_memory_utilization = std::nullopt; // Absolute KV-pool size in bytes (0 = unset). When > 0 it sizes the block // count directly and IGNORES gpu_memory_utilization, mirroring vLLM // CacheConfig.kv_cache_memory_bytes (cache.py:182,189). @@ -440,6 +455,13 @@ class LoadedEngine { // profile path (M3, not yet implemented) which falls back to 256. Throws // VLLM_ERR-shaped std::runtime_error when an absolute byte budget is smaller // than a single KV block. + // + // FIX-GPU-MEM-UTIL-INERT (#1165): the profile path also WARNS on stderr when + // it reaches knob 3 with an explicitly chosen gpu_memory_utilization, because + // that is the point at which the chosen fraction is discarded. Knobs 1 and 2 + // return first, so a caller who sized the pool with --num-blocks or + // --kv-cache-memory is never warned -- and under knob 2 that also mirrors + // vLLM, which ignores the fraction there (cache.py:189). static int ResolveNumBlocks(const EngineParams& params, const vllm::v1::KVCacheConfig& probe); // ROAD-V1-MEM M1: MakeKVCacheMaybeSpec with the block count resolved from the diff --git a/src/vllm/entrypoints/model_loader.cpp b/src/vllm/entrypoints/model_loader.cpp index 28348080b..f2f514d66 100644 --- a/src/vllm/entrypoints/model_loader.cpp +++ b/src/vllm/entrypoints/model_loader.cpp @@ -956,7 +956,38 @@ int LoadedEngine::ResolveNumBlocks(const EngineParams& params, // fraction can be turned into a block count. Until that lands, fall back to // the historical default so the default path is byte-identical. // TODO(ROAD-V1-MEM M3): profile run -> available_kv = free*util - non_kv. - return 256; + constexpr int kFallbackNumBlocks = 256; + // FIX-GPU-MEM-UTIL-INERT (#1165): this line is where an explicitly chosen + // fraction gets discarded, so this is where the engine has to say so. The + // flag is NOT refused: roadmap_v1.md:71 records the intent that it keeps + // vLLM's exact name and fraction semantics so a published vLLM launch line + // ports unchanged. What was wrong was accepting the value in silence, which + // left a user believing they had sized the KV pool when they had sized + // nothing. + // + // Only an EXPLICIT value warns. A default nobody set has nothing to report, + // and a line on every start is noise rather than a warning. + if (params.gpu_memory_utilization.has_value()) { + std::cerr + << "vllm.cpp: WARNING --gpu-memory-utilization " + << *params.gpu_memory_utilization + << " was accepted but did NOT size the KV cache.\n" + "vllm.cpp: The profile run that turns a free-memory fraction into " + "a block count is not\n" + "vllm.cpp: implemented yet (ROAD-V1-MEM M3, " + "https://github.com/mudler/vllm.cpp/issues/83).\n" + "vllm.cpp: The pool fell back to " + << kFallbackNumBlocks + << " blocks. To size it today, pass\n" + "vllm.cpp: --kv-cache-memory for an absolute KV budget, or " + "--num-blocks for an\n" + "vllm.cpp: exact block count.\n"; + // Unbuffered by the time the loader's next line lands, so the notice cannot + // be separated from the load it belongs to (same reason as the auto-fit + // INFO line in ResolveMaxModelLen). + std::cerr.flush(); + } + return kFallbackNumBlocks; } vllm::v1::KVCacheConfig LoadedEngine::MakeKVCacheResolved( diff --git a/src/vllm/entrypoints/openai/server_main.cpp b/src/vllm/entrypoints/openai/server_main.cpp index f1524cfe1..291e6bb21 100644 --- a/src/vllm/entrypoints/openai/server_main.cpp +++ b/src/vllm/entrypoints/openai/server_main.cpp @@ -181,7 +181,13 @@ struct Args { // engine (needs the M3 profile run; inert until then). --kv-cache-memory: an // absolute KV-pool size in bytes that sizes the block count directly (0 => // unset). - double gpu_memory_utilization = 0.92; + // + // TRI-STATE (FIX-GPU-MEM-UTIL-INERT, #1165), like enable_prefix_caching + // below: nullopt means the flag was NOT passed. It carries a value only when + // the user typed one, which is what lets the engine warn that a chosen + // fraction sized nothing without warning on a default nobody chose. A plain + // double pre-filled with 0.92 could not express the difference. + std::optional gpu_memory_utilization = std::nullopt; long long kv_cache_memory_bytes = 0; int max_model_len = 0; // 0 => config.max_position_embeddings int max_num_seqs = 32; // see model_loader.h: 8 clamped c8 batching. diff --git a/tests/vllm/entrypoints/test_loaded_engine_dense.cpp b/tests/vllm/entrypoints/test_loaded_engine_dense.cpp index 545a0d5fc..f0542b5bc 100644 --- a/tests/vllm/entrypoints/test_loaded_engine_dense.cpp +++ b/tests/vllm/entrypoints/test_loaded_engine_dense.cpp @@ -24,6 +24,9 @@ #include #include #include +#include +#include +#include #include #include @@ -243,6 +246,40 @@ SamplingParams Greedy(int max_tokens) { return sp; } +// FIX-GPU-MEM-UTIL-INERT (#1165): build ONE engine with std::cerr captured and +// return what it wrote. Scope-guarded, mirroring test_async_llm.cpp:130-144 — +// the restore must survive an exception out of the constructor, because two of +// the cases below build engines that log other things (the auto-fit INFO line) +// and a leaked rdbuf swap would silently redirect every later case. +class CerrRedirect { + public: + explicit CerrRedirect(std::streambuf* target) + : previous_(std::cerr.rdbuf(target)) {} + ~CerrRedirect() { std::cerr.rdbuf(previous_); } + CerrRedirect(const CerrRedirect&) = delete; + CerrRedirect& operator=(const CerrRedirect&) = delete; + + private: + std::streambuf* previous_; +}; + +// The engine is built INSIDE the capture and destroyed inside it too, so a +// notice emitted from any part of construction is seen. +std::string CerrOfEngineLoad(const HfConfig& c, const EngineParams& params) { + std::ostringstream captured; + { + CerrRedirect guard(captured.rdbuf()); + LoadedEngine eng(c, MakeDenseWeights(c), FreshFixture(), params); + std::cerr.flush(); + } + return captured.str(); +} + +// The one substring that identifies the notice. Deliberately NOT the whole +// message: the cases assert the individual facts separately, so a reworded +// sentence fails on the fact it dropped rather than on all of them at once. +constexpr const char* kInertNotice = "--gpu-memory-utilization"; + } // namespace // ─── 1. Arch-select: the FromModelDir dispatch decision ────────────────────── @@ -646,3 +683,83 @@ TEST_CASE("loaded_engine: an over-long prompt is REFUSED, not left waiting") { vllm::v1::InputValidationError); CHECK_FALSE(eng.engine().has_unfinished_requests()); } + +// ─── --gpu-memory-utilization must not be silently inert (#1165) ───────────── +// The flag is parsed, threaded to both engines, carried on the C ABI and then +// read by NOTHING: ResolveNumBlocks falls through to `return 256` under a +// TODO(ROAD-V1-MEM M3). Implementing the utilization path is #83 and is +// dgx-gated. What is gated HERE is only that the engine stops reporting +// success for a budget it discarded. +// +// These cases enter through the LOADER, not through the resolver: +// ResolveNumBlocks is private, and a test that called it would prove the +// function works rather than that anything reaches it. The chain under test is +// LoadedEngine ctor -> MakeKVCacheResolved -> ResolveNumBlocks +// (model_loader.cpp:1081-1083,972). + +TEST_CASE( + "loaded_engine: an EXPLICIT --gpu-memory-utilization says it did not size " + "the KV pool") { + const HfConfig c = MakeDenseConfig(); + EngineParams params; + params.gpu_memory_utilization = 0.85; // the flag a user actually types + + const std::string logged = CerrOfEngineLoad(c, params); + + // It fired at all. + REQUIRE(logged.find(kInertNotice) != std::string::npos); + // It names the value the caller chose, so a reader can tell WHICH knob is + // being reported when several are set. + CHECK(logged.find("0.85") != std::string::npos); + // It names what actually sized the pool instead. + CHECK(logged.find("256") != std::string::npos); + // It names the two flags that DO bind today, so the reader is left with an + // action rather than a complaint. + CHECK(logged.find("--kv-cache-memory") != std::string::npos); + CHECK(logged.find("--num-blocks") != std::string::npos); + // It names the row and the issue that own the real fix, so the notice cannot + // be mistaken for a permanent limitation. + CHECK(logged.find("ROAD-V1-MEM") != std::string::npos); + CHECK(logged.find("83") != std::string::npos); +} + +TEST_CASE( + "loaded_engine: an UNSET --gpu-memory-utilization is silent") { + // The noise case, and the one that fails if the notice is made + // unconditional. A default nobody chose has nothing to warn about: the pool + // resolves to 256 blocks, which is exactly what the default documents. + const HfConfig c = MakeDenseConfig(); + const EngineParams params; // gpu_memory_utilization untouched + + CHECK(CerrOfEngineLoad(c, params).find(kInertNotice) == std::string::npos); +} + +TEST_CASE( + "loaded_engine: --gpu-memory-utilization beside --kv-cache-memory is " + "silent, because vLLM ignores the fraction there too") { + // cache.py:189 @ 555967922: kv_cache_memory_bytes IGNORES + // gpu_memory_utilization. A caller who set both gets vLLM's exact semantics, + // so there is no lie to report. This case fails if the notice is hoisted + // above ResolveNumBlocks' early returns. + const HfConfig c = MakeDenseConfig(); + EngineParams params; + params.gpu_memory_utilization = 0.85; + // One block's worth of bytes for this model's own geometry, so knob 2 + // resolves to a servable pool rather than throwing. + params.kv_cache_memory_bytes = 1LL << 30; + + CHECK(CerrOfEngineLoad(c, params).find(kInertNotice) == std::string::npos); +} + +TEST_CASE( + "loaded_engine: --gpu-memory-utilization beside a --num-blocks override is " + "silent") { + // Same argument at knob 1 (vLLM num_gpu_blocks_override): the override sized + // the pool, so the fraction was not the thing that got discarded. + const HfConfig c = MakeDenseConfig(); + EngineParams params; + params.gpu_memory_utilization = 0.85; + params.num_blocks = 4; + + CHECK(CerrOfEngineLoad(c, params).find(kInertNotice) == std::string::npos); +}