Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .agents/engine-matrix.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ lifecycle are unchanged.
| `KV-MLA-SPEC` | Latent MLA KV specification | T2 | `vllm/v1/kv_cache_interface.py:363` | - | - | `planned: specs/mla-kv-spec.md` | `INVENTORIED` | - |
| `KV-CROSS-ENCODER-SPECS` | `CrossAttentionSpec` and `EncoderOnlyAttentionSpec` KV interface specs (`ATTN-ENCODER-CROSS` covers backends only); carried from porting-inventory §2 (T2) at the v1 fold | T2 | `vllm/v1/kv_cache_interface.py:710,717` | - | - | `planned: specs/encoder-cross-kv-specs.md` | `INVENTORIED` | - |
| `KV-SIZING` | GPU memory utilization and block-count overrides | T0 | `vllm/config/cache.py:68,87,168`; `tests/v1/core/test_kv_cache_utils.py:2224,2303` | fixed inputs `src/vllm/entrypoints/model_loader.cpp:117,129`; watermark `src/vllm/v1/core/kv_cache_manager.cpp:118` | watermark only `tests/vllm/v1/test_kv_cache_manager.cpp:298` | `planned: specs/kv-sizing.md` | `PARTIAL` | - |
| `KV-WARMUP-PROFILE` | Dummy runs, warmup, and startup memory profiling that derive the KV budget (`KV-SIZING` covers the sizing knobs only); carried from porting-inventory §3 (T0 there) at the v1 fold | T0 | `vllm/v1/worker/gpu/model_runner.py:504,647`; `vllm/v1/worker/gpu_worker.py:430` | - | - | `planned: specs/warmup-memory-profiling.md` | `INVENTORIED` | - |
| `KV-WARMUP-PROFILE` | Dummy runs, warmup, and startup memory profiling that derive the KV budget (`KV-SIZING` covers the sizing knobs only); carried from porting-inventory §3 (T0 there) at the v1 fold | T0 | `vllm/v1/worker/gpu_worker.py::determine_available_memory`; `vllm/v1/worker/gpu/model_runner.py::profile_run`; `vllm/v1/worker/gpu/model_runner.py::model_memory_usage` | - | - | `planned: specs/warmup-memory-profiling.md` | `INVENTORIED` | - |
| `ENG-EXPERT-STREAM` | Expert streaming from disk: bank-only routed-MoE weights paged into fixed contiguous Marlin slots after logical-expert→slot remap (low-concurrency capacity mode; surpass-track — inference-time disk expert paging is ABSENT in pinned vLLM) | T2 | absent in-pin: `vllm/model_executor/offloader/uva.py:21` (CPU-blanket UVA only), `vllm/model_executor/offloader/prefetch.py:557-560` (cpu-only); design reference antirez/ds4 (`ds4_metal.m`, `ds4_cuda.cu`, `ds4_ssd.c`); local dense-stride constraint `src/vt/cuda/marlin/libtorch_stable/moe/marlin_moe_wna16/marlin_template.h:543-550` | - | - | [expert-streaming.md](specs/expert-streaming.md) | `READY` | - |
| `ENG-WEIGHT-OFFLOAD` | Inference-time CPU weight offload mirror floor: UVA `cpu_offload_gb` per-parameter offload with pinned+zero-copy views and opt-in name-segment targeting (`cpu_offload_params`), plus layer-group `PrefetchOffloader`; v1-supported at the pin | T2 | `vllm/config/offload.py:23,34-44,47-76`; `vllm/model_executor/offloader/uva.py:64,80-108`; `vllm/model_executor/offloader/base.py:23-33,46-92,94-125,126-162`; install point `vllm/v1/worker/gpu_model_runner.py:939` (re-derived at the pin — the previously recorded `:445,913` are STALE and now unrelated lines); layer-wrap site `vllm/model_executor/models/utils.py:816,824`; loader re-offload `vllm/model_executor/model_loader/utils.py:160-193`; cudagraph seam `vllm/compilation/cuda_graph.py:310,324,359` + `breakable_cudagraph.py:379,387,421`; env `vllm/envs.py:278-279,1938-1943`; helpers `vllm/utils/platform_utils.py:51-57`, `vllm/utils/torch_utils.py:766-776`; upstream lineage: `offloader/base.py` header records adaptation from SGLang `srt/utils/offloader.py`; tests `tests/basic_correctness/test_cpu_offload.py:9-29`, `tests/quantization/test_cpu_offload.py:18-64` | W0a config surface: `include/vllm/config/offload.h`, `src/vllm/config/offload.cpp` (backend enum, both sub-configs, `Validate()` = 2 hard errors + 3 collected warnings, dot-anchored segment match, `int(gb*1024**3)` truncation, auto-selection order, layer grouping, JSON parse mirroring the `kv_transfer_config` precedent). UNREACHABLE: nothing constructs one yet. W0b wires it end to end: `include/vllm.h` `offload_config` (ABI v21), the C-API parse+`Validate()`+record in `src/capi/vllm_c.cpp`, `EngineParams::offload_config`, and the server `--offload-config` flag. Still UNREACHABLE by design: the config is validated and recorded, and no weight moves until W2/W5 Totality guard: `ModelFactory::supports_weight_offload` (defaults FALSE) + `RefuseUnsupportedWeightOffload` before any weight I/O + `VerifyWeightOffloadWasConsulted` after load; no model declares support yet, pinned by test. W2a decision: `include/vllm/model_executor/weight_offload_policy.h` + `src/vllm/model_executor/weight_offload_policy.cpp` (per-weight offload/not-targeted/budget-exhausted, running byte budget, `FromConfig` for the UVA arm only). Application seam CHOSEN as the LOADERS, beside `GgufKeepQuantPolicy::Route`, because a constructed `LoadedModel` has already allocated its device copy. W1 seam: `include/vllm/model_executor/weight_offloader.h` + `src/vllm/model_executor/weight_offloader.cpp` (interface, no-op default, process-global, factory), installed at `LoadedEngine::FromModelDir` and read at `ModelRegistry::Prepare` -- our analogue of `make_layers`, which this tree does not have | `tests/vllm/config/test_offload_config.cpp` 11/11 cases, 126/126 assertions; RED-first captured on a compiling stub (11/11 cases, 51/122 assertions RED, build rc=0 / 0 compile errors); mutation-proven 6/6 with compile status reported per mutation; W0b ABI round-trip `tests/capi/test_capi.cpp` "offload_config defaults to NULL and is parsed+validated (ABI v21)" 21/21 (7 refusal cases each asserted INVALID_ARGUMENT not MODEL_LOAD, plus the warning-is-not-a-refusal case), mutation-proven 2/2 (drop `Validate()` -> 3 red; warning-as-throw -> 1 red) | [weight-offload-uva.md](specs/weight-offload-uva.md) | `ACTIVE` | `CLAIM-WEIGHT-OFFLOAD-W0A` |
| `ENG-HYBRID-PLACEMENT` | Hybrid device placement: per-tensor-group device assignment resolved at model build, delivering routed-MoE expert COMPUTE on the CPU backend while attention/dense/router/norms stay on GPU (surpass-track — vLLM has the CPU MoE kernels but selects them platform-wide via `current_platform.is_cpu()`, so hybrid placement is ABSENT in pin). Moves compute toward the weights, the inverse of `ENG-WEIGHT-OFFLOAD`/`ENG-EXPERT-STREAM`; composes with `BACKEND-DISTRIBUTED-TP` sharding on an orthogonal axis and refuses where they conflict. Issue [#149](https://github.com/mudler/vllm.cpp/issues/149) (CPU-MoE half only) | T2 | absent in-pin (placement): `vllm/model_executor/layers/fused_moe/oracle/mxfp4.py:533`, `oracle/fp8.py:129`, `oracle/int8.py:53`, `oracle/int_wna16.py:111`, `oracle/unquantized.py:97,202`, `oracle/w4a8_int8.py:40` (all `current_platform.is_cpu()`); kernels present but platform-gated: `vllm/model_executor/layers/fused_moe/experts/cpu_moe.py`, `cpu_fused_moe.py:398,430`, `vllm/_custom_ops.py:3790,3803`; secondary oracle llama.cpp @ `237ad9b96` (`gateable = yes`): `common/arg.cpp:2451-2478` (`-ot`/`-cmoe`/`-ncmoe`), `common/common.h:1046-1054` (`LLM_FFN_EXPS_REGEX`), `src/llama-model-loader.cpp:1158-1160`, `src/llama-model.cpp:1032`, `include/llama.h:530`, auto-fit `common/fit.h:24` + `common/fit.cpp:457,485` (TP conflict refused at `:181`) | - | - | [hybrid-placement.md](specs/hybrid-placement.md) | `READY` | - |
Expand Down
Loading
Loading