fix(vllm): window long deduplicated KV loads

CHANGELOG.md

@@ -2,6 +2,18 @@
 
 ## Unreleased
 
+### vLLM long-context node-dedup load windows
+
+- Added opt-in `load_window_keys` and `load_window_min_keys` connector settings.
+  Large replicated loads can now publish bounded native GET results before the
+  same-host GPU dedup wait deadline, while short requests retain one native GET.
+- Defaults remain disabled. Values are validated at startup, preserve key/result
+  order, and retain one outer connector metric for the logical load.
+- On xb01-0064, a 1,000,000-token TP8 GLM-5.3 hot load with 128-key windows
+  reduced TTFT from 39.35 s to 22.92 s and reduced native remote fetches from
+  approximately eight copies to one, with zero dedup fallback. A 4,096-key
+  threshold kept the 65,536-token/C10 path unwindowed at 35,622 total tok/s.
+
 ### Adaptive RDMA receive capacity and bounded high-water reclaim
 
 - Replaced the eagerly committed monolithic RDMA receive segment with a lazy

docs/CONNECTORS.md

@@ -785,6 +785,9 @@ namespace/key 不一致是预期 cold miss。**空环 / MDS 不可达**可直接
 | `rail_affinity_fallbacks` | `1` | `1` | 相邻有序 fallback 数；`0`=严格单 rail，超出可用 rail 数时自动收敛 |
 | `load_async` | `True` | 普通 attention 保持 True；hybrid recurrent 模型设 `False` | `False` 在 forward 前同步完成 load，避免 recurrent-state compute 与远端 GPU 写重叠 |
 | `transfer_queue_capacity` | `256` | 保持默认，按压测调 | 每个 worker、每个方向的排队上限（`1..65536`）。满队列时非阻塞拒绝新任务：save 立即释放 finish/free fence，load 标记失败并重算；非法值启动即失败。 |
+| `recv_workers` | `1` | 从 `1` 起压测 | 共享有界 receive queue 的 GET worker 数（`1..32`）；仅在 queue wait 持续升高且后端仍有余量时增加。 |
+| `load_window_keys` | `0`（关闭） | 长上下文 replicated-MLA 按压测设置 | 单次 native GET window 的最大 key 数（`0..65536`）。窗口结果应能放入 `DFKV_NODE_DEDUP_GPU_ARENA_MB`，且在 `DFKV_NODE_DEDUP_WAIT_MS` 内完成，避免 follower rank 超时后重复读取同一批 KV。 |
+| `load_window_min_keys` | `0` | 高于常规请求 key 数 | 仅当请求 key 数达到此阈值时启用 `load_window_keys`（`0..65536`）；让短请求保持单次 GET，长上下文才窗口化。`load_window_keys=0` 时无效。 |
 | `enable_cross_layers_blocks` | `False` | 默认 False | 仅当引擎分页布局层内交错时开 |
 | `lookup_rpc_port` | ipc 自动 | 一般不设 | rank0 前缀查询 RPC，仅 socket 名冲突时设 |
 | `client_register` | `1`（MDS 发现时） | 默认即可 | MDS 客户端注册开关（`0` 关；env `DFKV_CLIENT_REGISTER=0` 等价，见 §1.1） |
@@ -795,6 +798,60 @@ DCP（decode context parallel）宽度/Rank **不是**本表键：连接器从 v
 （`get_dcp_group().world_size`），与 `tp_rank=-1`（replicated-MLA 存储坐标）一起进入
 canonical key metadata（§1.4），勿在 extra_config 手工设定。
 
+#### 3.4.1 `load_window_keys` / `load_window_min_keys` 取值最佳实践
+
+默认保持两个参数均为`0`。只有同机node-dedup日志出现follower `fallback`，
+或各rank累计`fetched`接近`TP宽度 × 逻辑key数`时，才启用窗口。已经能在等待
+期限内完成publish的请求不会因窗口获益，反而会增加native调用开销。
+
+两个参数的单位都是**dfkv key数，不是token数**。模型、KV dtype、block size、
+cache-group geometry、TP/DCP布局任一变化后，都要从代表性
+`batch_get_auto_sg` access log和connector geometry重新取得批次key数、结果
+bytes/key及耗时分布，不能照搬其他模型的数值。
+
+`load_window_keys`同时受两个上界约束：
+
+1. **arena上界**：先选保守的`sizing_bytes_per_key`（p99或geometry最大值）。
+   作为运维起点，让单窗口不超过`DFKV_NODE_DEDUP_GPU_ARENA_MB`的75%：
+   `window_keys <= floor(0.75 × arena_bytes / sizing_bytes_per_key)`。
+2. **等待上界**：单窗口fetch+publish的p99耗时应明显低于
+   `DFKV_NODE_DEDUP_WAIT_MS`，建议控制在等待时间的50%–70%以内。
+   拉长wait不能替代缩小过大的publish批次。
+
+取两个上界中的较小值，向下取32、64、128等便于比较的档位；一次只升一档。
+只有在`fallback=0`、累计`fetched`仍接近一份逻辑key、TTFT继续改善时才保留。
+窗口过小会增加调用次数；窗口过大可能使arena被覆盖或再次越过wait deadline。
+
+`load_window_min_keys`用来隔离长短请求：
+
+- 严格高于延迟敏感短请求的p99 key数；
+- 不高于必须窗口化的最小长请求key数；
+- 两类key数范围重叠时，优先拆分engine pool；否则必须接受并量化短请求开销，
+  不要填写一个无法由分布证明的阈值。
+
+`load_window_keys=0`时minimum无效。请求key数低于minimum，或请求本身不大于
+window时，仍只执行一次native GET；因此minimum只有大于window时才改变行为。
+
+一组**仅适用于本次GLM-5.3 TP8实测、不是通用默认值**的例子：单key约3.05 MB，
+arena 512 MiB，128-key窗口约391 MB，在1,000 ms wait内完成；1M请求约15,624
+keys，remote fetch从约8份收敛到1份。`load_window_min_keys=4096`让约1,024-key
+的短请求保持单GET：
+
+```json
+{
+  "recv_workers": 2,
+  "load_window_keys": 128,
+  "load_window_min_keys": 4096,
+  "load_async": true
+}
+```
+
+生产验收必须包含同数据、fresh target的两组热轮：
+
+- **长请求**：access log每批不超过window，`fallback=0`，累计`fetched`接近
+  逻辑key数而非`TP × key数`，failed/recompute均为0；
+- **短请求**：access log仍为单次native GET，吞吐和TTFT不超过业务验收阈值。
+
 连接器实现 vLLM `shutdown()` 生命周期钩子：先停止接单并取消排队任务，再等待当前
 native 操作完成、join 收发线程，最后仅关闭一次 native client。因而正常退出不依赖
 daemon 线程或进程终止；过载和退出期间都不会静默留下永久占用的 KV block。

integration/vllm/README.md

@@ -90,6 +90,8 @@ LMCache connector access logs, so one setting covers every integration. Format:
 | `load_async` | `True` | `True` returns `WAITING_FOR_REMOTE_KVS` and overlaps GPUDirect loads with unrelated model work. `False` performs each requested load synchronously in `start_load_kv`, before the forward pass. Use `False` for hybrid state-cache models when the engine cannot guarantee that remote writes target blocks disjoint from concurrent compute. |
 | `transfer_queue_capacity` | `256` | Maximum queued requests in each direction (`1..65536`). All receive workers consume one shared receive queue of this capacity; capacity is not multiplied by `recv_workers`. Submission is non-blocking: a full queue rejects new saves as completed (releasing finish/free fences) and rejects new loads as load errors (forcing recompute), so overload cannot grow memory or pin blocks indefinitely. Invalid or out-of-range values abort connector construction. |
 | `recv_workers` | `1` | Receive/load worker count (`1..32`). Workers consume the shared bounded receive queue and may execute independent native GETs concurrently. Invalid, boolean, or out-of-range values abort connector construction. |
+| `load_window_keys` | `0` (disabled) | Maximum keys per native GET window (`0..65536`). Use a value whose worst-case result bytes fit inside the node-dedup GPU arena. Windowing lets follower ranks consume published results before the dedup wait deadline instead of re-fetching a large replicated-MLA batch. |
+| `load_window_min_keys` | `0` | Apply `load_window_keys` only when the request contains at least this many keys (`0..65536`). Set a threshold to keep small, latency-sensitive loads as one native GET while windowing long-context loads. Has no effect when `load_window_keys=0`. |
 | `enable_cross_layers_blocks` | `False` | opt-in for engines whose paged layout interleaves layers within a block. Leave `False` unless you know the layout needs it. |
 | `lookup_rpc_port` | (ipc auto) | port for the rank-0 scheduler-side prefix-lookup RPC; set only if the default IPC socket name collides. |
 
@@ -110,8 +112,84 @@ with `vllm:dfkv_receive_active_workers`; all three are Prometheus histograms
 queue-wait quantiles, active-worker samples, dfkv GET latency, and failed/recompute
 counts at each setting rather than assuming more workers are faster. Startup
 must contain the evidence line
-`dfkv transfer queues: capacity=<N> per direction, recv_workers=<N>, overload=reject-new, shutdown=cancel-pending`;
-capture it together with the before/after Prometheus snapshots.
+`dfkv transfer queues: capacity=<N> per direction, recv_workers=<N>,
+load_window_keys=<N>, load_window_min_keys=<N>, overload=reject-new,
+shutdown=cancel-pending`; capture it together with the before/after Prometheus
+snapshots.
+
+For same-host replicated loads, a single native GET publishes deduplicated GPU
+results only after its storage fetch completes. If a large batch takes longer
+than `DFKV_NODE_DEDUP_WAIT_MS`, follower ranks time out and independently
+re-fetch it. Set `load_window_keys` so one window completes within that deadline
+and its result bytes fit `DFKV_NODE_DEDUP_GPU_ARENA_MB`. Then set
+`load_window_min_keys` above ordinary request sizes to avoid adding native-call
+overhead to short loads. Validate with `DFKV_CLIENT_NODE_DEDUP_LOG=1`: the
+windowed long-context path should report zero `fallback` and aggregate
+`fetched` counts near one logical copy, not one copy per TP rank.
+
+### Choosing load-window values
+
+Keep both settings at `0` unless same-host node-dedup logs show follower
+`fallback` or aggregate `fetched` counts approaching `TP size × logical keys`.
+Windowing adds native-call overhead and does not improve a load that already
+publishes before the follower deadline.
+
+Treat the values as **key counts, not token counts**. Derive batch key-count and
+result-byte distributions from representative `batch_get_auto_sg` access logs;
+use connector geometry or a conservative maximum for per-key sizing. Repeat
+this after changing model, KV dtype, block size, cache-group geometry, or TP/DCP
+layout.
+
+Choose `load_window_keys` with both constraints below:
+1. **Arena bound.** Choose a conservative `sizing_bytes_per_key` (p99 or the
+   geometry maximum). As an operational starting point, keep one window below
+   75% of `DFKV_NODE_DEDUP_GPU_ARENA_MB`:
+   `window_keys <= floor(0.75 × arena_bytes / sizing_bytes_per_key)`.
+2. **Deadline bound.** The p99 fetch-and-publish time of one window should be
+   comfortably below `DFKV_NODE_DEDUP_WAIT_MS` (target at most 50–70%). A
+   larger wait is not a substitute for a batch whose publication is too large.
+
+Start from the smaller bound, round down to a convenient value such as 32, 64,
+or 128, then increase one step at a time only while `fallback=0`, aggregate
+`fetched` stays near one logical copy, and TTFT improves. Smaller windows reduce
+publication latency but add calls; larger windows reduce call overhead but can
+overflow/lap the arena or cross the wait deadline.
+
+Choose `load_window_min_keys` only after separating the workload classes:
+
+- set it strictly above the p99 key count of latency-sensitive short loads;
+- keep it at or below the smallest long load that must be windowed;
+- if those ranges overlap, use separate engine pools or accept a measured
+  tradeoff rather than hiding it with an arbitrary threshold.
+
+When `load_window_keys=0`, the minimum has no effect. A batch also remains one
+native GET when it is below `load_window_min_keys` or no larger than
+`load_window_keys`; consequently, a threshold only changes behavior when it is
+larger than the window.
+
+Example from one GLM-5.3 TP8 deployment (not a portable default): the per-key
+result size used for sizing was about 3.05 MB and the dedup arena was 512 MiB.
+A 128-key window was about 391 MB, completed inside a 1,000 ms wait, and
+reduced a 15,624-key load from about eight remote copies to one.
+`load_window_min_keys=4096` kept
+approximately 1,024-key short loads on the single-GET path:
+
+```json
+{
+  "recv_workers": 2,
+  "load_window_keys": 128,
+  "load_window_min_keys": 4096,
+  "load_async": true
+}
+```
+
+Production acceptance requires two byte-identical hot tests:
+
+- **long path:** access logs show batches no larger than the window, zero
+  `fallback`, aggregate `fetched` near logical keys rather than `TP × keys`,
+  and no failed/recomputed keys;
+- **short path:** access logs show one native GET and throughput/TTFT do not
+  regress beyond the deployment's acceptance threshold.
 
 ## Reproducible external-cache benchmark