From 1569adc090b7420a691464ef080c728b65927be5 Mon Sep 17 00:00:00 2001 From: Ketor Date: Sun, 30 Aug 2026 03:50:22 +0800 Subject: [PATCH 1/2] fix(vllm): window long deduplicated KV loads --- CHANGELOG.md | 12 ++ docs/CONNECTORS.md | 3 + integration/vllm/README.md | 18 ++- integration/vllm/src/dfkv_vllm/worker.py | 128 ++++++++++++++++-- .../vllm/tests/test_worker_lifecycle.py | 56 ++++++++ 5 files changed, 206 insertions(+), 11 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 10ba61d..a8ae7fd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,18 @@ ## Unreleased +### vLLM long-context node-dedup load windows + +- Added opt-in `load_window_keys` and `load_window_min_keys` connector settings. + Large replicated loads can now publish bounded native GET results before the + same-host GPU dedup wait deadline, while short requests retain one native GET. +- Defaults remain disabled. Values are validated at startup, preserve key/result + order, and retain one outer connector metric for the logical load. +- On xb01-0064, a 1,000,000-token TP8 GLM-5.3 hot load with 128-key windows + reduced TTFT from 39.35 s to 22.92 s and reduced native remote fetches from + approximately eight copies to one, with zero dedup fallback. A 4,096-key + threshold kept the 65,536-token/C10 path unwindowed at 35,622 total tok/s. + ### Adaptive RDMA receive capacity and bounded high-water reclaim - Replaced the eagerly committed monolithic RDMA receive segment with a lazy diff --git a/docs/CONNECTORS.md b/docs/CONNECTORS.md index a519581..3834135 100644 --- a/docs/CONNECTORS.md +++ b/docs/CONNECTORS.md @@ -785,6 +785,9 @@ namespace/key 不一致是预期 cold miss。**空环 / MDS 不可达**可直接 | `rail_affinity_fallbacks` | `1` | `1` | 相邻有序 fallback 数；`0`=严格单 rail，超出可用 rail 数时自动收敛 | | `load_async` | `True` | 普通 attention 保持 True；hybrid recurrent 模型设 `False` | `False` 在 forward 前同步完成 load，避免 recurrent-state compute 与远端 GPU 写重叠 | | `transfer_queue_capacity` | `256` | 保持默认，按压测调 | 每个 worker、每个方向的排队上限（`1..65536`）。满队列时非阻塞拒绝新任务：save 立即释放 finish/free fence，load 标记失败并重算；非法值启动即失败。 | +| `recv_workers` | `1` | 从 `1` 起压测 | 共享有界 receive queue 的 GET worker 数（`1..32`）；仅在 queue wait 持续升高且后端仍有余量时增加。 | +| `load_window_keys` | `0`（关闭） | 长上下文 replicated-MLA 按压测设置 | 单次 native GET window 的最大 key 数（`0..65536`）。窗口结果应能放入 `DFKV_NODE_DEDUP_GPU_ARENA_MB`，且在 `DFKV_NODE_DEDUP_WAIT_MS` 内完成，避免 follower rank 超时后重复读取同一批 KV。 | +| `load_window_min_keys` | `0` | 高于常规请求 key 数 | 仅当请求 key 数达到此阈值时启用 `load_window_keys`（`0..65536`）；让短请求保持单次 GET，长上下文才窗口化。`load_window_keys=0` 时无效。 | | `enable_cross_layers_blocks` | `False` | 默认 False | 仅当引擎分页布局层内交错时开 | | `lookup_rpc_port` | ipc 自动 | 一般不设 | rank0 前缀查询 RPC，仅 socket 名冲突时设 | | `client_register` | `1`（MDS 发现时） | 默认即可 | MDS 客户端注册开关（`0` 关；env `DFKV_CLIENT_REGISTER=0` 等价，见 §1.1） | diff --git a/integration/vllm/README.md b/integration/vllm/README.md index f531f4f..103c857 100644 --- a/integration/vllm/README.md +++ b/integration/vllm/README.md @@ -90,6 +90,8 @@ LMCache connector access logs, so one setting covers every integration. Format: | `load_async` | `True` | `True` returns `WAITING_FOR_REMOTE_KVS` and overlaps GPUDirect loads with unrelated model work. `False` performs each requested load synchronously in `start_load_kv`, before the forward pass. Use `False` for hybrid state-cache models when the engine cannot guarantee that remote writes target blocks disjoint from concurrent compute. | | `transfer_queue_capacity` | `256` | Maximum queued requests in each direction (`1..65536`). All receive workers consume one shared receive queue of this capacity; capacity is not multiplied by `recv_workers`. Submission is non-blocking: a full queue rejects new saves as completed (releasing finish/free fences) and rejects new loads as load errors (forcing recompute), so overload cannot grow memory or pin blocks indefinitely. Invalid or out-of-range values abort connector construction. | | `recv_workers` | `1` | Receive/load worker count (`1..32`). Workers consume the shared bounded receive queue and may execute independent native GETs concurrently. Invalid, boolean, or out-of-range values abort connector construction. | +| `load_window_keys` | `0` (disabled) | Maximum keys per native GET window (`0..65536`). Use a value whose worst-case result bytes fit inside the node-dedup GPU arena. Windowing lets follower ranks consume published results before the dedup wait deadline instead of re-fetching a large replicated-MLA batch. | +| `load_window_min_keys` | `0` | Apply `load_window_keys` only when the request contains at least this many keys (`0..65536`). Set a threshold to keep small, latency-sensitive loads as one native GET while windowing long-context loads. Has no effect when `load_window_keys=0`. | | `enable_cross_layers_blocks` | `False` | opt-in for engines whose paged layout interleaves layers within a block. Leave `False` unless you know the layout needs it. | | `lookup_rpc_port` | (ipc auto) | port for the rank-0 scheduler-side prefix-lookup RPC; set only if the default IPC socket name collides. | @@ -110,8 +112,20 @@ with `vllm:dfkv_receive_active_workers`; all three are Prometheus histograms queue-wait quantiles, active-worker samples, dfkv GET latency, and failed/recompute counts at each setting rather than assuming more workers are faster. Startup must contain the evidence line -`dfkv transfer queues: capacity= per direction, recv_workers=, overload=reject-new, shutdown=cancel-pending`; -capture it together with the before/after Prometheus snapshots. +`dfkv transfer queues: capacity= per direction, recv_workers=, +load_window_keys=, load_window_min_keys=, overload=reject-new, +shutdown=cancel-pending`; capture it together with the before/after Prometheus +snapshots. + +For same-host replicated loads, a single native GET publishes deduplicated GPU +results only after its storage fetch completes. If a large batch takes longer +than `DFKV_NODE_DEDUP_WAIT_MS`, follower ranks time out and independently +re-fetch it. Set `load_window_keys` so one window completes within that deadline +and its result bytes fit `DFKV_NODE_DEDUP_GPU_ARENA_MB`. Then set +`load_window_min_keys` above ordinary request sizes to avoid adding native-call +overhead to short loads. Validate with `DFKV_CLIENT_NODE_DEDUP_LOG=1`: the +windowed long-context path should report zero `fallback` and aggregate +`fetched` counts near one logical copy, not one copy per TP rank. ## Reproducible external-cache benchmark diff --git a/integration/vllm/src/dfkv_vllm/worker.py b/integration/vllm/src/dfkv_vllm/worker.py index 6dcb730..29dacb3 100644 --- a/integration/vllm/src/dfkv_vllm/worker.py +++ b/integration/vllm/src/dfkv_vllm/worker.py @@ -252,6 +252,10 @@ def _logical_block_ids( MAX_TRANSFER_QUEUE_CAPACITY = 65536 DEFAULT_RECV_WORKERS = 1 MAX_RECV_WORKERS = 32 +DEFAULT_LOAD_WINDOW_KEYS = 0 +MAX_LOAD_WINDOW_KEYS = 65536 +DEFAULT_LOAD_WINDOW_MIN_KEYS = 0 +MAX_LOAD_WINDOW_MIN_KEYS = 65536 _TRANSFER_STOP = object() @@ -286,6 +290,79 @@ def _parse_recv_workers(value: Any) -> int: ) return workers +def _parse_load_window_keys(value: Any) -> int: + if isinstance(value, bool): + raise ValueError("load_window_keys must be an integer") + if isinstance(value, int): + window = value + elif isinstance(value, str) and value.strip().isdigit(): + window = int(value.strip()) + else: + raise ValueError("load_window_keys must be an integer") + if not 0 <= window <= MAX_LOAD_WINDOW_KEYS: + raise ValueError( + f"load_window_keys must be in [0, {MAX_LOAD_WINDOW_KEYS}], got {window}" + ) + return window + +def _parse_load_window_min_keys(value: Any) -> int: + if isinstance(value, bool): + raise ValueError("load_window_min_keys must be an integer") + if isinstance(value, int): + minimum = value + elif isinstance(value, str) and value.strip().isdigit(): + minimum = int(value.strip()) + else: + raise ValueError("load_window_min_keys must be an integer") + if not 0 <= minimum <= MAX_LOAD_WINDOW_MIN_KEYS: + raise ValueError( + "load_window_min_keys must be in " + f"[0, {MAX_LOAD_WINDOW_MIN_KEYS}], got {minimum}" + ) + return minimum + +def _load_windows( + length: int, + window_keys: int, + min_keys: int = 0, +) -> tuple[tuple[int, int], ...]: + if length < 0: + raise ValueError("length must be non-negative") + if length == 0: + return () + if length < min_keys or window_keys <= 0 or window_keys >= length: + return ((0, length),) + return tuple( + (start, min(start + window_keys, length)) + for start in range(0, length, window_keys) + ) + +def _batch_get_auto_sg_windowed( + client: Any, + keys: list[bytes], + seg_ptrs: list[list[int]], + seg_caps: list[list[int]], + window_keys: int, + min_keys: int, +) -> tuple[list[bool], list[int]]: + windows = _load_windows(len(keys), window_keys, min_keys) + if not windows: + return [], [] + if len(windows) == 1: + return client.batch_get_auto_sg(keys, seg_ptrs, seg_caps) + + hits: list[bool] = [] + lengths: list[int] = [] + for start, stop in windows: + window_hits, window_lengths = client.batch_get_auto_sg( + keys[start:stop], + seg_ptrs[start:stop], + seg_caps[start:stop], + ) + hits.extend(window_hits) + lengths.extend(window_lengths) + return hits, lengths + @dataclasses.dataclass class _ReceiveRequestState: @@ -817,6 +894,8 @@ def __init__( client_provider: Callable[[], Any] | None = None, queue_capacity: int = DEFAULT_TRANSFER_QUEUE_CAPACITY, recv_workers: int = DEFAULT_RECV_WORKERS, + load_window_keys: int = DEFAULT_LOAD_WINDOW_KEYS, + load_window_min_keys: int = DEFAULT_LOAD_WINDOW_MIN_KEYS, record_pool_sample: Callable[[str, int], None] | None = None, ): super().__init__( @@ -832,6 +911,10 @@ def __init__( queue_capacity=queue_capacity, ) self.recv_workers = _parse_recv_workers(recv_workers) + self.load_window_keys = _parse_load_window_keys(load_window_keys) + self.load_window_min_keys = _parse_load_window_min_keys( + load_window_min_keys + ) self.client_provider = client_provider self._invalid_block_ids_lock = threading.Lock() self._invalid_block_ids: set[int] = set() @@ -1161,9 +1244,10 @@ def _handle_request(self, req_meta: ReqMeta): rotated_block_ids = _rotate_list(block_id_list, rotation) descriptor_batch = SgDescriptorBatch.from_chunks(rotated_chunks) - # dfkv: one logical chunk is one batch key with its complete - # destination segment vector. libdfkv performs any HCA-width - # multi-WR windowing without changing object identity. + # One logical chunk remains one batch key with its complete + # destination segment vector. Optional key windows bound each + # native dedup publication; libdfkv still owns HCA-width multi-WR + # windowing within each object. client = self.client if client is None and self.client_provider is not None: client = self.client_provider() # lazy un-elide @@ -1176,17 +1260,23 @@ def _handle_request(self, req_meta: ReqMeta): load_get_start = time.perf_counter() try: - # The native logical operation publishes a hit only after every - # ordered WR window completes. Require the exact stored length; - # misses and either short or oversized objects fail the chunk - # closed and force vLLM to recompute it. + # Native node dedup publishes a fetched result only when the + # enclosing batch_get_auto_sg call completes. Bounded key + # windows let follower ranks consume large replicated loads + # before their wait deadline while preserving result order. + # Require the exact stored length; misses and either short or + # oversized objects fail the chunk closed and force vLLM to + # recompute it. if client is None: # no client (un-elide failed): miss -> recompute hits, lens = [False] * len(rotated_keys), [0] * len(rotated_keys) else: - hits, lens = client.batch_get_auto_sg( + hits, lens = _batch_get_auto_sg_windowed( + client, rotated_keys, descriptor_batch.ptrs, descriptor_batch.caps, + self.load_window_keys, + self.load_window_min_keys, ) if len(hits) != len(rotated_keys) or len(lens) != len(rotated_keys): raise RuntimeError( @@ -1348,11 +1438,23 @@ def __init__( self.recv_workers = _parse_recv_workers( extra.get("recv_workers", DEFAULT_RECV_WORKERS) ) + self.load_window_keys = _parse_load_window_keys( + extra.get("load_window_keys", DEFAULT_LOAD_WINDOW_KEYS) + ) + self.load_window_min_keys = _parse_load_window_min_keys( + extra.get( + "load_window_min_keys", + DEFAULT_LOAD_WINDOW_MIN_KEYS, + ) + ) logger.info( "dfkv transfer queues: capacity=%d per direction, recv_workers=%d, " - "overload=reject-new, shutdown=cancel-pending", + "load_window_keys=%d, load_window_min_keys=%d, overload=reject-new, " + "shutdown=cancel-pending", self.transfer_queue_capacity, self.recv_workers, + self.load_window_keys, + self.load_window_min_keys, ) self._close_lock = threading.Lock() self._closed = False @@ -1873,6 +1975,14 @@ def _repr_tensor(v: torch.Tensor | list[torch.Tensor]) -> torch.Tensor: ), recv_workers=getattr(self, "recv_workers", DEFAULT_RECV_WORKERS), record_pool_sample=self._record_kv_connector_pool_sample, + load_window_keys=getattr( + self, "load_window_keys", DEFAULT_LOAD_WINDOW_KEYS + ), + load_window_min_keys=getattr( + self, + "load_window_min_keys", + DEFAULT_LOAD_WINDOW_MIN_KEYS, + ), ) self.kv_recv_thread.start() ready_event_recving.wait() diff --git a/integration/vllm/tests/test_worker_lifecycle.py b/integration/vllm/tests/test_worker_lifecycle.py index a9bfb24..7681377 100644 --- a/integration/vllm/tests/test_worker_lifecycle.py +++ b/integration/vllm/tests/test_worker_lifecycle.py @@ -9,6 +9,10 @@ DfkvStoreWorker, KVCacheStoreSendingThread, KVTransferThread, + _batch_get_auto_sg_windowed, + _load_windows, + _parse_load_window_keys, + _parse_load_window_min_keys, _parse_transfer_queue_capacity, ) @@ -156,6 +160,58 @@ def test_queue_capacity_is_bounded_and_fail_closed(self): with self.assertRaises(ValueError): _parse_transfer_queue_capacity(value) + def test_load_window_keys_is_bounded(self): + self.assertEqual(_parse_load_window_keys(0), 0) + self.assertEqual(_parse_load_window_keys("64"), 64) + for value in (-1, 65537, True, 1.5, "auto"): + with self.subTest(value=value): + with self.assertRaises(ValueError): + _parse_load_window_keys(value) + + def test_load_window_min_keys_is_bounded(self): + self.assertEqual(_parse_load_window_min_keys(0), 0) + self.assertEqual(_parse_load_window_min_keys("4096"), 4096) + for value in (-1, 65537, True, 1.5, "auto"): + with self.subTest(value=value): + with self.assertRaises(ValueError): + _parse_load_window_min_keys(value) + + def test_load_windows_preserve_order_and_cover_exactly_once(self): + self.assertEqual(_load_windows(0, 64), ()) + self.assertEqual(_load_windows(3, 0), ((0, 3),)) + self.assertEqual(_load_windows(3, 8), ((0, 3),)) + self.assertEqual(_load_windows(10, 4), ((0, 4), (4, 8), (8, 10))) + self.assertEqual(_load_windows(10, 4, 11), ((0, 10),)) + self.assertEqual(_load_windows(10, 4, 10), ((0, 4), (4, 8), (8, 10))) + + def test_windowed_get_preserves_results_and_skips_small_batches(self): + class FakeClient: + def __init__(self): + self.calls = [] + + def batch_get_auto_sg(self, keys, ptrs, caps): + self.calls.append((list(keys), list(ptrs), list(caps))) + return [key[0] % 2 == 0 for key in keys], [cap[0] for cap in caps] + + keys = [bytes([index]) for index in range(10)] + ptrs = [[100 + index] for index in range(10)] + caps = [[200 + index] for index in range(10)] + + windowed = FakeClient() + hits, lengths = _batch_get_auto_sg_windowed( + windowed, keys, ptrs, caps, window_keys=4, min_keys=0 + ) + self.assertEqual([len(call[0]) for call in windowed.calls], [4, 4, 2]) + self.assertEqual(hits, [index % 2 == 0 for index in range(10)]) + self.assertEqual(lengths, [200 + index for index in range(10)]) + + below_threshold = FakeClient() + _batch_get_auto_sg_windowed( + below_threshold, keys, ptrs, caps, window_keys=4, min_keys=11 + ) + self.assertEqual(len(below_threshold.calls), 1) + self.assertEqual(below_threshold.calls[0], (keys, ptrs, caps)) + @unittest.skipUnless(HAVE_VLLM, "requires vllm (dfkv_vllm.worker imports it)") class WorkerCloseTest(unittest.TestCase): From 32389b31c6662ef0fadb3f7a1bb369d0bd2b7f65 Mon Sep 17 00:00:00 2001 From: Ketor Date: Sun, 30 Aug 2026 11:40:52 +0800 Subject: [PATCH 2/2] docs(vllm): add load-window sizing guidance --- docs/CONNECTORS.md | 54 ++++++++++++++++++++++++++++++++ integration/vllm/README.md | 64 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 118 insertions(+) diff --git a/docs/CONNECTORS.md b/docs/CONNECTORS.md index 3834135..63414aa 100644 --- a/docs/CONNECTORS.md +++ b/docs/CONNECTORS.md @@ -798,6 +798,60 @@ DCP（decode context parallel）宽度/Rank **不是**本表键：连接器从 v （`get_dcp_group().world_size`），与 `tp_rank=-1`（replicated-MLA 存储坐标）一起进入 canonical key metadata（§1.4），勿在 extra_config 手工设定。 +#### 3.4.1 `load_window_keys` / `load_window_min_keys` 取值最佳实践 + +默认保持两个参数均为`0`。只有同机node-dedup日志出现follower `fallback`， +或各rank累计`fetched`接近`TP宽度 × 逻辑key数`时，才启用窗口。已经能在等待 +期限内完成publish的请求不会因窗口获益，反而会增加native调用开销。 + +两个参数的单位都是**dfkv key数，不是token数**。模型、KV dtype、block size、 +cache-group geometry、TP/DCP布局任一变化后，都要从代表性 +`batch_get_auto_sg` access log和connector geometry重新取得批次key数、结果 +bytes/key及耗时分布，不能照搬其他模型的数值。 + +`load_window_keys`同时受两个上界约束： + +1. **arena上界**：先选保守的`sizing_bytes_per_key`（p99或geometry最大值）。 + 作为运维起点，让单窗口不超过`DFKV_NODE_DEDUP_GPU_ARENA_MB`的75%： + `window_keys <= floor(0.75 × arena_bytes / sizing_bytes_per_key)`。 +2. **等待上界**：单窗口fetch+publish的p99耗时应明显低于 + `DFKV_NODE_DEDUP_WAIT_MS`，建议控制在等待时间的50%–70%以内。 + 拉长wait不能替代缩小过大的publish批次。 + +取两个上界中的较小值，向下取32、64、128等便于比较的档位；一次只升一档。 +只有在`fallback=0`、累计`fetched`仍接近一份逻辑key、TTFT继续改善时才保留。 +窗口过小会增加调用次数；窗口过大可能使arena被覆盖或再次越过wait deadline。 + +`load_window_min_keys`用来隔离长短请求： + +- 严格高于延迟敏感短请求的p99 key数； +- 不高于必须窗口化的最小长请求key数； +- 两类key数范围重叠时，优先拆分engine pool；否则必须接受并量化短请求开销， + 不要填写一个无法由分布证明的阈值。 + +`load_window_keys=0`时minimum无效。请求key数低于minimum，或请求本身不大于 +window时，仍只执行一次native GET；因此minimum只有大于window时才改变行为。 + +一组**仅适用于本次GLM-5.3 TP8实测、不是通用默认值**的例子：单key约3.05 MB， +arena 512 MiB，128-key窗口约391 MB，在1,000 ms wait内完成；1M请求约15,624 +keys，remote fetch从约8份收敛到1份。`load_window_min_keys=4096`让约1,024-key +的短请求保持单GET： + +```json +{ + "recv_workers": 2, + "load_window_keys": 128, + "load_window_min_keys": 4096, + "load_async": true +} +``` + +生产验收必须包含同数据、fresh target的两组热轮： + +- **长请求**：access log每批不超过window，`fallback=0`，累计`fetched`接近 + 逻辑key数而非`TP × key数`，failed/recompute均为0； +- **短请求**：access log仍为单次native GET，吞吐和TTFT不超过业务验收阈值。 + 连接器实现 vLLM `shutdown()` 生命周期钩子：先停止接单并取消排队任务，再等待当前 native 操作完成、join 收发线程，最后仅关闭一次 native client。因而正常退出不依赖 daemon 线程或进程终止；过载和退出期间都不会静默留下永久占用的 KV block。 diff --git a/integration/vllm/README.md b/integration/vllm/README.md index 103c857..5298a66 100644 --- a/integration/vllm/README.md +++ b/integration/vllm/README.md @@ -127,6 +127,70 @@ overhead to short loads. Validate with `DFKV_CLIENT_NODE_DEDUP_LOG=1`: the windowed long-context path should report zero `fallback` and aggregate `fetched` counts near one logical copy, not one copy per TP rank. +### Choosing load-window values + +Keep both settings at `0` unless same-host node-dedup logs show follower +`fallback` or aggregate `fetched` counts approaching `TP size × logical keys`. +Windowing adds native-call overhead and does not improve a load that already +publishes before the follower deadline. + +Treat the values as **key counts, not token counts**. Derive batch key-count and +result-byte distributions from representative `batch_get_auto_sg` access logs; +use connector geometry or a conservative maximum for per-key sizing. Repeat +this after changing model, KV dtype, block size, cache-group geometry, or TP/DCP +layout. + +Choose `load_window_keys` with both constraints below: +1. **Arena bound.** Choose a conservative `sizing_bytes_per_key` (p99 or the + geometry maximum). As an operational starting point, keep one window below + 75% of `DFKV_NODE_DEDUP_GPU_ARENA_MB`: + `window_keys <= floor(0.75 × arena_bytes / sizing_bytes_per_key)`. +2. **Deadline bound.** The p99 fetch-and-publish time of one window should be + comfortably below `DFKV_NODE_DEDUP_WAIT_MS` (target at most 50–70%). A + larger wait is not a substitute for a batch whose publication is too large. + +Start from the smaller bound, round down to a convenient value such as 32, 64, +or 128, then increase one step at a time only while `fallback=0`, aggregate +`fetched` stays near one logical copy, and TTFT improves. Smaller windows reduce +publication latency but add calls; larger windows reduce call overhead but can +overflow/lap the arena or cross the wait deadline. + +Choose `load_window_min_keys` only after separating the workload classes: + +- set it strictly above the p99 key count of latency-sensitive short loads; +- keep it at or below the smallest long load that must be windowed; +- if those ranges overlap, use separate engine pools or accept a measured + tradeoff rather than hiding it with an arbitrary threshold. + +When `load_window_keys=0`, the minimum has no effect. A batch also remains one +native GET when it is below `load_window_min_keys` or no larger than +`load_window_keys`; consequently, a threshold only changes behavior when it is +larger than the window. + +Example from one GLM-5.3 TP8 deployment (not a portable default): the per-key +result size used for sizing was about 3.05 MB and the dedup arena was 512 MiB. +A 128-key window was about 391 MB, completed inside a 1,000 ms wait, and +reduced a 15,624-key load from about eight remote copies to one. +`load_window_min_keys=4096` kept +approximately 1,024-key short loads on the single-GET path: + +```json +{ + "recv_workers": 2, + "load_window_keys": 128, + "load_window_min_keys": 4096, + "load_async": true +} +``` + +Production acceptance requires two byte-identical hot tests: + +- **long path:** access logs show batches no larger than the window, zero + `fallback`, aggregate `fetched` near logical keys rather than `TP × keys`, + and no failed/recomputed keys; +- **short path:** access logs show one native GET and throughput/TTFT do not + regress beyond the deployment's acceptance threshold. + ## Reproducible external-cache benchmark `test/python/vllm_external_cache_benchmark.py` never resets caches, deploys