perf: adapt RDMA depth and recycle receive capacity

CHANGELOG.md

@@ -14,11 +14,26 @@
   `DFKV_RDMA_CONNECTION_MIN_BLOCK_BYTES` (default 256 KiB) as the floor.
 - Idle endpoint reuse selects the smallest sufficient class, and a full idle
   pool prefers a smaller returning QP over its largest retained QP.
+- Extended adaptive geometry to QP depth: scalar operations open depth-1 QPs,
+  while batched operations select the smallest sufficient power-of-two depth.
+  Pool reuse and resource accounting now match both block and depth classes,
+  with bounded per-class opened/active/idle metrics.
+- Replaced per-connection pull-read MR registration with type-2 Memory Windows
+  over shared per-PD chunk MRs. Hardware without Memory Window support retains
+  the exact-range MR fallback and exposes which path each connection used.
+- Affinitized growth chunks to their first rail/NUMA node, and added idle
+  shrinking for empty non-initial chunks (`DFKV_RDMA_RECV_CHUNK_IDLE_MS`,
+  default 60 s). The initial shared chunk is never released.
 - Changed watermark eviction to a persistent high/low hysteresis drain bounded
   by `DFKV_SLAB_EVICT_MAX_EXTENTS_PER_TICK` (default one). Whole extents clear
   `slots.tbl` in one contiguous write instead of one 64-byte pwrite per slot.
-- Added receive-pool growth/budget metrics and watermark active/tick/duration/
-  extent-clear metrics.
+- Added receive-pool growth/shrink/budget, pull Memory Window/fallback,
+  adaptive connection-class, and watermark active/tick/duration/clear metrics.
+- Reused one preallocated zero-record buffer for extent metadata clears,
+  removing allocation and repeated zero-fill work from the watermark lock.
+- Completed vLLM replicated-MLA rank convergence: converged client-rank mode
+  automatically enables native same-host GPU rendezvous, so one rank performs
+  each remote GET and CUDA IPC publishes identical bytes to TP followers.
 - On xb01-0064, 80 live 1 MiB data QPs under a 64 MiB logical ceiling used
   674 MiB across three lazy chunks and completed 10,000/10,000 PUTs; v2.23.3's
   fixed geometry exhausted an 8 GiB segment at roughly 15 such QPs. Sustained

CMakeLists.txt

@@ -237,6 +237,12 @@ if(DFKV_BUILD_TESTS)
               -p test_rail_affinity.py)
     set_tests_properties(python_vllm_rail_affinity PROPERTIES
       ENVIRONMENT "PYTHONPATH=${CMAKE_CURRENT_SOURCE_DIR}/integration/common/src:${CMAKE_CURRENT_SOURCE_DIR}/integration/vllm/src")
+    add_test(NAME python_vllm_client_ranks
+      COMMAND ${PYTHON3} -m unittest discover
+              -s ${CMAKE_CURRENT_SOURCE_DIR}/integration/vllm/tests
+              -p test_client_ranks.py)
+    set_tests_properties(python_vllm_client_ranks PROPERTIES
+      ENVIRONMENT "PYTHONPATH=${CMAKE_CURRENT_SOURCE_DIR}/integration/common/src:${CMAKE_CURRENT_SOURCE_DIR}/integration/vllm/src")
     # Shared telemetry layer: pure-python, no build/lib needed (OTel-push test
     # self-skips when the SDK is absent). Includes the vendored-copy drift guard.
     add_test(NAME python_telemetry

README.md

@@ -322,6 +322,7 @@ fabric selection and capacity explicit.
 | `--rdma-depth` | `4` (default) | Handshake window is `min(client, server)`. Pull-read connections lease `2 × depth × adaptive_slot_size`; a 1 MiB actual object uses about 8.4 MiB even when the logical ceiling is 64 MiB. |
 | `DFKV_RDMA_RECV_SEGMENT_SIZE` | `16 GiB` hard budget | Maximum process receive-pool commitment; it is no longer allocated eagerly. Size for worst-case live/churn connections, but startup pins only one chunk. |
 | `DFKV_RDMA_RECV_CHUNK_BYTES` | `256 MiB` | Initial and incremental receive-pool commitment. Increase only if measured connection classes are routinely larger; committed chunks never exceed the hard budget. |
+| `DFKV_RDMA_RECV_CHUNK_IDLE_MS` | `60000` | Empty non-initial chunks are released after this hold; `0` disables shrinking. Growth chunks are rail-affinitized and NUMA-bound, while the initial shared chunk is retained. |
 | `DFKV_RDMA_IDLE_MS` | `30000` for frequently replaced clients; otherwise default `600000` | Server dead-client reaper. A short interval bounds leaked receive-segment leases, but live clients must set `DFKV_RDMA_KEEPALIVE_MS` below this value or their next GET pays stale-QP recovery. |
 | `DFKV_RDMA_HEALTH_RECOVERY_SAMPLES` | `3` | Any healthy initialized rail keeps placement eligible; a partial loss stays online. Loss of the final healthy rail removes the node immediately. Only recovery from zero to nonzero waits the consecutive sample gate, then rejoins as `PARTIAL` or `ACTIVE`. |
 | `DFKV_RDMA_HEALTH_FILE` | unset | Diagnostic-only health input (`device port_state phys_state`, one per initialized/resolved rail, including an auto-discovered rail) for controlled fault injection. Production MUST leave this unset so health comes from sysfs. |
@@ -337,7 +338,7 @@ fabric selection and capacity explicit.
 | `DFKV_RDMA_DEV` | leave unset for one local HCA; set each host's actual stable local whitelist for multi-rail | The local list may differ in cardinality on GPU and CPU hosts. Peer-aware selection uses exact shared names from HLT1; configured tiers require an explicit list. |
 | `DFKV_RDMA_RAIL_TIERS` | unset for homogeneous hosts; e.g. `mlx5_0|mlx5_1;mlx5_2` for heterogeneous hosts | Every name must be present in `DFKV_RDMA_DEV`; leftmost is highest priority. GPU and CPU hosts may expose different subsets, but interoperating rail names must describe the same fabric. The client uses the highest tier in `local enabled ∩ peer healthy`; missing/incomplete peer HLT1 fails closed only when tiers are configured. Credits, NUMA, latency, and quarantine never cause tier overflow. |
 | `DFKV_RDMA_PRIMARY_DEV` | unset | Optional preferred rail within `DFKV_RDMA_DEV`. Admission uses other configured rails only when the primary cannot serve. SGLang `rail_affinity=true` sets a rank-local primary plus one neighboring fallback by default; `rail_affinity_fallbacks=0` disables the fallback. |
-| `DFKV_RDMA_DEPTH` | `4` (default) | Keep client/server defaults aligned for connector batch correctness. Throughput scaling comes from multiple pooled connections, not raising one QP's depth; lowering depth reduces receive-segment consumption only after validating the real engine workload. |
+| `DFKV_RDMA_DEPTH` | `4` ceiling (default) | Actual QPs use the smallest sufficient depth class: scalar=1, batches round up by requested window and server cap. Keep the ceiling aligned with the server; class metrics show the resulting mix. |
 | `DFKV_RDMA_MAX_BLOCK_BYTES` | exact logical object ceiling | Deterministic oversize guard only; it no longer sizes every data QP. |
 | `DFKV_RDMA_CONNECTION_MIN_BLOCK_BYTES` | `256 KiB` | Smallest data-QP class. Each operation rounds its actual largest object to a power-of-two class; idle reuse picks the smallest sufficient QP. |
 | `DFKV_RDMA_RAIL_ERROR_THRESHOLD` | `3` (default) | Consecutive client-local rail failures required to quarantine that rail; endpoint/peer failures do not contribute. |

docs/ARCHITECTURE.md

@@ -167,14 +167,15 @@ Production discovery uses MDS.
 - `DFKV_RDMA_MAX_BLOCK_BYTES` is the logical object safety ceiling. Each data
   connection advertises `next_power_of_two(max(actual operation bytes,
   DFKV_RDMA_CONNECTION_MIN_BLOCK_BYTES))`, capped by that ceiling.
-- The server validates each connection class against `--max-msg`, negotiates
-  `qd=min(client depth, server depth)`, and leases `qd` receive plus `qd` pull
-  slots from any committed chunk. Idle reuse chooses the smallest sufficient
-  class; lease ownership lasts until QP teardown or idle reclaim.
+- The client selects a second power-of-two class from the operation's requested
+  window: scalar QPs open at depth 1, while batches reuse/open the smallest
+  sufficient depth up to the client/server ceiling.
+- The server validates the block class, negotiates the depth class, and leases
+  that many receive plus pull slots from rail-affinitized chunks. Pull arenas
+  use type-2 Memory Windows over a shared chunk MR, with exact-MR fallback.
 - A data slot is `align4K(4096 + connection_class)`. The hard receive budget
-  covers worst-case live and pooled QPs, but resident/pinned memory follows the
-  observed connection high-water in `DFKV_RDMA_RECV_CHUNK_BYTES` increments.
-  Exhausting the hard budget rejects the connection without changing protocol.
+  covers worst-case live/pooled QPs; resident memory grows by chunk and empty
+  non-initial chunks return after `DFKV_RDMA_RECV_CHUNK_IDLE_MS`.
 - Client host/device pools are registered once per rail at declaration time.
   Re-declaring the same base with a larger size registers the larger extent; the
   registration call returns false/nonzero unless the full range is ready. Buffers

docs/CONNECTORS.md

@@ -84,11 +84,12 @@ tp_rank=..,ver=<lib>`（无 `role`——HiCache 是前缀 L3 缓存，无生产/
 |-----|------|------|------|
 | `DFKV_RDMA` | 一般路径未设 = TCP；**vLLM 直连无默认，必须 `1`** | 按连接器选择 | `1` 显式选择 native-verbs RDMA v2；请求 RDMA 后设备或协议不可用会失败，不会自动选择 TCP。`DfkvStoreConnector` 只接收 GPU 设备指针，构造时会关闭并拒绝任何非 RDMA handle。 |
 | `DFKV_RDMA_DEV` | 首个 `ACTIVE` 本地 HCA | 留空让两端各自选本地首口；多轨才显式写同 fabric 白名单 | 留空时 bootstrap 不发送设备名，client/server 可使用不同本地命名。逗号列表显式开启多轨，新连接在健康轨间轮转；显式设备名会发给 peer，故两端必须存在同名且互通的 fabric。设备名上限 **18 字节**（v2 bootstrap dev frame 限制），超长即 fail-fast 拒绝启动/建连，不会静默截断（见 `src/transport/dev_frame.h`）。 |
-| `DFKV_RDMA_DEPTH` | `4` | 两侧可不同，按容量选 | 握手协商 `min(client,server)`。每条 pull data QP 租 `2 × depth × adaptive_slot`；control buffer 始终有界。 |
+| `DFKV_RDMA_DEPTH` | `4` ceiling | 两侧保持上限一致 | scalar QP=depth1；batch按实际window选择最小power-of-two depth class，再由server cap钳制。 |
 | `DFKV_RDMA_MAX_BLOCK_BYTES` | 4 MiB 逻辑上限 | 覆盖最大合法对象 | 只做 deterministic oversize guard；不再让所有连接按最大值预留。 |
 | `DFKV_RDMA_CONNECTION_MIN_BLOCK_BYTES` | 256 KiB | 覆盖常见小块，保持默认起步 | 当前操作最大对象按 power-of-two 向上取 connection class；idle pool 选最小可满足 QP。 |
 | `DFKV_RDMA_RECV_SEGMENT_SIZE` | 16 GiB | 按 peak live/pooled QP 设 hard budget | server receive-pool 最大提交量；不再启动期全量申请。 |
 | `DFKV_RDMA_RECV_CHUNK_BYTES` | 256 MiB | 保持默认，除非常见 connection class 更大 | server 启动只提交一个 chunk，后续 allocation miss 按需增长，不超过 hard budget。 |
+| `DFKV_RDMA_RECV_CHUNK_IDLE_MS` | 60 s | 保持默认 | 空闲非初始chunk到期返还；`0`关闭。增长chunk绑定首次使用rail/NUMA。 |
 | `DFKV_RDMA_NUMA` | `0` | 显式多轨的大机可设 `1` | 建连按调用线程 NUMA 选 rail；动态 chunk 在实际租用它的 endpoint rail 上注册。 |
 | `DFKV_RDMA_MAX_PAYLOAD_BYTES` | 64 MiB（67108864） | — | 客户端单 value payload 上限（不得超过 server 侧同名上限） |
 | `DFKV_CUDA_PINNED_POOL_BYTES` | 64 MiB（67108864） | 按进程允许的 CUDA pinned host memory / memlock 设置 | vLLM CUDA GET publication 的进程级 bounce pool 预算。取正十进制字节数，最大 4 GiB；预算向下取整为完整 slot，且至少容纳一个 slot、最多 4096 个。非法或与 slot 不兼容的组合告警并回退整组默认值。slot 按需创建，因此实际 pinned high-water 不超过取整后的预算。 |
@@ -132,8 +133,10 @@ connection_class =
   min(logical_max,
       next_power_of_two(max(actual_bytes,
                             DFKV_RDMA_CONNECTION_MIN_BLOCK_BYTES)))
+depth_class = min(server_depth,
+                  next_power_of_two(max(actual_window, 1)))
 S_data = align4K(4096 + connection_class)
-B_connection = 2 × depth × S_data
+B_connection = 2 × depth_class × S_data
 ```
 
 同一 peer/rail 的 idle pool 选择最小可满足 class；没有才建新 QP。pool 满时，
@@ -586,9 +589,9 @@ server 启动只注册首个 receive chunk；新 data QP 租当前 operation cla
 L2-bypass 不需要独立 server 协议，但三项决定 v2 容量：
 
 - `DFKV_RDMA_RECV_SEGMENT_SIZE`：receive-pool hard budget；
-- `DFKV_RDMA_RECV_CHUNK_BYTES`：启动/增量提交粒度；
+- `DFKV_RDMA_RECV_CHUNK_BYTES` / `_IDLE_MS`：启动/增量提交粒度与空chunk保留期；
 - `DFKV_RDMA_MAX_BLOCK_BYTES`、`DFKV_RDMA_CONNECTION_MIN_BLOCK_BYTES` 与
-  `DFKV_RDMA_DEPTH`：分别决定逻辑上限、最小 class 和每 QP lease。
+  `DFKV_RDMA_DEPTH`：决定逻辑上限、最小block class和adaptive depth ceiling。
 
 #### 验证清单
 

docs/DEPLOY.md

@@ -362,11 +362,11 @@ journalctl -u dfkv -n 10 --no-pager
 > `kNoCompatibleRail` 均不增加 local rail error。cooldown 到期仍只准入一个真实
 > recovery probe。
 >
-> **v2 receive-pool 预算**：每条 data QP 按实际 connection class 计算
-> `slot=align4K(4096 + class)`，pull-read lease 为
-> `2 × depth × slot`。`DFKV_RDMA_RECV_SEGMENT_SIZE` 是 hard budget，
-> `DFKV_RDMA_RECV_CHUNK_BYTES` 是惰性提交粒度。上线看 committed/max/chunks、
-> used/free、growth/allocation failures；只有 hard budget 或注册失败才拒绝连接。
+> **v2 receive-pool 预算**：每条data QP按实际block/depth二维class计算
+> `slot=align4K(4096 + block_class)`，pull-read lease为
+> `2 × depth_class × slot`。`DFKV_RDMA_RECV_SEGMENT_SIZE`是hard budget，
+> `_CHUNK_BYTES`是惰性提交粒度，`_CHUNK_IDLE_MS`控制空chunk返还。
+> 上线看committed/max/chunks、used/free、growth/shrink/failure和MW/fallback。
 
 ### 3a. 每节点 tenant quota
 
@@ -445,7 +445,7 @@ flag 为 env facade）；未列 flag 的全部 env 均从源码排查就不误
 | `--put-inflight-limit` / `DFKV_PUT_INFLIGHT_LIMIT` | `0`=关 | 并发盘写上限，超出返回 kCacheFull 快速拒绝 |
 | `--tcp-max-conns` / `DFKV_TCP_MAX_CONNS` | `512`, 硬上限 4096 | cache TCP handler 上限；超限 accept 恒拒 |
 | `--tcp-io-timeout-s` / `DFKV_TCP_IO_TIMEOUT_S` | `60`, 硬上限 3600 | per-syscall RCVTIMEO（秒） |
-| `--rdma-depth` / `DFKV_RDMA_DEPTH` | `4` | server 提交 QP post 深度；与 client 协商取 `min` |
+| `--rdma-depth` / `DFKV_RDMA_DEPTH` | `4` | server ceiling；client scalar=1，batch按实际window选择depth class后再协商取min |
 | `--rdma-numa` / `DFKV_RDMA_NUMA` | `0` | NUMA-aware rail choice（off/1） |
 | `--rdma-idle-ms` / `DFKV_RDMA_IDLE_MS` | — | idle connection reaper tick |
 | `--rdma-op-timeout-ms` / `DFKV_RDMA_OP_TIMEOUT_MS` | `5000` | per-op RDMA deadline |
@@ -465,6 +465,7 @@ flag 为 env facade）；未列 flag 的全部 env 均从源码排查就不误
 |---|---|---|
 | `DFKV_RDMA_RECV_SEGMENT_SIZE` | `16 GiB` | server receive-pool hard budget；不再启动期全量分配 |
 | `DFKV_RDMA_RECV_CHUNK_BYTES` | `256 MiB` | server 启动与增量提交粒度 |
+| `DFKV_RDMA_RECV_CHUNK_IDLE_MS` | `60000` | 空闲非初始chunk返还延迟；`0`关闭缩容 |
 | `DFKV_RDMA_CONNECTION_MIN_BLOCK_BYTES` | `256 KiB` | client adaptive data-QP 最小 class；实际对象向上取 power-of-two |
 | `DFKV_RDMA_CONNECT_MS` | — | client：IB QP 建连超时 |
 | `DFKV_RDMA_IO_MS` | — | client：控制面帧读写超时 |

docs/METRICS.md

@@ -162,8 +162,10 @@ RDMA-listener scrape inventory。
 | `dfkv_rdma_recv_segment_used_bytes` / `free_bytes` | gauge | 已提交 chunk 中 lease 占用 / 空闲字节 |
 | `dfkv_rdma_recv_segment_largest_free_range_bytes` | gauge | 任一 chunk 最大连续 free range |
 | `dfkv_rdma_recv_segment_growths_total` / `growth_failures_total` | counter | 启动后 chunk 增长成功 / 因预算或分配失败 |
+| `dfkv_rdma_recv_segment_shrinks_total` / `released_bytes_total` / `chunk_idle_ms` | counter / gauge | 空闲缩容次数 / 已返还字节 / 非初始chunk空闲保留期 |
 | `dfkv_rdma_recv_segment_allocation_failures_total` | counter | grow 后仍无法满足的最终 allocation |
 | `dfkv_rdma_pull_connections` / `dfkv_rdma_legacy_connections` | gauge | 当前 pull-read / legacy responder-write connection 数 |
+| `dfkv_rdma_pull_memory_windows_total` / `dfkv_rdma_pull_mr_fallbacks_total` | counter | exact lease使用type-2 MW隔离 / 硬件不支持时回退per-connection MR |
 | `dfkv_rdma_connection_bytes{class=\"data|control\"}` | gauge | data/control connection 当前 lease 字节；应随 adaptive class 而非 logical max 增长 |
 | `dfkv_rdma_recv_segment_registered_rails` | gauge | 成功注册初始 receive chunk 的 rail 数；后续 chunk 按使用 rail 惰性注册 |
 | `dfkv_rdma_v2_ready` | gauge | 初始 receive chunk 与 rail anchor 是否就绪 |
@@ -337,6 +339,8 @@ C 客户端快照还含传输级指标（RDMA 构建）：
 | `dfkv_rdma_client_pool_mr_registrations_total` / `dfkv_rdma_client_pool_mr_registration_failures_total` | counter | shared-PD 上真实 `ibv_reg_mr` 次数 / 单 rail 显式 pool 注册失败；包含最终回滚的尝试 |
 | `dfkv_rdma_client_pool_mr_active_registrations` | gauge | 进程内仍有 endpoint 引用的 shared-PD MR generations；扩容成功后 anchor/空闲 endpoint 立即释放旧代，在飞旧 endpoint 到下次 acquire/close 才释放，确保旧 range 不中断 |
 | `dfkv_rdma_client_max_block_seen_bytes` / `dfkv_rdma_client_declared_max_block_bytes` / `dfkv_rdma_client_connection_min_block_bytes` | gauge | 实际请求高水位 / 逻辑安全上限 / adaptive QP 最小 class |
+| `dfkv_rdma_client_connection_class_opened_total{block_bytes,depth}` | counter | adaptive二维class累计新建data QP |
+| `dfkv_rdma_client_connection_class_active{block_bytes,depth}` / `connection_class_idle{block_bytes,depth}` | gauge | 当前执行中 / idle pool内的二维class分布；label集合受power-of-two class限制 |
 | `dfkv_rdma_client_oversize_rejects_total` | counter | 分配、注册或发帖前因超过声明上限而拒绝的操作 |
 | `dfkv_rdma_client_v2_probe_attempts_total` / `dfkv_rdma_client_v2_probe_failures_total` | counter | 必选 v2 bootstrap probe 尝试 / 失败 |
 | `dfkv_rdma_client_stale_pool_retries_total` | counter | pooled QP 失败后改用 fresh connection 的重试 |