Skip to content

perf: scale RDMA capacity and bound watermark reclaim - #359

Merged
ketor merged 1 commit into
dingodb:mainfrom
ketor:feat/rdma-adaptive-capacity-highwater
Aug 28, 2026
Merged

perf: scale RDMA capacity and bound watermark reclaim#359
ketor merged 1 commit into
dingodb:mainfrom
ketor:feat/rdma-adaptive-capacity-highwater

Conversation

@ketor

@ketor ketor commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Problem

Three multiplicative bottlenecks cap production scale:

  • the server eagerly allocates/registers one monolithic receive segment;
  • each client data QP declares the global maximum object size, regardless of its actual operation;
  • a high-water crossing drains synchronously to the low watermark and clears slot metadata one record at a time.

Change

  • Add a lazy chunked RecvSegmentPool: 256 MiB initial/growth chunks under the existing hard budget, with per-rail shared-PD registration and growth metrics.
  • Keep DFKV_RDMA_MAX_BLOCK_BYTES as a logical safety ceiling, but declare each data QP using the smallest power-of-two class covering the actual object. Idle-pool reuse now chooses the smallest sufficient QP.
  • Latch slab high-water reclaim until low watermark, cap work per tick (default one extent), and replace per-slot metadata writes with one contiguous extent clear.
  • Document sizing, rollout, and the new metrics. Wire new observable-contract tests.

No wire epoch or slab format change.

Evidence

  • Local: 807 tests passed, 0 failed.
  • ThreadSanitizer: 680 tests completed with no race report.
  • Real-verbs xb01-0064: adaptive class and lazy pool loopback tests passed.
  • 64-thread RDMA PUT: 10,000/10,000 with 80 live QPs using 673.7 MiB leases; fixed v2.23.3 exhausted an 8 GiB segment around 15 QPs.
  • 64 GiB steady high-water PUT: 5.39 -> 6.71 GB/s, p99 63.818 -> 24.142 ms, max 496.765 -> 50.796 ms.
  • GLM-5.3-Flash SGLang TP8: fresh target hit 2438/2438 KV + mamba keys over RDMA; 156k-token request 186.825 -> 2.723 s; 0 server errors.
  • GLM-5.3-Flash vLLM TP4+MTP5: fresh target hit 69/69 SG keys (72.75 MiB/rank); secret verification passed; 0 server errors.

@ketor
ketor added this pull request to the merge queue Aug 28, 2026
Merged via the queue into dingodb:main with commit 5275668 Aug 28, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant