Skip to content

perf: adapt RDMA depth and recycle receive capacity - #360

Merged
ketor merged 1 commit into
dingodb:mainfrom
ketor:feat/rdma-resource-watermark-v2
Aug 28, 2026
Merged

perf: adapt RDMA depth and recycle receive capacity#360
ketor merged 1 commit into
dingodb:mainfrom
ketor:feat/rdma-resource-watermark-v2

Conversation

@ketor

@ketor ketor commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Change

  • Select data QPs by two-dimensional (block class, depth class) geometry: scalar operations use depth 1, batches use the smallest sufficient power-of-two depth.
  • Export bounded per-class opened/active/idle metrics.
  • Complete replicated-MLA vLLM rank convergence by automatically enabling native CUDA-IPC same-host GET dedup; producer non-participants elide eager clients.
  • Replace per-connection pull ibv_reg_mr with type-2 Memory Windows over shared chunk MRs, retaining exact-MR fallback.
  • Affinitize growth chunks to their rail/NUMA node and release empty non-initial chunks after a configurable idle hold.
  • Reuse one preallocated extent-zero buffer in the bounded watermark path.

No wire, slab-format, or key-schema change.

Evidence

  • Local Release: 798 tests passed, 0 failed.
  • Local TSan: 797 tests passed, no race report.
  • xb01-0064 real RDMA: adaptive block/depth tests pass; 2,000 simultaneous depth-1 QPs complete in 20.9 s with no failures.
  • Pull isolation: 202/202 connections used type-2 Memory Windows, 0 MR fallbacks on ConnectX.
  • Chunk lifecycle: 3 chunks grew under 64-way 4 MiB traffic, then returned to the initial chunk; 512 MiB released.
  • 64 GiB watermark: 6,000/6,000 writes, 4 bounded ticks, max tick 2.871 ms, p99 30.830 ms, max 51.304 ms.
  • GLM-5.3-Flash SGLang TP8 fresh target: 813/813 KV + mamba hits; 52k-token request 205.854 -> 2.049 s.
  • GLM-5.3-Flash vLLM TP4+MTP5 converged mode: fresh target 24/24 keys and 50.49 MiB/rank hit; secret verification passed; active server QPs reduced from the previous 33 to 6.

Deliberate non-changes

  • Shared-CQ/fixed-worker rewrite rejected after the 2,000-QP real-verbs gate passed; no measured bottleneck justifies a high-risk Serve state-machine rewrite.
  • Lock-free watermark metadata I/O rejected because the bounded lock hold is 2.871 ms and a two-phase evicting state weakens the current persistence/concurrency proof.

@ketor
ketor added this pull request to the merge queue Aug 28, 2026
Merged via the queue into dingodb:main with commit 3db1627 Aug 28, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant