Skip to content

fix(vllm): support synchronous GPUDirect loads - #356

Merged
ketor merged 1 commit into
dingodb:mainfrom
ketor:fix/vllm-mla-gpu-dedup-default
Aug 27, 2026
Merged

fix(vllm): support synchronous GPUDirect loads#356
ketor merged 1 commit into
dingodb:mainfrom
ketor:fix/vllm-mla-gpu-dedup-default

Conversation

@ketor

@ketor ketor commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Summary

  • implement load_async=false for the vLLM connector
  • execute external GPUDirect loads in start_load_kv, before the request forward pass
  • keep the existing asynchronous WAITING_FOR_REMOTE_KVS path unchanged for load_async=true
  • validate the setting as a JSON boolean and document when synchronous mode is required

Problem

The async path issues remote GPU writes from get_finished() after model compute has already launched. On hybrid state-cache models, vLLM does not currently expose a per-block ownership fence proving those destination blocks are disjoint from concurrent recurrent-state compute.

On GLM-5.3-Flash TP8, 64K-input C10 external-cache replay consistently corrupted in-flight CUDA state:

  • C1 and C2: 20/20 successful
  • C10 async with node dedup enabled: 35/100 successful, then illegal memory access
  • C10 async with node dedup disabled: 21/100 successful, then CUDA/NVLink illegal access

The native client reported exact-length GET hits and zero I/O errors, so retrying transport or changing node dedup did not address the ownership race.

Validation

Hardware: xb01-0064, 8x B200. Model: GLM-5.3-Flash TP8+EP, no MTP. Workload: 100 prompts, 65,536 input tokens, 500 output tokens, concurrency 10.

With load_async=false:

  • 100/100 successful
  • 69,121 total tok/s, 523 output tok/s
  • 80,488 dfkv GET hits / 57,465,604,224 bytes read
  • zero GET misses and zero GET/PUT I/O errors
  • 156K-token restart round-trip also completed through dfkv; the target response contained the requested verification code

Tests:

  • worker lifecycle suite: 9/9 passed in the dedicated vLLM image

@ketor
ketor added this pull request to the merge queue Aug 27, 2026
Merged via the queue into dingodb:main with commit f1e474a Aug 27, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant