Skip to content

fix(vllm): support Qwen hybrid cache groups and TCP staging - #357

Merged
ketor merged 1 commit into
dingodb:mainfrom
ketor:fix/vllm-qwen-hybrid-tcp-staging
Aug 28, 2026
Merged

fix(vllm): support Qwen hybrid cache groups and TCP staging#357
ketor merged 1 commit into
dingodb:mainfrom
ketor:fix/vllm-qwen-hybrid-tcp-staging

Conversation

@ketor

@ketor ketor commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Problem

Qwen3.8-Flash-Next exposes align-mode Mamba groups with 400-token blocks while the attention cache uses 4-token blocks. The connector rejected this valid hybrid layout before startup even though its scheduler/worker path already resolves the LCM and preserves per-group geometry.

On xb01-0064, outbound GPUDirect PUT works but inbound GPUDirect GET is not usable on the selected rails. The vLLM connector hard-rejected libdfkv's existing host-staged TCP-to-CUDA publication path, leaving no correctness-first fallback.

Change

  • Remove the stale Mamba block-size equality gate; CrossAttentionSpec remains rejected.
  • Add explicit require_rdma=false to permit TCP host staging plus final CUDA publication. Default remains fail-closed RDMA.
  • Document hybrid load_async=false, static TCP endpoints, and the fallback tradeoff.
  • Add regression coverage for Qwen's 400-vs-4 group geometry and explicit TCP staging.

Verification

  • Qwen3.8-Flash-Next-FP8 TP4/EP4 vLLM startup succeeds with Mamba groups 400 vs attention block 4.
  • Targeted tests: 2 passed.
  • 0064 64K/500, 100 prompts, C10: cold 39,188.64 tok/s; cross-restart TCP-staged hot 65,969.65 tok/s (+68.3%), 100/100 requests.
  • Hot server evidence: 68,413 GET hits, 354,947,072,000 bytes read, 0 GET misses, 0 GET/PUT I/O errors.
  • 104,049-token source/target probe retained QWEN38-DFKV-9R7M after restart.

@ketor
ketor added this pull request to the merge queue Aug 28, 2026
Merged via the queue into dingodb:main with commit c9f0c09 Aug 28, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant