Skip to content

fix(vllm): preserve hybrid block tables for external KV - #354

Merged
ketor merged 1 commit into
dingodb:mainfrom
ketor:fix/vllm-glm53-connector-callbacks
Aug 27, 2026
Merged

fix(vllm): preserve hybrid block tables for external KV#354
ketor merged 1 commit into
dingodb:mainfrom
ketor:fix/vllm-glm53-connector-callbacks

Conversation

@ketor

@ketor ketor commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Problem

vLLM's current hybrid memory allocator passes only the blocks allocated in the current scheduler step through NewRequestData.block_ids. The dfkv connector treated that field as the complete request block table. On GLM-5.3-Flash (MLA/DSA/KDA hybrid cache), this caused asynchronous saves to fail with short block tables on TP ranks.

Stateful groups add two related constraints:

  • Their physical block tables are compact and retain only manager-selected state slots.
  • Equal scheduler/group block sizes do not mean that every logical chunk has a physical state slot.

The old uniform-block fast path therefore generated keys that could never be stored, and lookup convergence discarded otherwise complete sparse stateful prefixes.

Fix

  • Capture the complete per-group block table in update_state_after_alloc and use it for new-request metadata.
  • Derive store/load masks from each vLLM cache manager over the full aligned prefix.
  • Expand compact stateful block tables back to logical chunk positions before constructing GPUDirect descriptors.
  • Treat a contiguous prefix as complete after every object selected by that same manager mask exists.
  • Add regressions for scheduler-step-only block IDs, compact state tables, and equal-block-size stateful masks.

Validation

Runtime: vllm/vllm-openai:glm53-flash (g487ecf187), GLM-5.3-Flash native FP8, TP4, B200, dfkv v2.23.2 RDMA.

  • Unit regressions executed inside the dedicated vLLM image: FINAL_TESTS_PASS.
  • ruff check: all touched files passed.
  • Before fix: only TP0 stored; TP1-3 failed with block table has 0 ids.
  • After fix, no MTP: all 11 logical objects hit; 44 physical GETs across TP4, 40,728,192 bytes, zero failed keys.
  • After fix, MTP5: 8 logical objects hit; 32 physical GETs across TP4, about 170 MB read, zero dfkv I/O errors.
  • Cold cross-instance no-MTP request: 15,317 prompt tokens, 0.802 s end-to-end.

All tests used a source process followed by a separate cold target process.

@ketor
ketor added this pull request to the merge queue Aug 27, 2026
Merged via the queue into dingodb:main with commit de6e619 Aug 27, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant